When I checked my preview deploy (kambostrong.netlify.app) last week, I’m pretty sure it had X-ROBOTS-TAG: NOINDEX which is normal and good. I know this because Lightouse was complaining about it being present and affecting SEO.
However today, I’m checking it, and it’s not there - so the preview site seems to be getting indexed by google as well as the live production site… which is bad!
I thought this might be due to my custom _headers file which I added toady, but I’ve since removed that and noindex still is not present on the preview site.
For your reference, my headers file was simply:
/*
No-Vary-Search: params=(“utm_source” “utm_medium” “utm_campaign” “utm_content” “utm_term”)/_astro/*
Cache-Control: public, max-age=31536000, immutable
(but now it’s removed entirely for testing and noindex still isn’t being re-applied to the preview site).
This seems wild to me - who would want a dummy/preview site being indexed as well as their live version? I must be missing something?