Is there a way to completely override netlify’s default noindex HTTP header in deploy preview? I’m trying to change it to “all” but instead it creates “all, noindex”. Is there a way to change it to only “all” or something similar like “index”.
Hey there, @chidieberejoel
Welcome to the Netlify Forums! Have you read our Docs? This page outlines where custom headers can be used as well as syntax.
Let us know if this helps!
Yes, I read the docs. I’m trying to understand if I can have an X-Robots-Tag set to “all” or “index” (not “all, noindex”) in a deploy preview website since the default is noindex and it overrides “all” when both are used.
Hi @chidieberejoel ,
The X-Robots-Tag
is set to noindex
so that search engines don’t index deploy previews and therefore won’t serve them up as results in searches.
I’m not sure why you would want your deploy previews indexed.
I’m using the deploy site to run some SEO test before pushing to production
It seems as though the custom X-Robots-Tag
is prepended to that sent out by default.
Wonder if @Scott could shed some light on this.
There isn’t a way to override the inclusion of the x-robots-tag: noindex
header for two types of deploys:
- deploy previews
- deploy id specific URLs
We can enter a feature request to change this but it isn’t possible at this time.
Note, there is a type of URL which doesn’t do this automatically, the branch deploy URLs. Here is an example of using the deploy preview URL or the deploy id URL, they have two headers not one:
$ curl -svo /dev/null https://deploy-preview-1--trusting-jones-115d95.netlify.app/ 2>&1 | egrep 'x-robots-tag'
< x-robots-tag: all
< x-robots-tag: noindex
$ curl -svo /dev/null https://60caf29e9b4ce10007149288--trusting-jones-115d95.netlify.app/ 2>&1 | egrep 'x-robots-tag'
< x-robots-tag: all
< x-robots-tag: noindex
However, if I use the URL for the branch itself, then the noindex
line is missing:
$ curl -svo /dev/null https://branch-with-two-dashes--trusting-jones-115d95.netlify.app/ 2>&1 | egrep 'x-robots-tag'
< x-robots-tag: all
If you would like us to file this feature request to be able to override the x-robots-tag
headers for deploy id URLs and deploy preview URLs, please let us know.