Override X-Robots-Tag: noindex header in deploy preview

Is there a way to completely override netlify’s default noindex HTTP header in deploy preview? I’m trying to change it to “all” but instead it creates “all, noindex”. Is there a way to change it to only “all” or something similar like “index”.

Hey there, @chidieberejoel :wave:

Welcome to the Netlify Forums! Have you read our Docs? This page outlines where custom headers can be used as well as syntax.

Let us know if this helps!

Yes, I read the docs. I’m trying to understand if I can have an X-Robots-Tag set to “all” or “index” (not “all, noindex”) in a deploy preview website since the default is noindex and it overrides “all” when both are used.

Hi @chidieberejoel ,

The X-Robots-Tag is set to noindex so that search engines don’t index deploy previews and therefore won’t serve them up as results in searches.

I’m not sure why you would want your deploy previews indexed.

I’m using the deploy site to run some SEO test before pushing to production

It seems as though the custom X-Robots-Tag is prepended to that sent out by default.

Wonder if @Pie could shed some light on this.

There isn’t a way to override the inclusion of the x-robots-tag: noindex header for two types of deploys:

  • deploy previews
  • deploy id specific URLs

We can enter a feature request to change this but it isn’t possible at this time.

Note, there is a type of URL which doesn’t do this automatically, the branch deploy URLs. Here is an example of using the deploy preview URL or the deploy id URL, they have two headers not one:

$ curl -svo /dev/null https://deploy-preview-1--trusting-jones-115d95.netlify.app/  2>&1 | egrep 'x-robots-tag'
< x-robots-tag: all
< x-robots-tag: noindex
$ curl -svo /dev/null https://60caf29e9b4ce10007149288--trusting-jones-115d95.netlify.app/  2>&1 | egrep 'x-robots-tag'
< x-robots-tag: all
< x-robots-tag: noindex

However, if I use the URL for the branch itself, then the noindex line is missing:

$ curl -svo /dev/null https://branch-with-two-dashes--trusting-jones-115d95.netlify.app/  2>&1 | egrep 'x-robots-tag'
< x-robots-tag: all

If you would like us to file this feature request to be able to override the x-robots-tag headers for deploy id URLs and deploy preview URLs, please let us know.

1 Like