Cannot set X-Robots-Tag header on subfolders

Support
1

I’m trying to set certain paths on our website rasa.com to send an X-Robots-Tag: none header, but so far it’s not working, and I noticed another user here found that the tag seems to be controlled by netlify:
that is, set to "none" on deploy previews, and cleared on production deploys.

Could this be true? I’ve had luck setting it on only one other site, which is assets.rasa.com, but here only by applying it to the root i.e. /*.

2

Hi, @lunelson. I’m not seeing a header rule for this header on that site. How are you defining the rule (in _headers or netlify.toml)? Can you copy/paste the rule here which isn’t working?

3

Hi @luke, thanks for getting back, sorry I missed this. I had been testing it on a branch deploy and gave up on it, but maybe I did something wrong.

The rules I had created look like this; what I was trying to do was match a series of sitemap exclusion patterns we are using with the netlify sitemap plugin

[[headers]]
  for = "/carbon/*"
  [[headers.values]]
    X-Robots-Tag = "none"
[[headers]]
  for = "/carbon-bot-privacy-policy*"
  [[headers.values]]
    X-Robots-Tag = "none"
[[headers]]
  for = "/agreements*"
  [[headers.values]]
    X-Robots-Tag = "none"
[[headers]]
  for = "/product/welcome*"
  [[headers.values]]
    X-Robots-Tag = "none"
[[headers]]
  for = "/summit/thankyou*"
  [[headers.values]]
    X-Robots-Tag = "none"
[[headers]]
  for = "/thankyou*"
  [[headers.values]]
    X-Robots-Tag = "none"
[[headers]]
  for = "/google8972cda81756f8b1.html"
  [[headers.values]]
    X-Robots-Tag = "none"
[[headers]]
  for = "*/404*"
  [[headers.values]]
    X-Robots-Tag = "none"
4

Hi, @lunelson. I do see a syntax issue in the TOML above.

The [headers.values] lines should have a single square bracket enclosing them and not two brackets as in the example above. The correct syntax would be this:

[[headers]]
  for = "/carbon/*"
  [headers.values]
    X-Robots-Tag = "none"
[[headers]]
  for = "/carbon-bot-privacy-policy*"
  [headers.values]
    X-Robots-Tag = "none"
[[headers]]
  for = "/agreements*"
  [headers.values]
    X-Robots-Tag = "none"
[[headers]]
  for = "/product/welcome*"
  [headers.values]
    X-Robots-Tag = "none"
[[headers]]
  for = "/summit/thankyou*"
  [headers.values]
    X-Robots-Tag = "none"
[[headers]]
  for = "/thankyou*"
  [headers.values]
    X-Robots-Tag = "none"
[[headers]]
  for = "/google8972cda81756f8b1.html"
  [headers.values]
    X-Robots-Tag = "none"
[[headers]]
  for = "*/404*"
  [headers.values]
    X-Robots-Tag = "none"

If that doesn’t resolve the issue would you please send me the deploy id for the deploy where you tried to set the headers this way? The deploy id is the last path segment of the URL when viewing the deploy details for a deploy. For example, for this deploy details page below:

https://app.netlify.com/sites/example/deploys/55b8343352384f55f600000a

The deploy id is 55b8343352384f55f600000a. I don’t need the site subdomain, I just need the deploy id to find it.