X-Robots rules in netlify.toml are not adhered

Hi all,
I have a problem with my netlify.toml file. Either there is something wrong with my configuration (I guess) or Netlify is crazy (I doubt).

On the production website (live website), I can see a X-Robots-Tag = "noindex":

content-type: text/plain
date: Wed, 26 Jan 2022 13:52:27 GMT
referrer-policy: no-referrer
x-robots-tag: noindex
cache-control: max-age=0,no-cache,no-store,must-revalidate
x-xss-protection: 1; mode=block
location: https://example.com/
x-content-type-options: nosniff
x-frame-options: DENY
age: 0
server: Netlify

That’s the production website. In netlify.toml, which is in the production branch on GitHub, I have the following configuration:

[build]
  # Directory to change to before starting a build.
  # This is where we will look for package.json/.nvmrc/etc.
  # If not set, defaults to the root directory.
  # IMPORTANT: This setting takes precedence over GUI settings.
  base = "/"
  
  # Directory that contains the deploy-ready HTML files and assets generated by
  # the build, relative to the base directory (if set) or the root directory.
  # IMPORTANT: This setting takes precedence over GUI settings.
  publish = "_site"
  
  # Default build command.
  # IMPORTANT: This setting takes precedence over GUI settings.
  command = "jekyll build"
  
  # Directory that contains netlify functions
  # Default is YOUR_BASE_DIRECTORY/netlify/functions
  functions = "netlify/functions"



# PRODUCTION

[context.production] # "production" is a branch name

  # Directory to change to before starting a build.
  # This is where we will look for package.json/.nvmrc/etc.
  # If not set, defaults to the root directory.
  # IMPORTANT: This setting takes precedence over GUI settings.
  base = "/"

  # Set Jekyll environment for the development context
  # https://jekyllrb.com/docs/configuration/environments/
  environment = { JEKYLL_ENV = "production" }
  
  # Directory that contains the deploy-ready HTML files and assets generated by
  # the build, relative to the base directory (if set) or the root directory.
  # IMPORTANT: This setting takes precedence over GUI settings.
  publish = "_site"
  
  # Default build command.
  # IMPORTANT: This setting takes precedence over GUI settings.
  command = "jekyll build"
  
  # Directory that contains netlify functions
  # Default is YOUR_BASE_DIRECTORY/netlify/functions
  functions = "netlify/functions"

  [[headers]]
    # Define which paths this specific [[headers]] block will cover.
    for = "/*"

    [headers.values]
      # https://developers.google.com/search/docs/advanced/robots/robots_meta_tag?hl=en#xrobotstag
      X-Robots-Tag = "nocache"



# STAGING

[context.staging] # "staging" is a branch name

  # Directory to change to before starting a build.
  # This is where we will look for package.json/.nvmrc/etc.
  # If not set, defaults to the root directory.
  # IMPORTANT: This setting takes precedence over GUI settings.
  base = "/"

  # Set Jekyll environment for the development context
  # https://jekyllrb.com/docs/configuration/environments/
  environment = { JEKYLL_ENV = "staging" }
  
  # Directory that contains the deploy-ready HTML files and assets generated by
  # the build, relative to the base directory (if set) or the root directory.
  # IMPORTANT: This setting takes precedence over GUI settings.
  publish = "_site"
  
  # Default build command.
  # IMPORTANT: This setting takes precedence over GUI settings.
  command = "jekyll build"
  
  # Directory that contains netlify functions
  # Default is YOUR_BASE_DIRECTORY/netlify/functions
  functions = "netlify/functions"
  
  [[headers]]
    # Define which paths this specific [[headers]] block will cover.
    for = "/*"

    [headers.values]
      # https://developers.google.com/search/docs/advanced/robots/robots_meta_tag?hl=en#xrobotstag
      X-Robots-Tag = "noindex"
      
      # https://developer.mozilla.org/en-US/docs/Web/HTTP/Authentication
      Basic-Auth = "Foo:Bar"



# DEVELOPMENT

[context.development] # "development" is a branch name

  # Directory to change to before starting a build.
  # This is where we will look for package.json/.nvmrc/etc.
  # If not set, defaults to the root directory.
  # IMPORTANT: This setting takes precedence over GUI settings.
  base = "/"

  # Set Jekyll environment for the development context
  # https://jekyllrb.com/docs/configuration/environments/
  environment = { JEKYLL_ENV = "development" }
  
  # Directory that contains the deploy-ready HTML files and assets generated by
  # the build, relative to the base directory (if set) or the root directory.
  # IMPORTANT: This setting takes precedence over GUI settings.
  publish = "_site"
  
  # Default build command.
  # IMPORTANT: This setting takes precedence over GUI settings.
  command = "jekyll build"
  
  # Directory that contains netlify functions
  # Default is YOUR_BASE_DIRECTORY/netlify/functions
  functions = "netlify/functions"
  
  [[headers]]
    # Define which paths this specific [[headers]] block will cover.
    for = "/*"

    [headers.values]
      # https://developers.google.com/search/docs/advanced/robots/robots_meta_tag?hl=en#xrobotstag
      X-Robots-Tag = "noindex"
      
      # https://developer.mozilla.org/en-US/docs/Web/HTTP/Authentication
      Basic-Auth = "Foo:Bar"

As you can see, for production, the X-Robots-Tag = "nocache" is defined, whilst X-Robots-Tag = "noindex" is defined for staging and developement. The public website builds from production (Set in Netlify GUI).

Last week I had the problem of having the basic-auth on the production website, now it’s the X-Robots-Tag. Am I missing something? Is there an error in my configuration? What might cause these issues?

Thank you for your help.

Does this still hold true?

Redirects and Headers specified in netlify.toml are site-wide and cannot be configured specifically for some contexts.

Then, why is it working for the basic auth, but not for x-robots?

Hey @dafoobar,

That still holds true. Would you be able to share the site name?

The npm run build && cp _headers /public/_headers did not work for me. I am using Jekyll and ran into build errors with the appended command.

I’ve decided to go with:

command = "printf \"include:\n  - _headers\" >> _config.yml && jekyll build"

for the contexts that need the basic auth and a specific set of X-Robots rules.
In Jekyll, I’ve also added _headers to the exclude rule list in _config.yml, my includes are set below the the exclude rules, at the end of the document.

Not sexy but working.

Hi, @dafoobar. Thanks for the follow-up to share the solution you found.

If there are any other questions about this, please let us know.