X-Robots rules in netlify.toml are not adhered

Support
,
1

Hi all,
I have a problem with my netlify.toml file. Either there is something wrong with my configuration (I guess) or Netlify is crazy (I doubt).

On the production website (live website), I can see a X-Robots-Tag = "noindex":

content-type: text/plain
date: Wed, 26 Jan 2022 13:52:27 GMT
referrer-policy: no-referrer
x-robots-tag: noindex
cache-control: max-age=0,no-cache,no-store,must-revalidate
x-xss-protection: 1; mode=block
location: https://example.com/
x-content-type-options: nosniff
x-frame-options: DENY
age: 0
server: Netlify

That’s the production website. In netlify.toml, which is in the production branch on GitHub, I have the following configuration:

[build]
  # Directory to change to before starting a build.
  # This is where we will look for package.json/.nvmrc/etc.
  # If not set, defaults to the root directory.
  # IMPORTANT: This setting takes precedence over GUI settings.
  base = "/"
  
  # Directory that contains the deploy-ready HTML files and assets generated by
  # the build, relative to the base directory (if set) or the root directory.
  # IMPORTANT: This setting takes precedence over GUI settings.
  publish = "_site"
  
  # Default build command.
  # IMPORTANT: This setting takes precedence over GUI settings.
  command = "jekyll build"
  
  # Directory that contains netlify functions
  # Default is YOUR_BASE_DIRECTORY/netlify/functions
  functions = "netlify/functions"



# PRODUCTION

[context.production] # "production" is a branch name

  # Directory to change to before starting a build.
  # This is where we will look for package.json/.nvmrc/etc.
  # If not set, defaults to the root directory.
  # IMPORTANT: This setting takes precedence over GUI settings.
  base = "/"

  # Set Jekyll environment for the development context
  # https://jekyllrb.com/docs/configuration/environments/
  environment = { JEKYLL_ENV = "production" }
  
  # Directory that contains the deploy-ready HTML files and assets generated by
  # the build, relative to the base directory (if set) or the root directory.
  # IMPORTANT: This setting takes precedence over GUI settings.
  publish = "_site"
  
  # Default build command.
  # IMPORTANT: This setting takes precedence over GUI settings.
  command = "jekyll build"
  
  # Directory that contains netlify functions
  # Default is YOUR_BASE_DIRECTORY/netlify/functions
  functions = "netlify/functions"

  [[headers]]
    # Define which paths this specific [[headers]] block will cover.
    for = "/*"

    [headers.values]
      # https://developers.google.com/search/docs/advanced/robots/robots_meta_tag?hl=en#xrobotstag
      X-Robots-Tag = "nocache"



# STAGING

[context.staging] # "staging" is a branch name

  # Directory to change to before starting a build.
  # This is where we will look for package.json/.nvmrc/etc.
  # If not set, defaults to the root directory.
  # IMPORTANT: This setting takes precedence over GUI settings.
  base = "/"

  # Set Jekyll environment for the development context
  # https://jekyllrb.com/docs/configuration/environments/
  environment = { JEKYLL_ENV = "staging" }
  
  # Directory that contains the deploy-ready HTML files and assets generated by
  # the build, relative to the base directory (if set) or the root directory.
  # IMPORTANT: This setting takes precedence over GUI settings.
  publish = "_site"
  
  # Default build command.
  # IMPORTANT: This setting takes precedence over GUI settings.
  command = "jekyll build"
  
  # Directory that contains netlify functions
  # Default is YOUR_BASE_DIRECTORY/netlify/functions
  functions = "netlify/functions"
  
  [[headers]]
    # Define which paths this specific [[headers]] block will cover.
    for = "/*"

    [headers.values]
      # https://developers.google.com/search/docs/advanced/robots/robots_meta_tag?hl=en#xrobotstag
      X-Robots-Tag = "noindex"
      
      # https://developer.mozilla.org/en-US/docs/Web/HTTP/Authentication
      Basic-Auth = "Foo:Bar"



# DEVELOPMENT

[context.development] # "development" is a branch name

  # Directory to change to before starting a build.
  # This is where we will look for package.json/.nvmrc/etc.
  # If not set, defaults to the root directory.
  # IMPORTANT: This setting takes precedence over GUI settings.
  base = "/"

  # Set Jekyll environment for the development context
  # https://jekyllrb.com/docs/configuration/environments/
  environment = { JEKYLL_ENV = "development" }
  
  # Directory that contains the deploy-ready HTML files and assets generated by
  # the build, relative to the base directory (if set) or the root directory.
  # IMPORTANT: This setting takes precedence over GUI settings.
  publish = "_site"
  
  # Default build command.
  # IMPORTANT: This setting takes precedence over GUI settings.
  command = "jekyll build"
  
  # Directory that contains netlify functions
  # Default is YOUR_BASE_DIRECTORY/netlify/functions
  functions = "netlify/functions"
  
  [[headers]]
    # Define which paths this specific [[headers]] block will cover.
    for = "/*"

    [headers.values]
      # https://developers.google.com/search/docs/advanced/robots/robots_meta_tag?hl=en#xrobotstag
      X-Robots-Tag = "noindex"
      
      # https://developer.mozilla.org/en-US/docs/Web/HTTP/Authentication
      Basic-Auth = "Foo:Bar"

As you can see, for production, the X-Robots-Tag = "nocache" is defined, whilst X-Robots-Tag = "noindex" is defined for staging and developement. The public website builds from production (Set in Netlify GUI).

Last week I had the problem of having the basic-auth on the production website, now it’s the X-Robots-Tag. Am I missing something? Is there an error in my configuration? What might cause these issues?

Thank you for your help.

2

Does this still hold true?

Redirects and Headers specified in netlify.toml are site-wide and cannot be configured specifically for some contexts.

Then, why is it working for the basic auth, but not for x-robots?

3

Hey @dafoobar,

That still holds true. Would you be able to share the site name?

4

The npm run build && cp _headers /public/_headers did not work for me. I am using Jekyll and ran into build errors with the appended command.

I’ve decided to go with:

command = "printf \"include:\n  - _headers\" >> _config.yml && jekyll build"

for the contexts that need the basic auth and a specific set of X-Robots rules.
In Jekyll, I’ve also added _headers to the exclude rule list in _config.yml, my includes are set below the the exclude rules, at the end of the document.

Not sexy but working.

5

Hi, @dafoobar. Thanks for the follow-up to share the solution you found.

If there are any other questions about this, please let us know.