X-Robots-Tag = "noindex" - set this in netlify.toml file but am not seeing it applied yet

Hello

As the subject states - I’ve set X-Robots-Tag = "noindex" in a netlify.toml file but am not seeing the header applied in the browser.

This is a first for me in Netlify for either a branch deploy or targetted headers and I could do with a bit of advice. The branch deploy is going fine thus-far and I love the way this is managed, but I’m not sure I’m applying headers properly yet.

This is how I’ve written the netlify.toml file (not the actual domain btw):

[[headers]]
  for = "/thesite-preview.netlify.app/*"
  [headers.values]
    X-Frame-Options = "DENY"
    X-XSS-Protection = "1; mode=block"
    X-Robots-Tag = "noindex"

[[headers]]
  for = "/develop--thesite-preview.netlify.app/*"
  [headers.values]
    X-Frame-Options = "DENY"
    X-XSS-Protection = "1; mode=block"
    X-Robots-Tag = "noindex"
[[headers]]
  for = "/*--thesite-preview.netlify.app/*"
  [headers.values]
    X-Frame-Options = "DENY"
    X-XSS-Protection = "1; mode=block"
    X-Robots-Tag = "noindex"

Thus far I can’t see any of those header rules in the browser and so I have 2 questions:

  • is there anything glaringly wrong with the headers file syntax, or otherwise
  • if this is correct, can someone advise on what I should be looking for/how I can confirm the header is being applied?

Thanks in advance

question - did you redeploy after making changes?

Apoligies for the delay in replying.
Yes I did deploy, had tried various options but all showing nothing in headers. I got a bit kitchen sink about it and edited and deployed the headers to the following:

[[headers]]
  for = "thesite-preview.netlify.app"
  [headers.values]
    X-Frame-Options = "DENY"
    X-XSS-Protection = "1; mode=block"
    X-Robots-Tag = "noindex"
[[headers]]
  for = "/thesite-preview.netlify.app"
  [headers.values]
    X-Frame-Options = "DENY"
    X-XSS-Protection = "1; mode=block"
    X-Robots-Tag = "noindex"
[[headers]]
  for = "thesite-preview.netlify.app/*"
  [headers.values]
    X-Frame-Options = "DENY"
    X-XSS-Protection = "1; mode=block"
    X-Robots-Tag = "noindex"
[[headers]]
  for = "/thesite-preview.netlify.app/*"
  [headers.values]
    X-Frame-Options = "DENY"
    X-XSS-Protection = "1; mode=block"
    X-Robots-Tag = "noindex"

[[headers]]
  for = "develop--thesite-preview.netlify.app"
  [headers.values]
    X-Frame-Options = "DENY"
    X-XSS-Protection = "1; mode=block"
    X-Robots-Tag = "noindex"
[[headers]]
  for = "/develop--thesite-preview.netlify.app"
  [headers.values]
    X-Frame-Options = "DENY"
    X-XSS-Protection = "1; mode=block"
    X-Robots-Tag = "noindex"
[[headers]]
  for = "develop--thesite-preview.netlify.app/*"
  [headers.values]
    X-Frame-Options = "DENY"
    X-XSS-Protection = "1; mode=block"
    X-Robots-Tag = "noindex"
[[headers]]
  for = "/develop--thesite-preview.netlify.app/*"
  [headers.values]
    X-Frame-Options = "DENY"
    X-XSS-Protection = "1; mode=block"
    X-Robots-Tag = "noindex"

Unfortunately just getting the same - can’t see anything relating to noindex in browser headers.

Am I missing something or just trying for the wrong option? I’ve trawled through the docs etc and can’t see anything else.

Here’s screengrabs of headers in Firefox/Chrome:

Hi, @powellian. The rules are not working because you are trying to include domain names in your rules.

Header rules only consider the path component of the URL. There is no way for headers to apply to specific domain names only.

If you want the header applied for all paths, then the rule would look like this in netlify.toml format:

[[headers]]
  for = "/*"
  [headers.values]
    X-Frame-Options = "DENY"
    X-XSS-Protection = "1; mode=block"
    X-Robots-Tag = "noindex"

Or, in the _headers format, like this:

/*
  X-Frame-Options = "DENY"
  X-XSS-Protection = "1; mode=block"
  X-Robots-Tag = "noindex"

​Please let us know if there are other questions about this.

Hi @luke
Thanks for your response, totally missed it sorry for my tardy reply.
Actually I’d removed the preview branch for this as I needed to avoid duplicate indexing but will give it another go based on what you’ve written here.

I’ll be creating a uat branch and presumably it’ll resolve like this: uat--somename.netlify.app
So in [[headers]] do I write it like this?:

[[headers]]
  for = "/uat--*"
  ... ...
  X-Robots-Tag = "noindex"

Hi, @powellian. I would personally solve this by not using netlify.toml at all. Instead, use the _headers file and create a custom file specific to the branch.

In other words different branches will use different _headers files. If you want to return that header for all files in a deploy, this is the _headers file to use:

/*
  X-Robots-Tag = noindex

If there are other questions, please let us know.

Thank you for the reply @luke .

I had considered using _headers but when reading the docs here’s why I didn’t:

  1. I’m using Nuxt as my SSG - the publish directory for Nuxt projects is /dist - this dir is in .gitignore by default so _headers won’t be available to Netlify
  2. ignoring point 1 - the build command is npm run generate and this completely removes _headers from /dist - (locally at least so I’ll assume the same when deployed)
  3. point 2 is academic as point 1 means _headers is ignored from the repo anyway!
    (I’m not going to remove /dist from gitignore as it’s bound to get messy somewhere)

This apparent catch-22 is why I opted for netlify.toml but I’ll roll with whichever is best/reliable, assuming I can get it to work and/or get my head around how it is all wired up!

I’m aware this is straying into Nuxt territory and not just a Netlify thing, but I believe most SSG’s have a similar ‘destructive’ aspect in the build process and imagine Netlify would deal with that that in some way.

Am I either over-thinking this or missing something glaring in the _header documentation?

Hey there quick suggestion.

I’m not familar with NUXT however I know with Eleventy it’s possible to create a passThrough. This means you could put the _headers file at the root and tell your SSG to copy it over to /dist, this way Netlify still has access to the file and you are avoiding the use of netlify.toml like @luke suggested.

Once again, this is a feature we use with Eleventy but I’m not sure about NUXT and if they have a similar feature to this.

Hope you find a solution soon :slightly_smiling_face:

Kyle.

Hey @kylesloper
Thanks for inspiring me into looking further than the end of my nose :wink: - and presto, Nuxt has a whole module for what I need: netlify-files-module
Will hit that asap and write back with the result, thanks for the suggestion.

Actually I’ve been meaning to get into Eleventy for a while … does look good

Haha no worries :grinning:

Yea I really think you should, it kinda makes you think about how simple an SSG can be whilst being up there with the very best.

update
@luke I’m working in the _headers file and following the last thread suggestion from @kylesloper I’m also integrating a ‘proper’ workflow for nuxt to make this work.

I also thought about merge issues and hit on this: https://git-scm.com/book/en/v2/Customizing-Git-Git-Attributes#_merge_strategies

Seems to work so far in a local test setting - I’m getting the right _headers file in the Git main branch (indexing everything) - and with develop and uat--whatever keeping the noindex setting.

Long day … will test this more thoroughly tomorrow with deploys and write back.
Many thanks thus far.

1 Like

Hi, @powellian. I also wanted to mention that Nuxt allows you to put static files (like the _headers file) into a specific directory and then Nuxt will automatically copy those files to dist/ during the site build process.

1 Like

Hi @luke thanks for that tip, works like a charm and a bit more Nuxt knowledge.

I locally ran a production build rolling with a uat branch and can see ./static/_headers copying over into /dist - so all good locally. Here’s the _headers file:

/*
  X-Robots-Tag = "noindex"
  X-Frame-Options = "DENY"
  X-XSS-Protection = "1; mode=block"

Deployed to a Netlify test url for that branch - I assume this will deploy the ./static/_headers file same as my local test. Here’s the page: uat–headers-testing.netlify.app

However, I can’t see the headers being applied in either localhost or the uat test link and the file is def in Git. I tested _headers with/without quotes but same result - any chance you could cast your eye over that link and confirm/otherwise whether the headers are loading?

Here’s a grab of Chrome devtools from the uat branch url:

Hi, @powellian. There was an error processing the header rules as noted here:

https://app.netlify.com/sites/headers-testing/deploys/60797ef4b489600007b2706f

Quoting that page:

1 invalid header rule found

1 out of 1 header rule could not be processed. Check your site’s /_headers file for verification. You can learn more in the docs, or test your rules in the Netlify playground.

In /_headers:

  • /* X-Robots-Tag = “noindex” X-Frame-Options = “DENY” X-XSS-Protection = “1; mode=block”

There is documentation about the required syntax for this file here:

This the file used:

/*
  X-Robots-Tag = "noindex"
  X-Frame-Options = "DENY"
  X-XSS-Protection = "1; mode=block"

The correct syntax is found below:

/*
  X-Robots-Tag: noindex
  X-Frame-Options: DENY
  X-XSS-Protection: 1; mode=block

Would you please make that change and let us know if it still doesn’t work?

1 Like

@luke
That’s the one! all headers showing now.
I had tried it earlier without quotes but had left in the “=” instead of using “:” so obv it didn’t work.

Many thanks for your help on this one - also the syntax playground is a great helper, didn’t know about that til now.

2 Likes