Netlify website blocking indexing on google

Support
1

My site name is https://stratopsjobs.netlify.app/

I’m using nodejs. When I look at my lighthouse report it says this:

Search engines are unable to include your pages in search results if they don’t have permission to crawl them. Learn more.

Blocking Directive Source
x-robots-tag: noindex

The only place in my code with robots tag is the decap cms setup I am using. But don’t want that admin page indexed obviously.

Haven’t be able to solve this so I assume it’s something going on with netlify instead? Has anyone been able to solve this?

2

Hi, @asodifjsa. That header should only be present on non-production deploys. For example, if you use a permalink to a specific deploy id (the deploy id is part of the subdomain):

$ curl --compressed -svo /dev/null --stderr - https://658319b5f6cc2d00084e7bcb--stratopsjobs.netlify.app/  | egrep '^< '
< HTTP/2 200
< accept-ranges: bytes
< age: 0
< cache-control: public,max-age=0,must-revalidate
< cache-status: "Netlify Edge"; fwd=miss
< content-encoding: gzip
< content-type: text/html; charset=UTF-8
< date: Thu, 21 Dec 2023 20:59:01 GMT
< etag: "d5cf9a957aa82c18c8e098265b07fa4f-ssl-df"
< server: Netlify
< strict-transport-security: max-age=31536000; includeSubDomains; preload
< vary: Accept-Encoding
< x-nf-request-id: 01HJ72SAFXW27CQCQ4YEWBX665
< x-robots-tag: noindex
<

We can see the x-robots-tag: noindex header above. However, if you request the production deploy, it isn’t there anymore:

$ curl --compressed -svo /dev/null --stderr - https://stratopsjobs.netlify.app/  | egrep '^< '
< HTTP/2 200
< accept-ranges: bytes
< age: 0
< cache-control: public,max-age=0,must-revalidate
< cache-status: "Netlify Edge"; fwd=miss
< content-encoding: gzip
< content-type: text/html; charset=UTF-8
< date: Thu, 21 Dec 2023 21:01:57 GMT
< etag: "ac70d2a78bd780beb92e89fdd71287f3-ssl-df"
< link: <https://stratopsjobs.com/>; rel="canonical"
< server: Netlify
< strict-transport-security: max-age=31536000; includeSubDomains; preload
< vary: Accept-Encoding
< x-nf-request-id: 01HJ72YQ652J1REPJ8DN9CNDWH
<

So, this shouldn’t be a problem with production. You will see this for non-production deploys, though. However, as most people don’t want non-production deploys indexed, this should be a good thing in most cases.

If you see the header for non-production deploys or if you have any questions about this, please reply here anytime.

3

Thanks for your reply. When I push to production it looks like the lighthouse report is giving me a non production deploy URL in its report, which does indeed have the noindex.

Is this just how netlify’s lighthouse integration works? All deploys even production deploys will run a non production lighthouse report that always says it’s not indexed?

4

Hi, @asodifjsa. The integration works that way because the URL with the deploy id in the subdomain is the only way to guarantee that exact deploy is examined. So, the answer here is that, yes, this is unfortunately a limitation of that integration. The lighthouse reports are always going to say indexing isn’t allowed (because it is not for the URL being checked). However, if you manually run a lighthouse test on your production deploys, you will not have this issue. Basically, it is a false positive due to the limitation of the deploy id URLS.