Netlify ignoring edge functions and _headers when serving the Google Index Bot

Support
,
1

It seems that since mid January, Netlify is treating requests from the Google Search index bot differently, breaking sites in severe ways; my website has completely fallen out from the Google Index!

This graph from the Google Search console shows when the problem started appearing; around that time nothing about my site has changed:

photo_2025-03-21_23-14-17
photo_2025-03-21_23-14-171018×688 43.8 KB

For example for the main page of my site, https://www.joachim-breitner.de/blog, it would report “error due to redirections”. But there is no redirect happening!

The helpful people at the Google Search Community solved the puzzle: It seems that Netlify is behaving differently if the user agent is Googlebot Smartphone! In particular, it seems to be ignoring the edge functions and instead applies some other, contradicting URL normalization, which of course completely breaks the site. Compare

A very likely related, and also somewhat serious issue is that the _headers file seems to be ignored for that user agent, compare

where I am using _headers to set x-robots-tag: noindex, but if that setting isn’t effective precisely when Google is requesting the page, then that has a certain irony to it.

This very strongly points towards a problem within Netlify, and a severe one at that, as it can cause sites to becomes unfindable on Google. Is there a swift fix possible? Or would Netlify users somehow have to live with that?

2

The second links are wrong, they don’t set the user agent, so the difference isn’t shown. Should be

3

Given that Edge Functions are being advertised also for access control, e.g. at https://edge-functions-examples.netlify.app/, I wonder if this recent change at Netlify is actually a security issue?

4

Just in case someone looks at this now: I emergency-moved my site to different hoster (deno deploy) where this problem doesn’t exist, so that I can come back to the Google Index.

It would still be nice to be able to use Netlify for such sites, though.

5

You’ve enabled prerendering with Netlify: Build & deploy | Site configuration | joachim-breitner | Netlify. Have you tried disabling it?

6

Indeed! After turning pre-rendering off, the x-robots-tag: noindex shows up immediately. Good catch.

(Didn’t check the edge function using site, which I have now already moved to another hoster. If I have to move it back I’ll try :slight_smile: )

But should it be that way, that prerendering affects headers like this?
I also can’t remember enabling this (beta) feature?

7

Yes, prerendering uses its own set of headers. As for enabling the feature, we don’t automatically enable it, so it must have been done some time in the past.

8

Thanks for the info. That’s weird; I have about two dozen sites, and some of them are rather trivial (static HTML), so I am fairly certain that I did not change any settings from the default from when I created them (most in Feb 2022), yet all of them seem to have prerendering enabled. :thinking:

Anyways, I now know where to look if I observe issues :person_shrugging:. Thanks!