Home
Support Forums

Google Search redirect error started last week (no change to my site)

To summarize, asking us to explain messages inside of Google’s tools will rarely be successful as we have zero insight into those tools. We can troubleshoot Netlify itself. In order to troubleshoot the issue, we need some way to identify the incorrect HTTP responses and the x-nf-request-id header or the information is replaces are the two most common ways this can be done.

Copy pasting from the other thread:

As mentioned, the way to test this is by using Google Search Console which is the only authoritative way to test this as far as I can see. Like I mentioned before, even if Google’s implementation is incorrect their dominance in search makes their implementation the de-facto standard.

I would encourage all affected individuals to check the results for the urls in question by visiting Google Search Console and entering the url into the ‘URL Inspection’ tool which is third down in the left-hand navigation bar. You can then test any updates to your configuration by pressing the ‘test live url’ button.

If it works – wonderful. No further action needed.

If it fails, the SEO of your website is being negatively impacted and you’ll need to make changes to mitigate this.

Again, I believe it is the responsibility of Netlify to ensure that their default DNS setup is compatible with Google’s interpretation of the standards – regardless of whether or not it is correct. It’s the reality of their dominant position in the market.

Hi, @Ultra. Here is the blocker preventing this from being resolved:

  • There is no proof in either topic that Netlify is doing anything wrong.

Just show me the issue happening and I can get it fixed. That is all I need to help you.

I explained several different possible sets of information which would allow our support team to troubleshoot, which includes any of the following:

  • the x-nf-request-id header for a bad response
  • the client IP address, service IP address, URL, date, time, and timezone for a bad response
  • a HAR recording of a bad response
  • a curl command which will return a bad response

Any of those would allow me to research the issue.

Would you please send us that information (any one of those sets)? Do you have questions about what is required or how to gather that information?

Hi @Luke,

I’m confused with your stance that ‘there is no proof in either topic that Netlify is doing anything wrong’ .

That’s not my aim.

My concern is that people read these statements that show positive diagnostics results and believe that their site will perform well despite Google Search Console reporting otherwise.

It’s important you communicate to people that this isn’t the case.

Regardless of what Netlify’s or anybody else diagnostics say, if Google Search Console reports issues for their site, that’s a problem that will negatively impact their SEO.

Therefore, I’m recommending that people confirm that their urls are behaving in the way Goolge expects by confirming their results in Google Search Console.

This follows on from my own experience after following Netlify’s recommendations on setting up my own DNS that proved to cause Google Search Console to show configuration errors.

This seemed to be caused by Nettlify’s recommendation to use flattened CNAME records (ALIAS records) which, as you can verify below, has now been limited to Cloudflare customers only.

Original Instructions: https://web.archive.org/web/20210129101918if_/https://docs.netlify.com/domains-https/custom-domains/configure-external-dns/

Some DNS providers, such as Netlify DNS or NS1, have devised special record types to simulate CNAME-style domain resolution for apex domains. Find out if your provider supports this type of behavior, which might be labeled as CNAME flattening, ANAME records, or ALIAS records.

If your DNS provider supports one of these special record types (recommended), find and follow their instructions to point the apex domain directly to your Netlify subdomain, such as brave-curie-671954.netlify.app .

Current instructions: https://web.archive.org/web/20210129101918if_/https://docs.netlify.com/domains-https/custom-domains/configure-external-dns/

If you use Cloudflare as your DNS provider , it supports a special record type that also works well on the bare domain - Flattened CNAME records. This record type is recommended for your bare domain. You’d set the same record value as for your subdomains, such as brave-curie-671954.netlify.app.

What’s your evidence that this is the case? My experience has been exactly the opposite … except in one edge case, no matter what Google claims is wrong with my sites they still appear in search results as expected.

@luke I’m seeing this too and the fact that it is user agent specific should be a strong hint that this is something Netlify does server-side.

Example x-nf-request-id is f7bdb6dc-c836-48ff-80e6-3c40f077779b or 3c71401f-6a96-43c4-b6ed-c51ba2697780.

I am unable to provide you with a reproducible test case because Netlify behaves differently in repeated requests. Here’s a log of what I did:

First I tried a HTTP → HTTPS redirect:

$ curl --user-agent "Googlebot/2.1 (+http://www.google.com/bot.html)" -v http://poedit.net/support/
*   Trying 167.99.242.112...
* TCP_NODELAY set
* Connected to poedit.net (167.99.242.112) port 80 (#0)
> GET /support/ HTTP/1.1
> Host: poedit.net
> User-Agent: Googlebot/2.1 (+http://www.google.com/bot.html)
> Accept: */*
> 
< HTTP/1.1 301 Moved Permanently
< cache-control: public, max-age=0, must-revalidate
< content-length: 43
< content-type: text/plain
< date: Wed, 19 May 2021 14:19:31 GMT
< x-nf-request-id: 60bb1149-00bf-435c-9974-30e9e8a9278d
< location: https://poedit.net/support/
< server: Netlify
< age: 0
< 
Redirecting to https://poedit.net/support/
* Connection #0 to host poedit.net left intact

Then the same with HTTPS, and got a redirect loop back to the requested URL:

$ curl --user-agent "Googlebot/2.1 (+http://www.google.com/bot.html)" -v https://poedit.net/support/
*   Trying 167.99.242.112...
* TCP_NODELAY set
* Connected to poedit.net (167.99.242.112) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/cert.pem
  CApath: none
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-ECDSA-AES256-GCM-SHA384
* ALPN, server accepted to use h2
* Server certificate:
*  subject: CN=*.poedit.net
*  start date: Mar 24 20:37:42 2021 GMT
*  expire date: Jun 22 20:37:42 2021 GMT
*  subjectAltName: host "poedit.net" matched cert's "poedit.net"
*  issuer: C=US; O=Let's Encrypt; CN=R3
*  SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x7f8c20006600)
> GET /support/ HTTP/2
> Host: poedit.net
> User-Agent: Googlebot/2.1 (+http://www.google.com/bot.html)
> Accept: */*
> 
* Connection state changed (MAX_CONCURRENT_STREAMS updated)!
< HTTP/2 301 
< cache-control: public, max-age=0, must-revalidate
< content-length: 43
< content-type: text/plain
< date: Wed, 19 May 2021 14:19:31 GMT
< x-nf-request-id: f7bdb6dc-c836-48ff-80e6-3c40f077779b
< location: https://poedit.net/support/
< server: Netlify
< age: 11
< 
Redirecting to https://poedit.net/support/
* Connection #0 to host poedit.net left intact

And again:

$ curl --user-agent "Googlebot/2.1 (+http://www.google.com/bot.html)" https://poedit.net/support/
Redirecting to https://poedit.net/support/
$

So I tried as non-Googlebot:

curl  https://poedit.net/support/
<!doctype html>
<html lang="en">
<head>
  <meta charset="utf-8">
…etc…etc...

As you can see, there was wrong redirect for Googlebot user agent, but not for curl one.

I tried again, as Googlebot, to confirm, but this time, the problem did not occur:

$ curl --user-agent "Googlebot/2.1 (+http://www.google.com/bot.html)" https://poedit.net/support/ 
<!doctype html>
<html lang="en">
<head>
  <meta charset="utf-8">
…etc…etc...

After that initial successful fetch, I was unable to reproduce even with different URLs from the same domain (ones I know as problematic from Google Search Console).

Eventually, I thought I figured it out: I needed to make a plain HTTP request, followed by HTTPS one, to re-reproduce the bug. But that too isn’t perfectly reliable.

In the process, I noticed that sometimes curl returned data, not redirect, for http:// requests too! E.g. x-nf-request-id: ff83c7b5-5032-47f4-a74f-4609c64e72c3 - but this should never happen.

This leads me to think that something at Netlify changed recently-ish to use some short-lived shared cache for http:// and https:// That’s why it returns self-referencing redirect - it’s the previous http:// response. And that’s why it responds with body instead of redirect to a http:// request - it’s a cached response to the https:// request done shortly before it.

The misbehavior does seem short-lived, on a single-digit minutes TTL. But that’s enough to break Googlebot.

Finally, this is Googlebot-specific, the server does not behave this way for other user agents. That’s really not something that can be explained by DNS or by issues on Googlebot’s end, especially given the reproduction with curl above.

1 Like

@gregraven as mentioned in the other thread, any back links that are made to the site using the non-functioning url aren’t going to credit the site with domain authority. If you’re trying to boost your search ranking you need every backlink you can get. If a site is linked to by the NYT but they use the broken url, that’s some serious SEO juice being lost.

1 Like

@Ultra You’ve completely lost me. You seem to have gone from arguing that Google reports things are wrong to stating that your own URLs are wrong. If the URL is correct and links to a page on Netlify, I’m mystified as to what you think the problem is.

@gregraven I’ll try to make it clear: for many Netlify customers, myself included, search engine ranking is very important. Anything that damages that costs money and is something to be avoided.

2 Likes

@Ultra I’m one of them, but each of my many sites on Netlify are properly indexed by Google, to the best of my ability to verify as such.

@gregraven Do you see errors in Google Search Console?

@Ultra, yes, on some of them.

@gregraven ok, well for many people that’s not an acceptable situation. e.g. if their position on a Google SERP supports their livelihood.

1 Like

@Ultra IMHO, they should be more worried about saying something that speaks truth to power – which Google opposes – rather than whether they are hosting their files on Netlify.

Hi @luke,

FWIW, I’ve managed to reproduce it a couple of times using the ‘User-Agent Switcher and Manager’ Firefox extension (sent the HAR to you via email, also uploaded here: phonolyth.com_Archive [21-05-19 20-10-04].har).
My further attempts at this failed (or succeed, meaning I got HTTP 200 every time), but Google Search Console still reported redirect error…
Until I switched to incognito mode, where I’m now getting infinite 301s all the time.

Normal request & response:

GET / HTTP/2
Host: phonolyth.com
User-Agent: Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate, br
Connection: keep-alive
Cookie: paddlejs_checkout_variant={"inTest":true,"controlGroup":false,"isForced":false,"variant":"multipage-radio-payment"}
Upgrade-Insecure-Requests: 1
Pragma: no-cache
Cache-Control: no-cache
TE: Trailers


HTTP/2 200 OK
cache-control: public, max-age=0, must-revalidate
content-type: text/html; charset=UTF-8
date: Wed, 19 May 2021 13:27:42 GMT
etag: "ec3640e13c4b7fb249af67190553065c-ssl-df"
strict-transport-security: max-age=31536000
vary: Accept-Encoding
server: Netlify
content-encoding: br
x-nf-request-id: 220340c2-f9f5-4cc6-a3b5-c161224d1ee7
age: 7234
content-length: 6407
X-Firefox-Spdy: h2

Incognito request & response:

GET / HTTP/2
Host: phonolyth.com
User-Agent: Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate, br
DNT: 1
Connection: keep-alive
Upgrade-Insecure-Requests: 1
Pragma: no-cache
Cache-Control: no-cache
TE: Trailers


HTTP/2 301 Moved Permanently
cache-control: public, max-age=0, must-revalidate
content-length: 38
content-type: text/plain
date: Wed, 19 May 2021 15:25:41 GMT
x-nf-request-id: f4d36d93-9c14-4d07-bb4f-89bf80732141
location: https://phonolyth.com/
server: Netlify
age: 370
X-Firefox-Spdy: h2

(The only difference I see between requests is DNT vs Cookie header)

So I think @vslavik is onto something…

1 Like

Hi, @why.turov. Thank you for that HAR file in the support ticket! It was exactly what I needed to find the issue. (The x-nf-request-id headers above also would have done the trick and the additional examples are appreciated as well.

I also posted about this here:

I’m also editing my comments above about https://www.redirect-checker.org/index.php. It was giving correct information all along and it was our logging that incorrect information in it.

We will post another update here as soon as we know more.

3 Likes

I’m having a similar issue on my website, www.walkwiltonny.org. Infinite redirect with the google bot. I read another forum where it said it wasn’t possible in the UI to set an A record.

How can I ensure Google can crawl my netlify site?

Hello, @justingrant, @Ultra, @coelmay, @why.turov, @vslavik, @Stacey1845, and anyone else that reads this topic.

The underlying root cause has be resolved and this issue should be fixed now. Would you please reply here if you see any other issues with the redirects?

3 Likes

Hi @luke I’m still facing this issue. Some of the redirects are throwing these errors on the google search console but when I open the link manually it redirects properly. This is significantly reducing my impressions by almost 50%

Hello I am having this problem for my website https://www.marcin-zygan.com I checked it with redirect checker and all seems to be ok . But I cannot get my page indexed at all , I am getting redirect error in Google’s search console and I have no idea how to fix that . Any help much appreciated :slightly_smiling_face: Thank You

Hi @Souloo

I’ve responded here: