Google Search Console can no longer index site

Support
,
1

Hello,

My site: https://www.shyeditor.com/
My site netlify url: https://shyeditor-landing.netlify.app/

Today I encountered an issues where Google is suddenly unable to index my website. Attempting to access any page from the Google Search console results in an error “Failed: Robots.txt unreachable” and “Page cannot be indexed: Not available due to a site-wide issue”.
This worked perfectly yesterday and for the past 2 years at least, and I made no changes to either robots.txt or site access. The site link above is the most urgent example, but this is the case with all my sites deploted on netlify. Please help.

2

Netlify had an issue today: Netlify Status - Elevated API Latency and Error Rates Starting at 11:27 UTC. So if Google tried to visit your website during that time, chances are it might have failed.

3

I requested reindexing right now and the search console is still showing the same issue.

4

Google still cannot access my sites so now Google ads are disabled as well.

5

This is the report for the past 2 days (timings in UTC). Netlify has served all requests with a 200. If Google is reporting errors, it’s on Google’s end, not Netlify’s.

datetime status url ua
2025-05-08 13:02:01.723 200 https://www.shyeditor.com/robots.txt Googlebot-Image/1.0
2025-05-08 12:51:42.940 200 https://www.shyeditor.com/robots.txt Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
2025-05-08 11:48:59.290 200 https://www.shyeditor.com/robots.txt Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
2025-05-08 11:47:57.446 200 https://www.shyeditor.com/robots.txt Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
2025-05-08 11:45:19.262 200 https://www.shyeditor.com/robots.txt Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
2025-05-08 11:15:37.796 200 https://www.shyeditor.com/robots.txt Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
2025-05-08 05:57:46.138 200 https://www.shyeditor.com/robots.txt Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
2025-05-07 23:46:03.373 200 https://www.shyeditor.com/robots.txt Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.0.0 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
2025-05-07 23:45:53.149 200 https://www.shyeditor.com/robots.txt Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Googlebot/2.1; +http://www.google.com/bot.html) Chrome/86.0.0.0 Safari/537.36
2025-05-07 22:16:25.597 200 https://www.shyeditor.com/robots.txt Googlebot/2.1 (+http://www.googlebot.com/bot.html)
2025-05-07 22:16:25.038 200 https://www.shyeditor.com/robots.txt Googlebot/2.1 (+http://www.googlebot.com/bot.html)
2025-05-07 22:16:24.780 200 https://www.shyeditor.com/robots.txt Googlebot/2.1 (+http://www.googlebot.com/bot.html)
2025-05-07 11:50:39.374 200 https://www.shyeditor.com/robots.txt Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
2025-05-07 10:40:15.418 200 https://www.shyeditor.com/robots.txt Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
2025-05-07 08:23:09.711 200 https://www.shyeditor.com/robots.txt Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
2025-05-07 07:18:18.225 200 https://www.shyeditor.com/robots.txt Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
2025-05-07 05:24:34.023 200 https://www.shyeditor.com/robots.txt Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
2025-05-07 04:17:13.326 200 https://www.shyeditor.com/robots.txt Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
2025-05-07 03:05:02.055 200 https://www.shyeditor.com/robots.txt Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
2025-05-07 03:02:28.697 200 https://www.shyeditor.com/robots.txt Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
2025-05-07 02:00:53.422 200 https://www.shyeditor.com/robots.txt Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
2025-05-07 00:51:13.321 200 https://www.shyeditor.com/robots.txt Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
2025-05-06 23:40:49.482 200 https://www.shyeditor.com/robots.txt Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
2025-05-06 22:34:50.955 200 https://www.shyeditor.com/robots.txt Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
2025-05-06 21:10:27.429 200 https://www.shyeditor.com/robots.txt Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
2025-05-06 21:10:25.409 200 https://www.shyeditor.com/robots.txt Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
2025-05-06 20:04:44.138 200 https://www.shyeditor.com/robots.txt Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
2025-05-06 18:23:19.539 200 https://www.shyeditor.com/robots.txt Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
2025-05-06 16:20:05.850 200 https://www.shyeditor.com/robots.txt Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
2025-05-06 14:48:33.462 200 https://www.shyeditor.com/robots.txt Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
2025-05-06 13:26:43.893 200 https://www.shyeditor.com/robots.txt Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
7

We are having the same issue with our site. Site is online, but Google crawlers getting lots of 504 and 502 Bad Gateway errors. I’m replicating it in Chrome DevTools changing the user agent.

@amedvyedyev It seems your site is working fine again, right? I tested it and couldn’t get an error. Did you do anything to fix it?

@hrishikesh In our case, robots.txt loads, but not any regular page. Our site is wild-rider.com, built with Gatsby.

This is causing Google to disable our ads because they cannot reach our links, and they say it’s on our side.

Any clues??

Update: After disabling Netlify’s prerendering, things are back to normal. But does this mean that we won’t be able to use prerendering again?

9

@davidvm yes, prerendering is more or less going away.