Web pages returning "504 Gateway timeout" or "502 Bad gateway" in crawlers like Google Console, Ahrefs and Semrush

[hybrazil]

Hello, I’ve been noticing for about 3 months that all crawlers are having problems searching for URLs hosted on netlify, returning 504 or 502 errors even on online pages. All pages are working and online through my browser.

Does anyone know how to solve this? Because over time the pages are being marked as not indexed by Google

[nathanmartin]

@hybrazil This recent thread provided some proof that this issue may not be on Netlify’s side:
https://answers.netlify.com/t/google-search-console-can-no-longer-index-site/147118

[hrishikesh]

If you’re using prerendering, the first step would be to turn that off from your site settings.