Google Ad Crawler Failing With 504

Netlify site: https://monthly.com

Hello,

Every so often our Google Ads get rejected because our Netlify site returns 504 errors to their ad crawler.

Are you able to look in our logs to help us debug/resolve this issue? It’s been ongoing and has actually occurred during critical ad campaigns in the past.

This is the Google Ad policy that fails: Destination requirements - Advertising Policies Help

This is what Google tells us when we get notified:

Policy:	Destination not working
HTTP error:504, Platform: Desktop

Thanks so much,
Anthony

Hi @amorriscode,

I checked the logs and looks like that status code was received only by the bots like Bing, Google, Facebook. So, I checked and found out you’re using pre-rendering. Could you try to disable that and see if the problem persists?

@hrishikesh Can you provide a little more detail as to why you think that would resolve the problem? Shouldn’t pre-rendering make things load faster for the bots? Why would pre-rendering return 502?

Pre-rendered content is stored and treated differently than normal Netlify website cache. For example, the primary difference is that, the pre-rendered data is stored for 48 hours unlike the normal cache that’s invalidated as soon as you publish a new build - so sometimes bots continue to see outdated content.

I’m not saying it would solve the problem, it’s just that it might. I think it might because the 504 is only received by the user agents used by bots - no real user traffic for your site got 504 error code. Also, Netlify is a static file CDN + you’re not using Server Side Rendering - in this scenario, there’s no reason for a website to get 5xx errors as those are ‘server-side’ errors, but Netlify is merely sending the file requested by the browser back to it without any kind of processing.

Pre-rendering doesn’t make your website faster for bots (at least not significantly) - it just renders the JavaScript which makes the website ‘accessible’ to bots. Now-a-days, a few frameworks, use JavaScript-only routing and this might make it difficult for bots to see those pages if they don’t parse the JS. So, pre-rendering would keep a JS-parsed copy of the page ready to serve it to user user agents. So if you’re using static HTML pages, you don’t even need pre-rendering.

Thanks for all that info @hrishikesh.

I agree that it’s weird these bots are seeing 5XX but they are also in your logs, correct? Is it possible this is a larger Netlify pre-render bug? As you said, why is a ‘server-side’ error being surfaced by Netlify at all?

There are a few errors reported in pre-rendering by our automated tools, yes. Can’t call it a bug at the moment as the numbers are pretty normal.

Pre-rendering errors can be caused due to multiple errors - sometimes it’s a user error (something wrong with the code), sometimes the HTML file is too big and to store it in the cache, it takes more time than the client is ready to wait and so on and sometimes, our service might be degraded.

This is what I can see from our logs:

Sure, there are 504s, but compared to 200s it’s just 1%! So ~98%+ of the times, we’re serving the correct data. It would be very very hard to pin point why it failed when it did.

Did a recent change or something cause this?