Old HTML delivered to Google Bot

On our netlify hosted website we have an issue that old content is delivered to the google bot on some pages. This can be reproduced by using the User-Agent of the Google Bot for smartphones in the browser via a user agent switcher plugin:

Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) [Chrome/ *W.X.Y.Z* ‡](https://developers.google.com/search/docs/advanced/crawling/overview-google-crawlers?hl=de#user_agent_version) Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)

This is only the case on some pages. There is no issue when visiting with a “normal” User agent - so in all browsers we have the latest version. It would also not be an issue if this was just for five minutes after deployment, but we are now waiting for over a day.

I am not sure if we are missing a setting - since we don’t have this issue on most of our pages, I don’t think we are.

Let me know if you need more information to help us out with this issue.

Chris

@stylesuxx Welcome to the Netlify community.

Please help us help you by writing a good post!

All issues: Provide your Netlify site name (e.g. gifted-antelope-58b104.netlify.app).

DNS or SSL issues: Tell us your custom domain(s) (e.g. mycustomdomain.com), and how long it has been since you made changes to your DNS (DNS changes can take up to 48 hours). We can’t help if we don’t know your domain.

Build issues: Tell us about your workflow, including what SSG you are using.

If you are deploying via git, give us the URL of the repository.

Let us know what you have already tried and what the results were.

The better the post = the faster we can help!

I am deploying via git, it is a private repository though, so I can’t really link you to that.

What I have tried is re-deploying the site and clearing the build cache - it is a nextjs setup.

My workflow is like this: I push to master, netlify build is triggered, build completes successfully.

DNS changes have been made quite a long time ago and don’t seem to be part of the problem. I do not feel comfortable sharing any more information about my setup in public - is it possible to contact support in private?

Instead of relying on some 3rd party UA switcher tools, use Google Search Console’s Live Test feature and check if that’s indeed the case. I won’t think so and Netlify being a CDN doesn’t really check for user agents before serving your files.

Yes, that is how I found out, that there is an issue - Google Search console was complaining. Since I could not see an issue with a regular browser, I changed the UA to see if different UA’s get different content - which seems to be the case.

Sorry for not stating that more clearly in my initial post.

@stylesuxx Just so I’m clear: You’re publishing content to the Internet where anyone could find it, and you are implementing Google Search on your site so people can find it, but you cannot tell us the URLs of your site so we can help troubleshoot this issue?

@gregraven this page is for a customer - I was not sure, when I asked for help here - if he was OK with me publishing the URL, since the page has not yet been officially launched.

I provided (what I though was) enough information to get an answer to my question and this was also the reason why I asked if it was possible to contact support in private.

Anyway - the problem is solved for now. After some time (about two days), the google bot gets the same data as everyone else does, so it seems to me, that there are some UA checks in place before the content is being delivered.

Should the problem re-appear - which I am pretty sure it will - during the next deployments I will come back to you with a URL.