It would be great to use redirect rules to redirect customers to different sites based on their region:
- Canada customers to site.ca
- Everyone else to site.com
However, there is a big flaw that prevents us from using it: From what I can tell from the documentation, it doesn’t consider crawler bots like Googlebot, which will identify as coming from the USA.
Since it doesn’t ignore Googlebot and other crawlers, it will prevent Google from indexing the site.ca version. This also goes against best practices.
See Creating the Right Homepage for your International Users | Google Search Central Blog | Google Developers
I might be misunderstanding something, but this is the way we have implemented it earlier. What is your perspective on this? Something you have considered?
Hey Fred,
Welcome to our Forums! After a bit of searching, it looks like Googlebot’s docs mentions the use IPs outside of the US to crawl sites and that they want you to treat their US based crawlers as if they are American:
Googlebot crawls with IP addresses based outside the USA, in addition to the US-based IP addresses.
As we have always recommended, when Googlebot appears to come from a certain country, treat it like you would treat any other user from that country. This means that if you block USA-based users from accessing your content, but allow visitors from Australia to see it, your server should block a Googlebot that appears to be coming from the USA, but allow access to a Googlebot that appears to come from Australia.
If you’d like to specify your locale based pages, you can use our geo-based redirects. As for notifying Googlebot, it looks like you can signify an alternate language website by using HTML tags, sitemaps or HTTP headers. Upon a closer look, Netlify and Googlebot uses the same ISO standards to determine country/language, although we both have our quirks (see the attached urls). This should make it a bit easier to set up matching headers/redirects.
Our latest feature Edge Functions may also be an adequate solution for this use case. Here’s an example that might be useful to you: Serve localized content | Edge Functions on Netlify
2 Likes