Request for additional prerender user-agent strings (specifically for AI models)

Support
1

We use prerender.io for our site and are very happy with it and you working together with them.

Today they sent out an email about how prerendering helps with AI visibility (an important aspect in the travel space) so we checked which user agent strings triggered prerendering. Many of the important ones were merrily sent to prerender, but several significant ones were not. I checked in Claude to test and it complained about not being able to read our React SPA. ChatGPT had no issue with the site because its requests are sent to prerender.io.

Prerender.io sent us here: Recommended AI Crawler User Agents for Prerender.io and Netlify’s bot sent us here: [Support Guide] Understanding and debugging prerendering - #113 by hrishikesh That last one is 5 years old, but was closed not 3 weeks ago :sadface:

So, our request is: could you add the following AI-related bots to your user-agent strings that trigger prerendering? I’ve included the full strings as well for reference.

  • Anthropic-ai
    Mozilla/5.0 (compatible; anthropic-ai/1.0; +http://www.anthropic.com/bot.html)
  • ClaudeBot
    Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ClaudeBot/1.0; +claudebot@anthropic.com)
  • Claude-web
    Mozilla/5.0 (compatible; claude-web/1.0; +http://www.anthropic.com/bot.html)
  • Google-Extended
    Mozilla/5.0 (compatible; Google-Extended/1.0; +http://www.google.com/bot.html)
  • Amazonbot
    Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/600.2.5 (KHTML, like Gecko) Version/8.0.2 Safari/600.2.5 (Amazonbot/0.1; +https://developer.amazon.com/support/amazonbot)
  • Meta-ExternalAgent
    meta-externalagent/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler)
  • Meta-ExternalFetcher
    meta-externalfetcher/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler)
  • CCBot
    Mozilla/5.0 (compatible; CCBot/1.0; +http://www.commoncrawl.org/bot.html)
2

I’ll look into this, but it could take some time as I’d have to generate some reports for each user agent to make a case about this.

3

I have faith in you :smiley:

5

We are also interested in getting the recommended AI crawler user agents added as additional bots to the netlify integration.

6

What do you mean by Netlify Integration?

7

I mean that we enabled prerendering in our Netlify project configuration with an external service (prerender.io). And currently the Netlify CDN software that determs if a request is a crawler bot doesn’t seem to detect AI crawlers.

8

The above is already shipped, so those user agents should invoke a request to prerender.io.

9

Oh, okay. There wasn’t any final feedback wether this was implemented and the docs don’t include the crawlers that are detected. Is there a list of supported crawlers I can finde somewhere?

10

Ah, good to hear that this has been implemented!

And yes, what Hannes said: it would be nice to have a list of the current crawler substrings that are supported, like you guys did on the previous post ( [Support Guide] Understanding and debugging prerendering - #115 by fool )

11

The current list of user agents is:

"baiduspider",
"twitterbot",
"facebookexternalhit",
"facebot",
"rogerbot",
"linkedinbot",
"embedly",
"quora link preview",
"showyoubot",
"SocialFlow",
"Net::Curl::Simple",
"Snipcart",
"Googlebot",
"outbrain",
"pinterestbot",
"pinterest/0",
"slackbot",
"vkShare",
"W3C_Validator",
"redditbot",
"Mediapartners-Google",
"AdsBot-Google",
"parsely",
"DuckDuckBot",
"whatsapp",
"Hatena",
"Screaming Frog SEO Spider",
"bingbot",
"Sajaribot",
"DashLinkPreviews",
"Discordbot",
"RankSonicBot",
"lyticsbot",
"YandexBot/",
"YandexWebmaster/",
"naytev-url-scraper",
"newspicksbot/",
"Swiftbot/",
"mattermost",
"Applebot/",
"snapchat",
"viber",
"proximic",
"iframely/",
"upday",
"Google Web Preview",
"AhrefsBot/",
"AhrefsSiteAudit/",
"GoogleSites",
"PetalBot",
"Taboolabot/",
"Google-InspectionTool/",
"trueAnthem/",
"Mattermost-Bot/",
"MicrosoftPreview/",
"ZoomBot",
"Zendesk/External-Content",
"Discourse Forum Onebox v",
"Mastodon/",
"SiteAuditBot/",
"SemrushBot-BA",
"SemrushBot-SI/",
"SemrushBot-SWA/",
"SemrushBot-OCOB/",
"gptbot",
"chatgpt-user",
"oai-searchbot",
"perplexitybot",
"claudebot",
"dotbot",
"zoombot",
"petalbot",
"anthropic-ai/",
"ClaudeBot/",
"claude-web/",
"Google-Extended/",
"Amazonbot/",
"meta-externalagent/",
"meta-externalfetcher/",
"CCBot/",