New Prerender Extension - recognized user-agents?

Support
, ,
1

@elad.rosenheim the new Prerendering extension in early access is cool, and I appreciate the increased visibility available via logging, but what’s still unclear to me is which user-agent strings will trigger pre-rendering. In your post, you specify testing with Googlebot, but it would be great to have a comprehensive list available somewhere. As of now, the logs will only trigger to show when there was a prerendering hit, but not a miss, so I’m not entirely sure what user-agents are involved in some of these, but here’s what I’ve tested so far (mostly with my own app so I could see them come through in logs, but also using Alfenpinscher from your test app):

App Result Notes
Apple Messages (macOS 26 & iOS 26.1 beta) Hit user-agent: "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_1) AppleWebKit/601.2.4 (KHTML, like Gecko) Version/9.0.1 Safari/601.2.4 facebookexternalhit/1.1 Facebot Twitterbot/1.0"
Signal (Mac) Hit Interestingly shows up as user-agent: "WhatsApp/2"
Signal (iPhone) Miss
WhatsApp (Mac & iPhone) Miss
Instagram (Web & iPhone) Miss
Facebook Messenger (Web & iPhone) Miss
Facebook Sharing Debugger Miss

As you can see, for our use-cases, which are primarily social apps, it does not pre-render in most cases. It’s worth noting that this is a spot-check, and I would expect the full list to be much larger, and would appreciate some transparency into what that list is, so it can be discussed & feedback (like this) can be provided on it. Thanks!

1 Like
2

Hey!
You’re right that we haven’t published that information yet, and still considering how to make this more visible while being able to make changes and adjustments.

But I can already share with you something that we haven’t fully documented yet (not a secret - just didn’t document yet prior to release):

  • Recently (and this part is documented), our edge functions gained the ability to be triggered only when specific request headers with and values are matched. This is much preferable performance-wise vs. checking for headers inside the function, because unless the headers match, the function is not invoked at all.

  • To make matching by user-agents more practical (and help folks with writing their own edge function to match crawlers and agents, we’ve introduced a new request header available to edge functions: Netlify-Agent-Category. This has a fixed list of values that our edge sets based on the raw user-agent, with a category and optional sub-category, .e.g. browser, ai-agent;user, page-preview;general, crawler;ai, crawler;seo, crawler;general, crawler;social and more.

So the mapping we have is actually between user-agent strings and categories, and then the prerender “middleware” matches by category. Its actual matching is on ai-agent|page-preview|crawler categories (with any or no sub-category).

The full list of categories will surely be public. I’m not sure yet about the full list of values in all categories.

But I have an ask: can you share here (or in DM) a selection of user-agents that you expect to be pre-rendered?

1 Like
3

And while you’re here, check this new tool out: https://do-you-need-prerender.netlify.app/
It should give you an honest assessment of whether a given website should or shouldn’t use pre-rendering. (If it shouldn’t but pre-rendering is enabled, that only adds complexity).

1 Like
4

I’d expect at least the list that the old pre-rendering responded to (not sure if this is the most up-to-date list, but it generally worked in all cases we expected it to - we just are having problems with random 502s which is why we’re excited about this updated offering).

Cool! As expected, it recommends pre-rendering for our site

5

I’m on it, I see that some user-agents are missing in our new lists.
Can you re-check the Facebook Sharing Debugger? this should already work ok (and I’ve tested it)

6

Really? I see HTTP 206 Partial Content for your test app with pre-rendering enabled which indicates it’s only getting the partial version (for whatever reason, the “See exactly what our scraper sees for your URL” link isn’t working - it just shows “An error occured while fetching content” - so I don’t know what HTML is being returned, but I typically see 200 for pre-rendered pages on my production site using the old pre-rendering feature).

I also get a 206, and don’t see it show up in my Edge Function logs when I test one of my app’s URLs, which indicates to me that it isn’t triggering pre-rendering :thinking:

7

The status code 206 is because the share debugger is sending a Range request header, so Netlify responds with “here is the partial range you requested”. The page however is rendered in full.
As for why you don’t see it in the logs - I was able to see the edge function being called after clearing the prerender cache (in the extension settings), but I still don’t see the number of invocations I’d expect, so looking further.

8

I can’t rule out some correlation between the Range header and the caching behavior. However, I do see that most (>80%) of requests by this FB crawler don’t request a range.
I’d update you once I have more insight on this, and/or when we have updated user-agent mapping :slight_smile:

9

I know you’re investigating further, but fwiw, I just tried clearing the prerender cache and using the Sharing Debugger again, and I still don’t see it in my Edge Function logs

10

Ok,
So it turns out that the FB scraper sometimes sends an Accept header of */* and sometimes a more specific text/... values.
The extension, unlike the older service, was configured to look for concrete values rather than a catch-all */* to avoid some dubious cases, but seems that this is required. Please trigger a re-deploy of your test project to get the updated extension code.

Updating of user-agent mapping to include more values is in progress.

2 Likes
11

@swrobel we’ve updated the user-agents list. There are a few more minor agents we’d want to capture correctly, but all the relevant ones should work.
Tested with:

  • Facebook Sharing Debugger
  • Facebook (web)
  • WhatsApp on web and Android
  • Signal on Android

Note that the middleware would look at the Accept header and if it has a value but doesn’t include any of */*, text/html or text/plain, it will log a message to your edge function logs and skip pre-rendering. This is to avoid some past cases where crawlers & bots asked for images/JS/etc. which then went through pre-rendering

12

*Edit” My fault, i only just moved to netlify for hosting, I needed to add the redirect: “public/_redirects file”

I installed this extension yesterday, following the instructions. I have success with the homepage of my site, it is working correctly with: X-Prerendered - True, however its not working on all the other pages on the site: browser gives: Prerender failed.

am i missing some configuration? or do i need to change anything for all pages to get prerendered?

Thanks, this extensions shows a lot of promise.

13

ok, if you need us to look at specific pages that may fail pre-rendering, do tell!
Also note that in the extension settings you can turn on detailed reporting, which lists all requests that were completed/skipped/failed as the headless browser rendered the page.

This is usually a good way to check for resources that are slow to load and thus lead to timeouts - you can choose to skip any such resources that don’t actually affect the HTML body (which is what the crawlers are interested in mostly). In some cases, you might see some requests stay in “pending” state, which means that the function timed-out while these requests were still in flights. This might be due to very long API calls, or sometimes websocket requests that were not skipped by default.

14

I don’t want to open up a new topic, but I was wondering if it is possible to somehow use this extension (or basically to install the prerender extension functions) without having the project linked to a repository.
We build on another machines (Flutter) and we remotely upload the files to Netlify - with prerendering being the main reason we switched to Netlify.

16

Hi sorry if it is not directly related. I am testing ChatGPT and Perplexity and neither were able to get data directly from the example provided https://dogs-explorer-prerender.netlify.app/

image
image1394×694 32.6 KB

Also when I reviewed the source code on the browser it is not displaying the whole HTML.
I can only post one screenshot.

Update: I guess I have a hint. I have tested with curl:
curl -A googlebot https://dogs-explorer-prerender.netlify.app/
The content is displayed, but it is not working with GPTBot
curl -A GPTBot https://dogs-explorer-prerender.netlify.app/

Then could be that the example don’t have that?

17

(post deleted by author)

18

(post deleted by author)

19

Hey there! I made a little PS script to check if the prerenderer works with some known user-agents:

# =======================================

# PRERENDER TEST SCRIPT

# =======================================

$URL = "https://synkroai.app"




$UserAgents = @{

    "Googlebot"            = "Googlebot/2.1 (+http://www.google.com/bot.html)"

    "Googlebot-Mobile"     = "Mozilla/5.0 (Linux; Android 6.0; Nexus 5X) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.96 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"

    "Bingbot"              = "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"

    "DuckDuckBot"          = "DuckDuckBot/1.0; (+http://duckduckgo.com/duckduckbot)"

    "YandexBot"            = "Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)"

    "GPTBot"               = "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.0; +https://openai.com/gptbot)"

    "ClaudeBot"            = "Mozilla/5.0 (compatible; ClaudeBot/1.0; +https://anthropic.com/bot)"

    "PerplexityBot"        = "Mozilla/5.0 (compatible; PerplexityBot/1.0; +https://perplexity.ai/bot)"

    "Facebook"             = "facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php)"

    "Twitter"              = "Twitterbot/1.0"

    "LinkedIn"             = "Mozilla/5.0 (compatible; LinkedInBot/1.0; +http://www.linkedin.com)"

    "Slackbot"             = "Slackbot-LinkExpanding 1.0 (+https://api.slack.com/robots)"

    "Discord"              = "Mozilla/5.0 (compatible; Discordbot/2.0; +https://discordapp.com)"

    "Normal Browser"       = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 Chrome/120 Safari/537.36"

}




Write-Host "======================================="

Write-Host " SYNKRO PRERENDER TEST"

Write-Host " Testing: $URL"

Write-Host "=======================================`n"




foreach ($agent in $UserAgents.Keys) {

    $ua = $UserAgents[$agent]

    

    try {

        $body = curl.exe -A "$ua" -s "$URL" 2>&1

        

        if ($body -is [array]) {

            $body = $body -join "`n"

        }

        

        $length = $body.Length

        

        if ($length -gt 15000) {

            Write-Host "[$agent]  [OK] PRERENDERED ($length bytes)" -ForegroundColor Green

        } else {

            Write-Host "[$agent]  [FAIL] SPA ONLY ($length bytes)" -ForegroundColor Yellow

        }

    } catch {

        Write-Host "[$agent]  [ERROR] Failed to fetch: $_" -ForegroundColor Red

    }

}




Write-Host "`nDone!"

Seems that the extension is catching these prerender requests correctly, according to the edge function logs.

Console logs also imply this is working:

PS C:\Programs\Synkro> .\scripts\test-prerender.ps1

SYNKRO PRERENDER TEST
Testing: https://synkroai.app

[GPTBot]  [OK] PRERENDERED (39336 bytes)
[ClaudeBot]  [OK] PRERENDERED (39336 bytes)
[PerplexityBot]  [OK] PRERENDERED (39336 bytes)
[YandexBot]  [OK] PRERENDERED (39336 bytes)
[Twitter]  [OK] PRERENDERED (39336 bytes)
[Googlebot-Mobile]  [OK] PRERENDERED (39336 bytes)
[Googlebot]  [OK] PRERENDERED (39336 bytes)
[Facebook]  [OK] PRERENDERED (39336 bytes)
[Normal Browser]  [FAIL] SPA ONLY (3072 bytes)
[Bingbot]  [OK] PRERENDERED (39336 bytes)
[LinkedIn]  [OK] PRERENDERED (39336 bytes)
[DuckDuckBot]  [OK] PRERENDERED (39336 bytes)
[Discord]  [OK] PRERENDERED (39336 bytes)
[Slackbot]  [OK] PRERENDERED (39336 bytes)

Done!
PS C:\Programs\Synkro>

However, I’m not sure if the extension considers any non-default user-agent to be a crawler, so take this with a grain of salt!