# \[Support Guide\] Understanding and debugging prerendering

**URL:** <https://answers.netlify.com/t/support-guide-understanding-and-debugging-prerendering/150>\
**Category:** Support Guides\
**Tags:** pre-rendering, support-guide, netlify-cdn-adn\
**Created:** [March 8, 2019, 2:49pm UTC](https://answers.netlify.com/t/support-guide-understanding-and-debugging-prerendering/150 "2019-03-08T14:49:11Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![fool](https://sea1.discourse-cdn.com/netlify/user_avatar/answers.netlify.com/fool/32/37_2.png) [@fool](https://answers.netlify.com/u/fool)\
**Post date:** [March 8, 2019, 2:49pm UTC](https://answers.netlify.com/t/support-guide-understanding-and-debugging-prerendering/150/1 "2019-03-08T14:49:11Z")

</div>

_Last reviewed by Netlify Support in July 2025_

With the advent of [modern web hosting aka the JAMStack](https://jamstack.org), some robots and crawlers need “help” navigating client-side links. This problem first presented itself initially when navigation was implemented using Javascript instead of HTML, but these days the [OpenGraph standard](https://ogp.me/) and [unfurling](https://medium.com/slack-developer-blog/everything-you-ever-wanted-to-know-about-unfurling-but-were-afraid-to-ask-or-how-to-make-your-e64b4bb9254)[by social sharing services](https://medium.com/slack-developer-blog/everything-you-ever-wanted-to-know-about-unfurling-but-were-afraid-to-ask-or-how-to-make-your-e64b4bb9254) seems to be in many cases more important to our customers. (For more on this, see a further reading link we have added to the bottom of this post.) Prerendering is one solution to this problem.

Netlify provides a prerendering service [described in some detail in this article](https://docs.netlify.com/site-deploys/post-processing/prerendering/). This post aims to help if it doesn’t “just work” and you’d like to debug or understand how to test.

## Which requests are prerendered?

Prerendered content is only displayed in certain situations. Specifically it:

- will only be used for a Netlify website once prerendering is enabled on your site’s build & deploy settings page
- will only be returned in response to a web request when there is a [User-Agent](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/User-Agent) sent with the request that needs this special content, or a special URL is used (see below for more details around manual testing and the special URL).

There are a few dozen user agents that a request can present that will cause a prerendered response, for instance `Twitterbot` and `facebookexternalhit/1.0` . All crawlers that we are aware of which need prerendering receive this treatment automatically, but if yours seems to be missing, please mention it below so we can potentially add it to the list!

## Testing the service with your content

To test things out manually, you can also request a prerendered URL - see particularly the part about `_escaped_fragment_` in [our prerendering documentation](https://docs.netlify.com/site-deploys/post-processing/prerendering/).

So, to test prerendering, this is what I do. You can use any `User-Agent` that we prerender for, so I use `twitterbot` in this example.

`curl -A twitterbot https://www.sitename.com/`

or

`curl https://www.sitename.com/?_escaped_fragment_=`

This should show different output than not using a special User Agent or the `?_escaped_fragment_=` style URL, which will demonstrate that the service is in effect.

## Some caveats about the prerender service

1. Most websites using modern frameworks like next.js do not require prerendering these days. Please be sure it is needed before you configure it, since it will use more bandwidth to render sites for crawlers/bots and you’d want to make sure that is a good use of your money!

2. We cache the prerendered content for 24 to 48 hours, so you will want to make sure to test with different URLs each time if you are making changes and republishing a lot while you get things dialed in. Most importantly, this cache is **NOT** invalidated by a new deploy of your site!

3. You may also need to use `window.prerenderReady` in your javascript to indicate to our service when it is safe to consider the page fully rendered to take the snapshot for the services that need it, as mentioned [in the prerender.io docs here](https://docs.prerender.io/docs/11-best-practices). If this isn’t set, and we haven’t already determined that the page has loaded completely, we take a snapshot after 10 seconds regardless of the state of the page.

4. When testing with Google Webmaster tools, please be aware that there can be an issue using the "Fetch as Google" functionality. This tool does not check for the fragment meta tag and re-request the page with the ?_escaped\_fragment_= query parameter as it does in production: Googlebot itself does not have that issue. So, when using "Fetch as Google" you’ll need to append the `?_escaped_fragment_` query parameter on the end of the URL that you are testing, and that should show your prerendered page. If that works and if you have the following code snippet in the `<head>` of your HTML, then the real Googlebot will work just fine.

5. Another [astute customer discovered](https://answers.netlify.com/t/support-guide-understanding-and-debugging-prerendering/150/48) that some services, such as WhatsApp, [Facebook](https://developers.facebook.com/docs/sharing/webmasters/crawler/), and [Google](https://x.com/glenngabe/status/892844663778013184), only download the first N bytes of your html - so if you have a ton of stuff before those meta tags - it could be “missing” the parts that it needs to prerender correctly. That “astute customer discovered” link will show you one solution to this situation if you have it. You’ll see this as “twitter works correctly, whatsapp doesn’t” on the same URL to help narrow down your troubleshooting.

## Next Steps

That’s a lot of advice, but it’s possible that it still won’t resolve your issue. So what can you do next if your content doesn’t seem to be rendering correctly after enabling the prerender service? After checking out the below caveats to make sure it’s not related to one of those, here’s a bit more advice:

First, be aware that **your code** needs to provide the OpenGraph tags that are needed for social sharing services. A lot of frameworks and plugins (e.g. react-helmet) handle setting them for you, but “a wrong og tag” is generally not the fault of the prerender service but rather your code. You can examine the source code that we serve as described above to debug what is happening.

If instead you’re seeing errors (HTTP 5xx responses), then it is likely that your code is not rendering at all. Since we don’t expose prerender logs to customers, using a local copy of [Prerender.io’s prerender code](https://github.com/netlify/prerender) and following[the local-use instructions in its readme](https://github.com/netlify/prerender?tab=readme-ov-file#running-locally) is the best way to debug those issues in depth. This will show any syntax errors that are encountered as well as letting you examine the output including any OpenGraph tags.

Two more pieces of advice on sharing content on social services from folks not on Netlify’s tech Support team:

1. A customer wrote up his journey in [great detail with a lot of practical advice in this medium article here](https://medium.com/@corysmc/angular-seo-and-pre-rendering-with-netlify-c68b86a2ffd1) - it explains a lot about the world of rendering and unfurling and testing using third party tools. **If you’re struggling to understand why prerendering is needed and how prerendering works with social sharing articles, this is a great starting place.**
2. One of our Developer experience engineers @cassidoo recently wrote up some great implementation details which while tailored for next.js are very illustrative and generally informative for people working on their SEO/sharing setups. You can find [her amazing blog post here.](https://www.netlify.com/blog/2020/05/08/improve-your-seo-and-social-sharing-cards-with-next.js/) (Edit: Please note that in 2024 and beyond, and due to changes in the framework itself, most Next.js sites do _not_ require a prerendering service.)

---

<div class="post-metadata">

**Author:** ![bryancolligan](https://avatars.discourse-cdn.com/v4/letter/b/3ec8ea/32.png) [@bryancolligan](https://answers.netlify.com/u/bryancolligan)\
**Post date:** [May 18, 2019, 3:00pm UTC](https://answers.netlify.com/t/support-guide-understanding-and-debugging-prerendering/150/3 "2019-05-18T15:00:57Z")

</div>

This no longer works for the facebookexternalhit/1.1 useragent, or bing.  
It currently works for Googlebot and the Twitterbot/1.0 useragent.

Please help bring this back.

---

<div class="post-metadata">

**Author:** ![perry](https://sea1.discourse-cdn.com/netlify/user_avatar/answers.netlify.com/perry/32/16_2.png) [@perry](https://answers.netlify.com/u/perry)\
**Post date:** [May 20, 2019, 4:36pm UTC](https://answers.netlify.com/t/support-guide-understanding-and-debugging-prerendering/150/4 "2019-05-20T16:36:21Z")

</div>

Hi bryan, I’ve earmarked this for investigation.  
We’ll update this thread once we’ve looked into it!

---

<div class="post-metadata">

**Author:** ![alexisdray](https://sea1.discourse-cdn.com/netlify/user_avatar/answers.netlify.com/alexisdray/32/821_2.png) [@alexisdray](https://answers.netlify.com/u/alexisdray)\
**Post date:** [May 21, 2019, 11:30pm UTC](https://answers.netlify.com/t/support-guide-understanding-and-debugging-prerendering/150/5 "2019-05-21T23:30:59Z")

</div>

hi guys, i’m not sure if bryan is talking about the same thing, but your managed version of prerender seemd to suffer from this bug. (i’ve seen the same issue when examining as googlebot emulator) and workaround described in this issue also brings back facebook

> <https://github.com/prerender/prerender/issues/563>
>
> We've noticed an issue where the start of pre-rendered output string is of the f…ollowing format:
> 
> \`html\<html lang="en" \`
> 
> This is causing Facebook's scraper to not scrape properly, which makes sense as it's not valid html.
> 
> Looking through the source code, I notice in chrome.js the following lines (536-545):
> 
> \`\`\`
> }).then((response) =\> {
> 
> let doctype = '';
> if (response && response.result && response.result.description) {
> doctype = response.result.description;
> }
> 
> tab.prerender.content = doctype + tab.prerender.content;
> clearTimeout(parseTimeout);
> resolve();
> \`\`\`
> 
> Can you explain why the doctype is being appended like this as a string? How are we supposed to consume the \`content\` output of the prerender object?
> 
> Thanks

this is fixed in prerender 5.5.1

edit: netlify are already tracking this here:

> <https://github.com/netlify/prerender/issues/20>
>
> Hi,
> 
> I'm experiencing an issue where the \`\<!DOCTYPE html\>\` element is misinter…preted by the prerenderering engine and turned into a rouge string simply saying \`html\` and injected into the beginning of the html document. 
> 
> Anyone knows what is causing this? 
> 
> For example \`http://localhost:3000/https://www.netlify.com\`
> 
> yields
> 
> \`\`\`
> html\<html lang="en"\>\<head\>
> \<meta name="generator" content="Hugo 0.36.1"\>
> \<meta charset="utf-8"\>
> ...
> \`\`\`
> 
> When testing with my own sites locally everything is fine if I remove the \`\<!DOCTYPE html\>\` declaration in my html file.
> 
> The rouge string makes the html of my page malformed so that the Facebook crawler is tripped up, which is the reason I'm using prerendering in the first place.

---

<div class="post-metadata">

**Author:** ![futuregerald](https://sea1.discourse-cdn.com/netlify/user_avatar/answers.netlify.com/futuregerald/32/33_2.png) [@futuregerald](https://answers.netlify.com/u/futuregerald)\
**Post date:** [May 23, 2019, 5:59pm UTC](https://answers.netlify.com/t/support-guide-understanding-and-debugging-prerendering/150/6 "2019-05-23T17:59:21Z")

</div>

Hi @alexisdray,

Yea, we’re a few versions behind, but we definitely want to get our prerender service updated. Thanks for letting us know!

---

<div class="post-metadata">

**Author:** ![Yunikorn](https://avatars.discourse-cdn.com/v4/letter/y/59ef9b/32.png) [@Yunikorn](https://answers.netlify.com/u/Yunikorn)\
**Post date:** [May 27, 2019, 7:24am UTC](https://answers.netlify.com/t/support-guide-understanding-and-debugging-prerendering/150/7 "2019-05-27T07:24:04Z")

</div>

Another caveat about enabling Prerendering is that OG tags may not be read by Facebook crawler. It will say something like “the meta tags are in the body” because the HTML is messed up and FB crawler cannot identify the HEAD.

---

<div class="post-metadata">

**Author:** ![shane](https://avatars.discourse-cdn.com/v4/letter/s/dbc845/32.png) [@shane](https://answers.netlify.com/u/shane)\
**Post date:** [June 6, 2019, 3:08pm UTC](https://answers.netlify.com/t/support-guide-understanding-and-debugging-prerendering/150/8 "2019-06-06T15:08:30Z")

</div>

We are also experiencing this issue with Facebook. The tested curl results are suffering from the html string prepending the markup and when you look at the scrapped results from Facebook debugger its omitting the head completely and placing all the meta data/tags within the body. Therefore not finding any of the open graph tags.

Here is an example from the “what our scrapper sees” on Facebook Debugger:

```auto
<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN" "http://www.w3.org/TR/REC-html40/loose.dtd">
<html><body>
<p>html</p>
<meta http-equiv="content-type" content="text/html; charset=utf-8">
<meta charset="utf-8">

```

I’m Assuming by the funky p tagged html this is all stemming from the same issue.

However, I did find this response post within the react-helmet issues:

> <https://github.com/nfl/react-helmet/issues/26#issuecomment-474846173>
>
> Hi there, I've just started using Helmet to implement meta tags, so apps like Fa…cebook and Twitter can get some useful information about my page and put it into previews. However, I notice that there is a delay between when I render a page, and when the meta tags/title get set on the page. This seems to be causing Facebook and Twitter to miss the tags, as they are not present at the instant of page load. I'm wondering if I am doing something wrong, or there is a fix I am not aware of. 
> 
> I am currently using the recommended server-side technique - IE
> 
> \`\`\`
> str = React.renderToString(componentwithhelmet);
> Helmet.rewind();
> res.send(str)
> \`\`\`
> 
> I'm using express. And I have confirmed that when I hard code in the description (that is, it is helmet-independent), FB picks up the data. 
> 
> I really like the package, but if it doesn't take care of this kind of thing, I am afraid I'll have to roll my own architecture for tags like these. Any insight would be appreciated! 
> 
> Thanks,
> Sam

Stating that a GTM trigger was causing an iframe to be injected after the html tag and breaking things. We are using Helmet and GTM so I might dig a little on that, but thought I’d mention incase it sparked any ideas/thoughts.

Otherwise any idea on the status of an update/fixes for pre-rendering?

---

<div class="post-metadata">

**Author:** ![fool](https://sea1.discourse-cdn.com/netlify/user_avatar/answers.netlify.com/fool/32/37_2.png) [@fool](https://answers.netlify.com/u/fool)\
**Post date:** [June 7, 2019, 10:22pm UTC](https://answers.netlify.com/t/support-guide-understanding-and-debugging-prerendering/150/9 "2019-06-07T22:22:28Z")

</div>

Hey folks, good news! We shipped that fix for the rogue string `html` before your code yesterday at about 5pm Pacific and it has been running smoothly since.

---

<div class="post-metadata">

**Author:** ![shane](https://avatars.discourse-cdn.com/v4/letter/s/dbc845/32.png) [@shane](https://answers.netlify.com/u/shane)\
**Post date:** [June 8, 2019, 4:09am UTC](https://answers.netlify.com/t/support-guide-understanding-and-debugging-prerendering/150/10 "2019-06-08T04:09:33Z")

</div>

Netlify is Netli-fly 🕶

---

<div class="post-metadata">

**Author:** ![markasoftware](https://sea1.discourse-cdn.com/netlify/user_avatar/answers.netlify.com/markasoftware/32/1269_2.png) [@markasoftware](https://answers.netlify.com/u/markasoftware)\
**Post date:** [June 17, 2019, 4:47pm UTC](https://answers.netlify.com/t/support-guide-understanding-and-debugging-prerendering/150/11 "2019-06-17T16:47:05Z")

</div>

Pre-rendering appears to remove JavaScript tags from pages. That makes a lot of sense when the pre-rendered page and the original page share a URL (say, `my-site.com/post/5`), but when using the slightly old-fashioned #! URL structure which Google used to recommend (eg, `my-site.com/#!/post/5`), it breaks sites completely. Here’s what happens with a proper URL (no hash symbol):

- Normal browser: `my-site.com/post/5` loads with normal JS and looks normal.
- With GoogleBot: `my-site.com/post/5` is sent to the server, pre-rendered, and served without JavaScript.

Now with a hash:

- Normal browser: `my-site.com` is the portion of the URL sent to the server. JavaScript reads #!/post/5 portion of the URL and loads it dynamically.
- GoogleBot, without JavaScript: Sometimes GoogleBot tries to index without JavaScript. In this case, it rewrites the URL to use `_escaped_fragment_` and pre-rendering returns the correct page.
- GoogleBot, with JavaScript: Increasingly, GoogleBot prefers to use normal JavaScript like a client rather than `_escaped_fragment_`. I know this because my URLs are showing up as “not selected as canonical” in the Search Console despite the fact that the `_escaped_fragment_` versions look very different. The pre-rendering service only sees the [my-site.com](http://my-site.com) portion of the URL, so pre-renders the homepage regardless of the hash portion of the URL. Because JavaScript is removed, the hash portion of the URL is completely ignored and GoogleBot sees every page exactly the same way.

Possible solutions:

- Provide two pre-rendering modes:
  - Without hash; always remove JavaScript from HTML
  - With hash; only remove JavaScript from `_escaped_fragment_` URLs.

- Provide a checkbox to enable or disable `<script>`-tag removal.

Of course, the _real_ solution is probably for me to move away from the old-style `#!` URLs.

---

<div class="post-metadata">

**Author:** ![Dennis](https://sea1.discourse-cdn.com/netlify/user_avatar/answers.netlify.com/dennis/32/34_2.png) [@Dennis](https://answers.netlify.com/u/Dennis)\
**Post date:** [June 19, 2019, 12:41pm UTC](https://answers.netlify.com/t/support-guide-understanding-and-debugging-prerendering/150/13 "2019-06-19T12:41:31Z")

</div>

Yea, I think the real solution is indeed moving away from the #! routing style. One thing to try is adding `<meta name="fragment" content="!">` to your page to see if that helps as a possible short-term workaround.

---

<div class="post-metadata">

**Author:** ![sallar](https://avatars.discourse-cdn.com/v4/letter/s/3ab097/32.png) [@sallar](https://answers.netlify.com/u/sallar)\
**Post date:** [November 7, 2019, 12:35pm UTC](https://answers.netlify.com/t/support-guide-understanding-and-debugging-prerendering/150/14 "2019-11-07T12:35:20Z")

</div>

Hello there!

I have enabled Netlify prerendering in my dashboard, but all bot requests that are made to my site come back empty (zero length).

```
> User-Agent: facebookexternalhit/1.1
> Accept: */*
>
* Connection state changed (MAX_CONCURRENT_STREAMS == 150)!
< HTTP/2 200
< cache-control: public, max-age=0, must-revalidate
< content-type: text/plain; charset=utf-8
< date: Thu, 07 Nov 2019 12:00:04 GMT
< referrer-policy: same-origin
< x-bb-proxy: True
< x-bb-proxy-version: v2
< x-content-type-options: nosniff
< x-frame-options: DENY
< x-xss-protection: 1; mode=block
< content-length: 0
< age: 0
< server: Netlify
< x-nf-request-id: f2f3fa8b-8623-46aa-bb93-de72b7016997-846343
<
* Connection #0 to host [redacted].com left intact
* Closing connection 0

```

Am I doing something wrong?

---

<div class="post-metadata">

**Author:** ![Dennis](https://sea1.discourse-cdn.com/netlify/user_avatar/answers.netlify.com/dennis/32/34_2.png) [@Dennis](https://answers.netlify.com/u/Dennis)\
**Post date:** [November 8, 2019, 1:27pm UTC](https://answers.netlify.com/t/support-guide-understanding-and-debugging-prerendering/150/15 "2019-11-08T13:27:15Z")

</div>

Hi Sallar,

Did you try re-deploying your site after enabling prerendering? Give that a try and see if that helps.

---

<div class="post-metadata">

**Author:** ![system](https://sea1.discourse-cdn.com/netlify/user_avatar/answers.netlify.com/system/32/13_2.png) [@system](https://answers.netlify.com/u/system)\
**Post date:** [February 28, 2020, 8:26pm UTC](https://answers.netlify.com/t/support-guide-understanding-and-debugging-prerendering/150/23 "2020-02-28T20:26:51Z")

</div>

Automatically bumped on February 28, 2020, 8:26pm UTC



---

<div class="post-metadata">

**Author:** ![system](https://sea1.discourse-cdn.com/netlify/user_avatar/answers.netlify.com/system/32/13_2.png) [@system](https://answers.netlify.com/u/system)\
**Post date:** [April 26, 2020, 3:05am UTC](https://answers.netlify.com/t/support-guide-understanding-and-debugging-prerendering/150/26 "2020-04-26T03:05:41Z")

</div>

Automatically bumped on April 26, 2020, 3:05am UTC



---

<div class="post-metadata">

**Author:** ![sarfata](https://sea1.discourse-cdn.com/netlify/user_avatar/answers.netlify.com/sarfata/32/9872_2.png) [@sarfata](https://answers.netlify.com/u/sarfata)\
**Post date:** [April 27, 2020, 4:35pm UTC](https://answers.netlify.com/t/support-guide-understanding-and-debugging-prerendering/150/27 "2020-04-27T16:35:42Z")

</div>

> [@fool](#):
>
> There are a few dozen user agents that a request can present that will cause a prerendered response, for instance `Twitterbot` and `facebookexternalhit/1.0` . All crawlers that we are aware of which need prerendering receive this treatment automatically, but if yours seems to be missing, please mention it below so we can potentially add it to the list!

Could you guys add `SEMrushBot` ([Semrush Bot | Semrush](https://www.semrush.com/bot/))? Right now most of their tools do not work well on my site because they do not see what google sees.

Pre-rendering is awesome by the way. Thanks a lot!

---

<div class="post-metadata">

**Author:** ![perry](https://sea1.discourse-cdn.com/netlify/user_avatar/answers.netlify.com/perry/32/16_2.png) [@perry](https://answers.netlify.com/u/perry)\
**Post date:** [May 4, 2020, 5:05pm UTC](https://answers.netlify.com/t/support-guide-understanding-and-debugging-prerendering/150/28 "2020-05-04T17:05:05Z")

</div>

hey @sarfata! We haven’t forgotten about this, we will try and take a look.

---

<div class="post-metadata">

**Author:** ![fool](https://sea1.discourse-cdn.com/netlify/user_avatar/answers.netlify.com/fool/32/37_2.png) [@fool](https://answers.netlify.com/u/fool)\
**Post date:** [May 8, 2020, 6:58pm UTC](https://answers.netlify.com/t/support-guide-understanding-and-debugging-prerendering/150/29 "2020-05-08T18:58:39Z")

</div>

Hi @sarfata - you’re the first person to ask for this, and we generally don’t make these kinds of changes for a single customer without substantial research. In this case we’ll have to ask you to do some research. As you can see on that page you linked us to, there are a LOT of user agents they use:

> `User-agent: SemrushBot-SA`  
> `User-agent: SemrushBot-BA`  
> `User-agent: SemrushBot-SI`  
> `User-agent: SemrushBot-SWA`  
> `User-agent: SemrushBot-CT`  
> `User-agent: SemrushBot-BM`

Could you please contact their support team and confirm which of those need prerendered content? I think it is probably not all of them, but we will consider their advice once you ask for it and share it with us.

Thanks in advance for your help!

---

<div class="post-metadata">

**Author:** ![sarfata](https://sea1.discourse-cdn.com/netlify/user_avatar/answers.netlify.com/sarfata/32/9872_2.png) [@sarfata](https://answers.netlify.com/u/sarfata)\
**Post date:** [May 11, 2020, 4:22pm UTC](https://answers.netlify.com/t/support-guide-understanding-and-debugging-prerendering/150/30 "2020-05-11T16:22:12Z")

</div>

Hi @fool,

I have contacted SEMRush and pointed them to this thread so hopefully they can chime in here directly.

I am not sure what downsides there would be to add too many user-agent in this case. Their tools are all meant to emulate a search engine and report on what the search engine sees. I think just adding `SemrushBot` would be fine.

Thanks,  
thomas

---

<div class="post-metadata">

**Author:** ![fool](https://sea1.discourse-cdn.com/netlify/user_avatar/answers.netlify.com/fool/32/37_2.png) [@fool](https://answers.netlify.com/u/fool)\
**Post date:** [May 11, 2020, 10:31pm UTC](https://answers.netlify.com/t/support-guide-understanding-and-debugging-prerendering/150/31 "2020-05-11T22:31:26Z")

</div>

I agree that is likely to be their advice. However, when making changes that affect nearly 1,000,000 other customers we’d like to make sure 🙂

[Next page](https://answers.netlify.com/t/support-guide-understanding-and-debugging-prerendering/150.md?page=2)