[Support Guide] Understanding and debugging prerendering

Support Support Guides
, ,
1

Last Reviewed by Netlify Support in Aug 2023

With the advent of modern web hosting aka the JAMStack, some robots and crawlers need “help” navigating client-side links. This problem first presented itself initially when navigation was implemented using Javascript instead of HTML, but these days the OpenGraph standard and unfurling by social sharing services seems to be in many cases more important to our customers. (For more on this, see a further reading link we have added to the bottom of this post.) Prerendering is one solution to this problem.

Netlify provides a prerendering service described in some detail in this article. This post aims to help if it doesn’t “just work” and you’d like to debug or understand how to test.

Which requests are prerendered?

Prerendered content is only displayed in certain situations. Specifically it:

  • will only be used for a Netlify website once prerendering is enabled on your site’s build & deploy settings page
  • will only be returned in response to a web request when there is a User-Agent sent with the request that needs this special content, or a special URL is used (see below for more details around manual testing and the special URL).

There are a few dozen user agents that a request can present that will cause a prerendered response, for instance Twitterbot and facebookexternalhit/1.0 . All crawlers that we are aware of which need prerendering receive this treatment automatically, but if yours seems to be missing, please mention it below so we can potentially add it to the list!

Testing the service with your content

To test things out manually, you can also request a prerendered URL - see particularly the part about _escaped_fragment_ in our prerendering documentation.

So, to test prerendering, this is what I do. You can use any User-Agent that we prerender for, so I use twitterbot in this example.

curl -A twitterbot https://www.sitename.com/

or

curl https://www.sitename.com/?_escaped_fragment_=

This should show different output than not using a special User Agent or the ?_escaped_fragment_= style URL, which will demonstrate that the service is in effect.

Some caveats about the prerender service

  1. Most websites using modern frameworks like next.js do not require prerendering these days. Please be sure it is needed before you configure it, since it will use more bandwidth to render sites for crawlers/bots and you’d want to make sure that is a good use of your money!

  2. We cache the prerendered content for 24 to 48 hours, so you will want to make sure to test with different URLs each time if you are making changes and republishing a lot while you get things dialed in. Most importantly, this cache is NOT invalidated by a new deploy of your site!

  3. You may also need to use window.prerenderReady in your javascript to indicate to our service when it is safe to consider the page fully rendered to take the snapshot for the services that need it, as mentioned in the prerender.io docs here. If this isn’t set, and we haven’t already determined that the page has loaded completely, we take a snapshot after 10 seconds regardless of the state of the page.

  4. When testing with Google Webmaster tools, please be aware that there can be an issue using the "Fetch as Google" functionality. This tool does not check for the fragment meta tag and re-request the page with the ?escaped_fragment= query parameter as it does in production: Googlebot itself does not have that issue. So, when using "Fetch as Google" you’ll need to append the ?_escaped_fragment_ query parameter on the end of the URL that you are testing, and that should show your prerendered page. If that works and if you have the following code snippet in the <head> of your HTML, then the real Googlebot will work just fine.

        <meta name="fragment" content="!">

  5. Another astute customer discovered that some services, such as WhatsApp, Facebook, and Google, only download the first N bytes of your html - so if you have a ton of stuff before those meta tags - it could be “missing” the parts that it needs to prerender correctly. That “astute customer discovered” link will show you one solution to this situation if you have it. You’ll see this as “twitter works correctly, whatsapp doesn’t” on the same URL to help narrow down your troubleshooting.

Next Steps

That’s a lot of advice, but it’s possible that it still won’t resolve your issue. So what can you do next if your content doesn’t seem to be rendering correctly after enabling the prerender service? After checking out the below caveats to make sure it’s not related to one of those, here’s a bit more advice:

First, be aware that your code needs to provide the OpenGraph tags that are needed for social sharing services. A lot of frameworks and plugins (e.g. react-helmet) handle setting them for you, but “a wrong og tag” is generally not the fault of the prerender service but rather your code. You can examine the source code that we serve as described above to debug what is happening.

If instead you’re seeing errors (HTTP 5xx responses), then it is likely that your code is not rendering at all. Since we don’t expose prerender logs to customers, using a local copy of Netlify’s prerender code (forked from prerender.io, and different!) and following the local-use instructions in its readme is the best way to debug those issues in depth. This will show any syntax errors that are encountered as well as letting you examine the output including any OpenGraph tags.

Two more pieces of advice on sharing content on social services from folks not on Netlify’s tech Support team:

  1. A customer wrote up his journey in great detail with a lot of practical advice in this medium article here - it explains a lot about the world of rendering and unfurling and testing using third party tools. If you’re struggling to understand why prerendering is needed and how prerendering works with social sharing articles, this is a great starting place.
  2. One of our Developer experience engineers @cassidoo recently wrote up some great implementation details which while tailored for next.js are very illustrative and generally informative for people working on their SEO/sharing setups. You can find her amazing blog post here.
7 Likes