Netlify provides a prerendering service described in some detail in this article. This post aims to help if it doesn’t “just work” and you’d like to debug or understand how to test.
Which requests are prerendered?
Prerendered content is only displayed in certain situations. Specifically it:
- will only be used for a Netlify website once prerendering is enabled on your site’s build & deploy settings page
- will only be returned in response to a web request when there is a User-Agent sent with the request that needs this special content, or a special URL is used (see below for more details around manual testing and the special URL).
There are a few dozen user agents that a request can present that will cause a prerendered response, for instance
facebookexternalhit/1.0 . All crawlers that we are aware of which need prerendering receive this treatment automatically, but if yours seems to be missing, please mention it below so we can potentially add it to the list!
Testing the service with your content
To test things out manually, you can also request a prerendered URL - see particularly the part about
_escaped_fragment_ in our prerendering documentation.
So, to test prerendering, this is what I do. You can use any
User-Agent that we prerender for, so I use
twitterbot in this example.
curl -A twitterbot https://www.sitename.com/
This should show different output than not using a special User Agent or the
?_escaped_fragment_= style URL, which will demonstrate that the service is in effect.
Some caveats about the prerender service
We cache the prerendered content for 24 to 48 hours, so you will want to make sure to test with different URLs each time if you are making changes and republishing a lot while you get things dialed in. Most importantly, this cache is NOT invalidated by a new deploy of your site!
You may also need to use
When testing with Google Webmaster tools, please be aware that there can be an issue using the “Fetch as Google” functionality. This tool does not check for the fragment meta tag and re-request the page with the ?escaped_fragment= query parameter as it does in production: Googlebot itself does not have that issue. So, when using “Fetch as Google” you’ll need to append the
?_escaped_fragment_query parameter on the end of the URL that you are testing, and that should show your prerendered page. If that works and if you have the following code snippet in the
<head>of your HTML, then the real Googlebot will work just fine.
<meta name="fragment" content="!">
Another astute customer discovered that some services, such as WhatsApp, Facebook, and Google, only download the first N bytes of your html - so if you have a ton of stuff before those meta tags - it could be “missing” the parts that it needs to prerender correctly. That “astute customer discovered” link will show you one solution to this situation if you have it. You’ll see this as “twitter works correctly, whatsapp doesn’t” on the same URL to help narrow down your troubleshooting.
That’s a lot of advice, but it’s possible that it still won’t resolve your issue. So what can you do next if your content doesn’t seem to be rendering correctly after enabling the prerender service? After checking out the below caveats to make sure it’s not related to one of those, here’s a bit more advice:
First, be aware that your code needs to provide the OpenGraph tags that are needed for social sharing services. A lot of frameworks and plugins (e.g. react-helmet) handle setting them for you, but “a wrong og tag” is generally not the fault of the prerender service but rather your code. You can examine the source code that we serve as described above to debug what is happening.
If instead you’re seeing errors (HTTP 5xx responses), then it is likely that your code is not rendering at all. Since we don’t expose prerender logs to customers, using a local copy of Netlify’s prerender code (forked from prerender.io, and different!) and following the local-use instructions in its readme is the best way to debug those issues in depth. This will show any syntax errors that are encountered as well as letting you examine the output including any OpenGraph tags.
Two more pieces of advice on sharing content on social services from folks not on Netlify’s tech Support team:
- A customer wrote up his journey in great detail with a lot of practical advice in this medium article here - it explains a lot about the world of rendering and unfurling and testing using third party tools. If you’re struggling to understand why prerendering is needed and how prerendering works with social sharing articles, this is a great starting place.
- One of our Developer experience engineers @cassidoo recently wrote up some great implementation details which while tailored for next.js are very illustrative and generally informative for people working on their SEO/sharing setups. You can find her amazing blog post here.