[Support Guide] Understanding and debugging prerendering

Hello,
first of all, thanks for this great documentation and forum, everything is very useful.

I’m trying to fix this issue for my site (https://poralgoseempieza.holaluz.com/) but I can’t get it done. I tried running prerender locally and it returns a 401:

2021-02-16T09:11:15.247Z Starting Prerender
2021-02-16T09:11:15.249Z Starting Chrome
2021-02-16T09:11:15.260Z Prerender server accepting requests on port 3000
2021-02-16T09:11:15.780Z Started Chrome: HeadlessChrome/88.0.4324.150
2021-02-16T09:11:41.093Z getting https://poralgoseempieza.holaluz.com/
2021-02-16T09:11:41.102Z got 401 in 9ms for https://poralgoseempieza.holaluz.com/
2021-02-16T09:11:41.104Z method=GET status=401 url=/https://poralgoseempieza.holaluz.com prerender_url=https://poralgoseempieza.holaluz.com/ cache_type=CACHE_MISS timing=8 referrer="-" user_agent="Mozilla/5.0 (Macintosh; Intel Mac OS X 10.14; rv:85.0) Gecko/20100101 Firefox/85.0" request_id=unknown

Also I have enabled Prerendering for this site config but facebook scrapper still is broken.

thanks in advanced for your help,
cheers.

Hi, @adriamarcet. When you say the Facebook scraper is broken, what are you expecting to see and what is the actual response?

I ask because I’m seeing the prerendering working when I test here:

https://www.opengraph.xyz/url/https:%2F%2Fporalgoseempieza.holaluz.com%2F/

Hi @luke,
Sorry I didn’t print any results or response, I was referring to this Facebook Developer Tools (you need to be logged in).

Its response is Page Not Found on the snippet preview and 404 also as a response (see image attached). For now I think it has something to do with Facebook Cache since I created a new Netflify site and that one looks just fine trough the Developers’ tool.

Ps. The wrong canonical was fix 2 days ago but Facebook is still showing it as broken.

Hi, @adriamarcet. My best guess is that this is caused by the caching done by our prerender service (as opposed to caching at Facebook itself):

On the other hand, it could be Facebook caching if you see the problem at Facebook but not when you use opengraph.xyz to test.

My best guess is that the issue will resolve itself soon as it should only be cached for about 48 at most. You mentioned it has been two days so it should refresh soon. At the tool I linked to, I don’t see any 404 related messaging.

Last but not least, I want to mention that cannot use the Facebook Developer Tools. I do not have a Facebook login and that tool requires a login. I’ve attempted to create a “for business use only” Facebook account so I can use that tool but Facebook always shuts it down. (I won’t create a personal Facebook account because … :speak_no_evil:.)

We want to use the prerendering in our production react website but prerendering is in beta stage. Can we use it in our production react website?

Hi, @mamit. I believe that you are asking if you should use the prerendering for your production site because you certainly can use it. Nothing prevents you from using this feature with any site at Netlify.

Now, for the questions of if you should use it or not, that is ultimately up to you to decide. You might test enabling it on a non-production site and running tests if you have concerns. If you see issues with the tests, then you will know it is not right for your production site.

To summarize, my best suggestion would be to test the prerendering on some non-production site and then making a decision based on the results those tests.

Hi all, I’ve debug my site and netlify prerender got 400 on loading assets. In it’s github they told us to Note: you should proxy the request through your server (using middleware) so that any relative links to CSS/images/etc still work. If you don't, you'll see the relative links fail but the HTML output will still be useful for debugging purposes.. How to achieve that when we serve our SPA including assets using netlify?

My site that activate prerender is https://kpe-beta.netlify.app/

Hi, @3mp3ri0r. I don’t see any 400 responses for this site. If you are seeing 400 responses, where are you seeing them? Can you send us a URL to check to see the 400s? How do I find them?

If they are 400 responses from Netlify, I am not seeing any for this site. Not a single 400 response in the last 7 days. If you do get a 400 response from Netlify, please send us the x-nf-request-id header of that response.

There more information about this header here:

If that header isn’t available for any reason, please send the information it replaces (or as many of these details as possible). Those details are:

  • the complete URL requested
  • the IP address for the system making the request
  • the IP address for the CDN node that responded
  • the day and time of the request (with the timezone the time is in)

Hi @luke, this is netlify prerender log when I’m debuging my site from my computer: source.

URL that I tried to be prerendered from my computer: Krisma Perkasa Engineering - Pompa Industri dan Motor Elektrik (You’ll see wrong title here in this reply, it should be Alliance gear helical and bevel gearbox AK series | Krisma Perkasa Engineering if it prerendered correctly).

It looks like the team blind crawler does not get the prerendered page. I attached a link: https://www.factionfinancial.com/prediction/0508a90d-010e-4344-aa83-6988ae153973 Link gets pre-rendered properly on twitter and facebook but not on team blind.

Hi, @3mp3ri0r. I’m not sure what you are asking. There is no question in your last reply.

I requested the prerendered version of that URL with the following command locally:

curl -svo prerender-index.html -H "user-agent: twitterbot" \
 https://www.krismaperkasa.com/motor-elektrik/product/alliance-gear-helical-and-bevel-gearbox-AK-series

This is the title of the response:

<title>Alliance gear helical and bevel gearbox AK series | Krisma Perkasa Engineering</title>

So, that title does appear to be prerendering correctly when I test.

@jude234, our prerendering service only triggers when a user-agent matches the list for the prerendering service. We can add the user agent that the team blind crawler uses. However, before we can do so we need to know what that user agent is. I’ve tried finding that information but I couldn’t locate it. Do you know what user-agent header they send?

@luke I unfortunately do not. I tried to check their forums. I am guessing I should reach out to their dev team?

Hi, @jude234. Yes, contacting their developers sounds like a good way to get that information.

You can also ask them to contact our support team here at Netlify and we will work to enable prerendering for their service’s HTTP requests.

Any chance we can the Embedly crawler to the list?

“Mozilla/5.0 (compatible; Embedly/0.2; +http://support.embed.ly/)”.

Thanks!

I have an interesting issue with pre-rendering not working.

Here is the issue I’ve created yesterday.

To summarize here, pre-rendering seems to be broken with any url ending with a path-param containing a dot (“.”). I’m unable to reproduce locally using Netlify’s fork of prerender.

More details can be found in the issue link above.

Please help.

Hey there, @david_opbase :wave: I have looped in the Support team. Please hang tight! We will follow up on the thread you created yesterday :slight_smile:

Hey @fool, would you mind adding a couple bots from Ahrefs to the prerender list? Thank you for your time. Bot info.

  • Mozilla/5.0 (compatible; AhrefsBot/7.0; +http://ahrefs.com/robot/)
  • Mozilla/5.0 (compatible; AhrefsSiteAudit/6.1; +http://ahrefs.com/robot/)