I’d like to report a couple of prerendering issues I’m facing. I’ve been using prerender.io for the last 2 years for another site and I have a decent working knowledge on this topic. I’ve also read various posts here and I didn’t find any that directly fit. So here goes:
Google search console is reporting that the pages are not optimized for mobile - that’s impossible since the site is completely responsive across many different screensizes. On further checking, I see that the live test page screenshot is showing a larger screen view. Is it possible that netlify’s prerender headless browser is not determining that the incoming request is Googlebot Smartphone, and returns the desktop version instead? Goog says Source: Smartphone crawler, and hence is probably penalizing the site heavily for being “mobile unfriendly”.
Slack (and MS Teams) does not show the correct link previews, it just shows what is in index.html, rather than what has been inserted by Helmet, which is what should have been generated and cached by Netlify prerender. Every page created on the site has detailed OG tags, Twitter tags and old style meta tags. FB and Twitter and Whatsapp and Telegram etc show the rich previews just fine, but not Slack and MS Teams. This issue is there with prerender.io as well, BTW. (I’m checking with Todd as well)
My finding for MS Teams is that there is no known UA agent to look for in their case, so there’s no way to make a decision to prerender. I had a detailed discussion with MS Team L2 support team and they acknowledged the issue: Good day this concern is now raised as a Bug by our engineering team. Investigation has started but there’s no ETA yet that can be provide as of now. Depending on checks and fix needed, a bug case may take weeks or months before getting resolved. (this was back in Aug 2020)
But I understand that there is a UA string for Slack and hence a request from Slack will be responded with the prerendered version and a “proper” link preview would then be shown. But its not.
Really long delay in generating and replying with the prerendered page - I can see this in Facebook debugger which timed out for multiple links, in Goog search console “test live link”, etc. I don’t see such a large delay from prerender.io.
Any help or pointers appreciated. And if I can provide further info, please let me know. Thank you.
What a thorough post! Very sorry to be slow to answer, but your knowledge exceeds most of our customers and even our Support team already, so I’ll do my best as our prerendering “expert” (though the scare quotes are particularly relevant since my “expertise” is only relative to folks who don’t heavily use the service!)
I do not believe we do any different types of prerendering for mobile, no. Does prerender.io handle this differently? I’d be surprised to hear that, since we run a fork of prerender.io’s code… Definitely worth getting a feature request filed but would love to know if there is already a working implementation to reference in the request since that will make it a lot more likely that we’ll implement it in any foreseeable time frame.
We do require a known-in-advance user-agent and are happy to add additional ones to our config for prerendering as they come up, so let us know if you do ever hear of them picking one they can stick to!
But I understand that there is a UA string for Slack and hence a request from Slack will be responded with the prerendered version and a “proper” link preview would then be shown. But its not.
Didn’t quite understand this statement. Are you saying prerendering for slack isn’t working?
the best advice we have there is to optimize your pages to not be super heavy. It can take awhile to render big pages; it does not generally take awhile to render more optimized pages. If facebook is timing out I believe that means we’re talking “over 11 seconds” and while we do cache the result so a second fetch should get it from cache, I don’t know that they retry every time, so I can see the impact of having slower-to-render pages. It’s our intention that the cache is performant moreso than the rendering, since in general the timing doesn’t matter in these cases since humans are not awaiting the render result, but facebook has decided to cut those requests off and yeah, that’s not something I have a good workaround for right now. prerender.io does apparently run faster than us, which isn’t a surprise, since it’s a paid service whereas our prerendering is not, and thus…not a big target for our org.
Your best solution for the speed situation, if it is creating problems for you, may be to use prerender.io for your site.
Thanks for the very detailed reply - I wanted to update after I have more concrete stuff to respond back with.
I’ve parked for the moment and I’ll get back to it once #2 is solved.
Not that prerendering is not working, I’m sure it is, just that Slack is not showing the right link preview. It just shows the default link preview for ALL pages. I was following this up with prerender support, but they were unable to offer a solution. On my side, on further investigation into the raw prerendered HTML generated, I can see that the generated page has a default set of meta tags under < noscript > inserted via index.html, and then the actual relevant tags meta tags are inserted by Helmet - detailed OG tags, Twitter tags and old style meta tags. (This is of course as expected). Everyone is correctly ignoring the default tags under < noscript >, EXCEPT Slack. I think this is the source of the issue I’m facing and I’ve filed a bug ticket with Slack and I’ll update again when I hear back from them.
Yup, fully clear on this
Separately, on this item:
We do require a known-in-advance user-agent and are happy to add additional ones to our config for prerendering as they come up
Would you be willing to add our UA to your config or would that be too much to ask for? We are a tiny worm at the moment… so if you say no, I perfectly understand.
User agent token: mcbot/1.0
Full user agent string: User-Agent’: 'mcbot/1.0 Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.76 Safari/537.36
Ah, for #2 you need to adjust what you are setting - the og: tags must be wrong (or duplicated). You can probably look at your output using curl -a twitterbot https://yoururl and compare with what they say they want in their docs (not sure this is the best one, but seems relevant: Unfurling links in messages | Slack). What they show is about what your code produces rather than anything we “do” with that content
We can add UA if it is documented somewhere, globally applicable, and seems stable (changing the setup is tedious and we cannot do it frequently). This message alone, is not sufficient documentation, since this change will affect all customers so we need to see it in official writing from the bot producer for a commercial or popular service, to add it.
Since it definitely seemed to be an issue with < noscript > handling at Slack as far as link preview generation goes, I made a change to my code - I’ve removed the noscript node (with the default meta tags under it) at run time after React loads up, so that its not there in the prerendered HTML. With that, Slack link previews are working as expected!
Hiya, we are seeing a lot of speed issues with Netlify prerendering, and prerender.io has increased its prices recently - so it may not be suitable for us to move more sites over to them.
Is there a way to hook up a self hosted solution? Note that I’m not asking how to set up such a service - that I can do. Just how do I tell Netlify where to route the request when UA matches.
In the mail template you ask the following:
I want to configure an external prerendering service for site xyz. [Name of the prerendering service] [API token]
So can this accommodate only some named services, or can I provide our own? Eg instead of https://service.prerender.io/${url} can I provide our own URL?
I’ve recently done this for a customer where they created their own pre-rendering service in Python, I suppose and gave us a URL like the one you mentioned. So, it’s definitely possible and other users are doing it too.
