User-agent blocking?

[RTO]

Hey folks,

I have some issues with open graph previews loading properly on LinkedIn. The title/description come through fine, however the images don’t load when sharing posts, etc.

I’ve been on to the LinkedIn tech team who’ve advised that they appear to be being blocked (see user agent / response below).

The site itself builds/loads fine as far as I can see, and there are no issues on other social media sites (e.g. Twitter, Facebook, etc), e.g:

Page: https://cydea.com/blog/positive-security/
OG:image for that page: https://cydea.com/images/cydea-social-default.jpg

I can’t see anything that I have configured/set myself that would prevent access. Plus the title/description are being loaded OK. It just seems to relate to the OG:image.]

Any ideas? Does Netlify’s CDN block access to LinkedInBot’s user-agent? If so… how do I stop it doing so please?

Cheers,

Robin

** USER-AGENT **
LinkedInBot/1.0 (compatible; Mozilla/5.0; Jakarta Commons-HttpClient/3.1 +http://www.linkedin.com)

** RESPONSE **
See here: https://gist.github.com/rto/6137d29fe79e49df38e9dcea98e6bc52

[fool]

We do not block any user agents. We do prerender automatically for linkedin (if you have that feature enabled on the site, which you don’t seem to), perhaps your page does not prerender well? Here’s how to test, if you want to try that out:

[Common Issue] Understanding and debugging prerendering

With the advent of modern web hosting aka the JAMStack, some robots and crawlers need “help” navigating client-side links. This problem first presented itself initially when navigation was implemented using Javascript instead of HTML, but these days the OpenGraph standard and unfurling by social sharing services seems to be in many cases more important to our customers. (For more on this, see a further reading link we have added to the bottom of this post.) Prerendering is one solution to this p…

The response doesn’t come from our service - couldn’t say where it does come from without seeing the http response headers. I don’t get those results myself when using that user agent to connect via curl - I get the image (if I use the image URL) and the page (if I use the page URL).

Perhaps you could provide the reproduction steps for how you got that response in more detail?

[RTO]

Hey Chris - thanks for the quick and definitive response.

You’re right, I don’t have prerendering enabled. There is some content which is faded in, but no content loaded via JS (beyond Google Analytics, etc), it’s a fairly simple static HTML site. So I don’t think it should make a difference, but will take a look.

Good to hear you get the same results as me (i.e. they load fine in browser and via curl).

The response was provided to me by the LinkedIn agent - that’s all of it - so I’ll see if I can get some reproduction steps from them. Interesting/useful to know that it doesn’t come from Netlify though!

[luke]

@RTO, note that if you do enable the prerendering option (which might be the solution for this issue) a new deploy is needed for the changed prerendering setting to take effect.

Note, the new deploy doesn’t actually need to be different (the files can remain identical and the only change might be a comment line in a setting file). However, a new deploy (with changes or not) is always needed for this setting to start working.

If it doesn’t fix this issue though, please let us know and we’ll be happy to keep troubleshooting this.

[RTO]

Thanks Luke - makes sense - the evidence is pointing towards it being an issue at LinkedIn’s end, not Netlify, but thought it best to understand the handling of bot requests, etc. (And good to learn about your prerendering options too! :-))