Hello,
I write regarding activity I’ve seen in my Netlify site aramzs.xyz
I’ve searched around quite a bit but don’t seem to see a definitive answer. My site is static, all pre-built and rendered and uploaded to Netlify. However, occasionally I see LinkedIn, Facebook and sometimes other social sites being unable to read the data needed for a link card.
I have checked and can see crawls fail using Facebook and LinkedIn’s testing tools:
Facebook: Sharing Debugger - Meta for Developers
LinkedIn: Post Inspector
I recently have been testing an edge function and added logging there and see that the response sent to those sites’ tends to have weird inconsistent errors. They have received timeouts and 500 errors and at least once a 418 error. I can see logs thanks to my edge function that note the responses to the crawler user agents for social media for example:
“LinkedInBot/1.0 (compatible; Mozilla/5.0; Apache-HttpClient +http://www.linkedin.com)”
and
“Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_1) AppleWebKit/601.2.4 (KHTML, like Gecko) Version/9.0.1 Safari/601.2.4 facebookexternalhit/1.1 Facebot Twitterbot/1.0”
unusually error out on pages that are otherwise served normally.
What’s going on here? Is there some anti-bot measure built into Netlify that is going wrong?