I write regarding activity I’ve seen in my Netlify site aramzs.xyz
I’ve searched around quite a bit but don’t seem to see a definitive answer. My site is static, all pre-built and rendered and uploaded to Netlify. However, occasionally I see LinkedIn, Facebook and sometimes other social sites being unable to read the data needed for a link card.
I have checked and can see crawls fail using Facebook and LinkedIn’s testing tools:
I recently have been testing an edge function and added logging there and see that the response sent to those sites’ tends to have weird inconsistent errors. They have received timeouts and 500 errors and at least once a 418 error. I can see logs thanks to my edge function that note the responses to the crawler user agents for social media for example:
“Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_1) AppleWebKit/601.2.4 (KHTML, like Gecko) Version/9.0.1 Safari/601.2.4 facebookexternalhit/1.1 Facebot Twitterbot/1.0”
unusually error out on pages that are otherwise served normally.
What’s going on here? Is there some anti-bot measure built into Netlify that is going wrong?
I verified in Google Webmaster console that standard crawling appears to occasionally get 5xxed with no setting on my side encouraging that. What’s going on?
It looks like you’ve enabled Netlify’s prerendering feature. As this feature is in beta, we don’t recommend it for mission critical sites. Sporadic 500s can occur when using this feature. We’d instead recommend using a third-party service, such as Prerender.io or Headless-Render-API.com:
If you decide you’d like to migrate to one of these services, the support team can assist with configuring that for you.
Hi @sid.m ! If I turned on Prerendering I def didn’t intend to. I’ve turned it off. Since I’m building a static server-side-rendered site that is fully built by Netlify and published as individual HTML pages I shouldn’t need it right?