Bot gets timeout when crawling open graph image

Support
, , ,
#1

Hi there.

our service partner for recruiting (www.xing.de) is crawling our job urls (e.g. Fullstack Software Developer) from our website (hosted by you, netlify).
While crawling the url their bot is trying to load another image-url given from open-graph-image-meta-tag in order to download this file to their cdn.
e.g. <meta data-n-head="ssr" data-hid="og:image" property="og:image" content="https://www.wtfoxtrot.de/img/fullstack_eng.webp">

Unfortunately the bot (build with ruby) is receiving a Net::OpenTimeout exception.

But I can’t reproduce this issue. They said to me, they use these headers and that the request seems to be blocked when sending from their system.

curl -I -A "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:15.0) Xing Bot" -H "accept-encoding: gzip,deflate,identity" -H "accept: */*" -H "accept-charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7" -H "accept-language: en-us,en;q=0.5" https://www.wtfoxtrot.de/img/fullstack_eng.webp

So maybe you can have a look in the access.log of my site and scan for the user-agent in order to provide me some logging information and if you can think of any possible reason why this does not work.
Do you block some kind of origins?

PS:

Our site name is wtfoxtrot.

Thank you so much for your help!

#3

Hey @sven-wtfoxtrot :wave:

Our logs show one IP, 217.95.XXX.XXX, connecting seven times on 7th of Feb. Could you confirm that is your service partner’s bot?

1 Like
#4

Thank you Audrey.

I just asked them and will get back to you, when I get an answer.

1 Like
#5

Finally I got a result from the service partner.
The IPs are 185.169.112.0/22 and 185.169.113.0/22

Unfortunately not your mentioned ones. Can you put the above mentioned IPs on a allow-list?

#6

Hey @sven-wtfoxtrot,

I checked for those IP Addresses and it appears that they’re not on any of our block lists. Just in the past hour, we’ve already had 75 successful requests served to those addresses. I could confirm it’s the bot as it has the mentioned user agent.

So this could have something else to do other than blocks. But I can’t think of anything and the fact that there are working requests, I’m not sure how to debug this.