Is there a way to create a whitelist of IP addresses for crawlers?

Hi Netlify team,

I was wondering if there a way to whitelist a range of IP addresses?
We want to use a third-party crawler. It seems the crawler is blocked or there is some sort of limit. Could we remove the limits or whitelist specific IPs, please?

Site name: adoring-mclean-7177c8
Custom domain: dominodatalab.com

Best,
S

Hi, @maliMirkec. Once we know more about what is being used, we will be able to give more accurate answers.

What third-party crawler are you using and what IP address (or addresses) would it use to connect to our service?

This is the request from the third-party crawler.

Please ask your IT team this and to whitelist Caliperbot/Conductors IP - 162.246.176.1–162.246.183.254 and Deepcrawls IP address - (52.5.118.182)

Hi, @maliMirkec. We are not blocking access from those IP addresses nor do I find any evidence of us having blocked them historically either. I also see this site a part of a paid team type (a plan level above Starter) and so I’ve created a helpdesk ticket using the same email address used for this forum login.

Please reply to us here though if you don’t see the email about the helpdesk ticket.

Thank you! I’ll pass the information and then we will see if any extra actions are required.