Is there a way to create a whitelist of IP addresses for crawlers?

Support
, ,
#1

Hi Netlify team,

I was wondering if there a way to whitelist a range of IP addresses?
We want to use a third-party crawler. It seems the crawler is blocked or there is some sort of limit. Could we remove the limits or whitelist specific IPs, please?

Site name: adoring-mclean-7177c8
Custom domain: dominodatalab.com

Best,
S

#2

Hi, @maliMirkec. Once we know more about what is being used, we will be able to give more accurate answers.

What third-party crawler are you using and what IP address (or addresses) would it use to connect to our service?

#3

This is the request from the third-party crawler.

Please ask your IT team this and to whitelist Caliperbot/Conductors IP - 162.246.176.1–162.246.183.254 and Deepcrawls IP address - (52.5.118.182)

#6

Hi, @maliMirkec. We are not blocking access from those IP addresses nor do I find any evidence of us having blocked them historically either. I also see this site a part of a paid team type (a plan level above Starter) and so I’ve created a helpdesk ticket using the same email address used for this forum login.

Please reply to us here though if you don’t see the email about the helpdesk ticket.

#7

Thank you! I’ll pass the information and then we will see if any extra actions are required.