I am trying to use jetoctopus for SEO crawling. Everytime I run a scan from their platform I get an error saying that the website is blocking their IPs. Is there anyway to verify if IPs are getting blocked and, if so, how would I allow them?
Have you tried fixing the DNS entry for your CNAME? It’s currently:
|======================= dig CNAME(s) for =======================
| ----------------------- www.realciso.io -----------------------
| ------------ will be blank when using Netlify DNS -------------
realciso.io.
|================================================================
It’s supposed to point at mystifying-brahmagupta-78e5f3 dot netlify dot app.
Thanks for the reply. Ah I see. That is interesting. I wonder if they changed the setup. I had thought I initially setup the domains as instructed in the setup. In any case, I altered the www subdomain to point to my netlify.app subdomain, but the problem still persists. The crawler cannot reach the site.
Hey @perry, thanks for the link. I have my dns setup the way described in the link. It honestly does not feel like a DNS issue. The both naked and subdomain of the site, even before I made the change for www to point to the .netlify.app domain, have always resolved correctly.
There seems to me to be some acl somewhere that is blocking these handful of IPs, at least that is what the behavior resembles.
Yeah, I went to them first. They were not super helpful and basically said that the IPs were getting blocked, and that I needed to figure out what was up with Netlify. I am still trying to get more information out of them, but thought I would check here to see if I could get any more info on the netlify side. So I created this post.
Well thanks all for helping me with this. I tried to scan the site again today with jetoctopus and now it magically worked. So either the dns change worked and I didn’t wait long enough, or…gremlins?
HI, @soylentsnacks. It is hard to say what the issue was without more information (for example: the list of IP addresses that Jetoctopus is using or an x-nf-request-id response header for a failed connection attempt).
Thanks for letting us know it is working for you now and if there are any other questions, please let us know.