SEO Webcrawler getting blocked - how to resolve

Support
,
1

Netlify site name: mystifying-brahmagupta-78e5f3

Hey,

I am trying to use jetoctopus for SEO crawling. Everytime I run a scan from their platform I get an error saying that the website is blocking their IPs. Is there anyway to verify if IPs are getting blocked and, if so, how would I allow them?

Thanks,
Nick

2

@soylentsnacks Welcome to the Netlify community.

You’re going to have a tough time getting SEO on a netlify.app subdomain. Have you tried this with a custom subdomain?

3

@gregraven Yes, absolutely. I have a custom domain pointing at Netlify.

4

Did you want to share your custom domain name so we can check your set-up?

5

Sure thing.

6

Have you tried fixing the DNS entry for your CNAME? It’s currently:

|======================= dig CNAME(s) for =======================
| ----------------------- www.realciso.io -----------------------
| ------------ will be blank when using Netlify DNS -------------
realciso.io.
|================================================================

It’s supposed to point at mystifying-brahmagupta-78e5f3 dot netlify dot app.

7

Hey @gregraven,

Thanks for the reply. Ah I see. That is interesting. I wonder if they changed the setup. I had thought I initially setup the domains as instructed in the setup. In any case, I altered the www subdomain to point to my netlify.app subdomain, but the problem still persists. The crawler cannot reach the site.

8

Hi there! is this helpful at all?

9

Hey @perry, thanks for the link. I have my dns setup the way described in the link. It honestly does not feel like a DNS issue. The both naked and subdomain of the site, even before I made the change for www to point to the .netlify.app domain, have always resolved correctly.

There seems to me to be some acl somewhere that is blocking these handful of IPs, at least that is what the behavior resembles.

10

Have you contacted Jetoctopus?

11

Yeah, I went to them first. They were not super helpful and basically said that the IPs were getting blocked, and that I needed to figure out what was up with Netlify. I am still trying to get more information out of them, but thought I would check here to see if I could get any more info on the netlify side. So I created this post.

12

Well thanks all for helping me with this. I tried to scan the site again today with jetoctopus and now it magically worked. So either the dns change worked and I didn’t wait long enough, or…gremlins?

13

HI, @soylentsnacks. It is hard to say what the issue was without more information (for example: the list of IP addresses that Jetoctopus is using or an x-nf-request-id response header for a failed connection attempt).

Thanks for letting us know it is working for you now and if there are any other questions, please let us know.