ISP Routing Issue to Netlify Domains?

I’ve run into a real odd issue that seems by my ISPs fault and not that of netlify. I’m sure this is going to entail my having to call my ISP and go down that phone call support rabbit hole (which I’m dreading) but I figured before I do that, I’d ask if anyone here has had anything like this happen to them.

So this has been happening for the last 4 or 5 days. It all started when one day I noticed my domains hosted through netlify weren’t loading on my laptop. They were just timing out. Then I tried loading them on my phone, and same thing. Then I disconnected my phone from my home network and boom, everything started loading just fine (over AT&T 4G network). I even tethered my laptop to my phone and used its 4G connection and my netlify sites loaded just fine. But when I switched back to my home network, the sites started timing out again. “It must be a router issue or something,” I thought. So I reset my router, tried changing DNS settings (google’s 8.8.8.8, then cloudflare’s 1.1.1.1) but nothing fixed it.

As I started to Google for problems, I noticed that not only were my personal sites—which are hosted through netlify—timing out, but even netlify’s own sites (netlify.com) were timing out as well.

The weird thing is sometimes the sites will start loading. Then at times, they’ll start timing out again. When they were timing out, I tried running dig and on the domains that weren’t working and doing a curl but things were timing out:

❯ dig macosicongallery.com

; <<>> DiG 9.10.6 <<>> macosicongallery.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 56569
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
;; QUESTION SECTION:
;macosicongallery.com.        IN    A

;; ANSWER SECTION:
macosicongallery.com.    19    IN    A    104.248.78.24

;; Query time: 276 msec
;; SERVER: 8.8.8.8#53(8.8.8.8)
;; WHEN: Mon Oct 21 08:19:21 MDT 2019
;; MSG SIZE  rcvd: 65


~
❯ curl macosicongallery.com
curl: (7) Failed to connect to macosicongallery.com port 80: Operation timed out

~ 1m 25s

Strange.

Then I tried traceroute for another of my failing domains:

traceroute to jim-nielsen.com (104.248.78.23), 64 hops max, 52 byte packets
 1  192.168.86.1 (192.168.86.1)  11.183 ms  9.724 ms  17.699 ms
 2  * * *
 3  * * *
 4  * * *
 5  * * *
 6  * * *
 7  * * *
 8  * * *
 9  * * *
10  * * *
11  * * *
12  * * *
13  * * *
14  * * *
15  * * *
16  * * *
17  * * *
18  * * *
19  * * *

Just timing out.

Then I noticed that when I was connected to a VPN, the issue was resolved. Whenever I had my VPN connection on, all my personal domains as well as netlify’s own domains, loaded just fine.

So even connecting to answers.netlify.com isn’t possible from my home network unless I’m connected to my VPN (that’s how I was able to post this message).

So, in summary, I am not an network specialist but this appears to be some kind of routing issue with my ISP? My ISP is TDS. I suspect I am going to have to call them and ask to speak with someone in their network division about a routing issue to any netlify domains, but I figured I’d post this here and see if anyone has heard of such a case.

Sorry to hear about the trouble, Jim!

That is a weird one. I wouldn’t necessarily treat traceroute as the final source of truth or a good debugging tool - it is possible to configure your router as well as have things in your ISP’s land that can give inconsistent results. However, if you can traceroute to something else like 8.8.8.8 then your diagnosis is probably correct.

Like you I suspect that (assuming you can’t easily connect without your home router in the way) the next step will be talking with your ISP to see if there are blocks in place. There will be some challenges in your quest! Namely, that our CDN uses dozens of IP addresses and they change all the time.

Hopefully the block is IP-based and they can just remove it, but it might be the case for literally dozens of other IP’s too. I hope you’ll follow up here to let us know if they had anything “interesting” to say about it such as “we blocked because of XYZ” and my team will be happy to write to them to work through a better process there (like, “write to fraud@netlify.com if you have some problem with a site on our CDN as we address complaints quickly” or “that testing service you rely on seems misinformed, maybe we can work with THEM to fix things”).

I guess what I am trying to get to here is you should see if there are other IP’s serving your area that you can’t reach to get them all unblocked - they aren’t necessarily “network nearby” (different hosting providers for resiliency). Here’s a current list for the west coast US where I assume you are based on that IP address being returned for your hostname:

54.219.163.3  
54.183.247.139 
13.57.194.234  
104.248.78.23
104.248.78.24
206.189.73.52
35.230.62.222
35.197.55.186 
35.185.197.129
35.233.232.239
104.198.13.179
35.197.58.10
104.198.14.52

Let us know how it goes!

1 Like

Appreciate the feedback, thank you!

I’ve waited a couple more days to see if this issue just resolved itself, but unfortunately it did not.

I called my ISP today. Got bounced around a bit and finally talked to a guy who insisted that it wasn’t anything on their side. He said it was definitely something on my side, be it a modem or router setting of some sort. I mentioned that the problem goes away as soon as I connect to the VPN, and he said that was evidence it was a problem on my side, not the internet connection they were supplying to my house (unsure if that’s true?)

So, with all of that said, I’m a bit at a loss of what to do. I’m not a network expert by any means. Is there any chance that you could give me a few concrete things to try? So far I’ve:

  • Reset home router (google wifi)
  • Reset cable modem

But I still see the issue. I’m not exactly sure how I can troubleshoot here.

I guess what I am trying to get to here is you should see if there are other IP’s serving your area that you can’t reach to get them all unblocked

You gave me a list of IPs, but what should I do with those? Run traceroute on them as well? I tried the first one on the list and got nowhere:

traceroute to 54.219.163.3 (54.219.163.3), 64 hops max, 52 byte packets
 1  192.168.86.1 (192.168.86.1)  5.639 ms  7.170 ms  4.221 ms
 2  * * *
 3  stgrutfkhed12-lag12-75.network.tds.net (69.130.31.157)  23.545 ms  14.415 ms  14.983 ms
 4  h64-50-245-78.mdsnwi.tisp.static.tds.net (64.50.245.78)  25.595 ms  27.609 ms  26.792 ms
 5  99.82.177.90 (99.82.177.90)  25.526 ms  26.956 ms  26.721 ms
 6  * * *
 7  * * *
 8  54.239.41.177 (54.239.41.177)  139.147 ms  37.578 ms *
 9  * * *
10  54.240.243.156 (54.240.243.156)  44.769 ms * *
11  52.93.70.157 (52.93.70.157)  44.646 ms *
    54.240.243.216 (54.240.243.216)  40.603 ms
12  52.93.70.166 (52.93.70.166)  33.960 ms
    52.93.141.133 (52.93.141.133)  37.222 ms
    54.240.243.213 (54.240.243.213)  40.408 ms
13  54.240.243.63 (54.240.243.63)  33.881 ms
    54.240.243.23 (54.240.243.23)  36.299 ms
    54.240.243.83 (54.240.243.83)  32.901 ms
14  52.93.47.96 (52.93.47.96)  33.947 ms  35.048 ms
    72.21.222.220 (72.21.222.220)  37.370 ms
15  * * *
16  * * *
17  * * *
18  * * *
19  * * *
20  * * *
21  * * *
22  * * *
23  * * *
24  * * *
25  * * *
26  * * *
27  * * *
28  * * *
29  * * *
30  * * *
31  * * *
32  * * *
33  * * *
34  * * *
35  * * *
36  * * *
37  * * *
38  * * *
39  * * *
40  * * *
41  * * *
42  * * *
43  * * *
44  * * *
45  * * *

I still have intermittent issues accessing netlify.com or some of my domains hosted through netlify. What’s strange is sometimes, for example, I can access one my hosted sites but not another. And sometimes netlify.com will work, while one of my hosted sites won’t. There doesn’t seem to be any rhyme or reason as to which ones will work and when. The only thing that consistently works is connecting to my VPN provider and then I don’t have any issues at all. That allows me to continue to access and deploy sites through netlify, but it’s kind of a huge bummer that I have to remember to turn on my VPN every time to do it.

Here’s an example from today. I tried traceroute on my personal site and got nowhere:

traceroute to jim-nielsen.com (104.248.78.23), 64 hops max, 52 byte packets
 1  192.168.86.1 (192.168.86.1)  6.898 ms  5.020 ms  6.191 ms
 2  * * *
 3  * * *
 4  * * *
 5  * * *
 6  * * *
 7  * * *
 8  * * *
 9  * * *
10  * * *
11  * * *
12  * * *
13  * * *
14  * * *
15  * * *
16  * * *
17  * * *
18  * * *
19  * * *
20  * * *
21  * * *
22  * * *
23  * * *
24  * * *
25  * * *
26  * * *
27  * * *
28  * * *
29  * * *
30  * * *
31  192.168.86.1 (192.168.86.1)  11.651 ms !N * *
32  * * *
33  * * *
34  * * *
35  * * *
36  * * *

A minute or two later, I tried again and got this:

traceroute to jim-nielsen.com (206.189.73.52), 64 hops max, 52 byte packets
 1  192.168.86.1 (192.168.86.1)  5.055 ms  6.222 ms  5.248 ms
 2  * * *
 3  stgrutfkhed12-lag12-75.network.tds.net (69.130.31.157)  20.834 ms  14.367 ms  13.843 ms
 4  h64-50-245-76.mdsnwi.tisp.static.tds.net (64.50.245.76)  28.834 ms  25.319 ms  25.607 ms
 5  h64-50-243-202.mdsnwi.tisp.static.tds.net (64.50.243.202)  25.577 ms  30.944 ms  27.134 ms
 6  palo-b22-link.telia.net (62.115.119.90)  43.816 ms  36.390 ms  37.597 ms
 7  digitalocean-ic-336107-palo-b22.c.telia.net (213.248.99.163)  40.539 ms
    digitalocean-ic-336103-palo-b22.c.telia.net (213.248.99.161)  36.157 ms  45.827 ms
 8  * 138.197.249.187 (138.197.249.187)  39.211 ms *
 9  * * *
10  206.189.73.52 (206.189.73.52)  38.210 ms !Z  39.477 ms !Z  39.845 ms !Z

Then I tried with the VPN on:

traceroute to jim-nielsen.com (104.248.78.23), 64 hops max, 52 byte packets
 1  10.8.2.1 (10.8.2.1)  52.150 ms  59.042 ms  53.754 ms
 2  6bb6ea00.lon.100tb.com (107.182.234.66)  51.671 ms  51.578 ms  56.943 ms
 3  206.130.126.25.west-datacenter.net (206.130.126.25)  52.330 ms
    206.130.126.29.west-datacenter.net (206.130.126.29)  53.374 ms  56.283 ms
 4  * * salt-b1-link.telia.net (62.115.147.104)  54.805 ms
 5  palo-b22-link.telia.net (62.115.140.52)  69.409 ms  76.165 ms  73.507 ms
 6  digitalocean-ic-336103-palo-b22.c.telia.net (213.248.99.161)  70.821 ms
    digitalocean-ic-336107-palo-b22.c.telia.net (213.248.99.163)  73.228 ms
    digitalocean-ic-336103-palo-b22.c.telia.net (213.248.99.161)  71.393 ms
 7  * * *
 8  * * *
 9  104.248.78.23 (104.248.78.23)  73.165 ms !Z  71.517 ms !Z  72.898 ms !Z

Honestly, I have no idea what to try. My ISP says it’s not their issue. I’ve tried resetting everything at home. But for some reason, netlify-related sites still quite frequently will just time out through my home network (unless I’m connected to a VPN)

1 Like

I’d like to add that I’m also seeing this issue, and also use TDS as an ISP. Also, fine when trying a connection through VPN, but two of the IPs in that list are giving a continued problem: 104.248.78.23
104.248.78.24. The rest of the IPs work for my tests. I’ve filed a support ticket with TDS (today), and the issue seems to have persisted for about as long as you’ve noted here Jim.

2 Likes

I have the same issue when connecting from a VPN in Oregon, which uses TDS.net as one of it’s hops. When I test from my office in Washington, the 104.248.78.24 and 104.248.78.23 both work properly, but they are routing through level3.net (which is working). The other odd thing I noticed is that when using tracert from one of the impacted offices, we don’t get reverse-zone lookups for these IP addresses.

I have confirmed with our ISP who is owned by TDS.net that they are unable to reach the IP addresses mentioned above, yet we can reach them from our offices in WA and CA who do not use TDS.net as an ISP.

1 Like

I can’t post more than a single image per post, but here’s the test from Bend, Oregon (which uses TDS.net through Bend Broadband)

image

In WA using Level3.net for the same IP address

Kennewick, Washington

Additionally, I reached out to Bend Broadband (owned by TDS.net) and we tested the list of IPs provided by @fool to confirm that the IPs 104.248.78.23 and 104.248.78.24 are both blacklisted. We’ve opened an inquiry with their hosting team to get them unblocked. I might be worth having someone from netlify do the same

1 Like

I am also having this exact issue, frustrating because Netlify is my main platform for hosting. I have had TDS service for years and Netlify domains have been loading fine until recently… I called in to TDS and got the same answer, that they didnt have any blocks and it must be on my end. Sometimes I can get through and sometimes I can’t.

I appear to be having a similar issue. My ISP is SaskTel, I’m in Canada.

I got a report of one of my netlify sites being down, then double-checked and saw that I can’t access any of them, or even www.netlify.com itself.

Hmm, I can understand why this is frustrating. Are you able to verify that the sites are accessible via a different network, say mobile? Just want to rule things out.

Yes, in testing when using a VPN or Verizon (even via a tether on PC) the sites are all accessible. And to add to this, it only appears to be two IPs that fail (not all of them). 104.248.78.23 & 104.248.78.24

2 Likes

Are you able to verify that the sites are accessible via a different network, say mobile?

I can’t speak for everyone else, but as I mentioned in my original post, netlify-related sites work perfectly fine from mobile when they’re failing on my home WiFi (and if I switch to VPN then they start working)

We have a support engineer on the case and are going to investigate. This very frustrating :expressionless: He’ll be updating this thread once we have any insights to share. Thanks for patience while we take a look at things.

If any new insights come up for you all while we dig please keep us updated, appreciate it!

2 Likes

Awesome thanks @perry! I believe SaskTel (related to @web’s report) is also related to TDS Telecom, so I think it’s fair to assume it is related to this ISP

Quick look online reveals:

SaskTel International announced the largest software sale in its sixteen-year history – a contract was signed with TDS Telecom to provide the MARTENS® Service Provisioning and MAGIC (Modular Application Gateway Interface Connector) software products and related services.

surprise level = zero! :upside_down_face:

we’ll keep you posted as things develop.

Much appreciated :star_struck:, I’ve also got a ticket with TDS, and it looks like others might too, if I hear back I’ll report it here.

Thanks so much for taking the lead on that support case with them, Riley! We’re going to try reaching out too, but as I mentioned to you in the helpdesk our experience in the past with asking these types of providers to help has been “we won’t talk to you without an account ID”. Hoping that both our efforts can improve things!

1 Like

Is there anything I can do to work around the issue in the short term?

I have a couple low-impact sites hosted on Netlify, and am content to have more-or-less degraded service for those, and I’ve already migrated the one high-impact site that I hosted to another service. I’m not really in a position to follow up with TDS or SaskTel, or suggest that people choose a new ISP if they want to access the site.

For the record, SaskTel International seems to be quite a different company than SaskTel, despite the name, though a possible TDS link might still be the issue.

Here is a traceroute for one of my netlify-hosted sites, while we’re posting them:

traceroute to mostapt.com (104.248.78.23), 30 hops max, 60 byte packets
 1  homeportal (172.16.1.254)  6.374 ms  7.912 ms  7.842 ms
 2  * * *
 3  207-47-191-249.ngai.static.sasknet.sk.ca (207.47.191.249)  9.340 ms  10.020 ms  9.990 ms
 4  142.165.120.78 (142.165.120.78)  7.293 ms  7.261 ms  7.220 ms
 5  142.165.120.77 (142.165.120.77)  14.679 ms  15.150 ms  15.107 ms
 6  142.165.0.114 (142.165.0.114)  44.075 ms  43.507 ms  43.783 ms
 7  v716.core1.tor1.he.net (216.66.0.13)  39.282 ms  39.776 ms  39.756 ms
 8  100ge9-2.core2.chi1.he.net (184.105.80.5)  52.298 ms  52.909 ms  52.889 ms
 9  100ge3-1.core1.sjc2.he.net (184.105.213.113)  82.722 ms  82.182 ms  82.565 ms
10  * * *
11  * * *
12  * * *
13  * * *
14  104.248.78.23 (104.248.78.23)  80.644 ms !X  74.993 ms !X  75.415 ms !X

I spoke with TDS Internet Support this evening, as a representative of Netlify not a TDS customer. We (Netlify) now have a ticket opened with them about this. The support person at TDS took the matter seriously and read this community topic as well.

The issue is being escalated to their advanced internet support team and we should have another update about this sometime tomorrow. The link to this community topic as been included in the support ticket so the advanced team will see this topic as well.

Things are moving with this issue and I’m hoping we can get this resolved quickly now that all parties are in communication about it.

4 Likes

thanks for your work and the update, luke!