I am on https://johnathanguzman.netlify.app/.
There is no x-nf-request-id provided for this.
My Ip address is 2601:241:8701:a400:45a9:762e:627d:96a1.
I don’t know where the IP address for the cdn node is.
I made this request on Oct. 2 2020.
The time of request is 10:31am.
And the timezone is CST.
If you are able to check the IP address being returned for johnathanguzman.netlify.app, I’ fairly sure we will be able to confirm that this is not a Netlify controlled IP address.
Note, there are dozens of different IP address for this one domain name. Which IP address is returned will depend on the geographic location of the request making the DNS lookup. You can see this here:
Looking at some examples, for Lille, France one IP address is: 188.8.131.52. This is an IP address which routes to a system in or near Frankfurt, Germany.
For São Paulo, Brazil one IP address is: 184.108.40.206. This is an IP address in or near São Paulo.
When I look up the ownership of these IP addresses, I can confirm they are part of the networks of the cloud service providers (DigitalOcean, AWS, Packet, etc) our CDN nodes are built using. I also see that they are IP addresses we (meaning Netlify) currently control.
If you perform similar DNS lookups locally, I’m guessing that the IP address returning the ‘xfinity’ SSL certificate will be an IP address owned by your ISP.
So, how do you fix this? First, you might contact your ISP’s technical support ask them about why this DNS lookup isn’t being returned correctly by their DNS service.
Alternatively, you might just switch to a different DNS resolver and skip the your ISP’s DNS entirely. (There are even encrypted DNS services to prevent your ISP from observing or modifying your DNS lookups.) You might change local computer to use DNS from OpenNIC or Google. Please note those two projects are likely polar opposites in respect to how your data will be used when using them.
One last thing about changing the DNS service. If you do decide to do this, most home networks control this automatically for all devices in the household in the wireless router setup. If you pick a different set of IP addresses to use for DNS resolvers (like 220.127.116.11 and 18.104.22.168), you might change these defaults in your wireless router’s “DHCP” settings so all devices on your network will use those settings. (This can sometimes much faster than changing the setting on several devices manually. You may still need to reconnect the devices by turning wifi off/on or rebooting the devices.)
To sum up, either contact the ISP technical support about the DNS issue (if that is what it is). Again, if DNS is the issue, you might also just switch your DNS service.
If there are questions about any of this, please let us know.
Hi, @gpickett00. I don’t think we 100% confirmed the root cause here and there was no resolution reported.
From the SSL certificate returned and the behavior described, I’m personally 99.99% sure this was a case of DNS hijacking by an ISP:
To summarize, I believe @Jguz17’s ISP was sending him the wrong IP address for our domain name when the DNS query was sent to their DNS server. The SSL certificate was the SSL for this ISP’s webserver and because they redirected the DNS to their IP address and not the one for the real site at Netlify. The SSL was wrong because he was sent to the wrong server by his ISP. As the wiki page above mentions, this is unfortunately an all too common practice for even major ISPs.
This could be what is happening in your case but we need more information to be sure. We need the DNS lookup for the domain name and we also need to know what DNS server is answering. On many systems, you can get this information with nslookup. For example:
In the output above I can see for my query about the IP address for “example.netlify.app” the answer was two the IP addresses below:
The DNS server that returned the answer had the IP address of 22.214.171.124.
If you are experiencing DNS hijacking, I would recommend changing the IP address for your network’s DNS resolver:
You can also try contacting the technical support team for your ISP and ask them to correct the behavior of the DNS server which failing to return the authoritative DNS records for the zone.
Again, I’m not sure if this is even what is happening to you (but I do believe this was the issue for @Jguz17). Would you please confirm exactly the error message you are seeing and, if you think it is pertinent, would you also send us the nslookup output for the site with this issue?
It’s been a while since this thread has had any action, but I wanted to see if there was any solution to this issue. I am using Netlify for our company’s web applications and have heard of 3 instances of a similar issue. We have 3 apps, 2 are a subdomain of one domain name and the other one has it’s own domain name. All 3 are being affected. It seems to only be happening with Xfinity/Comcast users and in particular, since we are a Portland Oregon based company, the 3 occurrences have been in the northern half of the Portland area and just over the Washington state line. This issue is not happening south of Portland. The error that shows up for them is this: ERR_SSL_PROTOCOL_ERROR - I can include nslookup screen shots too if necessary. Thanks!
Hi, @cmonster52. I’m showing that as a successful page load and no SSL issues for that request id. Also, the x-nf-request-id doesn’t get sent if the browser closes the connection because of SSL errors which is why the other information was requested.
I’m asking for this information it will tell us which CDN node and exactly which HTTP response was incorrect. These details are important in tracking down the reason for the error.
Are you sure there was an SSL error for that exact x-nf-request-id? If so, did they explain how that got that header even though the SSL negotiation failed?
If it was actually a successful pageload, then I would still need the information for an instance where there was an SSL error to troubleshoot.
Hi, @luke . I am 100% certain that the ERR_SSL_PROTOCOL_ERROR happened on this exact x-nf-request-id. I did the request myself on my coworker’s computer. This has never happened when I load up our sites from my computer in Indianapolis IN, so this adds complexity to the situation. Here is a screenshot of what the browser loads when it’s failing …
As for which CDN node … I do not know exactly how to find this. If it is in the same “Response Header” section as the x-nf-request-id, there wasn’t any sort of record that supplied an ip address for the CDN. I have seen on successful pageloads, that the header “server” has a value of Netlify. Will you explain further on how to get this ip address please? Thanks
I have set 3 of my coworkers up with the Opera browser with the built-in vpn set to on and it allows them do their work. However, I’m pretty sure that our clients are experiencing this as well & therefore need to figure out what’s going on.
Hi, @cmonster52. I do not understand how you got that header:
If there is an SSL error, browsers will close the HTTP connection before any headers are sent. This is why I asked this:
I asked because it is normally impossible to get that header with that error. So, if you did get the header, you must have made some change to allow the HTTPS connection to continue with the invalid SSL certificate. I’m asking what you did to allow that header to be received.
I’m concerned that what occurred is that a second pageload was made that did return a header but that would only happen is SSL succeeded so then I would be looking for issues on the wrong CDN node.
Would you please explain more about how you managed to see that header when the browser should have closed the connection?
I need to know the IP address (which then allows me to know which CDN node caused this) that responded. I’ve tried to determine which browser you are using in the screenshot but I’m not sure which that is. I’ve looked for documentation for how to see the IP address for SSL errors using Chrome and I cannot find the documentation on how to do that.
Personally, I don’t ever troubleshoot SSL issue using browsers (I use a combination of curl and openssl on the command line) specifically because I find browsers themselves not at all useful for troubleshooting these types of issue.
Using curl for example, one can see the IP address returned in the first few lines if the -v (high verbosity) option is used, example below:
$ curl -sv https://admin.successionresource.com/login 2>&1 | head -n 5
* Trying 126.96.36.199...
* TCP_NODELAY set
* Connected to admin.successionresource.com (188.8.131.52) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
The line with “Trying 184.108.40.206…” is where curl shows the IP address.
Also that is a cloudflare IP address not a Netlify IP address. This means it will become much harder for us to troubleshoot as we cannot actually see your full connection. In other words, because you proxy to Netlify, everything downstream between us and Cloudflare and between Cloudflare and the browser becomes much harder (or impossible) for us to see. For example we have a support guide about how proxying to us from Cloudflare blocks SSL certificate provisioning here.
Our recommended DNS configuration for third-party DNS can be found here:
The the recommended DNS record would be the one below (and replacing <site-subdomain-here> with the actual site subdomain under netlify.app):
admin.successionresource.com. 1800 IN CNAME <site-subdomain-here>.netlify.app.
So, I’m not sure how to tell you how to gather the IP address. Personally, I’d just use curl but please use whatever solution you decide is best for you.
Again, unfortunately, even when you do send the IP address information it still may not help me as I still won’t be able to match that IP address to a CDN node at Netlify (because of the Cloudflare proxy).
You could still try to send me this information but I must be clear that while this site is proxied to Netlify from Cloudflare I am greatly limited when troubleshooting these SSL issues.
To reiterate, the information needed is:
the complete URL requested (and you did send this but it if you test using a different do let us know)
the IP address for the system making the request (you sent this before but if it is different)
the IP address that responded (even if it is Cloudflare)
the day, time, and timezone of the request
Yes, the x-nf-request-id can be used instead but I’ve very hesitant to use those for troubleshooting as I do not yet understand how that header is being gathered during testing.
Last but not least, you might consider not proxying to see if that itself will resolve the issue. Even if it doesn’t it will remove all the limitations that proxying adds to the troubleshooting process.
If there are questions about any of this, let us know and I’ll do my best to answer.