I have custom SSL/TLS certificates configured in the dashboard but the Netlify server stopped providing them to the clients today. Instead I now get the certificate for *.netlify.com.
My domains are https://ilin.dk and https://likr.xyz. It worked until yesterday.
hey there, it seems like both of those sites have SSL certs - can you explain a little bit more what your concerns are if they are still happening?
The error has been fixed since I reported it.
You are right that both sites have SSL certs, both are Cloudflare Origin certificates provided and signed by Cloudflare authority with expiration date set for 15 years in the future. Cloudflare is also set up as a caching frontend for both websites and it is proxying the requests in “Full (strict)” SSL mode (meaning that these certificates are basically seen only by them).
In the morning both of my websites stopped working and began reporting a Cloudflare-branded 526 error https://support.cloudflare.com/hc/en-us/articles/115003011431#526error. I checked the Netlify server using some online tool and it did not serve the correct (my custom) certificate.
I’m seeing this issue again right now.
hi @nifalse, thanks for your patience. I am asking one of our Support Engineers with DNS expertise to look into this when he comes on shift.
Hi, @nilfalse, the custom SSL certificate uploaded to our service is not correct and giving SSL errors.
$ curl -svo /dev/null --resolve likr.xyz:443:104.198.14.52 https://likr.xyz/
* Added likr.xyz:443:104.198.14.52 to DNS cache
* Hostname likr.xyz was found in DNS cache
* Trying 104.198.14.52...
* TCP_NODELAY set
* Connected to likr.xyz (104.198.14.52) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: /etc/ssl/cert.pem
CApath: none
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
} [222 bytes data]
* TLSv1.2 (IN), TLS handshake, Server hello (2):
{ [102 bytes data]
* TLSv1.2 (IN), TLS handshake, Certificate (11):
{ [1194 bytes data]
* TLSv1.2 (OUT), TLS alert, unknown CA (560):
} [2 bytes data]
* SSL certificate problem: unable to get local issuer certificate
* Closing connection 0
If the SSL certificate is ignored, the correct site does get served. The SSL certificate is the issue:
$ echo | openssl s_client -showcerts -servername ilin.dk -connect 104.198.14.52:443 | openssl x509 -text | head
depth=0 O = "CloudFlare, Inc.", OU = CloudFlare Origin CA, CN = CloudFlare Origin Certificate
verify error:num=20:unable to get local issuer certificate
verify return:1
depth=0 O = "CloudFlare, Inc.", OU = CloudFlare Origin CA, CN = CloudFlare Origin Certificate
verify error:num=21:unable to verify the first certificate
verify return:1
DONE
Certificate:
Data:
Version: 3 (0x2)
Serial Number:
79:97:aa:40:39:94:84:85:e4:23:64:28:f4:e4:32:03:e5:88:63:8c
Signature Algorithm: sha256WithRSAEncryption
Issuer: C=US, O=CloudFlare, Inc., OU=CloudFlare Origin SSL Certificate Authority, L=San Francisco, ST=California
Validity
Not Before: May 30 20:51:00 2020 GMT
Not After : May 27 20:51:00 2035 GMT
Note, we do not recommend proxying to Netlify from other services. There is more about this here:
To resolve this, you will likely need to get a different SSL certificate or to correct the existing certificate.
If there are other questions about this, please let us know.
Thanks for checking my issue.
That’s the right custom certificate. It’s issued by Cloudflare so they can verify it in order to establish a secure connection between your server and theirs. That’s what I want, it used to work until 2 days ago.
However this is not the certificate I’m now getting from a Netlify server close by (165.22.65.139):
$ echo | openssl s_client -showcerts -servername ilin.dk -connect 165.22.65.139:443 | openssl x509 -text | head -n11
depth=2 C = US, O = DigiCert Inc, OU = www.digicert.com, CN = DigiCert Global Root CA
verify return:1
depth=1 C = US, O = DigiCert Inc, CN = DigiCert SHA2 Secure Server CA
verify return:1
depth=0 C = US, ST = ca, L = San Francisco, O = "Netlify, Inc", CN = *.netlify.com
verify return:1
DONE
Certificate:
Data:
Version: 3 (0x2)
Serial Number:
0b:2a:4c:da:86:2f:17:e1:64:93:20:a3:f5:b1:b0:e3
Signature Algorithm: sha256WithRSAEncryption
Issuer: C=US, O=DigiCert Inc, CN=DigiCert SHA2 Secure Server CA
Validity
Not Before: Jun 15 00:00:00 2020 GMT
Not After : Aug 3 12:00:00 2021 GMT
Subject: C=US, ST=ca, L=San Francisco, O=Netlify, Inc, CN=*.netlify.com
I understand your point about proxying but unfortunately I want to be able to run some extra Cloudflare features on my domains, like Cloudflare Workers etc.
I checked with Cloudflare’s support and they see the following error in their logs:
upstream SSL certificate does not match"ilin.dk" or “ilin.netlify.app” while SSL handshaking to upstream
Here are some more details for that message:
"id": "5c594a66619e0772",
"timestamp": "2020-08-20T04:14:59Z",
"severity": "error",
"message": "upstream SSL certificate does not match\"ilin.dk\" or \"ilin.netlify.app\" while SSL handshaking to upstream",
"client": "35.205.199.49",
"request": "GET / HTTP/1.1",
"upstream": "https://157.230.103.136:443/",
"host": "ilin.dk",
"location": {
"colo": "FRA03",
}
"message": [
"upstream SSL certificate does not match\"@kibana-highlighted-field@ilin.dk@/kibana-highlighted-field@\" or \"ilin.netlify.app\" while SSL handshaking to upstream"
]
Checking the IP from that message 157.230.103.136 with openssl command from Luke’s post I can confirm that the certificate served there is also the one for *.netlify.com and not *.netlify.app or my custom one.
Did something change in your setup recently?
There is also a timestamp availabe above in case you want to look through your logs.
The Cloudflare origin certificate you have on your site should never have worked. I’ve seen dozens of customers try to use it and NONE have succeeded. I can’t find the link now, but Cloudflare’s own docs say that that certificate is for internal use of their systems only and will not work in situations like this.
You’ll see if you try to connect to our service using it, it will fail. This is not unique to Netlify or any specific browser:
$ curl -v https://likr.xyz --resolve likr.xyz:443:178.128.17.49
* Added likr.xyz:443:178.128.17.49 to DNS cache
* Hostname likr.xyz was found in DNS cache
* Trying 178.128.17.49:443...
* Connected to likr.xyz (178.128.17.49) port 443 (#0)
* ALPN, offering http/1.1
* SSL certificate problem: Invalid certificate chain
* Closing connection 0
curl: (60) SSL certificate problem: Invalid certificate chain
More details here: https://curl.haxx.se/docs/sslcerts.html
You’ll need a certificate from a third party which you purchase, OR you’ll need to proxy to likr.netlify.app instead.
That’s the limit of the tech support we can provide for you proxying from Cloudflare, due to all the reasons listed in this article: [Support Guide] What problems could occur when using Cloudflare in front of Netlify?
I see your point. However, the problem I’m reporting here is irrelevant to the certificate validity. The problem is I’m actually seeing a different certificate when doing the same curl command as you did:
$ curl -v https://likr.xyz --resolve likr.xyz:443:178.128.17.49
* Added likr.xyz:443:178.128.17.49 to DNS cache
* Hostname likr.xyz was found in DNS cache
* Trying 178.128.17.49...
* TCP_NODELAY set
* Connected to likr.xyz (178.128.17.49) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: /etc/ssl/cert.pem
CApath: none
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
* ALPN, server accepted to use h2
* Server certificate:
* subject: C=US; ST=ca; L=San Francisco; O=Netlify, Inc; CN=*.netlify.com
* start date: Jun 15 00:00:00 2020 GMT
* expire date: Aug 3 12:00:00 2021 GMT
* subjectAltName does not match likr.xyz
* SSL: no alternative certificate subject name matches target host name 'likr.xyz'
* Closing connection 0
* TLSv1.2 (OUT), TLS alert, close notify (256):
curl: (60) SSL: no alternative certificate subject name matches target host name 'likr.xyz'
More details here: https://curl.haxx.se/docs/sslcerts.html
curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.
As you can see my request was served with *.netlify.com cert, not my custom one.
And if I try to use another Netlify’s server IP address I can see my custom certificate (which is what I want):
$ curl -v https://likr.xyz --resolve likr.xyz:443:104.198.14.52
* Added likr.xyz:443:104.198.14.52 to DNS cache
* Hostname likr.xyz was found in DNS cache
* Trying 104.198.14.52...
* TCP_NODELAY set
* Connected to likr.xyz (104.198.14.52) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: /etc/ssl/cert.pem
CApath: none
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (OUT), TLS alert, unknown CA (560):
* SSL certificate problem: unable to get local issuer certificate
* Closing connection 0
curl: (60) SSL certificate problem: unable to get local issuer certificate
More details here: https://curl.haxx.se/docs/sslcerts.html
curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.
What I want is to make sure that all clients that hit Netlify’s servers requesting my website should get responses signed by my custom certificate since that is what I configured in the admin panel. It’s then the responsibility of my client apps and web browser to verify the certificate, not Netlify’s 
Thanks so much for that clarification! Sounds like a separate issue which the team is investigating, so this data point is quite helpful as your certificate fails far more often than the other reproduction case we saw.
…of course, if we did serve the right cert, it wouldn’t matter from a visitor point of view - their browser would block them from connecting - since that cert is broken anyway, right?
That’s right, my certificate is only valid for Cloudflare’s internal clients. Ordinary browsers should reject connections that present this cert.
Is there anything else I can help with?
Nope! We’ll follow up here as the team understands and helps fix the problem. It’s nothing you’ve caused or can affect from your side.
Looks like this has recently been fixed? Both of my websites seem to have started working some time yesterday.
Good eye! We have a fix in testing but it is not finalized yet so I had not followed up here since it could get reverted if it causes other problems. I don’t expect that, but I also am not willing to bet your life on it.
Update: We have deployed a permanent fix for this SSL issue.
We hope this solves the issue on your end. Please let us know if you receive further reports of insecure messages.