On 25 March, Netlify had a major service degradation affecting any customers using DNS “A” records pointing to our load balancer with IP address 104.198.14.52. We’ve changed our guidance to use our newer load balancer IP 75.2.60.5, but both will work for at least the next couple of months.
We’ve resolved the service degradation, but are opening this thread in case you have ongoing problems that you think may be related to the outage; we’ll monitor and respond as quickly as we can.
You do not need to migrate anything today, @GoNation . Both will keep working for the next couple of months at least, and before we make any future changes we’ll announce them broadly and via many channels including direct email.
I’m saying this because, if I try to open the site through the browser, it still accuses the certificate with an error … but if in private access it will go normal @fool
@fool is there anyway to subscribe to the status.netlify.com page via email? I’d like to be able to find out about outages like the one we just experienced before my clients do, but I don’t use twitter and I’m hardly nerdy enough to be into RSS feeds .
Otherwise: great efforts on the information flow and instructions on minimizing impact, that was a true life saver!
No, I fixed it 1 minute after your tweet. If it still isn’t working for you, try in another browser - some browsers “cache” SSL failures even though the page loads for folks who hadn’t seen the failure before.
We don’t have any immediate plans for email subscriptions there, Tom. Best practices might be to pipe RSS or twitter into slack (this is what we do, for our status and our partners) using the apps provided, or instead use a service like zapier.com or ifttt to see the tweets or monitor the RSS feeds and mirror into your preferred channel.
The old IP will eventually be deprecated. We’ll work with our customers to migrate before that happens, so if you didn’t already change your DNS, no need to change anything today.
I was trying to figure out how to prevent this issue from happening again to all of our customers because I believe this happened a few months back due to the same reason. Using Netlify DNS is not really ideal for many of our domains that use many AWS services. How come you guys don’t keep both online and provide multiple IPs so customers with more advanced knowledge can setup DNS failover at the least.
I considered setting up my own reverse proxy, but failing that over with SSL and stuff is a bit annoying with LetsEncrypt and what not so that idea is pretty much squashed.
We’ll be having our retrospective call tomorrow where we discuss the future of our network layout and we’ll publish a root cause analysis describing the measures we’re taking to protect you going forward in it. I’ll link that here for your consideration.
As to why we weren’t configured that way today, I can’t speak to that since it wasn’t my decision, but I suspect that RCA will shed some light on the history and show a future more compatible with your wishes.