Followups from 25 Mar 2021 Service Degradation

On 25 March, Netlify had a major service degradation affecting any customers using DNS “A” records pointing to our load balancer with IP address 104.198.14.52. We’ve changed our guidance to use our newer load balancer IP 75.2.60.5, but both will work for at least the next couple of months.

We’ve resolved the service degradation, but are opening this thread in case you have ongoing problems that you think may be related to the outage; we’ll monitor and respond as quickly as we can.

6 Likes

Should we be moving all our sites over to the new 75.2.60.5 or do we leave it at 104.198.14.52 I’m assuming to plan to depreciate 104.198.14.52 . ?

Hi could help me? There is a website that i getting ‘my connection is not secure’. Has anything to do with it?

Felipecss’s

You do not need to migrate anything today, @GoNation . Both will keep working for the next couple of months at least, and before we make any future changes we’ll announce them broadly and via many channels including direct email.

3 Likes

Sure thing, just got www.felipecss.com fixed for you. Your one other site with a custom domain was not broken :slight_smile:

Let me know if anything still is not working as you expect!

You guys are awesome guys. Do I need to wait a few hours?

Thank you. ok we will continue to use 104.198.14.52 in any new sites until you announce we need to switch

I’m saying this because, if I try to open the site through the browser, it still accuses the certificate with an error … but if in private access it will go normal @fool

@fool is there anyway to subscribe to the status.netlify.com page via email? I’d like to be able to find out about outages like the one we just experienced before my clients do, but I don’t use twitter and I’m hardly nerdy enough to be into RSS feeds :upside_down_face:.

Otherwise: great efforts on the information flow and instructions on minimizing impact, that was a true life saver! :raised_hands:

1 Like

@lipecss Try clearing your cache and flush your DNS: Pearson Support

1 Like

Do you know if the old IP will eventually be deprecated in favor of the new one? Or is it recommended to wait?

It worked, thank you friend you are a very good friend :heart_eyes:

1 Like

No problem, glad it helped!

1 Like

No, I fixed it 1 minute after your tweet. If it still isn’t working for you, try in another browser - some browsers “cache” SSL failures even though the page loads for folks who hadn’t seen the failure before.

We don’t have any immediate plans for email subscriptions there, Tom. Best practices might be to pipe RSS or twitter into slack (this is what we do, for our status and our partners) using the apps provided, or instead use a service like zapier.com or ifttt to see the tweets or monitor the RSS feeds and mirror into your preferred channel.

2 Likes

The old IP will eventually be deprecated. We’ll work with our customers to migrate before that happens, so if you didn’t already change your DNS, no need to change anything today.

Alright, I might go for the slack route, that seems viable enough. Thanks!

2 Likes

I was trying to figure out how to prevent this issue from happening again to all of our customers because I believe this happened a few months back due to the same reason. Using Netlify DNS is not really ideal for many of our domains that use many AWS services. How come you guys don’t keep both online and provide multiple IPs so customers with more advanced knowledge can setup DNS failover at the least.

I considered setting up my own reverse proxy, but failing that over with SSL and stuff is a bit annoying with LetsEncrypt and what not so that idea is pretty much squashed.

Your reverse proxy also would suffer the other shortcomings of proxying to our service described here, @jclusso : [Support Guide] Why not proxy to Netlify?

We’ll be having our retrospective call tomorrow where we discuss the future of our network layout and we’ll publish a root cause analysis describing the measures we’re taking to protect you going forward in it. I’ll link that here for your consideration.

As to why we weren’t configured that way today, I can’t speak to that since it wasn’t my decision, but I suspect that RCA will shed some light on the history and show a future more compatible with your wishes.