Followups from 25 Mar 2021 Service Degradation

@lipecss Try clearing your cache and flush your DNS: Pearson Support

1 Like

Do you know if the old IP will eventually be deprecated in favor of the new one? Or is it recommended to wait?

It worked, thank you friend you are a very good friend :heart_eyes:

1 Like

No problem, glad it helped!

1 Like

No, I fixed it 1 minute after your tweet. If it still isn’t working for you, try in another browser - some browsers “cache” SSL failures even though the page loads for folks who hadn’t seen the failure before.

We don’t have any immediate plans for email subscriptions there, Tom. Best practices might be to pipe RSS or twitter into slack (this is what we do, for our status and our partners) using the apps provided, or instead use a service like zapier.com or ifttt to see the tweets or monitor the RSS feeds and mirror into your preferred channel.

2 Likes

The old IP will eventually be deprecated. We’ll work with our customers to migrate before that happens, so if you didn’t already change your DNS, no need to change anything today.

Alright, I might go for the slack route, that seems viable enough. Thanks!

2 Likes

I was trying to figure out how to prevent this issue from happening again to all of our customers because I believe this happened a few months back due to the same reason. Using Netlify DNS is not really ideal for many of our domains that use many AWS services. How come you guys don’t keep both online and provide multiple IPs so customers with more advanced knowledge can setup DNS failover at the least.

I considered setting up my own reverse proxy, but failing that over with SSL and stuff is a bit annoying with LetsEncrypt and what not so that idea is pretty much squashed.

Your reverse proxy also would suffer the other shortcomings of proxying to our service described here, @jclusso : [Support Guide] Why not proxy to Netlify?

We’ll be having our retrospective call tomorrow where we discuss the future of our network layout and we’ll publish a root cause analysis describing the measures we’re taking to protect you going forward in it. I’ll link that here for your consideration.

As to why we weren’t configured that way today, I can’t speak to that since it wasn’t my decision, but I suspect that RCA will shed some light on the history and show a future more compatible with your wishes.

Good to hear. Looking forward to the more reliable methods going forward. Just responding to this because when I saw the status page again with “this is only for people not using Netlify DNS” it makes you feel like unless you use Netlify DNS you don’t get the highest level of redundancy since that hasn’t seemed to have these short comings lately. All good though. Glad everything is back online!

@tomrutgers You can also use an RSS to email service. For example, I’ve been using Blogtrottr for several years to follow a couple of feeds via email. Feedrabbit is another option, or you can create Zapier zaps that send email, as well.

2 Likes

During yesterday’s outage, a client called me saying their email was not working. Getting called by them less than once a year, I checked quickly to see if their site and domain was running ok or if maybe they had let it expire. The site didn’t open as usual and I was led to a bogus domain aftermarket page, or something like that. It was actually the kind of page you’re redirected to when a site is hacked. So I first thought maybe the domain had long expired, but when I checked it, it wasn’t expired at all, and its DNS was still pointing to Netlify.

So going into Netlify to check things up, that’s when I found out about the outage. So as advised, I went on to change the old A record to the new one. I noticed that the two domains that were using Netlify DNS were not working, so I changed the A record for them both. However, the A record that was set initially was 185.212.128.141, and not 104.198.14.52. So I wonder why this IP was redirecting the site to some bad website. Was there any kind of hack involved here maybe?

After changing the A record the site was still offline, so I went on to remove the domain from the site settings and add it again, and that seemed to make it work.

However, this didn’t explain why email was not working. I checked, and the MX records were missing in Netlify. I didn’t remember how this was setup before, as it had been a long time, but the MX records should have been there because email had been working all fine. I added the MX records to the DNS and email started working fine again, naturally. So my concern is that somehow, probably related to the outage, MX records were removed.

Also, there’s something else funny going on. One of the sites isn’t up yet, unidospodemos.cr. Netlify gives a warning saying to check DNS configuration, and to point DNS’s to p02.nsone.net, which is exactly what is set at the registrar since long ago, but a NS lookup for the domain says they are p05.nsone.net. I tried removing them and setting them again to dns1.p02.nsone.net etc, but the lookup still says p05, Netlify keeps giving the error, and the site is still offline, despite there has been enough time for propagation. I tried removing the domain from site settings too, but same issue continues.

EDIT: Sorry, I confused things, my post isn’t quite clear but cannot edit it anymore.

I changed the A record for domains that are not using Netlify DNS, no problem with those.

But for the 2 domains that are using Netlify DNS, they both were opening bogus pages, and this IP was set in the DNS (probably as an A record, but I don’t remember): 185.212.128.141. I removed it, removed the domain in site settings, added it again, and now they both are using the NETLIFY record. One of the sites started working fine, the other one hasn’t:

unidospodemos.cr is still not working. The DNS is set correctly, DNS check is picking it correctly, but Netlify isn’t yet.

MX records were not present for domain phoabogados.com, which led for email failing. I think they somehow disappeared.

Hey @davidvm,

Sorry to hear you’re having issues! I’ve done a quick lookup and I’m not seeing any NS, CNAME or A records for www.unidospodemos.cr. I’d expect to see NS records for www.unidospodemos.cr just like I do for unidospodemos.cr so I think you’ll want to add this at your DNS provider!

Hi @Scott, thanks,

I’m using Netlify’s DNS. The domain’s name servers are pointing to Netlify correctly, but Netlify doesn’t seem to be picking it up, I keep getting “Check DNS configuration”, but it’s pointing to dns1.p02.nsone.net, 2, 3 and 4 since long ago. It was working fine before the outage, though.

Sorry about that - just seeing that somehow those www/unidospodemos.cr records shown in our UI were somehow “ghost records” - appear in our config database but not our DNS service. I’ve re-added them correctly, and they are working better now.

Thanks a lot. Happy to report that it’s working fine now.

1 Like

Thank you for confirming this, @davidvm!

1 Like

This topic was automatically closed after 14 days. New replies are no longer allowed.