Home
Support Forums

Netlify DNS failing again, or hacked?

Continuing the discussion from Followups from 25 Mar 2021 Service Degradation:

My only 2 domains using Netlify DNS just started failing again, after the March 25th outage. Upon checking the DNS settings, I see that the records have been changed again, making the domains point to malicious sites.

Instead of the usual name.netlify.app NETLIFY records automatically set by the system for the domain, both domains were now pointing to this A record only: 5.45.72.68. If you browse to that IP, you’ll get the same site to which my domains are/were pointing. Any other records were removed or missing, including MX records, which I had just set again on March 25th.

Either your system is doing something wrong, which Chris @fool identified the last time, or my account was hacked and someone accessed it directly and changed the DNS. Is it possible for you to check the logs to see if there’s any activity related to that second scenario? The problem seems to have started today, probably not long ago.

The domains in question are unidospodemos.cr, which is still pointing to that bogus record so that you can review, and phoabogados.com, which I already changed the records back to what they should be.

I really appreciate you checking into this, so that we can avoid the issue happening again.

Thanks in advance.

David.

hey david, thanks for reaching out! we are going to take a look and get back to you when we can. Stay tuned.

1 Like

Hi David,

Thanks for your patience while our security team investigated.

We found in our audit logs that your User ID made those changes to the DNS records, and when we researched further, we strongly believe that it was an attacker who compromised your password. Thanks to your report, we found a handful of other user accounts with a similar pattern. We are working on a process to remove all malicious records and notify the other affected customers now.

We strongly recommend setting up two-factor authentication (2FA) to protect yourself against situations like this in the future.

1 Like

Thanks a lot Chris. I already changed my password and enabled 2FA. However, I’m not sure if I had already set a password before, as I always login using Github and I had no password for Netlify saved in my password manager. But there’s a chance I did in the past and just don’t remember. Anyway, I wonder how they could have got hold of my password. Is there a chance that they could have got it right from your system, or were just able to break in? I also reset Github, just in case.

I would be checking through a few third party services such as HaveIBeenPwned to see if your credentials have been compromised on another service, where your p/w may have been common across another service and ours.

There’s no risk or vulnerability at our end however there’s always a risk if common credentials are used. 2FA should help tremendously in these cases!

One more detail to share: we can be sure you did have a password set, because the attacker used it, and he couldn’t have accessed your account without it (unless he broke in via GitHub and set one, and then used it, which seems pretty unlikely. The session where he changed your DNS was initiated with password, though.)

I encountered a similar compromise tonight, different IP and domain than @davidvm reports, but similar circumstances. This was also an A record prepended in Netlify DNS. Happy to share specifics with staff. I’ve since rectified the issue and rotated security keys / authentication. Curious to know how this appears in audit logs. Credentials are unique to Netlify, so this could not come from elsewhere unless Netlify or Netlify DNS itself has been breached.

Hello there, @yann! Thank you so much for reaching out about this, and welcome to the Netlify Forums. We are going to take a look and get back to you as soon as we are able to!

Hi @yann I will escalate your message to the helpdesk where you can share the specifics more privately. Look for our email, and thank you for letting us know about this issue!

2 Likes

Just to feed back, we’d like to remind all users to ensure that 2FA is enabled where possible – particularly on your mailboxes. In fact, consider all these best practices!

1 Like

Thanks @Scott and @fool, good to know. I do seem to remember that I once set a simple password to be able to access from another location easily, and then probably forgot to change it. I’m curious about Yann’s case, though.

Hi there, @davidvm :wave:

Thanks for circling back. While I cannot speak to Yann’s private details, we will leave this thread open for now so that they can chime in should they choose to.

1 Like

Hi folks, thanks for the nudge @davidvm, I’m looping back on my experience here.

Netlify support engineers confirmed that the suspicious A record was in fact there when I first migrated the site to Netlify / Netlify DNS (Feb 2021). The site previously used Cloudflare DNS and their audit logs don’t go back far enough to see if or when this record was introduced.

There’s probably an unexciting explanation for the malicious record. Sorry I don’t have anything more definitive or conclusive than that.

Of course multi-factor authentication and frequently rotating API keys and passwords remains important, but this has also prompted me to maintain DNS monitoring across domains I control.

1 Like