Site going down every 10 minutes

Your Netlify Status can’t possibly be correct. Our Nextjs site has been going down every 10 minutes for hours already. We haven’t changed a thing on this site in a month. It was working very smoothly for a long time. Suddenly today every API route returns a 500 status. Site goes down. There is something wrong at Netlify. We have checked every single other API and it’s totally an issue at Netlify.

2 Likes

I was just about to post about this also. Our site has been up and down for the past hour.

@mrwiyums Thanks for confirming. Glad to know its just not us. Always strange when the Status page never shows any issues, but you know for sure there are issues. I kind of think the status is just manually updated. i don’t understand why they don’t have staff pinging the servers in some way.

Hello, @mrwiyums and @osseonews :wave:t6: welcome to the forums! I let our engineering team know. Thanks for bringing this to our attention. I will reach back out when I have an update from the team.

Please fix your status pages. Our application is affected and going down every 5 minutes. I shouldn’t need to google, your forums, visit the main forum page, just to see that the issue I’ve been diagnosing for the last hour is on your end.

2 Likes

Hi folks and sorry to hear about the trouble - I do understand how impactful misbehavior of our system is on your websites and that our status page not being up to date makes things harder to debug!

We did finally get a retroactive statuspage entry for the situation here: Netlify Status - HTTP 500 errors for visitors in Northeastern USA region on our Standard Edge CDN - could you please review to see if the timestamps line up with your experiences? Are your sites hosted on the Standard CDN (CNAME records to your sitename.netlify.app and using a load balancer of 75.2.60.5 would indicate this), and were affected visitors in the NYC area? This incident would have affected:

  • <10% of our traffic to the NYC area
  • it was only uncached traffic that was impacted
  • and that area is only a small part of our CDN and visitors outside of that area would have talked to data centers that were unaffected.

If you or your visitors are still experiencing the situation, this would likely have a different cause and we’d be happy to investigate further if you can share new request ID’s with us showing the problem.

We are still working to understand the root cause (so we can put safeguards in place to prevent recurrence), but we will be having our usual retrospective meeting and follow-up work that we have after every service incident to improve things, so I do want to acknowledge that our work isn’t done yet.

Please let me know if you have any questions, will be happy to work further with you around this or other issues.

This seems to fit our situation. However, we are running stuff from AWS in the Northeast, and those servers were the ones throwing the Netlify 500 errors. So basically it doesn’t have to be an individual person trying to access the page. If you have serverless functions running on aWS that need to hit Netlify then you are down also. Luckily, things seem back to normal for us again.

Hi @osseonews, glad to hear that! Our sincerest apologies for the inconvenience. I understand your frustrations.

FYI. We had customers all around the country reporting the issue from Washington State to Virginia.

Thanks for the candid feedback, @mrwiyums .

Could you please let me know some failing URL’s so I can further investigate? This is a sign that either you have misconfigured DNS for your site (and are mistakenly routing traffic across the country unnecessarily), or you perhaps have some other problems that were not related to this incident.

Just to follow up. We are now facing the same issue again with Netlify. Site goes down every 10 minutes. This is the same exact issue as last week. As per last week, suddenly today every API route returns a 500 status. Site goes down. There is something wrong at Netlify. We have checked every single other API and it’s totally an issue at Netlify. As per last week, we haven’t changed a thing. We never touch the code on this website anymore. It’s just that out of the blue Netlify servers just go down and throw 500 status on every API endpoint. Very disappointing.

1 Like

Same for us again also. Started around 8:40 AM EST.

1 Like

How can Netlify status not catch these errors? We now ping our API routes every 10 minutes to get a status and discover an error immediately. Netlify status says nothing. But, 100% Netlify is having issues. Same issues as last week.

Do I need to post another new topic about this? B/c it doesn’t seem like Netlify support responds to this thread anymore. Netlify status page, btw, is totally bogus. I don’t know how Netlify tracks the actual status.

Yes, for us, our first downtime was 8:36 AM EST

I posted one about 15 minutes ago. No response there yet either.

hi @osseonews and everyone

our team is already looking at this and will be posting a StatusPage soon. Apologies for the inconvenience, we’re already sorting this out :slight_smile:

Man this is killing us. Busiest time of year. Now our site is completely down with an: Error - Request ID: 01GM5ZD3FWMNT9R78R8ZKRRBYT

Jamstack should never go down. that’s why we moved it Jamstack. PHP, we saw this all the time.

Thank you. But shouldn’t your status page update automatically so WE KNOW you’re aware of the issue and looking at it. It’s been over an hour and the status page still says all is well. This is a common occurrence and frustration.

How does the status page say, “operational”. Our entire site is down!