All functions calls suddenly started failing

Support
1

Last Friday, all of a sudden all of our calls to lambda functions suddenly started failing with code 500. Only one call in maybe 20 ran successfully. This happened for all the endpoints, ever for the ones that don’t have an error condition that can return code 500. Our domain is https://www.dreamingspanish.com/

The logs showed nothing, even right after calling the endpoints.

This continued for several hours. Redeploying didn’t fix it. Redeploying with a clear cache didn’t fix it. Redeploying a previous version didn’t fix it.

In the end, I made a change in one of the endpoints that changed just one string literal and didn’t affect functionality, and that endpoint started working correctly. Making changes in all of the endpoints and deploying them fixed the situation.

What happened? Netlifystatus.com doesn’t show any disruptions for July 15. I didn’t get notified in any way that a problem was happening until users started sending emails in.

Can you find the source of the problem and give some assurance that the problem has been solved?

2

Hey there, @PabloRomanH :wave:

Thanks for reaching out, and welcome to the Netlify Forums! Sorry to hear you encountered this obstacle. Can you share a link to one of the deploy logs from the re-deploys where you tried to fix this? This will help us look into the occurrence further.

3

This is the log of a normal redeploy with clean cache: JustPaste.it - Share Text & Images the Easy Way

And here after introducing a change on each lambda function to force them to get updated (without any changes in functionality): JustPaste.it - Share Text & Images the Easy Way

5

Hi @PabloRomanH , it looks like you have a Helpdesk ticket opened for this issue. We’ll work with you from there!