Home
Support Forums

Occasional 500 Internal Server Errors

Netlify site: vlerick-prd.netlify.app

Our customers website went live on 27.09 and we noticed several 500 errors since then.

Monitor logs for www.vlerick.com:

History

29/09/2021 14:54:12 Down, HTTP/1.1 500 Internal Server Error
29/09/2021 14:54:12 Down, HTTP/1.1 500 Internal Server Error
29/09/2021 14:54:11 Down (Partial), HTTP/1.1 500 Internal Server Error
29/09/2021 14:54:11 Warning, HTTP/1.1 500 Internal Server Error
29/09/2021 14:54:08 Down, HTTP/1.1 500 Internal Server Error
29/09/2021 14:54:07 Warning, HTTP/1.1 500 Internal Server Error
29/09/2021 12:57:12 Up, 156 msec
29/09/2021 12:57:12 Up, 156 msec

History

29/09/2021 15:37:00 Down, HTTP/1.1 500 Internal Server Error
29/09/2021 15:36:53 Down, HTTP/1.1 500 Internal Server Error
29/09/2021 15:36:52 Warning, HTTP/1.1 500 Internal Server Error
29/09/2021 15:36:32 Down (Partial), HTTP/1.1 500 Internal Server Error
29/09/2021 15:36:32 Down, HTTP/1.1 500 Internal Server Error
29/09/2021 15:36:31 Warning, HTTP/1.1 500 Internal Server Error
29/09/2021 15:36:31 Warning, HTTP/1.1 500 Internal Server Error
29/09/2021 15:28:10 Edited, See the history for details.

History

29/09/2021 15:37:12 Down, HTTP/1.1 500 Internal Server Error
29/09/2021 15:37:12 Down, HTTP/1.1 500 Internal Server Error
29/09/2021 15:37:11 Down (Partial), HTTP/1.1 500 Internal Server Error
29/09/2021 15:37:11 Warning, HTTP/1.1 500 Internal Server Error
29/09/2021 15:37:07 Down, HTTP/1.1 500 Internal Server Error
29/09/2021 15:37:06 Warning, HTTP/1.1 500 Internal Server Error
29/09/2021 14:55:12 Up, 1.031 msec
29/09/2021 14:55:12 Up, 1.031 msec

History

4/10/2021 11:32:15 Down, HTTP/1.1 500 Internal Server Error
4/10/2021 11:32:15 Down, HTTP/1.1 500 Internal Server Error
4/10/2021 11:32:13 Down (Partial), HTTP/1.1 500 Internal Server Error
4/10/2021 11:32:13 Warning, HTTP/1.1 500 Internal Server Error
4/10/2021 11:32:09 Down, HTTP/1.1 500 Internal Server Error
4/10/2021 11:32:09 Warning, HTTP/1.1 500 Internal Server Error
4/10/2021 10:12:01 Up, 192 msec
4/10/2021 10:09:13 Up, 300 msec

After investigation we noticed the following:

  • Every Gatsby build the _redirects file is updated, but Netlify deploys, based on logs, upload only actually changed files. Logs from Gatsby build:

  • We noticed some of Netlify deploys contain no uploads:
    image

  • Before 500s errors two deploys happen at the same time or close - one with file uploads and second following without new uploads. An example of deploys that were followed by an error:

Questions:

  • Is it possible that two close deploys may cause a file to not be accessible for very short period resulting in a 500 error?

  • The _redirects file is always updated in builds but usually has no changes and is not included in uploads during deploys. Are there any actions performed for _redirects in this case, that can cause a 500 error?

Hiya @Tim_Refaerts_Vlaming and sorry to hear about the trouble! I can see some unexpected internal errors on our side that I have asked our team for help diagnosing and fixing. I wouldn’t expect that the errors are related to the recency of a deploy, and I also doubt they are related to redirects since I see some paths that aren’t redirected that seem affected.

But, the errors are also like nothing I’ve seen before so I have had to call in the folks who wrote the code to help me understand.

Will follow up as soon as we have news!

1 Like

As extra information, today due to build delays on Gatsby Cloud, we had multiple builds queued, going one after another, which after deploy to Netlify caused multiple 500s being picked up by our monitor.

hey there @Tim_Refaerts_Vlaming - i am trying to follow up on this for you.

1 Like

Hello. We received extra information from our client, that during recent 500s on 5.10, there were several updates to redirect files. Comparing the timing in our monitor of these errors to the timing at which the redirect files were updated and built, the timing matches.

thanks for that info - we are still discussing this info, and i’ll make sure this info gets passed along.

1 Like

Hi, I am coming back again as today we are receiving more and more of the 500 errors. Additional information:

  • At the time of the issue the browser shows nothing and it actually returns an error 500 and the source contains no HTML:

We can now confirm that:

  • This is not a sensor glitch

  • The customer doesn’t see an error

  • It is highly likely that, at the time of a deploy, the static page is missing for a while
    and that in that case Netlify is returning a 500 in stead of a 404

Any news on this?
The issue seems to worsen.

Hey there! Our backend team are still very invested in this. They’re eager to get a small-term fix out for you and we’ll loop back to you once this is ready to go. Then, we’ll look for the long-term fix!

@Rik – are you experiencing this on any of your sites too?

Good news @Tim_Refaerts_Vlaming - we’ve got our trial fix enabled on your site and we think it should improve the situation. Could you let us know what your monitor thinks after a couple of days?

@Rik if you are seeing HTTP 500 errors serving content you know to be present in your deploy and can load sometimes, we’ll happy to help debug and see if this fix will work for you. If there is something else going on (e.g. the status isn’t HTTP 500 on requests for valid content), let us know and we’ll split your question off into a separate thread where we can work - you’ll want to start with your netlify sitename so we can make sure we’re looking at what you’re working on :slight_smile:

1 Like

Hi Chris and Scott, sorry I didn’t made this clear, I’m working on this issue together with Tim. We’re colleagues :wink:
We’ll keep an eye on our monitors and let you know how the situation progresses.
Thanks for the fix!

@fool thanks for the efforts and trial fix.
Today (11/10) at 10:13 AM a new 500 error on a page.

Today (11/10) at 11:11 + 11:12 the homepage in both languages returned a 500 error

Could you have a check what went wrong on this one?

Fyi, this weekend we didn’t had any 500 error.

Thanks for letting us know! The developers of the fix mentioned that there is more work to be done, but they will be continuing with it today and this week, so please keep us apprised as additional occurrences come (if they do).

I understand you saw one this weekend but it sounds like you might quantify the frequency as reduced in total?

1 Like

This weekend we didn’t had any 500 error. The first occurrence was on monday. After the customer started to edit and publish content and new deploys to Netlify were requested.

@fool Our customer would like to have more insights on what causes the 500 error. Did you already found the exact route cause? If yes, can you explain what goes wrong?

Any view on a timeline to release a potential fix?
The customer flagged a few cases where end-users experienced a 500 error, so this is really impacting their business.

I couldn’t explain the cause effectively or completely, but it is related to the way we store and look up content. I’m not sure why it affects your site more than most everyone else’s, but the error messages in our internal logs helped our engineers figure that out.

I also do not have any planned ETA on the rest of the work. I got the impression it was on the longer term roadmap rather than immediate, but the developers are also waiting to hear back on what the change might have been in incidences based on what they’ve already done before working further on this.

So: I understand you saw one 500 error since I updated you - was that the only one?

1 Like

Hi Chris, no that was not the only 500 error we saw.
I created an export from our monitoring system to list them for 3 important pages
While since the 8th there seem to be less issues, the problem still persists

Hope this data helps you to track down the problem.
(since this software only allows me to list 6 url’s I replaced them with a description)

English Homepage: https://www.vlerick.com/en/
Dutch Homepage: https://www.vlerick.com/nl
Search page: https://www.vlerick.com/en/find-a-programme/?

Date Time Url
04/10/2021 11:32 English home page
04/10/2021 11:32 Dutch home page
05/10/2021 01:18 Search page
05/10/2021 08:46 Dutch home page
05/10/2021 13:21 English home page
05/10/2021 14:38 Search page
05/10/2021 14:39 English home page
05/10/2021 14:39 Dutch home page
05/10/2021 14:39 Dutch home page
05/10/2021 14:39 Search page
05/10/2021 14:40 English home page
05/10/2021 14:43 Search page
05/10/2021 14:44 English home page
05/10/2021 14:44 Dutch home page
05/10/2021 14:49 Search page
05/10/2021 15:02 Dutch home page
05/10/2021 15:02 Search page
05/10/2021 15:05 English home page
05/10/2021 15:05 Dutch home page
05/10/2021 15:07 Search page
05/10/2021 15:08 English home page
05/10/2021 15:08 Dutch home page
05/10/2021 15:09 Search page
05/10/2021 15:10 English home page
05/10/2021 15:10 Dutch home page
05/10/2021 15:43 Search page
05/10/2021 16:22 English home page
05/10/2021 16:22 Dutch home page
05/10/2021 16:26 Search page
06/10/2021 10:19 English home page
06/10/2021 10:19 Dutch home page
06/10/2021 10:27 Search page
06/10/2021 10:28 English home page
06/10/2021 10:28 Dutch home page
06/10/2021 17:22 Dutch home page
06/10/2021 17:23 Dutch home page
06/10/2021 17:24 Dutch home page
06/10/2021 18:14 Search page
06/10/2021 18:15 English home page
06/10/2021 18:15 Dutch home page
07/10/2021 01:18 Dutch home page
07/10/2021 08:54 English home page
07/10/2021 08:54 Dutch home page
07/10/2021 10:16 Search page
07/10/2021 10:17 Search page
07/10/2021 10:50 Search page
07/10/2021 10:51 English home page
07/10/2021 10:51 Dutch home page
07/10/2021 10:51 Search page
07/10/2021 11:52 English home page
07/10/2021 11:52 Dutch home page
07/10/2021 11:52 Search page
07/10/2021 11:55 English home page
07/10/2021 11:55 Dutch home page
07/10/2021 11:55 Search page
07/10/2021 11:56 Search page
07/10/2021 11:57 English home page
07/10/2021 12:16 English home page
07/10/2021 12:16 Dutch home page
07/10/2021 12:16 Search page
07/10/2021 12:19 Search page
07/10/2021 12:20 English home page
07/10/2021 12:20 Dutch home page
07/10/2021 12:22 Search page
07/10/2021 12:23 Search page
07/10/2021 12:25 English home page
07/10/2021 12:25 Dutch home page
07/10/2021 12:27 English home page
07/10/2021 12:27 Dutch home page
07/10/2021 12:29 Search page
07/10/2021 12:39 English home page
07/10/2021 12:39 Search page
07/10/2021 12:41 Dutch home page
07/10/2021 12:42 English home page
07/10/2021 12:42 Dutch home page
07/10/2021 12:47 Search page
07/10/2021 12:48 Search page
07/10/2021 12:49 Search page
07/10/2021 12:57 Search page
07/10/2021 13:11 Search page
07/10/2021 13:12 English home page
07/10/2021 13:13 English home page
07/10/2021 13:13 Dutch home page
07/10/2021 18:18 Search page
07/10/2021 18:19 English home page
07/10/2021 18:56 Search page
07/10/2021 18:57 Search page
07/10/2021 18:57 English home page
07/10/2021 18:58 English home page
07/10/2021 18:58 Search page
07/10/2021 18:58 Dutch home page
08/10/2021 01:19 Dutch home page
08/10/2021 09:14 English home page
08/10/2021 09:14 Dutch home page
08/10/2021 09:48 English home page
08/10/2021 09:48 Search page
08/10/2021 09:49 English home page
08/10/2021 09:49 Dutch home page
08/10/2021 11:28 English home page
08/10/2021 11:29 English home page
08/10/2021 11:30 English home page
08/10/2021 11:31 English home page
08/10/2021 11:32 English home page
08/10/2021 11:33 English home page
08/10/2021 11:34 English home page
08/10/2021 12:12 Search page
08/10/2021 12:13 English home page
08/10/2021 12:13 Dutch home page
08/10/2021 18:59 English home page
08/10/2021 18:59 Dutch home page
08/10/2021 18:59 Search page
08/10/2021 19:01 Search page
09/10/2021 01:18 Search page
11/10/2021 09:40 Search page
11/10/2021 11:11 English home page
11/10/2021 11:12 English home page
11/10/2021 11:12 Dutch home page
11/10/2021 13:10 English home page
11/10/2021 13:11 English home page
11/10/2021 13:11 Dutch home page
11/10/2021 13:11 Search page
11/10/2021 15:09 Dutch home page
11/10/2021 16:43 Search page
11/10/2021 17:03 English home page
11/10/2021 17:03 Dutch home page
12/10/2021 04:09 Dutch home page
12/10/2021 08:50 Dutch home page
13/10/2021 13:35 Search page
14/10/2021 01:18 English home page
14/10/2021 01:18 Dutch home page
14/10/2021 03:30 Dutch home page
14/10/2021 08:08 Search page
14/10/2021 09:38 Dutch home page
14/10/2021 10:20 English home page
14/10/2021 14:11 Dutch home page
14/10/2021 14:11 Search page
14/10/2021 14:12 Search page
14/10/2021 14:14 Dutch home page

By the way, times are CET

Thanks for those details, Rik! Our team has spent quite awhile working on this and while they think things are improved, they have figured out the proximate cause of your problems: you deploy manually, 5x, right in a row:

The prior deploys have not finished processing before the others start, leading to the 500’s at deploy time (as you helpfully pointed out :)).

Could you try…not doing that? Deploy only once? Or wait 10 minutes between deploys? Is kind of a waste to deploy repeatedly that fast, but you seem to do it quite regularly, which as I mentioned is the proximate cause of the problem.

We do have the actual cause sorted out, but it will take substantial work to fix. Work we do plan to do, but not in the immediate future.

Let me know if you think you can modify your deployment habits or not and I’ll pass that feedback on to our dev team to consider next steps.

To follow up, we have also enabled a feature flag for your account that should further improve performance even if you can’t change deploy patterns. Will be interested to hear how things go!