Need Support: Netlify functions sometimes receiving corrupted JSON despite valid json being sent. We've made no changes, this error is seemingly out of nowhere

slept · February 2, 2024, 10:49pm

I greatly appreciate the time and help… I was advised to create this post by the ask-netlify chat bot.

I’ve confirmed our backend is working as it should (tested in Postman), and the frontend is sending the data that it is supposed to be sending (confirmed this by using a json format verifier on the input that was sent before an instance of the error occurred in my browser).

the status code I am receiving is a 500 Internal Server Error, and there is no evidence of timeouts in the Netlify logs, just the json syntax errors from the corrupted json being received.

The site works fine locally. the error only occurs around every 1/10 requests (no consistencies regarding which requests fail) and has occurred with all Netlify functions. we do not use next.js. The api is Netlify functions (path includes /.netlify/).

I thought I had found my explanation yesterday morning when Netlify’s status page said that functions were experiencing degraded service, but now it reads operational and we are still experiencing the issue.

I have triple checked request headers and they are correct… it is just that the body is sometimes corrupted.

Our site deploys from GitHub with 11ty (GitHub’s status also reads healthy).

Site: apdata.netlify.app

slept · February 2, 2024, 11:28pm

This has been happening for a few days with no ascertainable cause (we haven’t updated dependencies nor our codebase since weeks before the occurrences began).

Below is an example of a log of the error with the event.body that the function received being logged as well (added a debug statement and created a preview deploy so I could see the corrupted json). Sometimes portions of the json are missing and sometimes weird stuff is added.

Feb 1, 08:15:15 AM: 4e99ae98 INFO Feb 1, 08:15:15 AM: 4e99ae98 ERROR Feb 1, 08:15:15 AM: 4e99ae98 ERROR Feb 1, 08:15:15 AM: 4e99ae98 ERROR Feb 1, 08:15:15 AM: 4e99ae98 ERROR Feb 1, 08:15:15 AM: 4e99ae98 ERROR Feb 1, 08:15:15 AM: 4e99ae98 ERROR Feb 1, 08:15:15 AM: 4e99ae98 ERROR Feb 1, 08:15:15 AM: 4e99ae98 ERROR Feb 1, 08:15:15 AM: 4e99ae98 ERROR Feb 1, 08:15:15 AM: 4e99ae98 ERROR Feb 1, 08:15:15 AM: 4e99ae98 ERROR Feb 1, 08:15:15 AM: 4e99ae98 ERROR Feb 1, 08:15:15 AM: 4e99ae98 ERROR Feb 1, 08:15:15 AM: 4e99ae98 ERROR Feb 1, 08:15:15 AM: 4e99ae98 ERROR Feb 1, 08:15:15 AM: 4e99ae98 ERROR Feb 1, 08:15:15 AM: 4e99ae98 ERROR Feb 1, 08:15:15 AM: 4e99ae98 ERROR Feb 1, 08:15:15 AM: 4e99ae98 ERROR {“NCA_id”:"35
SyntaxError: Unexpected end of JSON input
at JSON.parse ()
at getData (/var/task/api/querydata.js:9:28)
at handleRoute (/var/task/api/_shared/utils.js:204:16)
at /var/task/api/_shared/utils.js:212:44
at HandlerError.handleErrors (/var/task/api/_shared/utils.js:29:20)
at /var/task/api/_shared/utils.js:212:25
at /var/task/node_modules/@sentry/serverless/dist/awslambda.js:203:24
at Runtime.outerWrapper [as handler] (/var/task/api/_shared/utils.js:216:20)
at Runtime.handleOnceNonStreaming (file:///var/runtime/index.mjs:1173:29) SyntaxError: Unexpected end of JSON input
at JSON.parse ()
at getData (/var/task/api/querydata.js:9:28)
at handleRoute (/var/task/api/_shared/utils.js:204:16)
at /var/task/api/_shared/utils.js:212:44
at HandlerError.handleErrors (/var/task/api/_shared/utils.js:29:20)
at /var/task/api/_shared/utils.js:212:25
at /var/task/node_modules/@sentry/serverless/dist/awslambda.js:203:24
at Runtime.outerWrapper [as handler] (/var/task/api/_shared/utils.js:216:20)
at Runtime.handleOnceNonStreaming (file:///var/runtime/index.mjs:1173:29)

slept · February 2, 2024, 11:34pm

Here is what the json looks like at other times: {“id”:“881”}:“881”}

hrishikesh · February 3, 2024, 3:09pm

Mind sharing a HAR file withan example request that fails?

slept · February 3, 2024, 4:01pm

test.apdata.org.har (1.3 MB)

I have replaced all JWTs with ‘REDACTED’. This is all test data. Error is same for our test site/db and production.

Thank you so much for the response.

hrishikesh · February 3, 2024, 4:44pm

I’m not seeing the JSON in your function logs anymore. Have you removed it? All I’m seeing is Using test db and then the error. I was hoping to compare the data being sent and being logged in your functions.

slept · February 3, 2024, 4:54pm

I did remove it as it was redundant due to our use of sentry. Would you like me to add it back?

Edit: debug statement still present here: https://65bbc2d4ff5c7c0008f5c2d9--apdata.netlify.app/

This is an older preview. I just triggered the error a couple of times on that preview
(pr 36 Feb 1 at 8:12 AM).

The console.log statement was only added to the querydata function (most often invoked) but the error occurs across all functions.

corean · February 3, 2024, 6:01pm

I’ve encountered the same problem. The application was deployed six months ago and remained unchanged, functioning perfectly until the issue began occurring last month. The problem is intermittent, and occasionally, the request body (in JSON format) truncates certain characters in the middle, leading to JSON parsing errors in the serverless backend functions; I verified multiple times that the browser sends the correct JSON strings.

hrishikesh · February 4, 2024, 6:51pm

Thank you! One more thing before I can finally test this and possibly write an escalation for the devs if needed. Do you mind sharing the JWT required by the function? You can send it over in a DM. Right now, I’m trying to trigger the function, but it sends a 500 due to missing JWT.

hrishikesh · February 4, 2024, 6:52pm

@corean, just like @slept has been sharing some details, we’d also need something along the similar lines from your end.

slept · February 4, 2024, 6:55pm

I just swung a full cUrl copy of a request that failed over to you. Thank you so much again.

hrishikesh · February 4, 2024, 7:06pm

@slept, thank you! I have sent several requests to your API and I am continuously getting a 200 back. Yeah the issue is intermittent, but I was hoping to at least be able to reproduce it myself once. I’ll try a few more times, but I don’t think I’ll have much luck. But I’ll still ask the devs if they have changed something about this.

slept · February 4, 2024, 7:11pm

I was just able to reproduce the error (the curl I sent you was captured right before I sent it and resulted in a 500). It happens infrequently, but when our site requires multiple API calls it becomes frequent and hard to tolerate for our users. It happens at least 50% of the time when I load in certain pages.

hrishikesh · February 4, 2024, 7:14pm

Thank you for that info. I have passed that to the devs.

corean · February 5, 2024, 1:27am

Apologies for the delayed reply. While I can’t provide the entire logs, I managed to extract the request body string:

{
  ...
  body: '{"query":{"isUserContact":true},"select":"company representative email","options":{"includePermiss',
}

Given that the JSON string is generated by the frontend Axios library, it’s unlikely to contain an incomplete JSON structure.

The probable cause for the issue could be either the size of the body string or the multiple requests made by the frontend browser. However, in either case, it should work and wasn’t happening for years until the last month.

eduardoboucas · February 5, 2024, 10:03am

Hi all. I’m a Netlify engineer working on Functions.

I’m sorry to hear you’re experiencing this problem and I’d love to get to the bottom of it.

We have recently rolled out a change to how the request body is handled in functions. While I don’t think it’s the culprit here, since it shouldn’t have an effect on the actual body string that is sent to the function, I would like to rule that out for sure.

@slept I have turned off this change for your site, but unfortunately I’m unable to ascertain whether it has made a difference because the test request that @hrishikesh shared with me relies on a JWT that has already expired. Is there a chance you could provide a longer-term token (in private, of course) that we can use to troubleshoot this? If not, perhaps we could put together a simpler reproduction case that you can share with us?

@corean I could also try the same for your site, if you can provide your site ID via DM.

Thanks!

skippyyy · February 5, 2024, 2:27pm

Hello, I’m encountering the same issue on my website as the OP. The JSON payload sent by my React app via axios is occasionally getting malformed by the time it reaches my functions.

eduardoboucas · February 5, 2024, 2:38pm

Do you know when was the first time you saw the unexpected behaviour? Your post says “last month”, but if you happen to have a more concrete timeline that would be super helpful.

tomas1 · February 5, 2024, 2:52pm

Hello, looks like we are experiencing a similar issue.
We are proxying requests on domain.com/api/* to an external service using netlify.toml:

[[redirects]]
from = “/api/*”
to = “https://www.domain.com/api/:splat”
status = 200
headers = {X-Netlify-Hostname = “www.domain.com”}

The JSON body sent by the frontend seems to be malformed from time to time. The problem seems to occur more often when doing multiple calls at the same time.

This issue is first noticed a few hours ago (but could be older).
Since we are calling an external API I’m not able to log the actual malformed body received by the API server

hrishikesh · February 5, 2024, 2:54pm

Could you please let us know the actual URL so we can test it? Also, if you can’t see the received body, how were you able to determine that the request body is malformed? Did the API server send any such information that we can use?