Home
Support Forums

Function seems to be receiving malformed binary request data

Site name: https://infallible-elion-0e4df6.netlify.app
Repository: GitHub - Xenonym/netlify-binary-reprod

The site is a simple reproduction of an issue I am facing with sending requests with binary data to a Netlify function.

When you click Test, the site will send a fetch request with binary data to a Netlify function:

// As Base64: AAABAAABAAAAAAAAB2V4YW1wbGUDY29tAAABAAE=
const data = new Uint8Array([
  0, 0, 1, 0, 0, 1, 0, 0, 0, 0,
  0, 0, 7, 101, 120, 97, 109, 112, 108, 101,
  3, 99, 111, 109, 0, 0, 1, 0, 1
]);
async function reflect(contentType) {
  const response = await fetch('/.netlify/functions/reflect', {
    method: "POST",
      headers: {
        "Content-Type": contentType,
        "Content-Length": data.length
      },
      body: data
  });
  return response.text();
}

(The binary data is a DNS packet for an A query for example.com.)

The Netlify function simply returns whatever was sent to it:

export async function handler({ body }) {
  return {
    statusCode: 200,
    body,
  };
}

This function should return identical data from what is being passed in. However, when deployed, the function seems to receive malformed data:

Sent: AAABAAABAAAAAAAAB2V4YW1wbGUDY29tAAABAAE=
Recieved: AAABAAABAAAAAAAAB2V4YW1wbGUDY29tAAAB

The data received by the function is missing the last two bytes, which can also be observed from the Base64 encoded version ending with AB instead of the original AE=.

This happens both when using the Content-Type for DNS packets (application/dns-message) or for generic binary data (application/octet-stream).

Is this a bug with handling binary data in Netlify functions? Given that the function above does not do anything apart from what is being sent in, I am confused as to why the binary data received by the function is different from the one being sent by the client.

Your function needs to be:

export async function handler({body}) {
  return {
    statusCode: 200,
    body: Buffer.from(body, 'base64').toString('utf8')
  };
}

Because of:

Demo: https://quirky-mirzakhani-33e90a.netlify.app/

Note that, the above would return the decoded string so example com in your case.

Thanks for the reply @hrishikesh! However, the binary data is not a string, so comparing the sent and received data interpreted as UTF-8 may be misleading, since most of the bytes in the data are not actually UTF-8, apart from the domain name.

In your reprod site (https://quirky-mirzakhani-33e90a.netlify.app/), we can actually see that the Content-Length being sent and received is different, and the data receieved is shorter than what was sent by two bytes:

Assuming that the reflect function is largely the same, I am still not sure why the last two bytes will get lost when received by the function handler, especially when the function does nothing but return what was sent.

P.S. I note that for some reason the response is being returned with a Content-Type of image/x-icon, which was not set by the function handler in the first place.

Well okay, I now return the data as Uint8Array and see the problem better, here are the 2 arrays:

[0, 0, 1, 0, 0, 1, 0, 0, 0, 0, 0, 0, 7, 101, 120, 97, 109, 112, 108, 101, 3, 99, 111, 109, 0, 0, 1, 0, 1] // sent
[0, 0, 1, 0, 0, 1, 0, 0, 0, 0, 0, 0, 7, 101, 120, 97, 109, 112, 108, 101, 3, 99, 111, 109, 0, 0, 1] // received

So the 0, 1 from the end seem to be missing. It seems to be an extra space that’s lost.

However, have you tried this with real files? I’ve used the similar method for files and it seems to work fine.

Do you have a more complex example where the problem is more prominent?

@hrishikesh

Do you mean trying to submit with a form e.g. multipart/form-data? I imagine that would work, since form-data would be encoded, but the issue with sending binary data would remain: the handler shouldn’t be missing bytes from the request in the first place.

Not too sure what you mean here, but I have updated https://infallible-elion-0e4df6.netlify.app/ to take in an arbitrary size of base64 encoded data. The issue seems to replicate with 4096 bytes of random data (crypto.randomBytes(4096).toString('base64')) at least.

No, in the past, I was able to send files to functions by base64 encoding them on the client-side and sending the data as JSON. The files were received correctly and didn’t cause any corruption.

I’ve filed an issue for investigation.

1 Like

Hey @hrishikesh! I was wondering if there was any update to this issue?

Hey @Xenonym,

I just had a talk with the team handling Functions and looks like this is being escalated and awaiting response from another team at the moment. We’d update this thread when stuff moves around.