Netlify Functions: issue uploading Blob to S3

Support
1

Hello,

I am trying to upload an audio file to S3 through a Netlify Function. The problem is that I pass a playable Blob of type audio/mpeg to the Function, but when the Function receives the request, it gets an un-playable string of type text/plain.

Here is the code of making the request to the Netlify Function:

const uploadAudio = async audioBlob => {
    // "audioBlob" is a Blob of type "audio/mpeg" that I tested that I can play as mp3
    try {
        const response = await fetch("/.netlify/functions/upload-audio", {
            method: "POST",
            body: audioBlob
        });

        if (!response) {
            return null;
        }

        const JSONresp = await response.json();

        return JSONresp.data;
    } catch (error) {
        console.log("Error uploading audio", error);
    }
};

This is the Function itself, when it gets there, the former “Blob” is only a string now:

const AWS = require("aws-sdk");
const s3 = new AWS.S3({
    accessKeyId: process.env.MY_AWS_ACCESS_KEY_ID,
    secretAccessKey: process.env.MY_AWS_SECRET_ACCESS_KEY
});

export async function handler(event) {
    try {
        // At this point, event.body is just a long string, not an audio Blob anymore
        const audioBlob = event.body;
        const params = {
            Bucket: "audio",
            Key: "uploads/usertest/filetest.mp3",
            Body: audioBlob,
            ACL: "public-read",
            ContentType: "audio/mpeg",
            ContentDisposition: "inline"
        };

        let putObjectPromise = await s3.upload(params).promise();
        let location = putObjectPromise.Location;

        return {
            statusCode: 200,
            body: JSON.stringify({ data: location })
        };
    } catch (error) {
        return {
            statusCode: 400,
            body: JSON.stringify(error)
        };
    }
}

How do I get the Blob from the client and send to AWS S3 as a proper Blob of type “audio/mpeg”?

Thanks!

2

I’m running into the same issue. I’ve tried to use:

Buffer.from( event.body, "utf8" )

… but, the file is corrupted. When I do a byte-by-byte comparison of the original file and the uploaded file, it’s a few-bytes different. I am not sure what is going on.

Cross-linking my post here: Baffled by Binary event.body in Lambda Function

3

My current understanding is that Netlify Functions does not support any binary data format, so it just makes everything into a string in JSON.

I created a workaround that I have no idea if it is a good way to solve the problem, nor I’m aware of the trade-offs, but it is working for me. There might be unnecessary steps in it too.

I first transform the Blob (with a mime type of audio/mpeg) generated by the MediaRecorder API into an ArrayBuffer object:

const arrayBuffer = await new Response(audioBlob).arrayBuffer();

This step is necessary because later, inside the Netlify Function, I have to decode the information back into binary data to save on S3, but I am using NodeJS and Node cannot handle a Blob, but it can handle a Buffer.

Then I transform the arrayBuffer into a regular string

const audioBufferStringified = String.fromCharCode(
    ...new Uint8Array(arrayBuffer)
  );

so I can pass it in a JSON to the Netlify Function.

const response = await fetch(
      `/.netlify/functions/upload-audio?user=${user}&filename=${filename}`,
      {
        headers: { Accept: "application/json" },
        method: "POST",
        body: audioBufferStringified
      }
    );

Inside the Netlify Function, I map the string back into an ArrayBuffer object

const audioUint8Array = Uint8Array.from(
      [...audioBufferStringified].map(ch => ch.charCodeAt())
    ).buffer;

and then transform the ArrayBuffer into a Buffer that keeps the mime type audio/mpeg.

const audioBuffer = Buffer.from(audioUint8Array);

This Buffer I can store directly in AWS S3 which can be played later as a valid MP3 file.

const resourceKey = `uploads/${user}/${filename}.mp3`;
const params = {
  Bucket: "bucketname",
  Key: resourceKey,
  Body: audioBuffer,
  ACL: "public-read",
  ContentType: "audio/mpeg",
  ContentDisposition: "inline"
};

const putObjectPromise = await s3.upload(params).promise();

The whole process is:
blob → arraybuffer → string (request to Netlify Function) string → arraybuffer → buffer

I would appreciate comments on any issue with this code that I am missing.

4

Oh, super interesting! I’ve been looking through their documentation and I don’t see anything about it not allowing for binary data :frowning: I just assumed that was what the isBase64Encoded stuff would have been about. Thanks for keying me onto this issue – sounds like there’s no way I can do this, unless I alter the client-side code (as you have). Unfortunately, in my context, that’s not super easy to do.

What you have done in your code is a bit beyond my experience. My only thought that it might be easier to convert the blob to a base64 string on the client-side. Then, on the Lambda side, you might be able to just do Buffer.from( audio_string, "base64" ), rather than dealing with typed-arrays. But, that’s just a guess on my part.

5

An update for my solution above.

As I wrote above, if the audio recording is larger than a few seconds it quickly runs into a RangeError: Maximum call stack size exceeded.

The solution is to have a more efficient method to reduce the binary data to a string. This is how I did:

In the client, instead of just turning the arrayBuffer into a string, I do this:

const reducedAudioUint8Array = new Uint8Array(arrayBuffer).reduce(
    (data, byte) => data + String.fromCharCode(byte),
    ""
);

const reducedAudioUint8ArrayBase64Encoded = btoa(reducedAudioUint8Array);

So I pass reducedAudioUint8ArrayBase64Encoded as the request body.

Then, in the Netlify Function, I decode it back like this:
(note that the atob() function is not available at NodeJS as it is at the browser, I used an npm package for it)

const reducedAudioUint8ArrayBase64Encoded = event.body;

const reducedAudioUint8ArrayBase64Decoded = atob(
  reducedAudioUint8ArrayBase64Encoded
);

const audioUint8Array = Uint8Array.from(
  [...reducedAudioUint8ArrayBase64Decoded].map(ch => ch.charCodeAt())
).buffer;

const audioBuffer = Buffer.from(audioUint8Array);

Then the audioBuffer is a playable MP3 file that I upload to AWS S3.

Netlify Functions still have a request size limit (I believe it is 6MB) that is unavoidable. From my tests, above 5 minutes the request start returning a PayloadTooLargeError.

This limit could probably be solved using multiple requests, but I am ok with that limit for my use case. I will limit the audio length for the users at 4 minutes.