[Functions] Issues parsing images from multipart/form-data

dmmulroy · August 29, 2019, 6:16pm

I opened an issue on the netlify-lambda repo here: Issue parsing multipart/form-data · Issue #180 · netlify/netlify-lambda · GitHub

But the gist is that I’m trying to parse an image upload from a function/lambda event and it doesn’t appear to be working. I’ve tried using busboy and also aws-lambda-multipart-parser. I can get a resulting buffer from both, but it seems to be malformed each time. I’ve tried checking the mime type with file-type and also trying to load the parsed image buffer into Jimp and get the following error: Could not find MIME for buffer (null). So it seems like, at least from these experiences, that part of the form data is getting malformed/dropped along the way.

dmmulroy · August 29, 2019, 6:18pm

I can only put two links in a post so here are two follow up replies with the packages referenced in the OP.

dmmulroy · August 29, 2019, 6:18pm

fool · September 3, 2019, 7:35pm

Could you tell us more about your workflow? For instance, if you are using an event-triggered function you wouldn’t have any uploaded file data in the body - it would be a URL to the data. But perhaps you mean you wrote a function that accepts POSTs and you’re having trouble parsing the data directly off the wire?

dmmulroy · September 3, 2019, 9:54pm

Yup, it’s the latter! I’m posting directly to an endpoint. The errors I wrote about in the OP occur while running on Netlify/deployed to Lambda. Also when I use Netlify Dev the . parse.on('file', cb) never even occurs.

Here is where the post request occurs:

github.com

dmmulroy/partyfy.party/blob/beta/src/components/ImageUpload.vue#L47


      
            this.partyImageData = null;
          
          
  const fileReader = new FileReader();
          
          
  fileReader.onload = e => {
              this.imageData = e.target.result;
            };
          
          
  fileReader.readAsDataURL(this.file);
          },
          async onSubmit() {
            try {
              const formData = new FormData();
              const fileReader = new FileReader();
          
          
    formData.append("image", this.file, this.file.name);
          
          
    const partyImage = await fetch("http://134.122.23.177/partyfy", {
                method: "POST",
                body: formData,
                mode: "no-cors"

and the lambda function is here: https://github.com/dmmulroy/partyfy.party/blob/beta/.netlify/functions/partyfy.js

futuregerald · September 5, 2019, 3:12pm

Hi,

I don’t actually think that will work right now. I’ve opened an issue regarding adding support for multi-part requests to Netlify functions, and added a link to this thread so we’ll let you know once we have more info. Sorry about that @dmmulroy!

dmmulroy · September 5, 2019, 3:29pm

Ahhh, bummer! Thanks for letting me know though and linking back to this issue for future updates!

ryanditjia · September 10, 2019, 6:02am

Man, I wish I had stumbled upon this discussion before spending hours.

Wrote a post in AWS subreddit asking for help before finding this thread, here it is: Reddit - Dive into anything

In my case, PDFs and JPGs were malformed (slight filesize difference, and not openable).

According to the the library I’ve had the most success with (GitHub - francismeynard/lambda-multipart-parser: This nodejs module will parse the multipart-form containing files and fields from the AWS lambda event object. It works very well parsing binary and text files.), a config has to be enabled:

Important Please make sure to enable the “Use Lambda Proxy integration” in API Gateway method Integration request.

If decided not to enable it for some reason, make sure to pass the required Lambda event parameters in Integration Request → Mapping Templates section, such as body, headers and isBase64Encoded flag.

Sample Lambda and API Gateway implementation with Cloudformation can be found in here.

dmmulroy · September 10, 2019, 2:53pm

I ended up switching to straight up using API Gateway and Lambda and got lambda-multipart - npm working. Although my lambda function takes >29 seconds on images >650kb so I’m getting time outs so I’m rewriting from Node → Go

ryanditjia · September 10, 2019, 3:32pm

I imagined 650kb to be trivial for node to handle, didn’t think it would’ve required rewriting to Go. Well, I didn’t imagine sending email with attachment to be a hassle too. PS: I’m a newer developer, devs on Twitter always talk about how easy it was to set up Rails back in the day. Sorry for ranting

dmmulroy · September 10, 2019, 3:46pm

It’s not so much the image itself it’s what I’m doing with it once I parse it . v. serious business

atymic · October 8, 2019, 5:55am

I also spend some hours stuck on this

Any updates?

dmmulroy · October 8, 2019, 1:14pm

I haven’t heard or seen any action from Netlify on their side for this - as mentioned above I’m using API Gateway + Lambda directly since their doesn’t seem to be support yet for it from netlify.

Edit: To the folks at Netlify: if the code for this open source, I’d be more than happy to help hack on this and try to get a PR up

fool · October 9, 2019, 5:32pm

We don’t have any immediate plans to change the way our functions are implemented around multipart submissions, if that’s what you’re asking?

We will follow up here if things change, but my suggestion would be don’t wait for a config change on our side but find a different flow to use:

if you want to use our functions implementation, work within today’s featureset.
if you need a different setup, you could use our proxying to your separate deploy of functions in your own AWS (or serverless.com, or whatever service) account.

dmmulroy · October 9, 2019, 5:45pm

If there is no plan to support images via multipart submissions straight off the request I’d love to at least see that added to the docs as I’m not the only one who has been caught off guard by this.

perry · October 9, 2019, 9:59pm

good point, @dmmulroy - i’ll chat with docs about this. thanks for pointing that out.

s-r-aman · October 12, 2019, 8:48am

Hey! @dmmulroy! I am also making a netlify function to accept a form with two images, I have converted them into base64 format. They are working properly in my local environment.

But in production I am getting a base64 encoded body, with the event.isBase64Encoded being true. None of that was happeining in local environment. And worst part is the body after decoding is partiall of what is sent.

Can you please help in this.

futuregerald · October 14, 2019, 8:47pm

Hi,

We don’t use the lambda API gateway at Netlify, we run our own proxy. That said, you should be able to BASE64 encode your data and then decode it from within the function. Do you have a simple example repo that shows how the files are being uploaded and how you’re decoding them? You mentioned them being cut off, are you sending them back to the frontend?

piducancore · January 21, 2020, 11:23pm

I encountered this issue with a GraphQL server running on Netlify functions.

On the client I have a Gatsby site with apollo-upload-client set up, and on the server I’m using graphql-kungfu, which uses apollo-server and graphql-upload under the hood.

The images are transfered but somehow “corrupted” (slightly different file size and unreadable).
I’ve also tested with plain text files and this doesn’t happen.

I wonder if this has anything to do with what they say here about setting API Gateway’s Binary Media Types?

I’m no expert and I’ve only used lambda functions on Netlify, so I’d appreciate any information.

marcus · January 27, 2020, 8:20pm

Welcome @piducancore!

Since we don’t support the binary media types setting on Netlify Functions you’d need to find a way to send the binary data by using plaintext.
The usual way this works is by encoding the payload via Base64.

The way the GraphQL multipart upload spec is designed (apollo-upload-client implements it) i don’t think you’ll be able to use it without adjustments on Netlify.

I don’t really have a solution to your problem, but will suggest not using upload-client but rather implementing file uploads just via fields in your mutation where the base64 encoded images go.