Trouble with handling files in Netlify Function

Hi @Coderr,

It’s because the event.body is base64 encoded.

You need to use Buffer.from(event.body, 'base64').toString('utf8').

Full code:

import { initializeApp } from 'firebase/app'
import { getStorage, ref, uploadString } from 'firebase/storage'

const Busboy = require('busboy')

exports.handler = async event => {

  return new Promise(resolve => {

    const fields = {}

    const busboy = new Busboy({
      headers: event.headers
    })

    busboy.on('file', (fieldname, filestream, filename, _, mimeType) => {
      filestream.on('data', data => {
        fields[fieldname] = {
          content: data,
          filename,
          type: mimeType
        }
      })
    })

    busboy.on('field', (fieldName, value) => {
      fields[fieldName] = value
    })

    busboy.on('finish', () => {
      resolve(fields)
    })

    busboy.write(Buffer.from(event.body, 'base64').toString('utf8'))

  }).then(formData => {

    return uploadString(ref(getStorage(initializeApp({
      /* firebase stuff */
    })), formData.name), formData.file.content.toString('base64'), 'base64').then(() => {

      return {
        body: JSON.stringify(true),
        statusCode: 200
      }

    })

  })

}

Note: This is my form:

<form enctype = "multipart/form-data" method = "POST" name = "fileForm">
  <p>
    <label>
      File name:
      <input type = "text" name = "name"/>
    </label>
  </p>
  <p>
    <label>
      File:
      <input type = "file" name = "file"/>
    </label>
  </p>
  <p>
    <button>
      Send
    </button>
  </p>
</form>

So if your form has different fields and names (it most likely will), you’d have to make adjustments in your serverless code.

After adding Buffer.from(event.body, ‘base64’).toString(‘utf8’) in busboy.write, I am getting below timed out error. Is the PDF content working on your above code?

{"errorMessage":"Task timed out after 10.00 seconds","errorType":"TimeoutError","stackTrace":["new TimeoutError (lib/node_modules/netlify-cli/node_modules/lambda-local/build/lib/utils.js:112:28)

This time I tried a simple 20 KB YML file as that was readily available on my Desktop. I was able to get the file in my Firebase Storage and it was MD5 perfect. So I assumed it would work for other files too.

How big is the PDF that you’re trying to use?

Also, I think the error you’ve got is in Netlify CLI. Netlify CLI doesn’t encode the event.body as base64 (yet - I’ll be filing as issue for that later today). So, in Netlify CLI, you have to use it like before, the above code would only work in Production.

If you wish to support both the environments, you would have to do something like:

if (event.isbase64Encoded) {
  busboy.write(Buffer.from(event.body, 'base64').toString('utf8'))
} else {
  busboy.write(event.body)
}

OR one liner (might not work):

busboy.write(event.isbase64Encoded ? Buffer.from(event.body, 'base64').toString('utf8') : event.body)

I will try the above. I also used simple pdf file size like 50kb.

Still pdf is corrupted. Actually by default ‘isBase64Encoded’ is coming as false when checked the console log of event.

May be this causes issue. → Netlify server functions unable to handle multipart/form-data - #4 by step

As I said, in CLI it’s always going to be false.

The PDF you’re trying to use would be useful to test if you still think it’s not working. Without that, I don’t have any way to reproduce what you’re saying.

According to the file that was sent and tested, yes there seems to be a problem in handling non-text based files. So images, PDFs or any other file type for that matter that’s not plain-text would not work. I’ll see if there’s something that can be done about this.

Yes, both images and pdf are not working. Maybe they need to include multipart/form data in the new change mentioned here → Changed behavior in function body encoding

So, I tested this and I’ve 2 options to offer:

  1. Continue using multipart/form-data:

Yes, you could continue using it, but with a catch. If you only need to upload a file with the form (and no other form fields), this would do the trick. In the above code, simply change it to Buffer.from(event.body, 'base64'). I repeat, there should not be any other data except the file. Not multiple files either, just 1 file! If you need multiple files, you’d have to send each file to a different serverless function. You can use the same function, but you’d have to invoke it again for each file. For form fields, you could use FormData(), but make sure that you exclude the file from it.

  1. Switch to application/json:

Yes, submitting a form with application/json might seem weird, but it works. However, there’s some extra client-side work that’s required. You need to convert the file to base64 and add each form field separately to the JSON payload. You could do it programatically too, but yeah that’s the workflow. Here’s an example:

const form = document.forms.fileForm
form.onsubmit = event => {
  event.preventDefault()
  new Promise((resolve, reject) => {
    const reader = new FileReader()
    reader.readAsDataURL(form.elements.file.files[0])
    reader.onload = () => resolve(reader.result)
    reader.onerror = error => reject(error)
  }).then(file => {
    fetch('/.netlify/functions/getData/', {
      body: JSON.stringify({
        file: file,
        name: form.elements.name.value
      }),
      headers: {
        'content-type': 'application/json'
      },
      method: 'post',
    }).then(response => {
      if (response.ok) {
        return response.json()
      } else {
        throw response.statusText
      }
    }).then(data => {
      console.log(data)
    }).catch(error => {
      console.log(error)
    })
  })
}

Finally, in the serverless function, no need to use Busboy anymore:

import { initializeApp } from 'firebase/app'
import { getDownloadURL, getStorage, ref, uploadString } from 'firebase/storage'

exports.handler = async event => {

  const payload = JSON.parse(event.body)

  return uploadString(ref(getStorage(initializeApp({
    /* firebase stuff */
  })), payload.name), payload.file.split(',')[1], 'base64').then(file => {

    return getDownloadURL(file.ref).then(url => {

      return {
        body: JSON.stringify({
          url: url
        }),
        statusCode: 200
      }

    })

  })

}

Basically, the file’s base64 content exists as: payload.file.split(',')[1]. The file gets encoded as follows:

data:application/pdf;base64,actual-base64-content

And thus, we need to split it to get the actual base64 content. That’s required at least to upload the file to Firebase, it might not be required for other providers - but if you wish to convert the base64 to some other format like Buffer or something else, you’d again have to use split(',')[1].

Using the JSON way, you get both, the form fields and the file - MD5 perfect.

Test it here:

https://stoic-brown-d4563d.netlify.app/

Name is supposed to be the file name that you’d like it to be saved as.

You’d get the download URL of the file in console or if the console trims it out, you could always check it in Network tab.

Hi Folks!

I’m struggling to get a similar netlify function working. I’m trying to upload an image to Cloudflare Images I too, followed that netlify blog post, and I think I’m close to getting it working, but I’m struggling to upload an image, I get the error returned from cloudflare ERROR 9422: Decode error: image failed to be decoded: Uploaded image must have image/jpeg or image/png content type

So what I’m asking for help with, is how to convert that uploaded file buffer into a jpeg/png?

My complete code is in this gist, and I got it working easily with a plain express version (code here) – any help gratefull received!

 const fields = await parseMultipartForm(event)

  const data = new FormData()
  // data.append('file', fields.uploadedImage.content.toString('base64'), { type: 'image/jpeg' })
  data.append('file', Buffer.from(fields.uploadedImage.content, 'base64'))

  await axios({
    method: 'post',
    url: `https://api.cloudflare.com/client/v4/accounts/${process.env.CLOUDFLARE_ACCOUNT_ID}/images/v1`,
    headers: {
      Authorization: `Bearer ${process.env.CLOUDFLARE_API_TOKEN}`,
      'Content-Type': 'multipart/form-data; boundary=' + data._boundary,
    },
    data
  })
    .then((response) => {
      return {
        statusCode: 200,
        body: JSON.stringify({ response }),
      }
    })
    .catch((error) => {
      console.log({ error })
      return {
        statusCode: 500,
        body: JSON.stringify({ error }),
      }
    })

I can see that Cloudflare needs the content type as that.

But you seem to be sending this.

Hi, yeah it needs to be sent as that, otherwise it returbs ERROR 5415: Images must be uploaded as a form, not as raw image data. Please use multipart/form-data format

I’m sure it’s something to do with encoding the image buffer into a jpeg to attach to the formdata

In case this is any use, this is what I get from my fields when I send it an image

{
  "content": {
    "type": "Buffer",
    "data": [239,191,189,239,191,189,239,191,189,239,191,.....etc]
  },
  "filename": {
    "filename": "temp-join-us.jpg",
    "encoding": "7bit",
    "mimeType": "image/jpeg"
  }
}

Are you sure adding a boundary yourself is the correct way though? Not sure how this could be handled in Functions, but at least in browsers, you should avoid setting the content-type header for multipart to work.

It turns out I just needed to amend that last line from that blog post function to busboy.write(Buffer.from(event.body, 'base64'))

All working! For future searchers, here is my working netlify function to cloudflare image script

import Busboy from 'busboy'
import axios from 'axios'
import FormData from 'form-data'

function parseMultipartForm(event) {
  return new Promise((resolve) => {
    const fields = {}

    const busboy = Busboy({
      headers: event.headers,
    })

    busboy.on('file', (fieldname, filestream, filename, _, mimeType) => {
      filestream.on('data', (data) => {
        fields[fieldname] = {
          content: data,
          filename,
          type: mimeType,
        }
      })
    })

    busboy.on('field', (fieldName, value) => {
      fields[fieldName] = value
    })

    busboy.on('finish', () => {
      resolve(fields)
    })

    // This was the bastard!
    busboy.write(Buffer.from(event.body, 'base64'))
  })
}

exports.handler = async function (event, context) {
  try {

    // Get url param
    const url = event.queryStringParameters.url

    // Parse multipart form for image
    const { image } = await parseMultipartForm(event)

    // Create a new form to send to Cloudflare Images and append our image
    const data = new FormData()
    data.append('file', image.content, image.filename.filename)

    // Upload to cloudflare
    const upload = await axios({
      method: 'post',
      url: `https://api.cloudflare.com/client/v4/accounts/${process.env.CLOUDFLARE_ACCOUNT_ID}/images/v1`,
      headers: {
        Authorization: `Bearer ${process.env.CLOUDFLARE_API_TOKEN}`,
        ...data.getHeaders(),
      },
      data
    })
      .then((response) => response.data)
      .catch((error) => error)

    return {
      statusCode: 200,
      body: JSON.stringify({ url, upload }),
    }

  } catch (error) {
    console.error(error)
    return {
      statusCode: 500,
      body: JSON.stringify({ error }),
    }
  }
}

1 Like

Thank you so much for coming back and sharing this with us, @felixthehat! I am glad you got everything working.

Happy building :netliconfetti:

1 Like

Busboy had some breaking changes. I’ve managed to get it working by modifying the netlify blog code:

function parseMultipartForm(event) {
  return new Promise((resolve) => {

    const fields = {};
    const bb = busboy({
      headers: event.headers
    });

    bb.on("file", (name, file, info) => {
      const {filename, encoding, mimeType } = info;
        file.on("data", (data) => {
          if (!fields[name])
            fields[name] = []

          fields[name].push({
            filename,
            type: mimeType,
            content: data,
          });
        })
      }
    );

    bb.on("field", (fieldName, value) => {
      fields[fieldName] = value;
    });

    bb.on("close", () => {
      resolve(fields)
    });

    bb.end(Buffer.from(event.body, 'base64'));
  });

The important changes

  • busboy no longer a class, instead its just an exported function
  • on file paramerter info is a single object now and needs to be destructed to access
  • finish event was renamed to close
  • To manually write into the stream we need to call end instead of write (usually in node its req.pipe)
  • Had to convert the event.body to a Buffer with base 64 encoding.
  • Also slightly modified the file section to allow for multiple files in the same field
1 Like

I too was stuck with this problem for several hours. The solution was as follows (using the new busboy 1.6, assuming base64Encoded is true).

BAD:

const bodyStr = Buffer.from(event.body, 'base64').toString();
bb.end(bodyStr);

GOOD:

const bodyBuf = Buffer.from(event.body, 'base64');
bb.end(bodyBuf);

So you must pass a raw Buffer to busboy. Passing an equivalent binary string won’t work for some reason. Maybe this is a problem on busboy’s side…

Hi, I’m running into timeout issue with either Bussboy or JSON.parse.

that’s my function

const pointId: string = event.queryStringParameters!['id']!;

    console.log('parsing');

    const { base64Content, mediaType } = JSON.parse(event.body!);

    console.log('creating file');

    const cover = XataFile.fromBase64(base64Content, {
      mediaType
    });

    if (cover.size! > 3e6) {
      return { statusCode: 400, body: 'File size exceeded.' };
    }

    if (
      !['image/png', 'image/jpeg', 'image/gif', 'image/bmp'].includes(
        cover.mediaType
      )
    ) {
      return { statusCode: 400, body: 'Unsupported file type.' };
    }

    const savedPoint = await client.db.points.update(pointId, {
      cover,
    });

    return {
      statusCode: 200,
      body: JSON.stringify(savedPoint?.cover),
    };

I’m getting

parsing
{"level":"error","message":"End - Error:"}
{"level":"error","message":"{\n\t\"errorMessage\": \"Task timed out after 10.00 seconds\",\n\t\"errorType\": \"TimeoutError\",\n\t\"stackTrace\": [\n\t\t\"new TimeoutError (/Users/anvlkv/.npm-global/lib/node_modules/netlify-cli/node_modules/lambda-local/build/lib/utils.js:119:28)\",\n\t\t\"Context.<anonymous> (/Users/anvlkv/.npm-global/lib/node_modules/netlify-cli/node_modules/lambda-local/build/lib/context.js:110:19)\",\n\t\t\"listOnTimeout (node:internal/timers:569:17)\",\n\t\t\"processTimers (node:internal/timers:512:7)\"\n\t]\n}"}
Response with status 500 in 10197 ms.

Or

Response with status 200 in 9228 ms.

Both are with fairly small files (72KB).

I suspect this may have to do with slow connection, but I couldn’t find a way of disabling timeout when running netlify dev.

Is there a nice and performant way of uploading file to xata that I couldn’t find?