How to Parallelize Edge Function work for ~40,000 image uploads within the same minute?

Support
,
1

First, need to admit I’m relatively new to using edge functions, so I’m not entirely surprised to learn that on my first time using them at any reasonable scale, something went wrong when the demand scaled up to my use case.

Here’s what’s going on: There is an event where ~8000 people are at the same location, and when they use our application, there is a 1 minute window where they all need to take 1 ~ 5 photos of something special happening where they are. I then use these photos to provide a special on-site experience for those users who participate. Considering that the whole experience is 2 minutes maximum, a huge part of the problem is parallelization of ingesting all of these images at the same time.

The edge function is designed to do this work:

  • HTTP application/cbor request is received and parsed
  • The incoming message contains some metadata, plus a ~3.5MB Uint8Array of image data
  • The images and the metadata are uploaded to an S3 bucket
  • The service responds to the user with a session ID that can be used to send additional images and metadata to the edge function in subsequent requests and tie the session segments together

Things seem to go well for about the first 768 uploads, but after that, my Netlify Log starts displaying these messages:

An edge function has exceeded the memory limit. Refer to https://ntl.fyi/edge-function-limits for more information.

Reading that page, I eventually find my way to the Edge Functions limits -> Operation limits section, and find the limit: “Memory per set of deployed edge functions: 512 MB”

This is something that I’m trying to understand; What exactly is a set of deployed edge functions? What defines this boundary? Is this per Netlify site? Or is this per edge location that’s determined to be closest to the user?

Is there something I can do to easily get more sets of deployed edge functions to handle these requests in parallel?

Should I solve this by using Lambda Functions instead of Edge Functions, and do they provide the automagic “parallelization for free” that I was hoping for?

Most importantly, what should I be doing differently to handle this amount of throughput in this short timescale using Netlify services?

Thank you very much for your consideration, I’ve loved working with Netlify for much of this project so far.

2

Could you let us know the site for which this happens?

4

Sadly, I cannot share the link to the site publicly. Since this is a time sensitive event, we change the URL for each new event, and it is not meant to be accessible after the fact. I might be able to send it to you privately. But the rest of the questions still stand - What is the most appropriate path to take?

5

The devs mentioned:

I’d expect this to work. Without the domain(s) to inspect then I don’t see what more we can do. :confused: Even if it is an older set we can check it out.

So we’re essentially blocked from helping here without a URL to test.