I want to run a CRON job every hour. This CRON job does a relatively complex task for each user. There could be thousands of users. I want this CRON job to finish within a few minutes each hour. Is there a way to spin up multiple of the same Netlify Function to achieve this?
Hiya @alext162 ,
Interesting question, but no. You’ll get the standard (1Gb) amount of memory with the function, and while you can run as many copies as you want, each only runs for up to 10 or 26 at most seconds, and there is no “inter-function” communication, to allow them to work together on a problem. Even background functions - available to Pro+ teams - run for only 15 min, but won’t be able to talk amongst themselves while running to coordinate the work.
I think you’ll want a different platform than ours for this kind of workflow (honestly, I wouldn’t try it in a lambda at all even without our limitations, if it is substantial compute and coordination, such as you seem to describe).
Sorry I don’t have better news for you today.
Hello! I successfully resolved the problem through a straightforward approach. I have set up one lambda function that orchestrates the events, which is executed as a cron job on an hourly basis, alongside another lambda function dedicated to processing these events. In the scheduler function, I group the events into batches and initiate a fetch request to the processing lambda for each batch. This method allows me to achieve the desired horizontal scaling since each fetch request triggers a new instance of the processing lambda, enabling the rapid processing of thousands of events in a brief span of time.
Hi, thank so much for writing back in and sharing your solution with the community.