I have a new feature where we need to process some task in the background. These tasks can take up to 2 minutes. However, I have to process 300,000 tasks to be processed. Imagine there is a list in our DB with 300,000 entries. If we process them in series, it’ll take forever. So we need to process all of them in parallel. How do we approach this? Does Netlify Background function handle this scenario? i.e. can we call this function 300,000 times with different parameters?
@ramram I know this doesn’t answer your direct question, but I’m curious to know if you’re processing all entries, every time, or if you’re only processing all the records once initially?
If you’re only doing them once initially, can you pre-process them offline instead of utilizing cloud resources?
We can’t do that. This needs to be done everyday essentially and the quantity of Tasks would vary between 10-20K to 100K or more.
@ramram Netlify functions are AWS Lambda’s and as I understand it from answers in other threads, the default concurrency limit is 1,000.
I’m not sure if the same limit applies to the background functions, but a Netlify staff member will be able to advise.