we’re currently running our site on Gatsby Cloud, where we have booked the “Professional Plan”. A whole build without cache takes about an hour there.
Admittedly, our project has a substantial amount of sites (about 20-25k in total) and a comparatively large number of images that need to be processed. All of these images are pre-optimized with most of them matching their end-resolution almost exactly before being queried in Gatsby. From our and the Gatsby communities experience, that’s whats causing memory spikes, build crashes and long build times in most cases when there is a user error.
We’re currently using the 30-Day-trial of Netlifys Pro-Plan and already have a project, where the maximum build time is increased to (about?) an hour. Still, when building with more than 4 CPU-Cores, the build will crash with an OOM-Error, when building with just 4 cores, the build will run into a timeout.
Are there any other optimizations we could do to make this work, or are we forced to opt for the Enterprise-Program with a matching price-tag or search for another hosting provider?
Hi hrishikesh,
increasing the available memory and/or parallel image processing, like Gatsby Cloud did, would certainly be the better solution to this problem, since increasing build times does not solve the root cause of this issue.
If those other options are not on the table, what would be the maximum allowed build time? Our project already runs at a maximum of 1hr.
Hey @audrey ,
thank you that may help getting the site off the ground, so we’d definitely like to give it a shot!
But is more like treating the symptoms but not the disease. Would it not be more cost effective and sustainable, also for Netlify as a company/service, to implement something like Gatsby Cloud already has? I’m mainly referring to parallel image processing. Maybe I’m being naive here, but from our point of view, you (Netlify) just bought a service (Gatsby Cloud) that has a great solution to this problem, but seem hesitant to implement this yourself. Which in return is causing customers like us to have massive problems.
Please don’t make incorrect/false assumptions. I have said this to several people (not sure if you were one of those users in the helpdesk). But, any kind of changes that would require hardware-level changes especially for the build pod are not planned at the moment. The build network of the two platforms differs significantly, and it’s simply not possible to migrate all of that into one platform. It is indeed not the best thing, but given the budget and timeline constraints, it’s not possible to aim for such a cross-platform migration - considering Netlify builds not just Gatsby but other sites as well, and due to any hardware changes, they should not start breaking.
Hey @hrishikesh,
yes, I was one of these people in the helpdesk and I certainly do understand your struggles, no hard feelings!
But please consider our situation:
The #1 selling point of and reason for using Gatsby Cloud was parallel image processing.
In fact, the “Resolving out of memory issues”-guide says to use Gatsby Cloud with parallel image processing, to this day!
Our site builds just fine on Gatsby Clouds Pro plan wich is very cost-efficient.
Netlify made the conscious decision to buy this platform and advertised a smooth, hassle free transition/migration.
Our site does not in fact migrate smoothly into Netlify. Far from it to be honest.
The proposed “solution” is to book the Enterprise-Plan with more capable machines.
Keep in mind, our site builds well on Gatsby Cloud with a mere 8 Gigs of RAM and very limited CPU-power.
Now we’re looking at a about 50 to 100-fold increase in cost, jumping from Gatsby Cloud Professional to Netlify Enterprise. Yes admittedly Gatsby Cloud was pretty cheap for what it offered and maybe a 2 to 4-fold increase in cost would be appropriate. But going from a few dozen to possibly several thousand $ per month is a very bitter pill to swallow…
