Build failing with large number of files

[altqna.com]

Hi,

Site: https://golang-altqna.netlify.app/

I am trying to deploy a fairly large static site. The whole site (if deployed) will have millions of html pages. The data is generated from the stack overflow data dumps. The goal is to create a read-only static site for select programming languages. The page size of a page (A question and its answers) is usually < 10kb. However, sometimes it can be 100kb or so.

My current process:

1. Run custom script to generate html files  (local system) into `public` folder
2. Push to Gitlab (Auto publishing is on. Deploys from main are published automatically).

I was testing with 67000 html pages (questions for golang). Initially I generated the whole 67k pages and pushed it to Gitlab. It triggered a deploy and failed with the following error

Failed during stage 'building site': Failed to execute deploy: timed out while waiting to enter states [prepared, ready]

I tried generating and pushing in batches of 1000 html files. The deploy was successful till about 14k pages. Never after that.

Notes:

• I am on the free plan. Is there some limitation that I am unaware of?
• I have used about 71/300 build minutes till now.

Full build logs for 15k files:

1:27:47 AM: Build ready to start
1:27:49 AM: build-image version: d2c6dbeac570350a387d832f64bc980dc964ad65 (focal)
1:27:49 AM: build-image tag: v4.8.0
1:27:49 AM: buildbot version: 4c861aef7b38d135b002320990f147f009a73d13
1:27:49 AM: Fetching cached dependencies
1:27:49 AM: Starting to download cache of 40.8MB
1:27:50 AM: Finished downloading cache in 471.553226ms
1:27:50 AM: Starting to extract cache
1:27:51 AM: Finished extracting cache in 1.416584304s
1:27:51 AM: Finished fetching cache in 1.892906175s
1:27:51 AM: Starting to prepare the repo for build
1:27:52 AM: Preparing Git Reference refs/heads/main
1:27:56 AM: Parsing package.json dependencies
1:27:57 AM: No build steps found, continuing to publishing
1:27:57 AM: Starting to deploy site from 'public'
1:27:58 AM: Creating deploy tree asynchronously
1:27:59 AM: Creating deploy upload records
1:28:06 AM: Starting post processing
1:28:06 AM: Post processing - HTML
1:32:59 AM: Creating deploy upload records
1:32:59 AM: Failing build: Failed to build site
1:32:59 AM: Failed during stage 'building site': Failed to execute deploy: timed out while waiting to enter states [prepared, ready]
1:33:00 AM: Finished processing build request in 5m10.524461115s

[nathanmartin]

According to this post from mid last year (Large site on Netlify - what are the limits?), I don’t believe there are limits outside of the standard build minutes / build time and bandwidth.

A Netlify rep can probably give a more recent update though.

[perry]

hiya,

lets get you reading through this, first:

[Support Guide] Making sure your builds use appropriate resources for Netlify's build system

updated in Dec 2022 Hopefully, your builds at Netlify “just work”, once you have things configured. That’s the intention and the common case, but for those times when your reality falls a bit short of our dream, this article talks in detail about some of the limitations that customers run into while building and how to work around them. The operation of Netlify’s build system is pretty well documented in several places such as: a somewhat conversational “how it works” guide. our official do…

and then if you’re still facing the issue, we can take another look!