Build failing with large number of files

Support
,
1

Hi,

Site: https://golang-altqna.netlify.app/

I am trying to deploy a fairly large static site. The whole site (if deployed) will have millions of html pages. The data is generated from the stack overflow data dumps. The goal is to create a read-only static site for select programming languages. The page size of a page (A question and its answers) is usually < 10kb. However, sometimes it can be 100kb or so.

My current process:

1. Run custom script to generate html files  (local system) into `public` folder
2. Push to Gitlab (Auto publishing is on. Deploys from main are published automatically).

I was testing with 67000 html pages (questions for golang). Initially I generated the whole 67k pages and pushed it to Gitlab. It triggered a deploy and failed with the following error

Failed during stage 'building site': Failed to execute deploy: timed out while waiting to enter states [prepared, ready]

I tried generating and pushing in batches of 1000 html files. The deploy was successful till about 14k pages. Never after that.

Notes:

  • I am on the free plan. Is there some limitation that I am unaware of?
  • I have used about 71/300 build minutes till now.

Full build logs for 15k files:

1:27:47 AM: Build ready to start
1:27:49 AM: build-image version: d2c6dbeac570350a387d832f64bc980dc964ad65 (focal)
1:27:49 AM: build-image tag: v4.8.0
1:27:49 AM: buildbot version: 4c861aef7b38d135b002320990f147f009a73d13
1:27:49 AM: Fetching cached dependencies
1:27:49 AM: Starting to download cache of 40.8MB
1:27:50 AM: Finished downloading cache in 471.553226ms
1:27:50 AM: Starting to extract cache
1:27:51 AM: Finished extracting cache in 1.416584304s
1:27:51 AM: Finished fetching cache in 1.892906175s
1:27:51 AM: Starting to prepare the repo for build
1:27:52 AM: Preparing Git Reference refs/heads/main
1:27:56 AM: Parsing package.json dependencies
1:27:57 AM: No build steps found, continuing to publishing
1:27:57 AM: Starting to deploy site from 'public'
1:27:58 AM: Creating deploy tree asynchronously
1:27:59 AM: Creating deploy upload records
1:28:06 AM: Starting post processing
1:28:06 AM: Post processing - HTML
1:32:59 AM: Creating deploy upload records
1:32:59 AM: Failing build: Failed to build site
1:32:59 AM: Failed during stage 'building site': Failed to execute deploy: timed out while waiting to enter states [prepared, ready]
1:33:00 AM: Finished processing build request in 5m10.524461115s
2

According to this post from mid last year (Large site on Netlify - what are the limits?), I don’t believe there are limits outside of the standard build minutes / build time and bandwidth.

A Netlify rep can probably give a more recent update though.

3

hiya,

lets get you reading through this, first:

and then if you’re still facing the issue, we can take another look!