I am trying to deploy a fairly large static site. The whole site (if deployed) will have millions of html pages. The data is generated from the stack overflow data dumps. The goal is to create a read-only static site for select programming languages. The page size of a page (A question and its answers) is usually < 10kb. However, sometimes it can be 100kb or so.
My current process:
1. Run custom script to generate html files (local system) into `public` folder 2. Push to Gitlab (Auto publishing is on. Deploys from main are published automatically).
I was testing with 67000 html pages (questions for golang). Initially I generated the whole 67k pages and pushed it to Gitlab. It triggered a deploy and failed with the following error
Failed during stage 'building site': Failed to execute deploy: timed out while waiting to enter states [prepared, ready]
I tried generating and pushing in batches of 1000 html files. The deploy was successful till about 14k pages. Never after that.
- I am on the free plan. Is there some limitation that I am unaware of?
- I have used about 71/300 build minutes till now.
Full build logs for 15k files:
1:27:47 AM: Build ready to start 1:27:49 AM: build-image version: d2c6dbeac570350a387d832f64bc980dc964ad65 (focal) 1:27:49 AM: build-image tag: v4.8.0 1:27:49 AM: buildbot version: 4c861aef7b38d135b002320990f147f009a73d13 1:27:49 AM: Fetching cached dependencies 1:27:49 AM: Starting to download cache of 40.8MB 1:27:50 AM: Finished downloading cache in 471.553226ms 1:27:50 AM: Starting to extract cache 1:27:51 AM: Finished extracting cache in 1.416584304s 1:27:51 AM: Finished fetching cache in 1.892906175s 1:27:51 AM: Starting to prepare the repo for build 1:27:52 AM: Preparing Git Reference refs/heads/main 1:27:56 AM: Parsing package.json dependencies 1:27:57 AM: No build steps found, continuing to publishing 1:27:57 AM: Starting to deploy site from 'public' 1:27:58 AM: Creating deploy tree asynchronously 1:27:59 AM: Creating deploy upload records 1:28:06 AM: Starting post processing 1:28:06 AM: Post processing - HTML 1:32:59 AM: Creating deploy upload records 1:32:59 AM: Failing build: Failed to build site 1:32:59 AM: Failed during stage 'building site': Failed to execute deploy: timed out while waiting to enter states [prepared, ready] 1:33:00 AM: Finished processing build request in 5m10.524461115s