Netlify CLI can not deploy extremely large site

Hello, I’m trying to migrate one of my websites to hugo. It has around 250k files in it. I’m stuck at the deployment phase. Locally I get this error:

Meanwhile any recommendation is welcome. I just have way too many posts (+130k) and each post has a data json with itself. Maybe I can embed json data directly to posts md files directly instead of using GetJSON.

  • we need to know your netlify site name. Example: stupefied-carson-b343df.netlify.app

Full logs:

10:14:59 AM: Build ready to start
10:15:00 AM: build-image version: 2cee85eb7f808bf3b6e87378c5307f9411f0a332
10:15:00 AM: build-image tag: v3.8.0
10:15:00 AM: buildbot version: 7ed3dd8fac64067d9cac4a8ffcf331d1669e61a3
10:15:00 AM: Fetching cached dependencies
10:15:01 AM: Starting to download cache of 111.0MB
10:15:03 AM: Finished downloading cache in 2.507168692s
10:15:03 AM: Starting to extract cache
10:15:06 AM: Finished extracting cache in 3.147810479s
10:15:06 AM: Finished fetching cache in 5.685499101s
10:15:06 AM: Starting to prepare the repo for build
10:15:09 AM: Preparing Git Reference refs/heads/master
10:15:39 AM: Parsing package.json dependencies
10:15:39 AM: Starting build script
10:15:40 AM: Installing dependencies
10:15:40 AM: Python version set to 2.7
10:15:40 AM: Started restoring cached node version
10:15:42 AM: Finished restoring cached node version
10:15:43 AM: v12.18.0 is already installed.
10:15:43 AM: Now using node v12.18.0 (npm v6.14.4)
10:15:43 AM: Started restoring cached build plugins
10:15:43 AM: Finished restoring cached build plugins
10:15:43 AM: Attempting ruby version 2.7.2, read from environment
10:15:44 AM: Using ruby version 2.7.2
10:15:45 AM: Using PHP version 5.6
10:15:45 AM: Installing Hugo 0.83.1
10:15:45 AM: hugo v0.83.1-5AFE0A57+extended linux/amd64 BuildDate=2021-05-02T14:38:05Z VendorInfo=gohugoio
10:15:45 AM: Started restoring cached go cache
10:15:45 AM: Finished restoring cached go cache
10:15:45 AM: go version go1.14.4 linux/amd64
10:15:45 AM: go version go1.14.4 linux/amd64
10:15:45 AM: Installing missing commands
10:15:45 AM: Verify run directory
10:15:45 AM: ​
10:15:45 AM: ────────────────────────────────────────────────────────────────
10:15:45 AM:   Netlify Build                                                 
10:15:45 AM: ────────────────────────────────────────────────────────────────
10:15:45 AM: ​
10:15:45 AM: ❯ Version
10:15:45 AM:   @netlify/build 12.10.0
10:15:45 AM: ​
10:15:45 AM: ❯ Flags
10:15:45 AM:   deployId: 60d8419357384e0008d73dac
10:15:45 AM: ​
10:15:45 AM: ❯ Current directory
10:15:45 AM:   /opt/build/repo
10:15:45 AM: ​
10:15:45 AM: ❯ Config file
10:15:45 AM:   /opt/build/repo/netlify.toml
10:15:45 AM: ​
10:15:45 AM: ❯ Context
10:15:45 AM:   production
10:15:45 AM: ​
10:15:45 AM: ────────────────────────────────────────────────────────────────
10:15:45 AM:   1. Build command from Netlify app                             
10:15:45 AM: ────────────────────────────────────────────────────────────────
10:15:45 AM: ​
10:15:45 AM: $ hugo
10:15:45 AM: Start building sites …
10:17:50 AM: ​
10:17:50 AM: ────────────────────────────────────────────────────────────────
10:17:50 AM:   "build.command" failed                                        
10:17:50 AM: ────────────────────────────────────────────────────────────────
10:17:50 AM: ​
10:17:50 AM:   Error message
10:17:50 AM:   Command was killed with SIGKILL (Forced termination): hugo
10:17:50 AM: ​
10:17:50 AM:   Error location
10:17:50 AM:   In Build command from Netlify app:
10:17:50 AM:   hugo
10:17:50 AM: ​
10:17:50 AM:   Resolved config
10:17:50 AM:   build:
10:17:50 AM:     command: hugo
10:17:50 AM:     commandOrigin: ui
10:17:50 AM:     environment:
10:17:50 AM:       - HUGO_VERSION
10:17:50 AM:     publish: /opt/build/repo/public
10:17:50 AM:     publishOrigin: ui
10:17:50 AM:   functions:
10:17:50 AM:     '*': {}
10:17:50 AM: Caching artifacts
10:17:50 AM: Started saving build plugins
10:17:50 AM: Finished saving build plugins
10:17:50 AM: Started saving pip cache
10:17:50 AM: Finished saving pip cache
10:17:50 AM: Started saving emacs cask dependencies
10:17:50 AM: Finished saving emacs cask dependencies
10:17:50 AM: Started saving maven dependencies
10:17:50 AM: Finished saving maven dependencies
10:17:50 AM: Started saving boot dependencies
10:17:50 AM: Finished saving boot dependencies
10:17:50 AM: Started saving rust rustup cache
10:17:50 AM: Finished saving rust rustup cache
10:17:50 AM: Started saving go dependencies
10:17:50 AM: Finished saving go dependencies
10:17:50 AM: Build failed due to a user error: Build script returned non-zero exit code: 2
10:17:50 AM: Creating deploy upload records
10:17:50 AM: Failing build: Failed to build site
10:17:50 AM: Failed during stage 'building site': Build script returned non-zero exit code: 2
10:17:50 AM: Finished processing build request in 2m50.231230047s

This thread should help: Hugo is killed when building

Thanks for the reply. As it was suggested, I built the site in my local machine (took around 6h) and then I ran netlify deploy from netlify CLI to push public directory of hugo site. I received this:

netlify deploy
Please provide a publish directory (e.g. "public" or "dist" or "."): fe_hugo
? Publish directory fe_hugo/public
Deploy path:        fe_hugo/public
Configuration path: fe_hugo/netlify.toml
Deploying to draft URL...
✔ Finished hashing 591724 files
⠧ CDN diffing files... ›   Warning: JSONHTTPError:  500
 ›   Warning: 
 ›   {
 ›     "name": "JSONHTTPError",
 ›     "status": 500,
 ›     "json": {
 ›       "error": "Document exceeds allowed max BSON size. The max is 16777216. (on 
 ›   db-origin-gc-dsm-1.node.gc-dsm.nf.internal)"
 ›     }
 ›   }
 ›
⠇ CDN diffing files...    JSONHTTPError: Internal Server Error

Can you see what is wrong with the setup? Is it the sheer size of the website?

Based on this error, and given you have almost 600,000 files, I would say yes.

1 Like

Unfortunately I do not think Netlify CLI can deploy this large of a site. You are going to have to look into other ways like through Github/Gitlab.

2 Likes

There is perhaps a good use-case here for On-demand Builders (docs).

1 Like