Netlify CLI can not deploy extremely large site

Support
,
1

Hello, I’m trying to migrate one of my websites to hugo. It has around 250k files in it. I’m stuck at the deployment phase. Locally I get this error:

Meanwhile any recommendation is welcome. I just have way too many posts (+130k) and each post has a data json with itself. Maybe I can embed json data directly to posts md files directly instead of using GetJSON.

  • we need to know your netlify site name. Example: stupefied-carson-b343df.netlify.app

Full logs:

10:14:59 AM: Build ready to start
10:15:00 AM: build-image version: 2cee85eb7f808bf3b6e87378c5307f9411f0a332
10:15:00 AM: build-image tag: v3.8.0
10:15:00 AM: buildbot version: 7ed3dd8fac64067d9cac4a8ffcf331d1669e61a3
10:15:00 AM: Fetching cached dependencies
10:15:01 AM: Starting to download cache of 111.0MB
10:15:03 AM: Finished downloading cache in 2.507168692s
10:15:03 AM: Starting to extract cache
10:15:06 AM: Finished extracting cache in 3.147810479s
10:15:06 AM: Finished fetching cache in 5.685499101s
10:15:06 AM: Starting to prepare the repo for build
10:15:09 AM: Preparing Git Reference refs/heads/master
10:15:39 AM: Parsing package.json dependencies
10:15:39 AM: Starting build script
10:15:40 AM: Installing dependencies
10:15:40 AM: Python version set to 2.7
10:15:40 AM: Started restoring cached node version
10:15:42 AM: Finished restoring cached node version
10:15:43 AM: v12.18.0 is already installed.
10:15:43 AM: Now using node v12.18.0 (npm v6.14.4)
10:15:43 AM: Started restoring cached build plugins
10:15:43 AM: Finished restoring cached build plugins
10:15:43 AM: Attempting ruby version 2.7.2, read from environment
10:15:44 AM: Using ruby version 2.7.2
10:15:45 AM: Using PHP version 5.6
10:15:45 AM: Installing Hugo 0.83.1
10:15:45 AM: hugo v0.83.1-5AFE0A57+extended linux/amd64 BuildDate=2021-05-02T14:38:05Z VendorInfo=gohugoio
10:15:45 AM: Started restoring cached go cache
10:15:45 AM: Finished restoring cached go cache
10:15:45 AM: go version go1.14.4 linux/amd64
10:15:45 AM: go version go1.14.4 linux/amd64
10:15:45 AM: Installing missing commands
10:15:45 AM: Verify run directory
10:15:45 AM: ​
10:15:45 AM: ────────────────────────────────────────────────────────────────
10:15:45 AM:   Netlify Build                                                 
10:15:45 AM: ────────────────────────────────────────────────────────────────
10:15:45 AM: ​
10:15:45 AM: ❯ Version
10:15:45 AM:   @netlify/build 12.10.0
10:15:45 AM: ​
10:15:45 AM: ❯ Flags
10:15:45 AM:   deployId: 60d8419357384e0008d73dac
10:15:45 AM: ​
10:15:45 AM: ❯ Current directory
10:15:45 AM:   /opt/build/repo
10:15:45 AM: ​
10:15:45 AM: ❯ Config file
10:15:45 AM:   /opt/build/repo/netlify.toml
10:15:45 AM: ​
10:15:45 AM: ❯ Context
10:15:45 AM:   production
10:15:45 AM: ​
10:15:45 AM: ────────────────────────────────────────────────────────────────
10:15:45 AM:   1. Build command from Netlify app                             
10:15:45 AM: ────────────────────────────────────────────────────────────────
10:15:45 AM: ​
10:15:45 AM: $ hugo
10:15:45 AM: Start building sites …
10:17:50 AM: ​
10:17:50 AM: ────────────────────────────────────────────────────────────────
10:17:50 AM:   "build.command" failed                                        
10:17:50 AM: ────────────────────────────────────────────────────────────────
10:17:50 AM: ​
10:17:50 AM:   Error message
10:17:50 AM:   Command was killed with SIGKILL (Forced termination): hugo
10:17:50 AM: ​
10:17:50 AM:   Error location
10:17:50 AM:   In Build command from Netlify app:
10:17:50 AM:   hugo
10:17:50 AM: ​
10:17:50 AM:   Resolved config
10:17:50 AM:   build:
10:17:50 AM:     command: hugo
10:17:50 AM:     commandOrigin: ui
10:17:50 AM:     environment:
10:17:50 AM:       - HUGO_VERSION
10:17:50 AM:     publish: /opt/build/repo/public
10:17:50 AM:     publishOrigin: ui
10:17:50 AM:   functions:
10:17:50 AM:     '*': {}
10:17:50 AM: Caching artifacts
10:17:50 AM: Started saving build plugins
10:17:50 AM: Finished saving build plugins
10:17:50 AM: Started saving pip cache
10:17:50 AM: Finished saving pip cache
10:17:50 AM: Started saving emacs cask dependencies
10:17:50 AM: Finished saving emacs cask dependencies
10:17:50 AM: Started saving maven dependencies
10:17:50 AM: Finished saving maven dependencies
10:17:50 AM: Started saving boot dependencies
10:17:50 AM: Finished saving boot dependencies
10:17:50 AM: Started saving rust rustup cache
10:17:50 AM: Finished saving rust rustup cache
10:17:50 AM: Started saving go dependencies
10:17:50 AM: Finished saving go dependencies
10:17:50 AM: Build failed due to a user error: Build script returned non-zero exit code: 2
10:17:50 AM: Creating deploy upload records
10:17:50 AM: Failing build: Failed to build site
10:17:50 AM: Failed during stage 'building site': Build script returned non-zero exit code: 2
10:17:50 AM: Finished processing build request in 2m50.231230047s
2

This thread should help: Hugo is killed when building

3

Thanks for the reply. As it was suggested, I built the site in my local machine (took around 6h) and then I ran netlify deploy from netlify CLI to push public directory of hugo site. I received this:

netlify deploy
Please provide a publish directory (e.g. "public" or "dist" or "."): fe_hugo
? Publish directory fe_hugo/public
Deploy path:        fe_hugo/public
Configuration path: fe_hugo/netlify.toml
Deploying to draft URL...
✔ Finished hashing 591724 files
⠧ CDN diffing files... ›   Warning: JSONHTTPError:  500
 ›   Warning: 
 ›   {
 ›     "name": "JSONHTTPError",
 ›     "status": 500,
 ›     "json": {
 ›       "error": "Document exceeds allowed max BSON size. The max is 16777216. (on 
 ›   db-origin-gc-dsm-1.node.gc-dsm.nf.internal)"
 ›     }
 ›   }
 ›
⠇ CDN diffing files...    JSONHTTPError: Internal Server Error

Can you see what is wrong with the setup? Is it the sheer size of the website?

5

Based on this error, and given you have almost 600,000 files, I would say yes.

1 Like
6

Unfortunately I do not think Netlify CLI can deploy this large of a site. You are going to have to look into other ways like through Github/Gitlab.

2 Likes
7

There is perhaps a good use-case here for On-demand Builders (docs).

1 Like