Deploy timeout with "timed out while waiting to enter states"

Hi there

We tried to deploy over 60000 pages but failed with an unknown error below:

Error message
5:54:34 AM:   Error: Deploy did not succeed: Failed to execute deploy: timed out while waiting to enter states [prepared, ready]

What does it mean and can we do anything? The build is always success locally.

As I have sent you an email as well but have received no response at all, please forgive me for posting here as well,

site-id: affectionate-lamarr-c83264
full log:

2:04:27 AM: Build ready to start
2:04:29 AM: build-image version: d2c6dbeac570350a387d832f64bc980dc964ad65 (focal)
2:04:29 AM: build-image tag: v4.8.0
2:04:29 AM: buildbot version: e58b6be665675c0f99b33132a8c1eec1f775eba1
2:04:29 AM: Fetching cached dependencies
2:04:29 AM: Starting to download cache of 589.6MB
2:04:39 AM: Finished downloading cache in 10.249720107s
2:04:39 AM: Starting to extract cache
2:04:52 AM: Finished extracting cache in 12.31500737s
2:04:52 AM: Finished fetching cache in 22.624511593s
2:04:52 AM: Starting to prepare the repo for build
2:04:52 AM: Preparing Git Reference refs/heads/master
2:04:59 AM: Parsing package.json dependencies
2:05:00 AM: 
​
❯ Initial build environment
baseRelDir: true
branch: master
context: production
cwd: /opt/build/repo
featureFlags: []
mode: buildbot
repositoryRoot: /opt/build/repo
siteId: 36dc8a72-bda5-48a7-8c6f-f1aff776587d
​
❯ UI build settings
baseRelDir: true
build:
  command: nuxt generate
  environment:
    - NETLIFY_BUILD_DEBUG
    - ONEGRAPH_AUTHLIFY_TOKEN
  publish: dist
​
❯ Resolved build environment
branch: master
buildDir: /opt/build/repo
configPath: /opt/build/repo/netlify.toml
context: production
env: []
​
❯ Resolved config
build:
  command: nuxt generate
  commandOrigin: ui
  environment:
    - NETLIFY_BUILD_DEBUG
    - ONEGRAPH_AUTHLIFY_TOKEN
  publish: /opt/build/repo/dist
  publishOrigin: ui
headers:
  - for: /*
    values:
      Access-Control-Allow-Methods: GET, OPTIONS
      Access-Control-Allow-Origin: '*'
headersOrigin: config

2:05:00 AM: Starting build script
2:05:00 AM: Installing dependencies
2:05:00 AM: Python version set to 2.7
2:05:01 AM: Started restoring cached node version
2:05:02 AM: Finished restoring cached node version
2:05:03 AM: v16.14.2 is already installed.
2:05:03 AM: Now using node v16.14.2 (npm v8.5.0)
2:05:03 AM: Started restoring cached build plugins
2:05:03 AM: Finished restoring cached build plugins
2:05:04 AM: Attempting ruby version 2.7.2, read from environment
2:05:05 AM: Using ruby version 2.7.2
2:05:05 AM: Using PHP version 8.0
2:05:06 AM: No npm workspaces detected
2:05:06 AM: Started restoring cached node modules
2:05:06 AM: Finished restoring cached node modules
2:05:06 AM: Started restoring cached go cache
2:05:06 AM: Finished restoring cached go cache
2:05:06 AM: go version go1.16.5 linux/amd64
2:05:06 AM: go version go1.16.5 linux/amd64
2:05:06 AM: Installing missing commands
2:05:06 AM: Verify run directory
2:05:08 AM: ​
2:05:08 AM: ────────────────────────────────────────────────────────────────
2:05:08 AM:   Netlify Build                                                 
2:05:08 AM: ────────────────────────────────────────────────────────────────
2:05:08 AM: ​
2:05:08 AM: ❯ Version
2:05:08 AM:   @netlify/build 27.0.1
2:05:08 AM: ​
2:05:08 AM: ❯ Flags
2:05:08 AM:   apiHost: api.netlify.com
2:05:08 AM:   baseRelDir: true
2:05:08 AM:   branch: master
2:05:08 AM:   buildId: 6268261b9839151525053e4c
2:05:08 AM:   buildbotServerSocket: /tmp/netlify-buildbot-socket
2:05:08 AM:   cacheDir: /opt/build/cache
2:05:08 AM:   cachedConfigPath: /tmp/netlify_config.json
2:05:08 AM:   context: production
2:05:08 AM:   cwd: /opt/build/repo
2:05:08 AM:   deployId: 6268261b9839151525053e4e
2:05:08 AM:   edgeFunctionsDistDir: /tmp/edge-6268261b9839151525053e4e
2:05:08 AM:   featureFlags:
2:05:08 AM:     - buildbot_build_go_functions
2:05:08 AM:     - edge_functions_cache_cli
2:05:08 AM:     - zisi_parse_isc
2:05:08 AM:   framework: nuxt
2:05:08 AM:   functionsDistDir: /tmp/zisi-6268261b9839151525053e4e
2:05:08 AM:   mode: buildbot
2:05:08 AM:   nodePath: /opt/buildhome/.nvm/versions/node/v16.14.2/bin/node
2:05:08 AM:   repositoryRoot: /opt/build/repo
2:05:08 AM:   saveConfig: true
2:05:08 AM:   sendStatus: true
2:05:08 AM:   siteId: 36dc8a72-bda5-48a7-8c6f-f1aff776587d
2:05:08 AM:   statsd:
2:05:08 AM:     host: 10.65.19.17
2:05:08 AM:     port: 8125
2:05:08 AM:   testOpts:
2:05:08 AM:     silentLingeringProcesses: ''
2:05:08 AM: ​
2:05:08 AM: ❯ Current directory
2:05:08 AM:   /opt/build/repo
2:05:08 AM: ​
2:05:08 AM: ❯ Config file
2:05:08 AM:   /opt/build/repo/netlify.toml
2:05:08 AM: ​
2:05:08 AM: ❯ Resolved config
2:05:08 AM:   build:
2:05:08 AM:     command: nuxt generate
2:05:08 AM:     commandOrigin: ui
2:05:08 AM:     environment:
2:05:08 AM:       - NETLIFY_BUILD_DEBUG
2:05:08 AM:       - ONEGRAPH_AUTHLIFY_TOKEN
2:05:08 AM:     publish: /opt/build/repo/dist
2:05:08 AM:     publishOrigin: ui
2:05:08 AM:   headers:
2:05:08 AM:     - for: /*
      values:
        Access-Control-Allow-Methods: GET, OPTIONS
        Access-Control-Allow-Origin: '*'
  headersOrigin: config
​
2:05:08 AM: ❯ Context
2:05:08 AM:   production
2:05:08 AM: ​
2:05:08 AM: ────────────────────────────────────────────────────────────────
2:05:08 AM:   1. Build command from Netlify app                             
2:05:08 AM: ────────────────────────────────────────────────────────────────
2:05:08 AM: ​
2:05:08 AM: $ nuxt generate
5:43:36 AM: ​
5:43:36 AM: (build.command completed in 3h 38m 28.4s)
5:43:36 AM: ​
5:43:36 AM: ────────────────────────────────────────────────────────────────
5:43:36 AM:   2. Deploy site                                                
5:43:36 AM: ────────────────────────────────────────────────────────────────
5:43:36 AM: ​
5:43:36 AM: ​
5:43:36 AM: ❯ Uploaded config
5:43:36 AM:   [build]
5:43:36 AM: ​
5:43:36 AM:   [[headers]]
5:43:36 AM:     for = "/*"
5:43:36 AM:       [headers.values]
5:43:36 AM:       Access-Control-Allow-Origin = "*"
5:43:36 AM:       Access-Control-Allow-Methods = "GET, OPTIONS"
5:43:36 AM:   #    Access-Control-Allow-Headers = "X-Requested-With, X-HTTP-Method-Override, Content-Type, Accept"
5:43:36 AM: ​
5:43:36 AM: ❯ Uploaded headers
5:43:36 AM:   No headers
5:43:36 AM: ​
5:43:36 AM: ❯ Uploaded redirects
5:43:36 AM:   No redirects
5:43:36 AM: ​
5:43:36 AM: Starting to deploy site from 'dist'
5:49:26 AM: Creating deploy tree asynchronously
5:49:31 AM: Creating deploy upload records
5:54:34 AM: ​
5:54:34 AM: ────────────────────────────────────────────────────────────────
5:54:34 AM:   Internal error during "Deploy site"                           
5:54:34 AM: ────────────────────────────────────────────────────────────────
5:54:34 AM: ​
5:54:34 AM:   Error message
5:54:34 AM:   Error: Deploy did not succeed: Failed to execute deploy: timed out while waiting to enter states [prepared, ready]
5:54:34 AM: ​
5:54:34 AM:   Error location
5:54:34 AM:   During Deploy site
5:54:34 AM:       at handleDeployError (file:///opt/buildhome/node-deps/node_modules/@netlify/build/src/plugins_core/deploy/buildbot_client.js:87:18)
5:54:34 AM:       at deploySiteWithBuildbotClient (file:///opt/buildhome/node-deps/node_modules/@netlify/build/src/plugins_core/deploy/buildbot_client.js:68:12)
5:54:34 AM:       at processTicksAndRejections (node:internal/process/task_queues:96:5)
5:54:34 AM:       at async coreStep (file:///opt/buildhome/node-deps/node_modules/@netlify/build/src/plugins_core/deploy/index.js:45:5)
5:54:34 AM:       at async fireCoreStep (file:///opt/buildhome/node-deps/node_modules/@netlify/build/src/steps/core_step.js:39:9)
5:54:34 AM: Creating deploy upload records
5:54:34 AM:       at async tFireStep (file:///opt/buildhome/node-deps/node_modules/@netlify/build/src/time/main.js:20:59)
5:54:34 AM:       at async runStep (file:///opt/buildhome/node-deps/node_modules/@netlify/build/src/steps/run_step.js:88:7)
5:54:34 AM:       at async pReduce.index (file:///opt/buildhome/node-deps/node_modules/@netlify/build/src/steps/run_steps.js:91:11)
5:54:34 AM:       at async runSteps (file:///opt/buildhome/node-deps/node_modules/@netlify/build/src/steps/run_steps.js:51:7)
5:54:34 AM:       at async runBuild (file:///opt/buildhome/node-deps/node_modules/@netlify/build/src/core/main.js:612:7)
5:54:35 AM: Failed during stage 'building site': Build script returned non-zero exit code: 4 (https://ntl.fyi/exit-code-4)
5:54:34 AM: ​
5:54:34 AM:   Resolved config
5:54:34 AM:   build:
5:54:34 AM:     command: nuxt generate
5:54:34 AM:     commandOrigin: ui
5:54:34 AM:     environment:
5:54:34 AM:       - NETLIFY_BUILD_DEBUG
5:54:34 AM:       - ONEGRAPH_AUTHLIFY_TOKEN
5:54:34 AM:     publish: /opt/build/repo/dist
5:54:34 AM:     publishOrigin: ui
5:54:34 AM:   headers:
5:54:34 AM:     - for: /*
      values:
        Access-Control-Allow-Methods: GET, OPTIONS
        Access-Control-Allow-Origin: '*'
  headersOrigin: config
Caching artifacts
5:54:34 AM: Started saving node modules
5:54:34 AM: Finished saving node modules
5:54:34 AM: Started saving build plugins
5:54:34 AM: Finished saving build plugins
5:54:34 AM: Started saving pip cache
5:54:34 AM: Finished saving pip cache
5:54:34 AM: Started saving emacs cask dependencies
5:54:34 AM: Finished saving emacs cask dependencies
5:54:34 AM: Started saving maven dependencies
5:54:34 AM: Finished saving maven dependencies
5:54:34 AM: Started saving boot dependencies
5:54:34 AM: Finished saving boot dependencies
5:54:34 AM: Started saving rust rustup cache
5:54:34 AM: Finished saving rust rustup cache
5:54:34 AM: Started saving go dependencies
5:54:34 AM: Finished saving go dependencies
5:54:34 AM: Build failed due to an internal system error: Build script returned non-zero exit code: 4
5:54:35 AM: Failing build: Failed to build site
5:54:35 AM: Finished processing build request in 3h50m6.013552439s

hey there,

so the lines above are definitely the problem - while we have maybe 1-2 sites (enterprise sites) on our network that take that long to build, that is a VERY long build time. This definitely indicates that something isn’t working right.

some quick thoughts on this:

I’d work through these guides and see if there is anything helpful to bring down the build time. I am also wondering if there is some kind of process somewhere that isn’t actually building but idling in the background, and eventually timing out.

While we can bump up the build time to an hour or an hour and half, I don’t think we can go as high as 3 hours without getting you talking with a different team to see if a custom build solution is right for you.

Another option could be to build locally and push the built files Not ideal from a workflow perspective, but, this will at least get a build up while you take the time to see if you can optimize your build somehow to bring build times down.

give these resources a try, and let us know!