Deploy timeout with "timed out while waiting to enter states"

[nisarahot]

Hi there

We tried to deploy over 60000 pages but failed with an unknown error below:

Error message
5:54:34 AM:   Error: Deploy did not succeed: Failed to execute deploy: timed out while waiting to enter states [prepared, ready]

What does it mean and can we do anything? The build is always success locally.

As I have sent you an email as well but have received no response at all, please forgive me for posting here as well,

site-id: affectionate-lamarr-c83264
full log:

2:04:27 AM: Build ready to start
2:04:29 AM: build-image version: d2c6dbeac570350a387d832f64bc980dc964ad65 (focal)
2:04:29 AM: build-image tag: v4.8.0
2:04:29 AM: buildbot version: e58b6be665675c0f99b33132a8c1eec1f775eba1
2:04:29 AM: Fetching cached dependencies
2:04:29 AM: Starting to download cache of 589.6MB
2:04:39 AM: Finished downloading cache in 10.249720107s
2:04:39 AM: Starting to extract cache
2:04:52 AM: Finished extracting cache in 12.31500737s
2:04:52 AM: Finished fetching cache in 22.624511593s
2:04:52 AM: Starting to prepare the repo for build
2:04:52 AM: Preparing Git Reference refs/heads/master
2:04:59 AM: Parsing package.json dependencies
2:05:00 AM: 
​
❯ Initial build environment
baseRelDir: true
branch: master
context: production
cwd: /opt/build/repo
featureFlags: []
mode: buildbot
repositoryRoot: /opt/build/repo
siteId: 36dc8a72-bda5-48a7-8c6f-f1aff776587d
​
❯ UI build settings
baseRelDir: true
build:
  command: nuxt generate
  environment:
    - NETLIFY_BUILD_DEBUG
    - ONEGRAPH_AUTHLIFY_TOKEN
  publish: dist
​
❯ Resolved build environment
branch: master
buildDir: /opt/build/repo
configPath: /opt/build/repo/netlify.toml
context: production
env: []
​
❯ Resolved config
build:
  command: nuxt generate
  commandOrigin: ui
  environment:
    - NETLIFY_BUILD_DEBUG
    - ONEGRAPH_AUTHLIFY_TOKEN
  publish: /opt/build/repo/dist
  publishOrigin: ui
headers:
  - for: /*
    values:
      Access-Control-Allow-Methods: GET, OPTIONS
      Access-Control-Allow-Origin: '*'
headersOrigin: config

2:05:00 AM: Starting build script
2:05:00 AM: Installing dependencies
2:05:00 AM: Python version set to 2.7
2:05:01 AM: Started restoring cached node version
2:05:02 AM: Finished restoring cached node version
2:05:03 AM: v16.14.2 is already installed.
2:05:03 AM: Now using node v16.14.2 (npm v8.5.0)
2:05:03 AM: Started restoring cached build plugins
2:05:03 AM: Finished restoring cached build plugins
2:05:04 AM: Attempting ruby version 2.7.2, read from environment
2:05:05 AM: Using ruby version 2.7.2
2:05:05 AM: Using PHP version 8.0
2:05:06 AM: No npm workspaces detected
2:05:06 AM: Started restoring cached node modules
2:05:06 AM: Finished restoring cached node modules
2:05:06 AM: Started restoring cached go cache
2:05:06 AM: Finished restoring cached go cache
2:05:06 AM: go version go1.16.5 linux/amd64
2:05:06 AM: go version go1.16.5 linux/amd64
2:05:06 AM: Installing missing commands
2:05:06 AM: Verify run directory
2:05:08 AM: ​
2:05:08 AM: ────────────────────────────────────────────────────────────────
2:05:08 AM:   Netlify Build                                                 
2:05:08 AM: ────────────────────────────────────────────────────────────────
2:05:08 AM: ​
2:05:08 AM: ❯ Version
2:05:08 AM:   @netlify/build 27.0.1
2:05:08 AM: ​
2:05:08 AM: ❯ Flags
2:05:08 AM:   apiHost: api.netlify.com
2:05:08 AM:   baseRelDir: true
2:05:08 AM:   branch: master
2:05:08 AM:   buildId: 6268261b9839151525053e4c
2:05:08 AM:   buildbotServerSocket: /tmp/netlify-buildbot-socket
2:05:08 AM:   cacheDir: /opt/build/cache
2:05:08 AM:   cachedConfigPath: /tmp/netlify_config.json
2:05:08 AM:   context: production
2:05:08 AM:   cwd: /opt/build/repo
2:05:08 AM:   deployId: 6268261b9839151525053e4e
2:05:08 AM:   edgeFunctionsDistDir: /tmp/edge-6268261b9839151525053e4e
2:05:08 AM:   featureFlags:
2:05:08 AM:     - buildbot_build_go_functions
2:05:08 AM:     - edge_functions_cache_cli
2:05:08 AM:     - zisi_parse_isc
2:05:08 AM:   framework: nuxt
2:05:08 AM:   functionsDistDir: /tmp/zisi-6268261b9839151525053e4e
2:05:08 AM:   mode: buildbot
2:05:08 AM:   nodePath: /opt/buildhome/.nvm/versions/node/v16.14.2/bin/node
2:05:08 AM:   repositoryRoot: /opt/build/repo
2:05:08 AM:   saveConfig: true
2:05:08 AM:   sendStatus: true
2:05:08 AM:   siteId: 36dc8a72-bda5-48a7-8c6f-f1aff776587d
2:05:08 AM:   statsd:
2:05:08 AM:     host: 10.65.19.17
2:05:08 AM:     port: 8125
2:05:08 AM:   testOpts:
2:05:08 AM:     silentLingeringProcesses: ''
2:05:08 AM: ​
2:05:08 AM: ❯ Current directory
2:05:08 AM:   /opt/build/repo
2:05:08 AM: ​
2:05:08 AM: ❯ Config file
2:05:08 AM:   /opt/build/repo/netlify.toml
2:05:08 AM: ​
2:05:08 AM: ❯ Resolved config
2:05:08 AM:   build:
2:05:08 AM:     command: nuxt generate
2:05:08 AM:     commandOrigin: ui
2:05:08 AM:     environment:
2:05:08 AM:       - NETLIFY_BUILD_DEBUG
2:05:08 AM:       - ONEGRAPH_AUTHLIFY_TOKEN
2:05:08 AM:     publish: /opt/build/repo/dist
2:05:08 AM:     publishOrigin: ui
2:05:08 AM:   headers:
2:05:08 AM:     - for: /*
      values:
        Access-Control-Allow-Methods: GET, OPTIONS
        Access-Control-Allow-Origin: '*'
  headersOrigin: config
​
2:05:08 AM: ❯ Context
2:05:08 AM:   production
2:05:08 AM: ​
2:05:08 AM: ────────────────────────────────────────────────────────────────
2:05:08 AM:   1. Build command from Netlify app                             
2:05:08 AM: ────────────────────────────────────────────────────────────────
2:05:08 AM: ​
2:05:08 AM: $ nuxt generate
5:43:36 AM: ​
5:43:36 AM: (build.command completed in 3h 38m 28.4s)
5:43:36 AM: ​
5:43:36 AM: ────────────────────────────────────────────────────────────────
5:43:36 AM:   2. Deploy site                                                
5:43:36 AM: ────────────────────────────────────────────────────────────────
5:43:36 AM: ​
5:43:36 AM: ​
5:43:36 AM: ❯ Uploaded config
5:43:36 AM:   [build]
5:43:36 AM: ​
5:43:36 AM:   [[headers]]
5:43:36 AM:     for = "/*"
5:43:36 AM:       [headers.values]
5:43:36 AM:       Access-Control-Allow-Origin = "*"
5:43:36 AM:       Access-Control-Allow-Methods = "GET, OPTIONS"
5:43:36 AM:   #    Access-Control-Allow-Headers = "X-Requested-With, X-HTTP-Method-Override, Content-Type, Accept"
5:43:36 AM: ​
5:43:36 AM: ❯ Uploaded headers
5:43:36 AM:   No headers
5:43:36 AM: ​
5:43:36 AM: ❯ Uploaded redirects
5:43:36 AM:   No redirects
5:43:36 AM: ​
5:43:36 AM: Starting to deploy site from 'dist'
5:49:26 AM: Creating deploy tree asynchronously
5:49:31 AM: Creating deploy upload records
5:54:34 AM: ​
5:54:34 AM: ────────────────────────────────────────────────────────────────
5:54:34 AM:   Internal error during "Deploy site"                           
5:54:34 AM: ────────────────────────────────────────────────────────────────
5:54:34 AM: ​
5:54:34 AM:   Error message
5:54:34 AM:   Error: Deploy did not succeed: Failed to execute deploy: timed out while waiting to enter states [prepared, ready]
5:54:34 AM: ​
5:54:34 AM:   Error location
5:54:34 AM:   During Deploy site
5:54:34 AM:       at handleDeployError (file:///opt/buildhome/node-deps/node_modules/@netlify/build/src/plugins_core/deploy/buildbot_client.js:87:18)
5:54:34 AM:       at deploySiteWithBuildbotClient (file:///opt/buildhome/node-deps/node_modules/@netlify/build/src/plugins_core/deploy/buildbot_client.js:68:12)
5:54:34 AM:       at processTicksAndRejections (node:internal/process/task_queues:96:5)
5:54:34 AM:       at async coreStep (file:///opt/buildhome/node-deps/node_modules/@netlify/build/src/plugins_core/deploy/index.js:45:5)
5:54:34 AM:       at async fireCoreStep (file:///opt/buildhome/node-deps/node_modules/@netlify/build/src/steps/core_step.js:39:9)
5:54:34 AM: Creating deploy upload records
5:54:34 AM:       at async tFireStep (file:///opt/buildhome/node-deps/node_modules/@netlify/build/src/time/main.js:20:59)
5:54:34 AM:       at async runStep (file:///opt/buildhome/node-deps/node_modules/@netlify/build/src/steps/run_step.js:88:7)
5:54:34 AM:       at async pReduce.index (file:///opt/buildhome/node-deps/node_modules/@netlify/build/src/steps/run_steps.js:91:11)
5:54:34 AM:       at async runSteps (file:///opt/buildhome/node-deps/node_modules/@netlify/build/src/steps/run_steps.js:51:7)
5:54:34 AM:       at async runBuild (file:///opt/buildhome/node-deps/node_modules/@netlify/build/src/core/main.js:612:7)
5:54:35 AM: Failed during stage 'building site': Build script returned non-zero exit code: 4 (https://ntl.fyi/exit-code-4)
5:54:34 AM: ​
5:54:34 AM:   Resolved config
5:54:34 AM:   build:
5:54:34 AM:     command: nuxt generate
5:54:34 AM:     commandOrigin: ui
5:54:34 AM:     environment:
5:54:34 AM:       - NETLIFY_BUILD_DEBUG
5:54:34 AM:       - ONEGRAPH_AUTHLIFY_TOKEN
5:54:34 AM:     publish: /opt/build/repo/dist
5:54:34 AM:     publishOrigin: ui
5:54:34 AM:   headers:
5:54:34 AM:     - for: /*
      values:
        Access-Control-Allow-Methods: GET, OPTIONS
        Access-Control-Allow-Origin: '*'
  headersOrigin: config
Caching artifacts
5:54:34 AM: Started saving node modules
5:54:34 AM: Finished saving node modules
5:54:34 AM: Started saving build plugins
5:54:34 AM: Finished saving build plugins
5:54:34 AM: Started saving pip cache
5:54:34 AM: Finished saving pip cache
5:54:34 AM: Started saving emacs cask dependencies
5:54:34 AM: Finished saving emacs cask dependencies
5:54:34 AM: Started saving maven dependencies
5:54:34 AM: Finished saving maven dependencies
5:54:34 AM: Started saving boot dependencies
5:54:34 AM: Finished saving boot dependencies
5:54:34 AM: Started saving rust rustup cache
5:54:34 AM: Finished saving rust rustup cache
5:54:34 AM: Started saving go dependencies
5:54:34 AM: Finished saving go dependencies
5:54:34 AM: Build failed due to an internal system error: Build script returned non-zero exit code: 4
5:54:35 AM: Failing build: Failed to build site
5:54:35 AM: Finished processing build request in 3h50m6.013552439s

[perry]

@nisarahot

2:05:08 AM: ────────────────────────────────────────────────────────────────
2:05:08 AM:   1. Build command from Netlify app                             
2:05:08 AM: ────────────────────────────────────────────────────────────────
2:05:08 AM: ​
2:05:08 AM: $ nuxt generate
5:43:36 AM: ​
5:43:36 AM: (build.command completed in 3h 38m 28.4s)

hey there,

so the lines above are definitely the problem - while we have maybe 1-2 sites (enterprise sites) on our network that take that long to build, that is a VERY long build time. This definitely indicates that something isn’t working right.

some quick thoughts on this:

[Support Guide] Hung or stuck, but seemingly successful build

Last review by Netlify Support Team on December 2022 When a deploy gets stuck with no apparent build failure, it typically means that while our robots did start building your code, it didn’t wrap up some process within the 15 minute time limit our build environment requires. Why can this happen? More on that in a sec, but first I want to mention that our build environment requires all processes, not just the main ones started by your build to complete. If even one thing gets stuck, the proces…

[Support Guide] How long should Netlify builds take?

You should count on your builds having only 15 minutes to complete their build command. The additional time for processing and uploading will be added to that and do not need to complete within that initial 15 minutes, and in some cases for very large sites or if our system is busy, they may stretch out to minutes or even an hour (the latter only for tens of thousands of new files in a deploy). Around unexpected timeouts, I’d suggest some reading here: [Support Guide] Hung or stuck, but seemi…

[Support Guide] How can I optimize my Netlify build time?

Last reviewed by Netlify Support - August, 2023 If you are interested in learning more about how our bots approach building sites, these three articles are a great place to start: How our Build Bots build sites The last major upgrade to our build environment and perhaps least-loved but most useful, how to debug your builds So you might think, it’s great that Netlify builds every commit from my repos…but isn’t that a bit wasteful? The answer is, it depends, but we can at least give you…

[Support Guide] Build Performance Pro-tips

Last Reviewed By Netlify Support Staff: May 2023 You’ve gotten your site to build, perhaps via our Support Guide on debugging site builds. Congrats! What’s next? Now is a great time to make your build FASTER! While there is no “one size fits all” for how long your build should take, less is more here. The faster you can deploy a site, the faster you can work on the next version. This guide will review some best practices and offer some guidance on speeding up your builds.There are three hig…

I’d work through these guides and see if there is anything helpful to bring down the build time. I am also wondering if there is some kind of process somewhere that isn’t actually building but idling in the background, and eventually timing out.

While we can bump up the build time to an hour or an hour and half, I don’t think we can go as high as 3 hours without getting you talking with a different team to see if a custom build solution is right for you.

Another option could be to build locally and push the built files Not ideal from a workflow perspective, but, this will at least get a build up while you take the time to see if you can optimize your build somehow to bring build times down.

give these resources a try, and let us know!