Builds shows Completed but Deploy is still Processing

nisarahot · April 21, 2022, 5:48am

Hi, there.

I have deployed a site with a large number of files. It is taking a long time because of the large number of files (about 1G in total).

After 24 hours or so, Builds shows Complete, but when I go to “Go to deploy details” and go to the deploy log, it is still in Processing and the last output in the log is

Post processing - HTML

The last output in the log is “Post processing - HTML”. What is going on here?

And the build time limit has long been exceeded, but the process has not been canceled. What steps of the process consume the “build time”?

Thanks.

hillary · April 21, 2022, 10:10pm

Hey there, @nisarahot

Great questions! Sorry that you are encountering difficulties here. Can you please share a bit more information so that we can look into this further?

A link to your site
You full build log

Additionally, here is a great resource on optimizing your build time: [Support Guide] How can I optimize my Netlify build time?

nisarahot · April 22, 2022, 2:34am

It takes 48hours since build start, still processing, looks like freezing.
I think it’s indeed taking too long.

Thanks in advance.

sitename: affectionate-lamarr-c83264
build log:

12:49:55 PM: Build ready to start
12:49:57 PM: build-image version: d2c6dbeac570350a387d832f64bc980dc964ad65 (focal)
12:49:57 PM: build-image tag: v4.8.0
12:49:57 PM: buildbot version: e58b6be665675c0f99b33132a8c1eec1f775eba1
12:49:57 PM: Fetching cached dependencies
12:49:57 PM: Starting to download cache of 589.4MB
12:50:10 PM: Finished downloading cache in 12.67752556s
12:50:10 PM: Starting to extract cache
12:50:21 PM: Finished extracting cache in 10.801951929s
12:50:21 PM: Finished fetching cache in 23.559228901s
12:50:21 PM: Starting to prepare the repo for build
12:50:21 PM: Preparing Git Reference refs/heads/master
12:50:28 PM: Parsing package.json dependencies
12:50:30 PM: Starting build script
12:50:30 PM: Installing dependencies
12:50:30 PM: Python version set to 2.7
12:50:30 PM: Started restoring cached node version
12:50:32 PM: Finished restoring cached node version
12:50:32 PM: v16.14.2 is already installed.
12:50:33 PM: Now using node v16.14.2 (npm v8.5.0)
12:50:33 PM: Started restoring cached build plugins
12:50:33 PM: Finished restoring cached build plugins
12:50:33 PM: Attempting ruby version 2.7.2, read from environment
12:50:34 PM: Using ruby version 2.7.2
12:50:34 PM: Using PHP version 8.0
12:50:35 PM: No npm workspaces detected
12:50:35 PM: Started restoring cached node modules
12:50:35 PM: Finished restoring cached node modules
12:50:35 PM: Started restoring cached go cache
12:50:35 PM: Finished restoring cached go cache
12:50:35 PM: go version go1.16.5 linux/amd64
12:50:35 PM: go version go1.16.5 linux/amd64
12:50:35 PM: Installing missing commands
12:50:35 PM: Verify run directory
12:50:37 PM: 
12:50:37 PM: ────────────────────────────────────────────────────────────────
12:50:37 PM:   Netlify Build                                                 
12:50:37 PM: ────────────────────────────────────────────────────────────────
12:50:37 PM: 
12:50:37 PM: ❯ Version
12:50:37 PM:   @netlify/build 27.0.1
12:50:37 PM: 
12:50:37 PM: ❯ Flags
12:50:37 PM:   baseRelDir: true
12:50:37 PM:   buildId: 625f82e3fda6cb22ebdcce26
12:50:37 PM:   deployId: 625f82e3fda6cb22ebdcce28
12:50:37 PM: 
12:50:37 PM: ❯ Current directory
12:50:37 PM:   /opt/build/repo
12:50:37 PM: 
12:50:37 PM: ❯ Config file
12:50:37 PM:   /opt/build/repo/netlify.toml
12:50:37 PM: 
12:50:37 PM: ❯ Context
12:50:37 PM:   production
12:50:37 PM: 
12:50:37 PM: ────────────────────────────────────────────────────────────────
12:50:37 PM:   1. Build command from Netlify app                             
12:50:37 PM: ────────────────────────────────────────────────────────────────
12:50:37 PM: 
12:50:37 PM: $ nuxt generate
2:57:15 PM: 
2:57:15 PM: (build.command completed in 2h 6m 38s)
2:57:15 PM: 
2:57:15 PM: ────────────────────────────────────────────────────────────────
2:57:15 PM:   2. Deploy site                                                
2:57:15 PM: ────────────────────────────────────────────────────────────────
2:57:15 PM: 
2:57:15 PM: Starting to deploy site from 'dist'
2:57:20 PM: Creating deploy tree asynchronously
2:57:20 PM: Creating deploy upload records
2:57:49 PM: 49601 new files to upload
2:57:49 PM: 0 new functions to upload
3:21:50 PM: Site deploy was successfully initiated
3:21:50 PM: 
3:21:50 PM: (Deploy site completed in 24m 35s)
3:21:50 PM: 
3:21:50 PM: ────────────────────────────────────────────────────────────────
3:21:50 PM:   Netlify Build Complete                                        
3:21:50 PM: ────────────────────────────────────────────────────────────────
3:21:50 PM: 
3:21:50 PM: (Netlify Build completed in 2h 31m 13.2s)
3:21:51 PM: Caching artifacts
3:21:51 PM: Started saving node modules
3:21:51 PM: Finished saving node modules
3:21:51 PM: Started saving build plugins
3:21:51 PM: Finished saving build plugins
3:21:51 PM: Started saving pip cache
3:21:51 PM: Finished saving pip cache
3:21:51 PM: Started saving emacs cask dependencies
3:21:51 PM: Finished saving emacs cask dependencies
3:21:51 PM: Started saving maven dependencies
3:21:51 PM: Finished saving maven dependencies
3:21:51 PM: Started saving boot dependencies
3:21:51 PM: Finished saving boot dependencies
3:21:51 PM: Started saving rust rustup cache
3:21:51 PM: Finished saving rust rustup cache
3:21:51 PM: Started saving go dependencies
3:21:51 PM: Finished saving go dependencies
3:21:51 PM: Build script success
3:22:59 PM: Failed to save cache, continuing with build
3:22:59 PM: Finished processing build request in 2h33m2.235138904s
3:26:52 PM: Starting post processing
3:26:52 PM: Post processing - HTML

hrishikesh · April 24, 2022, 3:44pm

Hey @nisarahot,

We’ve tried to cancel the build from our end, which should probably get you unblocked. Could you try to see if you’re still having issues?

nisarahot · April 24, 2022, 11:40pm

@hrishikesh

Thanks for the response. I manually canceled and try again but it fails with “Configuration error” occurs. What does it mean?

Deploy did not succeed: [PUT /sites/{site_id}/deploys/{deploy_id}][422] updateSiteDeploy default  &{Code:422 Message:Build is already done}

Full log is here:

11:31:11 PM: Build ready to start
11:31:12 PM: build-image version: d2c6dbeac570350a387d832f64bc980dc964ad65 (focal)
11:31:12 PM: build-image tag: v4.8.0
11:31:12 PM: buildbot version: e58b6be665675c0f99b33132a8c1eec1f775eba1
11:31:12 PM: Fetching cached dependencies
11:31:12 PM: Starting to download cache of 589.4MB
11:31:21 PM: Finished downloading cache in 8.26215874s
11:31:21 PM: Starting to extract cache
11:31:31 PM: Finished extracting cache in 10.311095045s
11:31:31 PM: Finished fetching cache in 18.650012991s
11:31:31 PM: Starting to prepare the repo for build
11:31:31 PM: Preparing Git Reference refs/heads/master
11:32:02 PM: Parsing package.json dependencies
11:32:03 PM: 

❯ Initial build environment
baseRelDir: true
branch: master
context: production
cwd: /opt/build/repo
featureFlags: []
mode: buildbot
repositoryRoot: /opt/build/repo
siteId: 36dc8a72-bda5-48a7-8c6f-f1aff776587d

❯ UI build settings
baseRelDir: true
build:
  command: nuxt generate
  environment:
    - NETLIFY_BUILD_DEBUG
    - ONEGRAPH_AUTHLIFY_TOKEN
  publish: dist

❯ Resolved build environment
branch: master
buildDir: /opt/build/repo
configPath: /opt/build/repo/netlify.toml
context: production
env: []

❯ Resolved config
build:
  command: nuxt generate
  commandOrigin: ui
  environment:
    - NETLIFY_BUILD_DEBUG
    - ONEGRAPH_AUTHLIFY_TOKEN
  publish: /opt/build/repo/dist
  publishOrigin: ui
headers:
  - for: /*
    values:
      Access-Control-Allow-Methods: GET, OPTIONS
      Access-Control-Allow-Origin: '*'
headersOrigin: config

11:32:03 PM: Starting build script
11:32:03 PM: Installing dependencies
11:32:03 PM: Python version set to 2.7
11:32:03 PM: Started restoring cached node version
11:32:05 PM: Finished restoring cached node version
11:32:05 PM: v16.14.2 is already installed.
11:32:06 PM: Now using node v16.14.2 (npm v8.5.0)
11:32:06 PM: Started restoring cached build plugins
11:32:06 PM: Finished restoring cached build plugins
11:32:06 PM: Attempting ruby version 2.7.2, read from environment
11:32:07 PM: Using ruby version 2.7.2
11:32:07 PM: Using PHP version 8.0
11:32:08 PM: No npm workspaces detected
11:32:08 PM: Started restoring cached node modules
11:32:08 PM: Finished restoring cached node modules
11:32:08 PM: Started restoring cached go cache
11:32:08 PM: Finished restoring cached go cache
11:32:08 PM: go version go1.16.5 linux/amd64
11:32:08 PM: go version go1.16.5 linux/amd64
11:32:08 PM: Installing missing commands
11:32:08 PM: Verify run directory
11:32:09 PM: 
11:32:09 PM: ────────────────────────────────────────────────────────────────
11:32:09 PM:   Netlify Build                                                 
11:32:09 PM: ────────────────────────────────────────────────────────────────
11:32:09 PM: 
11:32:09 PM: ❯ Version
11:32:09 PM:   @netlify/build 27.0.1
11:32:09 PM: 
11:32:09 PM: ❯ Flags
11:32:09 PM:   apiHost: api.netlify.com
11:32:09 PM:   baseRelDir: true
11:32:09 PM:   branch: master
11:32:09 PM:   buildId: 62655f2ebe3a432cd77f3fba
11:32:09 PM:   buildbotServerSocket: /tmp/netlify-buildbot-socket
11:32:09 PM:   cacheDir: /opt/build/cache
11:32:09 PM:   cachedConfigPath: /tmp/netlify_config.json
11:32:09 PM:   context: production
11:32:09 PM:   cwd: /opt/build/repo
11:32:09 PM:   deployId: 62655f2ebe3a432cd77f3fbc
11:32:09 PM:   edgeFunctionsDistDir: /tmp/edge-62655f2ebe3a432cd77f3fbc
11:32:09 PM:   featureFlags:
11:32:09 PM:     - buildbot_build_go_functions
11:32:09 PM:     - edge_functions_cache_cli
11:32:09 PM:     - zisi_parse_isc
11:32:09 PM:   framework: nuxt
11:32:09 PM:   functionsDistDir: /tmp/zisi-62655f2ebe3a432cd77f3fbc
11:32:09 PM:   mode: buildbot
11:32:09 PM:   nodePath: /opt/buildhome/.nvm/versions/node/v16.14.2/bin/node
11:32:09 PM:   repositoryRoot: /opt/build/repo
11:32:09 PM:   saveConfig: true
11:32:09 PM:   sendStatus: true
11:32:09 PM:   siteId: 36dc8a72-bda5-48a7-8c6f-f1aff776587d
11:32:09 PM:   statsd:
11:32:09 PM:     host: 10.65.154.174
11:32:09 PM:     port: 8125
11:32:09 PM:   testOpts:
11:32:09 PM:     silentLingeringProcesses: ''
11:32:09 PM: 
11:32:09 PM: ❯ Current directory
11:32:09 PM:   /opt/build/repo
11:32:09 PM: 
11:32:09 PM: ❯ Config file
11:32:09 PM:   /opt/build/repo/netlify.toml
11:32:09 PM: 
11:32:09 PM: ❯ Resolved config
11:32:09 PM:   build:
11:32:09 PM:     command: nuxt generate
11:32:09 PM:     commandOrigin: ui
11:32:09 PM:     environment:
11:32:09 PM:       - NETLIFY_BUILD_DEBUG
11:32:09 PM:       - ONEGRAPH_AUTHLIFY_TOKEN
11:32:09 PM:     publish: /opt/build/repo/dist
11:32:09 PM:     publishOrigin: ui
11:32:09 PM:   headers:
11:32:09 PM:     - for: /*
      values:
        Access-Control-Allow-Methods: GET, OPTIONS
        Access-Control-Allow-Origin: '*'
  headersOrigin: config

11:32:09 PM: ❯ Context
11:32:09 PM:   production
11:32:09 PM: 
11:32:09 PM: ────────────────────────────────────────────────────────────────
11:32:09 PM:   1. Build command from Netlify app                             
11:32:09 PM: ────────────────────────────────────────────────────────────────
11:32:09 PM: 
11:32:10 PM: $ nuxt generate
1:42:17 AM: 
1:42:17 AM: (build.command completed in 2h 10m 7.2s)
1:42:17 AM: 
1:42:17 AM: ────────────────────────────────────────────────────────────────
1:42:17 AM:   2. Deploy site                                                
1:42:17 AM: ────────────────────────────────────────────────────────────────
1:42:17 AM: 
1:42:17 AM: 
1:42:17 AM: ❯ Uploaded config
1:42:17 AM:   [build]
1:42:17 AM: 
1:42:17 AM:   [[headers]]
1:42:17 AM:     for = "/*"
1:42:17 AM:       [headers.values]
1:42:17 AM:       Access-Control-Allow-Origin = "*"
1:42:17 AM:       Access-Control-Allow-Methods = "GET, OPTIONS"
1:42:17 AM:   #    Access-Control-Allow-Headers = "X-Requested-With, X-HTTP-Method-Override, Content-Type, Accept"
1:42:17 AM: 
1:42:17 AM: ❯ Uploaded headers
1:42:17 AM:   No headers
1:42:17 AM: 
1:42:17 AM: ❯ Uploaded redirects
1:42:17 AM:   No redirects
1:42:17 AM: 
1:42:17 AM: Starting to deploy site from 'dist'
1:42:22 AM: Creating deploy tree asynchronously
1:42:23 AM: 
1:42:23 AM: ────────────────────────────────────────────────────────────────
1:42:23 AM:   Configuration error                                           
1:42:23 AM: ────────────────────────────────────────────────────────────────
1:42:23 AM: 
1:42:23 AM:   Error message
1:42:23 AM:   Deploy did not succeed: [PUT /sites/{site_id}/deploys/{deploy_id}][422] updateSiteDeploy default  &{Code:422 Message:Build is already done}
1:42:23 AM: 
1:42:23 AM:   Resolved config
1:42:23 AM:   build:
1:42:23 AM:     command: nuxt generate
1:42:23 AM:     commandOrigin: ui
1:42:23 AM:     environment:
1:42:23 AM:       - NETLIFY_BUILD_DEBUG
1:42:23 AM:       - ONEGRAPH_AUTHLIFY_TOKEN
1:42:23 AM:     publish: /opt/build/repo/dist
1:42:23 AM:     publishOrigin: ui
1:42:23 AM:   headers:
1:42:24 AM:     - for: /*
      values:
        Access-Control-Allow-Methods: GET, OPTIONS
        Access-Control-Allow-Origin: '*'
  headersOrigin: config
Caching artifacts
1:42:24 AM: Started saving node modules
1:42:24 AM: Finished saving node modules
1:42:24 AM: Started saving build plugins
1:42:24 AM: Finished saving build plugins
1:42:24 AM: Started saving pip cache
1:42:24 AM: Finished saving pip cache
1:42:24 AM: Started saving emacs cask dependencies
1:42:24 AM: Finished saving emacs cask dependencies
1:42:24 AM: Started saving maven dependencies
1:42:24 AM: Finished saving maven dependencies
1:42:24 AM: Started saving boot dependencies
1:42:24 AM: Finished saving boot dependencies
1:42:24 AM: Started saving rust rustup cache
1:42:24 AM: Finished saving rust rustup cache
1:42:24 AM: Started saving go dependencies
1:42:24 AM: Finished saving go dependencies
1:42:24 AM: Build failed due to a user error: Build script returned non-zero exit code: 2
1:42:24 AM: Failing build: Failed to build site
1:42:24 AM: Finished processing build request in 2h11m11.658321967s

audrey · April 27, 2022, 2:42am

Hi @nisarahot ,

It appears you were able to self resolve the issue as we see several successful builds since your last post

Looking into the site’s most recent failed deploy, we saw a pre-process timeout error. We’ve increased that timeout which should help with the next build. Let us know if that’s not the case

nisarahot · April 28, 2022, 12:13am

Thank you @andrew!

We tried to deploy again but it failed with the same error.

Is the new timeout enough?
We are trying to deploy over 70000 pages and over 200000 pages finally.

Thanks in advance.
2022年4月27日 11:52 +0900、audrey via Netlify Support Forums netlify@discoursemail.comのメール:

audrey · April 29, 2022, 3:10am

One thing that will help is to disable form detection on your site here: Netlify App

With each deploy, Netlify scans new and changed HTML files for forms to enable handling submissions. Since you’re not using Netlify Forms, you can disable form detection to speed up deploy time. Let us know how it goes!

nisarahot · May 4, 2022, 11:56am

Thank you @audrey !

First, we changed the Form settings and tried to build, but the results did not change.

Next, in e-mail communications @hrishikesh upgraded the timeout of the deploy stage to 4 times more than before and then we didn’t get the error above but an another error was occured.

9:54:15 PM: Failed during stage 'building site': Command was cancelled
9:54:15 PM: Execution cancelled
9:54:15 PM: Error running command: Command was cancelled
9:54:15 PM: Failing build: Failed to build site

Seems like we’re almost there. Again, our goal is to deploy 200,000 pages (current build is 63,000 pages around), so please configure on that assumption.

This problem seems to be a factor on the Netlify server side that is consuming a lot of my build time. I would be happy to have this resolved soon. Thanks.

audrey · May 7, 2022, 4:04am

Hey @nisarahot !

Our developers were able to ship a fix yesterday that should help speed up processing times on large builds. Would you please try re-deploying and let us know if it helped or not?

nisarahot · May 8, 2022, 7:50am

Thank you @audrey,

We tried to deploy again, but “nuxt generate” is not yet complete after 24 hours.

Previously nuxt generate was complete in about 4 hours, what does that mean?

hrishikesh · May 8, 2022, 4:21pm

Try disabling the quiet mode: Nuxt - The build Property, to see what Nuxt could be doing.

nisarahot · May 9, 2022, 3:29am

Thank you @hrishikesh, okay, we started to build again with disabling the quiet mode, but it would probably output over 60000 lines so I’m concerned that it’s hard to read on the deploy page.

I will let you know when I get the results.

nisarahot · May 10, 2022, 11:11am

@hrishikesh @audrey The build has now been about 30 hours and still not completed nor failed. I am going to cancel a little later.

There are so many lines it is difficult to read from the deploy page, but if the logs are logged and shown correctly it looks like the generate was timeout after 3 hours.

Remind that nuxt generate succeeds in local builds and even completed on Netlify before the fix.

perry · May 11, 2022, 4:38pm

hi nisarahot,

we have been talking about this internally, and are wondering whether this is more of an issue with your build needing too much memory.

I’m still trying to get some more eyes on this, but for now, could you look through this post please?

nisarahot · May 12, 2022, 7:20am

Thank you @perry , I read that post, and unlike Gatsby, it seems impossible to reduce the number of “changed” pages with the Nuxt.js mechanism.

And about memory, I wonder why was nuxt generate successful more than once before the fix.

It’s been about 20 days and my client is getting a bit frustrated, so I’m sincerely waiting to hear from you.

gualter · May 13, 2022, 2:13pm

Hello @nisarahot

I can see that you were able to make a successful build today. Did you make any change on your end? the build ended in about 1 hour.

nisarahot · May 14, 2022, 12:04pm

Hello @gualter

Because we had to launch before we could resolve it, thesuccessful build was temporarily limited to generate only 10000+ pages, and some other configurations were also very limited.

No other changes were made, and the build we want to achive should be the same result if you haven’t made any changes.

I will revert to the original configuration later before a next test deploy, but please let me know it is better to do so sooner.

hrishikesh · May 15, 2022, 7:47pm

Hey @nisarahot,

Sorry for all these back-and-forth. We’d be discussing this with our engineers, most likely tomorrow. So, we’d keep you posted about the updates.

hrishikesh · May 19, 2022, 10:39am

Hey @nisarahot,

We got a chance to discuss this, and it turns out that you’d probably be disappointed. The problem for the “stuck” builds was because the builds ran out of memory. The memory limits of the container simply exhausted while trying to deploy.

While the engineers are working to optimise the larger problem of OOM issues, the immediate solution for you would probably be deploying via CLI.