Builds getting stuck every time

alberto98fx · December 15, 2021, 5:05pm

Hi @hrishikesh @luke , I share some deploys IDs that got stuck:

61ba1afa4937e1491f6fcb00
61ba1b41d9e13b3fa70a2635

Original:
61ba19b05791c8000743c890

Retries of c890 here ^ :

OK: 61ba1d528d5b4341e8aafd14
OK: 61ba1d96fae0db447c801590
NOT OK (Didn’t even start, stuck at Build Ready to start for 3 mins) : 61ba1db981f95742e78a50b7
NOT OK (Failed like above ^): 61ba1db981f95742e78a50b7
Cleared cache and retried, OK: 61ba1ec6272c30443a3d8dfc
OK: 61ba1eebe9ea254556f4541d

NOT OK, stuck at Hugo build like @champagnealexandre 61ba1f354937e153e86fc52d

This is the VERY SAME deployment.
Cleared cache, restarted many times and got issues most of the times

OK total: 4
NOT OK total: 3
Out of 7 builds 3 had issues.

This is annoying, please help us
Thanks!

champagnealexandre · December 16, 2021, 11:05am

@hrishikesh there you go

champagnealexandre · December 16, 2021, 3:46pm

@hrishikesh you can compare with this repo, another hugo site I’m maintaining, which doesn’t have a similar problem

markphelps · December 16, 2021, 5:33pm

@hrishikesh I did clear the cache after installing the problem and am still experiencing the same issue:

hrishikesh · December 17, 2021, 2:31pm

Interesting, I tried that repo and the build works fine every time.

Check it out here:

Was there a specific change after which you started noticing their behaviour?

champagnealexandre · December 17, 2021, 2:49pm

Humm this is indeed weird! Some configuration in our netlify account got messed up or something then? If the repo works…

Have you tried modifying a file from the repo before rebuilding?

And no, no specific change at all…

hrishikesh · December 17, 2021, 2:56pm

I had not tried it, but now I did. I edited this file:

https://github.com/Hrishikesh-K/alexandrechampagne.io/blob/master/layouts/section/blog.html

The site was built. However, I don’t see any changes on the site. Is that expected?

Could you link us to the first deploy which had this problem?

champagnealexandre · December 18, 2021, 2:43pm

That’s expected. You’d have to change files in the content folder. But I’m guessing this is unrelated then.

Looking at the deploy log, I suppose problems started with this deploy: Netlify App

hrishikesh · December 19, 2021, 3:03pm

Just curious, since all of you’re using Hugo, does this problem exist with using a different Hugo version (possibly the latest v0.91.0)?

champagnealexandre · December 20, 2021, 11:26am

No idea, I’m new to Hugo — do you know how to update the version deployed on Netlify?

alberto98fx · December 21, 2021, 3:37pm

Yes, I tried upgrading and downgrading. Doesn’t matter

hrishikesh · December 22, 2021, 1:32pm

That’s usually possible by setting an environment variable (HUGO_VERSION) in UI or the TOML file. But based on @alberto98fx’s comment, I don’t think it’s making a difference. But if you feel inclined to test it for yourself and are able to get different results do let us know.

alberto98fx · December 22, 2021, 1:34pm

Hi @hrishikesh,
Got any news concerning this? I know this is an odd issue, but it’s a bit annoying that each time you make a Git push you need to go to the console and make sure the build didn’t actually get stuck
Thanks for any update you can give us

hrishikesh · December 22, 2021, 1:51pm

I am personally failing to reproduce this and find any explanation. So, I’ve reached out to Hugo to try and understand what can cause this, so we can come towards a solution. If someone wants to keep up with that discussion, it’s here:

In the meantime, there’s a workaround. We can disable Git-based deploys for your sites. So, you could trigger builds via API. But, instead of having to call the API manually, you can use GitHub Actions directly, or run a Node.js or Bash script in there to always trigger a build by clearing cache. You need to make a call to this endpoint:

fetch('https://api.netlify.com/api/v1/sites/<site-id>/builds', {
  body: {
    clear_cache: true
  },
  headers: {
    authorization: 'Bearer <token>'
  },
  method: 'post'
})

champagnealexandre · December 23, 2021, 10:54am

Thanks for the reply @hrishikesh !

Let us know how this turns out.

hrishikesh · December 23, 2021, 10:55am

Just for a short update, this might be a Hugo bug and it’s being fixed.

Related issue:

github.com/gohugoio/hugo

Hugo stuck on "Start building sites …"

opened 05:28PM - 21 Jan 21 UTC

closed 02:09PM - 23 Dec 21 UTC

ybizeul

Bug

### What version of Hugo are you using (`hugo version`)? <pre> $ hugo versio…n Hugo Static Site Generator v0.80.0-792EF0F4 linux/amd64 BuildDate: 2020-12-31T13:37:58Z </pre> ### Does this issue reproduce with the latest release? Yes ### Context Hugo running as a container in Gitlab pages, container base is `klakegg/hugo:debian` but it doesn't seem to matter. ### Description Every once in a while (actually, every single time when run in Gitlab) hugo will be stuck on `Start building sites …` message. I was able to run in `--debug`and got the following : ``` INFO 2021/01/21 17:14:55 Using config file: Start building sites … INFO 2021/01/21 17:14:55 syncing static files to /public/ ``` Then I wanted to reproduce the issue home and it was a little bit trickier, it happens less often, but running site generation in a `while`loop does the trick. So here is what I was able to catch with `strace`, it's stuck with repeating `FUTEX_WAIT_PRIVATE` ``` [...] newfstatat(AT_FDCWD, "/public/", {st_mode=S_IFDIR|0755, st_size=104, ...}, 0) = 0 futex(0x3134988, FUTEX_WAIT_PRIVATE, 0, NULL) = 0 futex(0x3134988, FUTEX_WAIT_PRIVATE, 0, NULL) = 0 futex(0x3134988, FUTEX_WAIT_PRIVATE, 0, NULL) = 0 futex(0xc000058548, FUTEX_WAKE_PRIVATE, 1) = 1 futex(0x3134988, FUTEX_WAIT_PRIVATE, 0, NULL) = 0 futex(0x3134988, FUTEX_WAIT_PRIVATE, 0, NULL^Cstrace: Process 101 detached ``` I'm running `hugo`with the following flags to test : ``` hugo --debug --noChmod --noTimes -d ../public ``` Also, the site is bound to the container from a local directory I'm not sure what other parts of strace would be relevant.

Fix PR:

champagnealexandre · December 24, 2021, 10:49am

@hrishikesh the fix has been merged to hugo’s master branch, I’ll let you know in my next website updates how it goes

thanks again for the follow up !

hrishikesh · December 24, 2021, 11:13am

Other users facing this issue, could you all confirm if it’s all good?

champagnealexandre · January 3, 2022, 7:52pm

I have tried updating the Hugo version Netlify uses by replacing

HUGO_VERSION = "0.86.1"

with

HUGO_VERSION = "0.91.2"

in my netlify.toml but now the build completely fails.

Log:

2:49:45 PM: Build ready to start
2:49:47 PM: build-image version: 3fa6e2800752667ac56af498951f923e208f7404 (xenial)
2:49:47 PM: build-image tag: v3.10.3
2:49:47 PM: buildbot version: dc5f8eb56e4839fed589a9cad4576ae4b1140fda
2:49:47 PM: Fetching cached dependencies
2:49:47 PM: Starting to download cache of 162.1MB
2:49:48 PM: Finished downloading cache in 1.073849061s
2:49:48 PM: Starting to extract cache
2:49:52 PM: Finished extracting cache in 4.101287581s
2:49:52 PM: Finished fetching cache in 5.210793282s
2:49:52 PM: Starting to prepare the repo for build
2:49:53 PM: Preparing Git Reference refs/heads/master
2:49:54 PM: Parsing package.json dependencies
2:49:55 PM: Starting build script
2:49:55 PM: Installing dependencies
2:49:55 PM: Python version set to 2.7
2:49:55 PM: Started restoring cached node version
2:49:58 PM: Finished restoring cached node version
2:49:59 PM: v12.18.0 is already installed.
2:50:00 PM: Now using node v12.18.0 (npm v6.14.4)
2:50:00 PM: Started restoring cached build plugins
2:50:00 PM: Finished restoring cached build plugins
2:50:00 PM: Attempting ruby version 2.7.1, read from environment
2:50:01 PM: Started restoring cached ruby version
2:50:01 PM: Finished restoring cached ruby version
2:50:03 PM: Using ruby version 2.7.1
2:50:03 PM: Using PHP version 5.6
2:50:03 PM: Installing Hugo 0.91.2
2:50:05 PM: hugo v0.91.2-1798BD3F+extended linux/amd64 BuildDate=2021-12-23T15:33:34Z VendorInfo=gohugoio
2:50:05 PM: Started restoring cached go cache
2:50:05 PM: Finished restoring cached go cache
2:50:05 PM: go version go1.14.4 linux/amd64
2:50:05 PM: go version go1.14.4 linux/amd64
2:50:05 PM: Installing missing commands
2:50:05 PM: Verify run directory
2:50:07 PM: 
2:50:07 PM: ────────────────────────────────────────────────────────────────
2:50:07 PM:   Netlify Build                                                 
2:50:07 PM: ────────────────────────────────────────────────────────────────
2:50:07 PM: 
2:50:07 PM: ❯ Version
2:50:07 PM:   @netlify/build 26.1.0
2:50:07 PM: 
2:50:07 PM: ❯ Flags
2:50:07 PM:   baseRelDir: true
2:50:07 PM:   buildId: 61d3535934fc990007a3e39d
2:50:07 PM:   deployId: 61d3535934fc990007a3e39f
2:50:07 PM: 
2:50:07 PM: ❯ Current directory
2:50:07 PM:   /opt/build/repo
2:50:07 PM: 
2:50:07 PM: ❯ Config file
2:50:07 PM:   /opt/build/repo/netlify.toml
2:50:07 PM: 
2:50:07 PM: ❯ Context
2:50:07 PM:   production
2:50:07 PM: 
2:50:07 PM: ────────────────────────────────────────────────────────────────
2:50:07 PM:   1. build.command from netlify.toml                            
2:50:07 PM: ────────────────────────────────────────────────────────────────
2:50:07 PM: 
2:50:07 PM: $ hugo --gc --minify -b $URL
2:50:07 PM: Start building sites …
2:50:07 PM: hugo v0.91.2-1798BD3F+extended linux/amd64 BuildDate=2021-12-23T15:33:34Z VendorInfo=gohugoio
2:50:07 PM: ERROR 2022/01/03 19:50:07 render of "page" failed: execute of template failed: template: _default/single.html:5:3: executing "_default/single.html" at <partial "site_head" .>: error calling partial: "/opt/build/cache/hugo_cache/modules/filecache/modules/pkg/mod/github.com/wowchemy/wowchemy-hugo-modules/wowchemy@v0.0.0-20210601183557-01a25ecaecdd/layouts/partials/site_head.html:195:13": execute of template failed: template: partials/site_head.html:195:13: executing "partials/site_head.html" at <getenv "WC_POST_CSS">: error calling getenv: access denied: "WC_POST_CSS" is not whitelisted in policy "security.funcs.getenv"; the current security configuration is:
2:50:07 PM: [security]
2:50:07 PM:   enableInlineShortcodes = false
2:50:07 PM:   [security.exec]
2:50:07 PM:     allow = ['^dart-sass-embedded$', '^go$', '^npx$', '^postcss$']
2:50:07 PM:     osEnv = ['(?i)^(PATH|PATHEXT|APPDATA|TMP|TEMP|TERM)$']
2:50:07 PM:   [security.funcs]
2:50:07 PM:     getenv = ['^HUGO_']
2:50:07 PM:   [security.http]
2:50:07 PM:     methods = ['(?i)GET|POST']
2:50:07 PM:     urls = ['.*']
2:50:07 PM: ERROR 2022/01/03 19:50:07 render of "page" failed: execute of template failed: template: _default/single.html:5:3: executing "_default/single.html" at <partial "site_head" .>: error calling partial: "/opt/build/cache/hugo_cache/modules/filecache/modules/pkg/mod/github.com/wowchemy/wowchemy-hugo-modules/wowchemy@v0.0.0-20210601183557-01a25ecaecdd/layouts/partials/site_head.html:195:13": execute of template failed: template: partials/site_head.html:195:13: executing "partials/site_head.html" at <getenv "WC_POST_CSS">: error calling getenv: access denied: "WC_POST_CSS" is not whitelisted in policy "security.funcs.getenv"; the current security configuration is:
2:50:07 PM: [security]
2:50:07 PM:   enableInlineShortcodes = false
2:50:07 PM:   [security.exec]
2:50:07 PM:     allow = ['^dart-sass-embedded$', '^go$', '^npx$', '^postcss$']
2:50:07 PM:     osEnv = ['(?i)^(PATH|PATHEXT|APPDATA|TMP|TEMP|TERM)$']
2:50:07 PM:   [security.funcs]
2:50:07 PM:     getenv = ['^HUGO_']
2:50:07 PM:   [security.http]
2:50:07 PM:     methods = ['(?i)GET|POST']
2:50:07 PM:     urls = ['.*']
2:50:07 PM: ERROR 2022/01/03 19:50:07 render of "page" failed: execute of template failed: template: _default/single.html:5:3: executing "_default/single.html" at <partial "site_head" .>: error calling partial: "/opt/build/cache/hugo_cache/modules/filecache/modules/pkg/mod/github.com/wowchemy/wowchemy-hugo-modules/wowchemy@v0.0.0-20210601183557-01a25ecaecdd/layouts/partials/site_head.html:195:13": execute of template failed: template: partials/site_head.html:195:13: executing "partials/site_head.html" at <getenv "WC_POST_CSS">: error calling getenv: access denied: "WC_POST_CSS" is not whitelisted in policy "security.funcs.getenv"; the current security configuration is:
2:50:07 PM: [security]
2:50:07 PM:   enableInlineShortcodes = false
2:50:07 PM:   [security.exec]
2:50:07 PM:     allow = ['^dart-sass-embedded$', '^go$', '^npx$', '^postcss$']
2:50:07 PM:     osEnv = ['(?i)^(PATH|PATHEXT|APPDATA|TMP|TEMP|TERM)$']
2:50:07 PM:   [security.funcs]
2:50:07 PM:     getenv = ['^HUGO_']
2:50:07 PM:   [security.http]
2:50:07 PM:     methods = ['(?i)GET|POST']
2:50:07 PM:     urls = ['.*']
2:50:07 PM: ERROR 2022/01/03 19:50:07 render of "page" failed: execute of template failed: template: _default/single.html:5:3: executing "_default/single.html" at <partial "site_head" .>: error calling partial: "/opt/build/cache/hugo_cache/modules/filecache/modules/pkg/mod/github.com/wowchemy/wowchemy-hugo-modules/wowchemy@v0.0.0-20210601183557-01a25ecaecdd/layouts/partials/site_head.html:195:13": execute of template failed: template: partials/site_head.html:195:13: executing "partials/site_head.html" at <getenv "WC_POST_CSS">: error calling getenv: access denied: "WC_POST_CSS" is not whitelisted in policy "security.funcs.getenv"; the current security configuration is:
2:50:07 PM: [security]
2:50:07 PM:   enableInlineShortcodes = false
2:50:07 PM:   [security.exec]
2:50:07 PM:     allow = ['^dart-sass-embedded$', '^go$', '^npx$', '^postcss$']
2:50:07 PM:     osEnv = ['(?i)^(PATH|PATHEXT|APPDATA|TMP|TEMP|TERM)$']
2:50:07 PM:   [security.funcs]
2:50:07 PM:     getenv = ['^HUGO_']
2:50:07 PM:   [security.http]
2:50:07 PM:     methods = ['(?i)GET|POST']
2:50:07 PM:     urls = ['.*']
2:50:07 PM: Error: Error building site: failed to render pages: render of "page" failed: execute of template failed: template: _default/single.html:5:3: executing "_default/single.html" at <partial "site_head" .>: error calling partial: "/opt/build/cache/hugo_cache/modules/filecache/modules/pkg/mod/github.com/wowchemy/wowchemy-hugo-modules/wowchemy@v0.0.0-20210601183557-01a25ecaecdd/layouts/partials/site_head.html:195:13": execute of template failed: template: partials/site_head.html:195:13: executing "partials/site_head.html" at <getenv "WC_POST_CSS">: error calling getenv: access denied: "WC_POST_CSS" is not whitelisted in policy "security.funcs.getenv"; the current security configuration is:
2:50:07 PM: [security]
2:50:07 PM:   enableInlineShortcodes = false
2:50:07 PM:   [security.exec]
2:50:07 PM:     allow = ['^dart-sass-embedded$', '^go$', '^npx$', '^postcss$']
2:50:07 PM:     osEnv = ['(?i)^(PATH|PATHEXT|APPDATA|TMP|TEMP|TERM)$']
2:50:07 PM:   [security.funcs]
2:50:07 PM:     getenv = ['^HUGO_']
2:50:07 PM:   [security.http]
2:50:07 PM:     methods = ['(?i)GET|POST']
2:50:07 PM:     urls = ['.*']
2:50:07 PM: Total in 629 ms
2:50:07 PM: 
2:50:07 PM: ────────────────────────────────────────────────────────────────
2:50:07 PM:   "build.command" failed                                        
2:50:07 PM: ────────────────────────────────────────────────────────────────
2:50:07 PM: 
2:50:07 PM:   Error message
2:50:07 PM:   Command failed with exit code 255: hugo --gc --minify -b $URL
2:50:07 PM: 
2:50:07 PM:   Error location
2:50:07 PM:   In build.command from netlify.toml:
2:50:07 PM:   hugo --gc --minify -b $URL
2:50:07 PM: 
2:50:07 PM:   Resolved config
2:50:07 PM:   build:
2:50:07 PM:     command: hugo --gc --minify -b $URL
2:50:07 PM:     commandOrigin: config
2:50:07 PM:     environment:
2:50:07 PM:       - HUGO_VERSION
2:50:07 PM:       - HUGO_ENABLEGITINFO
2:50:07 PM:       - HUGO_ENV
2:50:07 PM:     publish: /opt/build/repo/public
2:50:07 PM:     publishOrigin: config
2:50:08 PM: Caching artifacts
2:50:08 PM: Started saving build plugins
2:50:08 PM: Finished saving build plugins
2:50:08 PM: Started saving pip cache
2:50:08 PM: Finished saving pip cache
2:50:08 PM: Started saving emacs cask dependencies
2:50:08 PM: Finished saving emacs cask dependencies
2:50:08 PM: Started saving maven dependencies
2:50:08 PM: Finished saving maven dependencies
2:50:08 PM: Started saving boot dependencies
2:50:08 PM: Finished saving boot dependencies
2:50:08 PM: Started saving rust rustup cache
2:50:08 PM: Finished saving rust rustup cache
2:50:08 PM: Started saving go dependencies
2:50:08 PM: Finished saving go dependencies
2:50:08 PM: Build failed due to a user error: Build script returned non-zero exit code: 2
2:50:08 PM: Creating deploy upload records
2:50:08 PM: Failing build: Failed to build site
2:50:08 PM: Failed during stage 'building site': Build script returned non-zero exit code: 2
2:50:09 PM: Finished processing build request in 21.978541552s

champagnealexandre · January 3, 2022, 8:02pm

Okay seems like the problem was again due to Hugo. They introduced breaking changes in the latest update (see this post for a quick explanation).

Adding the following lines to config.yaml fixes the problem:

security:
  funcs:
    getenv:
      - ^HUGO_
      - ^WC_

My first commit built successfully on Netlify — however this problem happened intermittently for me. Hope this is indeed fixed!

Topic		Replies	Views
Deploy stuck in "Building" Support building , deployment	1	59	July 11, 2024
Builds are getting stuck Support building	4	256	September 3, 2023
Deploying site stuck on "Building" state Support building	1	370	September 3, 2023
Trouble deploying stuck building Support building , deployment , netlify-newbie	1	964	February 24, 2023
Build gets stuck downloading Node v10 Support building	3	919	September 16, 2020

Builds getting stuck every time

Related topics