Scheduled functions crashes deploys

Support
, ,
1

From 16:51 (past two hours), all of my builds have been unsuccessful. Everything up until step 7 - deploy site looks normal. The feedback from step 7 and on in the build log (build ID: 6228e09649f56216fb093ea0) is:

6:38:40 PM: ────────────────────────────────────────────────────────────────
6:38:40 PM:   7. Deploy site                                                
6:38:40 PM: ────────────────────────────────────────────────────────────────
6:38:40 PM: ​
6:38:40 PM: Starting to deploy site from 'out'
6:38:44 PM: Creating deploy upload records
6:38:40 PM: Creating deploy tree 
6:38:41 PM: ​
6:38:41 PM: ────────────────────────────────────────────────────────────────
6:38:41 PM:   Internal error during "Deploy site"                           
6:38:41 PM: ────────────────────────────────────────────────────────────────
6:38:44 PM: Failed during stage 'building site': Build script returned non-zero exit code: 4 (https://ntl.fyi/exit-code-4)
6:38:41 PM: ​
6:38:41 PM:   Error message
6:38:41 PM:   Error: Deploy did not succeed: Failed to execute deploy: [PUT /sites/{site_id}/deploys/{deploy_id}][500] updateSiteDeploy default  &{Code:0 Message:}
6:38:41 PM: ​
6:38:41 PM:   Error location
6:38:41 PM:   During Deploy site
6:38:41 PM:       at handleDeployError (file:///opt/buildhome/node-deps/node_modules/@netlify/build/src/plugins_core/deploy/buildbot_client.js:87:18)
6:38:41 PM:       at deploySiteWithBuildbotClient (file:///opt/buildhome/node-deps/node_modules/@netlify/build/src/plugins_core/deploy/buildbot_client.js:68:12)
6:38:41 PM:       at runMicrotasks (<anonymous>)
6:38:41 PM:       at processTicksAndRejections (node:internal/process/task_queues:96:5)
6:38:41 PM:       at async coreStep (file:///opt/buildhome/node-deps/node_modules/@netlify/build/src/plugins_core/deploy/index.js:45:5)
6:38:41 PM:       at async fireCoreStep (file:///opt/buildhome/node-deps/node_modules/@netlify/build/src/steps/core_step.js:39:9)
6:38:41 PM:       at async tFireStep (file:///opt/buildhome/node-deps/node_modules/@netlify/build/src/time/main.js:20:59)
6:38:41 PM:       at async runStep (file:///opt/buildhome/node-deps/node_modules/@netlify/build/src/steps/run_step.js:88:7)
6:38:41 PM:       at async pReduce.index (file:///opt/buildhome/node-deps/node_modules/@netlify/build/src/steps/run_steps.js:91:11)
6:38:41 PM:       at async runSteps (file:///opt/buildhome/node-deps/node_modules/@netlify/build/src/steps/run_steps.js:51:7)
6:38:41 PM: ​
6:38:41 PM:   Resolved config
6:38:41 PM:   build:
6:38:41 PM:     command: CI= npm run build && npm run export
6:38:41 PM:     commandOrigin: config
6:38:41 PM:     environment:
6:38:41 PM:       - ADMIN_auth_provider_x509_cert_url
6:38:41 PM:       - ADMIN_auth_uri
6:38:41 PM:       - ADMIN_client_email
6:38:41 PM:       - ADMIN_client_id
6:38:41 PM:       - ADMIN_client_x509_cert_url
6:38:41 PM:       - ADMIN_private_key
6:38:41 PM:       - ADMIN_private_key_id
6:38:41 PM:       - ADMIN_project_id
6:38:41 PM:       - ADMIN_token_uri
6:38:41 PM:       - ADMIN_type
6:38:41 PM:       - DISCORD_CLIENT_ID
6:38:41 PM:       - DISCORD_SECRET
6:38:41 PM:       - FIREBASE_apiKey
6:38:41 PM:       - FIREBASE_appId
6:38:41 PM:       - FIREBASE_authDomain
6:38:41 PM:       - FIREBASE_measurementId
6:38:41 PM:       - FIREBASE_messagingSenderId
6:38:41 PM:       - FIREBASE_projectId
6:38:41 PM:       - FIREBASE_storageBucket
6:38:41 PM:       - ONEGRAPH_AUTHLIFY_TOKEN
6:38:41 PM:       - SENTRY_AUTH_TOKEN
6:38:41 PM:       - SENTRY_ORG
6:38:41 PM:       - SENTRY_PROJECT
6:38:41 PM:       - NETLIFY_NEXT_PLUGIN_SKIP
6:38:41 PM:       - NODE_VERSION
6:38:41 PM:     publish: /opt/build/repo/out
6:38:41 PM:     publishOrigin: config
6:38:41 PM:   functions:
6:38:41 PM:     '*':
6:38:41 PM:       external_node_modules:
6:38:41 PM:         - firebase-admin
6:38:41 PM:         - node-fetch
6:38:41 PM:         - axios
6:38:41 PM:         - date-fns
6:38:41 PM:       node_bundler: esbuild
6:38:41 PM:     triggerAutomations:
6:38:41 PM:       schedule: '@hourly'
6:38:41 PM:   functionsDirectory: /opt/build/repo/functions
6:38:41 PM:   plugins:
6:38:41 PM:     - inputs: {}
6:38:41 PM:       origin: ui
6:38:41 PM:       package: '@netlify/plugin-nextjs'
6:38:41 PM:     - inputs: {}
6:38:41 PM:       origin: config
6:38:41 PM:       package: '@sentry/netlify-build-plugin'
6:38:41 PM:   redirects:
6:38:41 PM:     - from: /api/*
      status: 200
      to: /.netlify/functions/:splat
  redirectsOrigin: config
Caching artifacts
6:38:41 PM: Started saving node modules
6:38:41 PM: Finished saving node modules
6:38:41 PM: Started saving build plugins
6:38:41 PM: Finished saving build plugins
6:38:41 PM: Started saving pip cache
6:38:41 PM: Finished saving pip cache
6:38:41 PM: Started saving emacs cask dependencies
6:38:41 PM: Finished saving emacs cask dependencies
6:38:41 PM: Started saving maven dependencies
6:38:41 PM: Finished saving maven dependencies
6:38:41 PM: Started saving boot dependencies
6:38:41 PM: Finished saving boot dependencies
6:38:41 PM: Started saving rust rustup cache
6:38:41 PM: Finished saving rust rustup cache
6:38:41 PM: Started saving go dependencies
6:38:41 PM: Finished saving go dependencies
6:38:44 PM: Build failed due to an internal system error: Build script returned non-zero exit code: 4
6:38:44 PM: Failing build: Failed to build site
6:38:44 PM: Finished processing build request in 4m36.496725047s

At first I assumed it was related to Netlify Status - Increased errors and latency affecting multiple services but that now states resolved, so maybe not.

1 Like
2

Similar issue here. Also thought it was related to the latest incident but attempts to build continue to error.

CleanShot 2022-03-09 at 09.50.19
CleanShot 2022-03-09 at 09.50.191920×1274 206 KB

3

Builds also continue to fail for me after the incident has been resolved.


────────────────────────────────────────────────────────────────
11:10:39 AM:   Internal error during "Deploy site"                           
11:10:39 AM: ────────────────────────────────────────────────────────────────
11:10:39 AM: ​
11:10:39 AM:   Error message
11:10:39 AM:   Error: Deploy did not succeed: Failed to execute deploy: [PUT /sites/{site_id}/deploys/{deploy_id}][500] updateSiteDeploy default  &{Code:0 Message:}
11:10:39 AM: ​
11:10:39 AM:   Error location
11:10:39 AM:   During Deploy site
11:10:39 AM:       at handleDeployError (file:///opt/buildhome/node-deps/node_modules/@netlify/build/src/plugins_core/deploy/buildbot_client.js:87:18)
11:10:39 AM:       at deploySiteWithBuildbotClient (file:///opt/buildhome/node-deps/node_modules/@netlify/build/src/plugins_core/deploy/buildbot_client.js:68:12)
11:10:39 AM:       at runMicrotasks (<anonymous>)
11:10:40 AM: Creating deploy upload records
11:10:39 AM:       at processTicksAndRejections (node:internal/process/task_queues:96:5)
11:10:39 AM:       at async coreStep (file:///opt/buildhome/node-deps/node_modules/@netlify/build/src/plugins_core/deploy/index.js:45:5)
11:10:39 AM:       at async fireCoreStep (file:///opt/buildhome/node-deps/node_modules/@netlify/build/src/steps/core_step.js:39:9)
11:10:39 AM:       at async tFireStep (file:///opt/buildhome/node-deps/node_modules/@netlify/build/src/time/main.js:20:59)
11:10:39 AM:       at async runStep (file:///opt/buildhome/node-deps/node_modules/@netlify/build/src/steps/run_step.js:88:7)
11:10:39 AM:       at async pReduce.index (file:///opt/buildhome/node-deps/node_modules/@netlify/build/src/steps/run_steps.js:91:11)
11:10:40 AM: Failed during stage 'building site': Build script returned non-zero exit code: 4 (https://ntl.fyi/exit-code-4)
11:10:39 AM:       at async runSteps (file:///opt/buildhome/node-deps/node_modules/@netlify/build/src/steps/run_steps.js:51:7)
11:10:39 AM: ​
11:10:39 AM:   Resolved config
11:10:39 AM:   build:
11:10:39 AM:     environment:
11:10:39 AM:       - DATABASE_URL
11:10:39 AM:     publish: /opt/build/repo/public
11:10:39 AM:     publishOrigin: config
11:10:39 AM:   functionsDirectory: /opt/build/repo/my_functions
11:10:39 AM:   redirects:
11:10:39 AM:     - from: /wiggle
11:10:39 AM:       status: 200
11:10:39 AM:       to: <redacted>
11:10:39 AM:   redirectsOrigin: config
4

Seems to be an issue if you have scheduled functinos

From Netlifystatus.com

Failing builds

Investigating - We are investigating reports of builds failing on sites that use scheduled functions. To mitigate, please temporarily disable or remove the scheduled functions to continue building.
Mar 9, 19:05 UTC

5

Hey folks, thanks for reaching out here.

We have opened an incident and updated our status page to reflect this. For updates on what is happening, please follow along here: https://www.netlifystatus.com/

6

Hey folks! More analysis completed and status page has been updated, but here’s the details you need to know:

  • only deploys using scheduled functions should be impacted since 14:18 UTC
  • older deploys using scheduled functions are still working as expected
  • new deploys can be successfully completed in case you remove the scheduled functions (which are a beta feature) from your code
  • team is working on both a fix and an easier way to disable the feature without code changes to make this easier to work around.
1 Like
7

Hi, @hillary redirect me from a new topic but the issue we’re having does not seem to be related to this at all.
Our builds are not failing and out deploys are stating as completed, there are no schedules. Still our Branch subdomain deploys are a blank page.

8

heya @silentbob - then this is probably a different issue. I’ll work with you in the other thread on it shortly!

1 Like
9

Just to close the loop here:

  • scheduled functions are working again at the moment; you should not need to remove them.
  • in case you do get that same error during upload: 6:38:41 PM: Error: Deploy did not succeed: Failed to execute deploy: [PUT /sites/{site_id}/deploys/{deploy_id}][500] updateSiteDeploy default &{Code:0 Message:} - it is possible they broke again - please let us know here and we can clear it up again.
  • we have a permanent fix written and in test now that we will deploy in the next few days to ensure that things work reliably for scheduled functions users going forward.
2 Likes