Netlify lambda deploys started timing out with no relevant code change

Support
, ,
1

For the past couple of days, my builds have been timing out (“Build exceeded maximum allowed runtime”) in the “Packaging Functions from built-lambda directory” step of the deploy.

Until a couple of days ago, this step took about 3 minutes. Now it is timing out after ~20 minutes. There have been no lambda changes in my code that would cause this delay – it sort of just came out of nowhere. Deploying previous states does not resolve this issue.

The hanging step prints “Packaging Functions from built-lambda directory”, then lists all of the JS bundles for my lambdas (e.g. “subscribe.js”). Then it just stops and hangs for 20 minutes until it times out. It formerly completed this step in ~3 minutes.

Furthermore, running yarn build executes locally with no issues. This step also completes in the deploy job on Netlify.

My site is simple-subscriptions.netlify.app. Here is an example deploy: Netlify App

Can you help me understand what’s going on here and how to prevent the hanging / timeout?

2 Likes
2

Hi @gwitwer,

This appears to be a known issue at the moment as several additional customers are also seeing it. The developers have already been notified of this and we’d let you know how things develop.

1 Like
4

@hrishikesh any update on this? We currently cannot fix a production bug, rollback, or deploy any new features.

5

I have a client that is suffering from this issue as well, the page is being rebuilt many times each day and now most of the builds fail or use about 15-30 minutes to build, up from about 3-6 minutes. Would love to be updated if there’s any developments.

6

I seem to be facing the same issue - Build exceeding allowed time when it was deploying fine until now. @gwitwer did you find a workaround? I’m stuck trying to push a hotfix.

@hrishikesh, is there a way to update the frontend files without touching the lambdas? I’m ok to disable the lambdas but the last thing I want is that they’re gone altogether from prod.

7

I am facing the same problem right now. I have not modified the code of my lambda functions however they have started to exceed the time limit to be packed. Any update about it?

8

Is there a way to opt-in some accounts on a specific version of @netlify/build? That would be handy in such cases.

9

Hi all! We can’t reproduce the issue on our side, but we can check if the recent upgrade to the build system has caused it. Can you run the following CLI command?

netlify api createSiteBuild --data='{ "site_id": "SITE_ID", "body": { "image": "8258ffacd94158bc8a8cc9efea33d47497e80cbd-focal" }}'

Where the SITE_ID is the API ID from the site’s setting page (looks something like 5a89dc85-2fab-439c-86ed-cfcb3b3c7ccc)

If this works and reduces the build time, it helps a lot in identifying what caused the issue. Thank you!

2 Likes
11

Thanks @ivaaan, it did fix it for me. See build log at Netlify App.

12

Hi @tibotiber.

We have reverted the version of Netlify Build for all customers, so you should be able to run builds normally.

Let us know if that works for you.

Thanks for your patience!

2 Likes
13

Hi @eduardoboucas, yes I can confirm it works as per normal now.

Thanks both of you and the team for the quick resolution and rollback. Really appreciate it.

1 Like
14

@tibotiber I’m glad it’s working again!

Sadly, I’m still unable to reproduce the issue myself, which is complicating the resolution. Is there any chance you could share a simplified reproduction case with us, or the built functions you’re using on your site along with your package.json and netlify.toml?

I’m sorry to be asking this, and I completely understand if you’d rather not share your files, but there’s something about your specific use case that I’m currently unable to replicate in my test scenarios.

If you’re comfortable doing this, I can reach out via direct message.

Thanks!

1 Like
17

The issue appears to be fixed. Thank you!

18

@gwitwer Thanks for confirming.

Would you be able to share your function and config files, by any chance? That would really help us get to the root cause.

If you’re okay with this, I can reach out via direct message.

Thanks in advance!

1 Like
19

@eduardoboucas Feel free to reach out, I’m fine with sharing.

20

Hi @eduardoboucas, I should be able to share the functions (nothing sensitive or strategic) and config files. Feel free to DM me. Hopefully this can help with a resolution.

Besides that, do you have any idea what might go wrong, can we try to identify commonalities between our setups? For instance, here are some specifics that others might confirm / disaffirm:

  1. i’m using netlify-lambda
  2. below are my dependencies
  "dependencies": {
    "@sentry/node": "^6.4.1",
    "@sentry/tracing": "^6.4.1",
    "auth0": "^2.20.0",
    "axios": "^0.19.0",
    "change-case": "^3.1.0",
    "firebase-admin": "^8.2.0",
    "fs-extra": "^8.1.0",
    "gltf-pipeline": "^2.1.3",
    "graphql": "^14.5.8",
    "graphql-request": "^1.8.2",
    "graphql-tag": "^2.10.1",
    "moment": "^2.24.0",
    "postmark": "^2.5.0",
    "ramda": "^0.27.0",
    "stripe": "^8.129.0",
    "twilio": "^3.39.1",
    "urlencode": "^1.1.0",
    "winston": "^3.3.3"
  },
  "devDependencies": {
    "@babel/cli": "^7.4.4",
    "@babel/core": "^7.4.5",
    "@babel/plugin-transform-async-to-generator": "^7.4.4",
    "@babel/preset-env": "^7.4.5",
    "babel-watch": "^7.0.0",
    "eslint": "^5.12.0",
    "eslint-config-standard": "^12.0.0",
    "eslint-plugin-import": "^2.14.0",
    "eslint-plugin-node": "^7.0.1",
    "eslint-plugin-promise": "^4.0.1",
    "eslint-plugin-standard": "^4.0.0",
    "lint-staged": "^8.2.1",
    "netlify-lambda": "^1.6.3",
    "prettier-eslint": "^9.0.0",
    "prettier-eslint-cli": "^4.7.1"
  }
  1. below are my babel & webpack config
{
  "presets": [
    [
      "@babel/preset-env",
      {
        "targets": {
          "node": "12"
        }
      }
    ]
  ],
  "plugins": ["@babel/plugin-transform-async-to-generator"]
}

const webpack = require('webpack')

module.exports = {
  optimization: { minimize: false },
  plugins: [
    new webpack.DefinePlugin({
      'global.GENTLY': false
    })
  ]
}
21

hi there, thank you all for your patience.

has anything changed with the slowness? we have made some changes and implemented a fix. Do let us know if anything has improved (or not) today.