Build Fails – Invalid AWS Lambda parameters?

[patrick.schneider]

Getting this error while pushing a new deploy. Never had this issue before and I am really perplexed about the error message.

What the console says:

7:32:23 AM: Function Dir: /opt/build/repo/src/functions
7:32:23 AM: TempDir: /tmp/zisi-5da6aad657a7ce9601ac89f4
7:32:24 AM: Prepping functions with zip-it-and-ship-it 0.3.1
7:34:06 AM: [ { path: '/tmp/zisi-5da6aad657a7ce9601ac89f4/create-user-ad.zip',
7:34:06 AM:     runtime: 'js' },
7:34:06 AM:   { path:
7:34:06 AM:      '/tmp/zisi-5da6aad657a7ce9601ac89f4/create-user-message-respond.zip',
7:34:06 AM:     runtime: 'js' },
7:34:06 AM:   { path: '/tmp/zisi-5da6aad657a7ce9601ac89f4/create-user-message.zip',
7:34:06 AM:     runtime: 'js' },
7:34:06 AM:   { path: '/tmp/zisi-5da6aad657a7ce9601ac89f4/fetch-ad.zip',
7:34:06 AM:     runtime: 'js' },
7:34:06 AM:   { path: '/tmp/zisi-5da6aad657a7ce9601ac89f4/fetch-ads.zip',
7:34:06 AM:     runtime: 'js' },
7:34:06 AM:   { path: '/tmp/zisi-5da6aad657a7ce9601ac89f4/fetch-category-ads.zip',
7:34:06 AM:     runtime: 'js' },
7:34:06 AM:   { path:
7:34:06 AM:      '/tmp/zisi-5da6aad657a7ce9601ac89f4/fetch-conversation-messages.zip',
7:34:06 AM:     runtime: 'js' },
7:34:06 AM:   { path: '/tmp/zisi-5da6aad657a7ce9601ac89f4/fetch-home-ads.zip',
7:34:06 AM:     runtime: 'js' },
7:34:06 AM:   { path: '/tmp/zisi-5da6aad657a7ce9601ac89f4/fetch-user-ads.zip',
7:34:06 AM:     runtime: 'js' },
7:34:06 AM:   { path:
7:34:06 AM:      '/tmp/zisi-5da6aad657a7ce9601ac89f4/fetch-user-conversation.zip',
7:34:06 AM:     runtime: 'js' },
7:34:06 AM:   { path:
7:34:06 AM:      '/tmp/zisi-5da6aad657a7ce9601ac89f4/fetch-user-conversations.zip',
7:34:06 AM:     runtime: 'js' },
7:34:06 AM:   { path:
7:34:06 AM:      '/tmp/zisi-5da6aad657a7ce9601ac89f4/fetch-user-notifications.zip',
7:34:06 AM:     runtime: 'js' },
7:34:06 AM:   { path: '/tmp/zisi-5da6aad657a7ce9601ac89f4/fetch-user.zip',
7:34:06 AM:     runtime: 'js' },
7:34:06 AM:   { path: '/tmp/zisi-5da6aad657a7ce9601ac89f4/logout.zip',
7:34:06 AM:     runtime: 'js' },
7:34:06 AM:   { path: '/tmp/zisi-5da6aad657a7ce9601ac89f4/search-ads.zip',
7:34:06 AM:     runtime: 'js' },
7:34:06 AM:   { path: '/tmp/zisi-5da6aad657a7ce9601ac89f4/token.zip',
7:34:06 AM:     runtime: 'js' },
7:34:06 AM:   { path:
7:34:06 AM:      '/tmp/zisi-5da6aad657a7ce9601ac89f4/update-user-notifications.zip',
7:34:06 AM:     runtime: 'js' } ]
7:34:06 AM: Prepping functions complete
7:34:06 AM: Caching artifacts
7:34:06 AM: Started saving node modules
7:34:06 AM: Finished saving node modules
7:34:06 AM: Started saving yarn cache
7:34:06 AM: Finished saving yarn cache
7:34:06 AM: Started saving pip cache
7:34:06 AM: Finished saving pip cache
7:34:06 AM: Started saving emacs cask dependencies
7:34:06 AM: Finished saving emacs cask dependencies
7:34:06 AM: Started saving maven dependencies
7:34:06 AM: Finished saving maven dependencies
7:34:06 AM: Started saving boot dependencies
7:34:06 AM: Finished saving boot dependencies
7:34:06 AM: Started saving go dependencies
7:34:06 AM: Finished saving go dependencies
7:34:11 AM: Build script success
7:34:11 AM: Starting to deploy site from 'dist'
7:34:11 AM: Creating deploy tree 
7:34:11 AM: 0 new files to upload
7:34:11 AM: 17 new functions to upload
7:36:16 AM: Failed to upload file: &{Name:create-user-message-respond Sum:18066c9d6f6c2b03ed6a5266bcb83ad57e228eb48dfdd991aef36080ce52f3f3 Runtime:js Size:<nil> Path: Buffer:0xc0001a11a0}
7:36:16 AM: Failed to upload file: &{Name:fetch-home-ads Sum:0268001bba0330f0751469e0fe0e03aa25ca76591535a939904248cce6c7b6a8 Runtime:js Size:<nil> Path: Buffer:0xc0001a17d0}
7:36:22 AM: Failed to upload file: &{Name:fetch-user-ads Sum:25162720a7f9c11785786e91a8f21a160039a4e02bcce6a68c0bfde4f38a4ea0 Runtime:js Size:<nil> Path: Buffer:0xc0001a1860}
7:36:23 AM: Failed to upload file: &{Name:fetch-category-ads Sum:2eaf489b9504fa2974774672a7084ae27acade8a215b30769e0299e42a4a1b57 Runtime:js Size:<nil> Path: Buffer:0xc0001a1740}
7:36:27 AM: Failed to upload file: &{Name:fetch-conversation-messages Sum:50622dd3a5745daefa49a153ee91ceef4ac3dda115f49a4673fccf8cd69e47ca Runtime:js Size:<nil> Path: Buffer:0xc0001a1410}
7:36:28 AM: Failed to upload file: &{Name:fetch-ads Sum:a5b1f86dfefa4fb02608043d4d0ec8fe537426f6c48e3e28e1ffdc1ba7beb4b7 Runtime:js Size:<nil> Path: Buffer:0xc0001a13e0}
7:36:31 AM: Failed to upload file: &{Name:create-user-message Sum:5594ae4956a6bb5d73a15de6e9b88d189f82d8cf7bad12419182c9e7cdc4ca11 Runtime:js Size:<nil> Path: Buffer:0xc0001a1080}
7:36:33 AM: Failed to upload file: &{Name:create-user-ad Sum:efcaedf0f9d36ba363dd58abe58c3ae55f80612012727138a9626798a090350b Runtime:js Size:<nil> Path: Buffer:0xc0001a0f90}
7:36:49 AM: Failed to upload file: &{Name:fetch-user Sum:a4c09411a7595f45aba084fd37a17fe1788ae6957295ef7e8fca160a1c98b5ac Runtime:js Size:<nil> Path: Buffer:0xc0001a19e0}
7:36:59 AM: Failed to upload file: &{Name:fetch-ad Sum:0c4dd28236013d3eac8b35761ede21a68251a62a4f72bc2babc97cf08cd812ce Runtime:js Size:<nil> Path: Buffer:0xc0001a16b0}
7:36:59 AM: Failing build: Failed to deploy site
7:36:59 AM: failed during stage 'deploying site': Failed to execute deploy: Upload cancelled: fetch-ad
7:36:59 AM: Finished processing build request in 6m58.421920962s

Then I tried to deploy with the Netlify CLI, there the console says:

→ netlify deploy --dir=public --functions=./src/functions
Deploy path:        /Git/FOOBAR/public
Functions path:     /Git/FOOBAR/src/functions
Configuration path: /Git/FOOBAR/netlify.toml
Deploying to draft URL...
✔ Finished hashing 33 files and 17 functions
✔ CDN requesting 0 files and 17 functions
▮ (14/17) Uploading search-ads... ›   Warning: JSONHTTPError: undefined 422
 ›   Warning: 
 ›   {
 ›      "name": "JSONHTTPError",
 ›      "status": 422,
 ›      "json": {
 ›        "errors": {
 ›          "function_validation": "Invalid AWS Lambda parameters. Reserved keys used in this request",
 ›          "code": "InvalidParameterValueException"
 ›        }
 ›      }
 ›   }
 ›
JSONHTTPError: Unprocessable Entity
    at NetlifyAPI.uploadDeployFunction (~/.config/yarn/global/node_modules/netlify/src/open-api/index.js:162:15)
    at runMicrotasks (<anonymous>)

I checked my used env variables and double-checked with https://docs.aws.amazon.com/lambda/latest/dg/lambda-environment-variables.html – nothing identical here.

I retried the deploy with a clean cache – didn’t work.

I’ve removed the node_modules and yarn.lock file, rebuild everything locally – no problem.

I checked the size of my environment variables, but the biggest one has 3,136 bytes – so it should be ok.

I even removed all my functions, created the hello-world example with the Netlify-CLI – so it’s a standardized fresh function which must work – but still the same error.

So what might this be, even when a function without node dependencies fails? Thanks in advance for your suggestions!

[patrick.schneider]

Thanks to Chris McCraw from the Netlify Support Team I now know, that the restriction of the maximum size of environment variables concatenated all together (e.g. VAR1=val1,VAR2=val2) should not be bigger than 4,096 bytes.

So problem is solved!

[perry]

thanks for posting the solution! This is super helpful for other users - maybe if you see someone posting about the same issue, you can share your fixes with them

[rob]

@perry We just spent a whole day tracking down a deployment issue that turned out to be caused by this issue. The deployment logs on the netlify site did not shed any light on the real root cause and we only found it after doing a local build and deploy from the netlify-cli.

Once we saw the extra error information in the console output from the local deploy attempt we googled it and the only thing we found mentioning it was this post, so thank you!

That said, this should be documented in Netlify and possibly mentioned in the environment variables blog post @swyx wrote a little while back. We have a lot of environment variables and to get around this limitation was a real effort. We had to reduce the size of variable names and the size of values where possible (build the URLs in code with one key value from the env var instead of just putting the whole URL in the env var for instance). Had we known that we had an upper limit of 4096 bytes for all variables combined we would have utilized a different solution.

Now that we have compressed the sizes of the env var names and eliminated some that were optional, we have gotten our total size down to 3200 bytes which considering we are still actively developing, I suspect only buys us time until this becomes an issue again after adding more env vars. If you have any recommendations for how to work around this when the legitimate need is for more than 4096 bytes worth of env vars, I would love to discuss that with someone.

Maybe putting a counter of how close you are to the limit in the site environment config section would be a good idea but at the least document it on the environment variables docs page as a start. Oh yeah the deploy logs on app.netlify.com should show the real reason the functions fail too.

Some background references from AWS are Using Lambda environment variables - AWS Lambda

Using Lambda environment variables - AWS Lambda

In the second link above is where AWS says the total of all environment variables cannot exceed 4KB.

Thanks for the awesome service!
Rob

EDIT: Added a feature request for allowing functions to specify a subset of environment variables to specify in the Lambda creation rather than using all available.

Environment Vars Feature requests / Filter env vars to include in Lambda environment parameter

Feature request Allow a Netlify function to have an override configuration that specifies which env vars to include in the lambda environment parameter when Netlify creates the Lambda on AWS. If there are >4096 env vars for the entire site but only 1KB needed for a function, the function configuration can select just those env vars that make up the 1KB and thus the Lambda deploy would not fail and thus the whole site deploy would not fail. Original issue that prompted this feature idea We rec…

[perry]

rob, thanks for all of this detailed information - super appreciate it. I’m going to touch base with our docs team so we can take another look at this and at the very least clarify the 4096 byte limit!

[wesbos]

I ran into this today too, was able to rename a few things and squeak in, but this limit seems really low because:

1. We can’t specify which env variables go to functions - half of them were not needed, but they all go along
2. We can’t specify between preview / production env variables, so many of my vars are duplicated with _PREVIEW tacked onto the end.

Would love to see something similar to how Vercel does their env variables.

If anyone wants to know how many chars they have used, (assuming each char is a byte), you can use this code in your console to count:

Array.from(document.querySelectorAll('.floating-labels input')).map(x => x.value).join('').length

[fool]

Thanks for that suggestion and script, Wes! We don’t set the 4k limit - AWS does. Your comments about lack of ability to control what is sent are something we are trying to find a solution to, though!

FYI at the moment, variables set in netlify.toml are NOT sent to functions, if you do want to separate into sent and not sent, but this of course blocks the use of context-specific variables at all via netlify.toml, as you also point out.

[wesbos]

Thanks for the followup! Envionmental variables in netlify.toml will be public facing / in a git repo though, right? I’m just looking at it right now and I could put NODE_VERSION in there, which frees up a little space

And yes the limit is from AWS, and it’s reasonable at 4k if we can control which values are sent to functions / context specific.

[fool]

Well, if the repo is public anyway, yes, that is true.

As far as workarounds, I do understand it’s not awesome, but may unblock someone less savvy than you to use the pattern I mention in this post (specifically the SECOND instance of #3 in the numbered lists). “save a short token, use it to fetch the longer one during build”. You do then have to, well, “deploy” it, during build - it wouldn’t be an environment variable, but embedded in a file or something similar. But, that keeps your token out of the public repo AND out of the variables sent with the function.

[Support Guide] How do I access private repositories in the build environment?

Last reviewed by Netlify Support on December 2022 The link from your repository to our service only authenticates us against the “main” repo that you first configured, and won’t automatically allow access to submodules even if they also belong to you and you’ve granted Netlify access to those repositories. In case your submodule is public, you can change the URI schema for it to start with https://, and it should work without further adjustment (so, in .gitmodules, change git@github.com… to h…

[romain1304]

I spent a total of 6 hours debugging that error before i realize that one my cloud function file was empty… once this file deleted everything worked fine again…
this error was misleading me… i suppose it should be updated…

[hillary]

Hey there, @romain1304 Thank you for sharing this with us! I will share your feedback with the appropriate folks.

[akang10]

I just ran into this issue and solved it by removing some unused environment variables… looks like we had run over the size limit as well. Very misleading error message.

[perry]

totally agree that it is misleading. I promise there is an active intituative to make this better, we just have to get some other work completed first.

[justintemps]

Hi, this looks like an old discussion, but we’ve been having this issue all day and I don’t think the length of env vars have anything to do with it, we’re nowhere near the limit.

Interestingly, this is happening because of content triggered deploys to a statically generated website, but the latest git commit is the same. At one point the code was deploying fine and then it wasn’t, but the code itself didn’t change, HEAD is still where it was when deploys were working.

In some builds, the deploy logs say…

5:41:33 PM:   Deploy did not succeed with HTTP Error 400: [PUT /deploys/{deploy_id}/functions/{name}][400] uploadDeployFunction default  &{Code:400 Message:Failed to create function on AWS Lambda: invalid parameter for lambda creation: Invalid AWS Lambda parameters used in this request.}

In others the deploy logs aren’t available at all…

5:53:38 PM: [ERROR] Deploy logs are currently unavailable. We are working on resolving the issue.

[Melvin]

Hi @justintemps,

This may be due to Node.js 14 deprecation on AWS Lambda. If you’re using Node.js for your site, we may be attempting to deploy your Functions also in that same Node.js version. However, AWS Lambda has now started blocking uploads for that Node.js version, which is what causing this issue.

Could you try to update your Node.js version to 18 or 20?

[justintemps]

Ok, that sounds about right, we’ll try updating that and tell you how it goes.

[justintemps]

Yup! Updating the AWS JS Runtime to Node 18 fixed it. Thank you!