Home
Support Forums

Environment Vars Feature requests / Filter env vars to include in Lambda environment parameter

Feature request
Allow a Netlify function to have an override configuration that specifies which env vars to include in the lambda environment parameter when Netlify creates the Lambda on AWS. If there are >4096 env vars for the entire site but only 1KB needed for a function, the function configuration can select just those env vars that make up the 1KB and thus the Lambda deploy would not fail and thus the whole site deploy would not fail.

Original issue that prompted this feature idea
We recently ran into the 4KB limitation of environment variables in the Lambda functions. I’ve written about it in this support thread:

Build Fails – Invalid AWS Lambda parameters? - #4 by rob.

Why is this bad?
The entire site deployment was broken because of this issue and we had no warning of the impending problem. We had no idea how close we were to this limit before-hand to proactively manage it. Since the total size of ALL site-wide env vars is counted and we had exceeded the 4096 size, we were left with no option but to eliminate/reduce env vars despite that not being the best design decision for the app and despite the fact that many were not even referenced in the Lambda functions.

Caveat
We rely upon a /functions/lib folder for shared local util libraries that we write and use across many of our netlify functions. At this time, I believe this only works if the netlify functions live in the root of the functions folder. If we use the structure of putting a netlify function in it’s own directory inside of the functions folder then the the lib folder is no longer accessible. (I may be remembering this wrong). If that’s the case, then placement of a function’s env vars override could need to not rely upon the folder type configuration for a function. Instead it may be best to put this into the netlify.toml somewhere.

2 Likes

This is a great feature request. I think the minimum viable mitigation is to tell you the problem, since we can detect that your env vars are too big when the deploy fails - but as you point out, you might not be aware that there is such a limit, or that you are hitting it - at least we could inform you in the build logs!

Your suggestion is more elegant but also much more intensive to implement.

Nonetheless, I don’t write the code, so I will suggest both and we’ll follow up here if either is implemented. Thanks so much for taking the time to suggest this so thoughtfully!

My pleasure and if any part of it’s open source I’d be happen to take a crack at it.

Thanks for the quick reply.

thanks for the offer to help, rob! Those two repo’s aren’t public, but I’ll mention your kind offer in the issue.

We’ll update this post as soon as we have something new!

It would be great if the input fields for environment variables would be bigger (or could expand), especially the value fields. We use the Netlify API to create them, but the way we use it in combination with GatsbyJS, we need to create single values that are JSON strings that can become quite long. Editing existing environment values for apps is something we do surprisingly often, and it would be very helpful if the web interface would allow us to do this more easily instead of us having to scroll inside a tiny input field or having to open up a separate text editor to copy paste it to and fro just to be able to see more text at once.

I am facing the same problem. How do I hide environment variables from the functions deployment step? I cannot deploy my functions, because the variables are too large, but need to be available during the build step in order to be serialized into the lambda function source.

Variables declared in the netlify.toml file are NOT sent to / used by functions, so that may be a way around this for you, Andreas.

1 Like

That’s an idea except then both the encrypted file and the encryption key are in the same repository. Then we might as well store all of the secret values in the repo which is what we don’t want. So it gets around the one problem by creating another. Can’t think of a way to adjust this to make it work.

I guess that you have a couple other options:

  • build two sites - one with the functions, one without. Maybe the functions don’t need so much to build on their own that they overrun the AWS limits?
  • build or pay for a secrets service that you can query during build. Then you need only one token set in our UI, to access that service.

We are talking a bit internally about the right way to productize a solution, and I’ve added this thread to the ones to notify in case we do come up with a better, native solution. But as a pragmatist, I’m not going to suggest that what we might build tomorrow is a solution to the problem you see today.

1 Like

Right we have built a secret service and put the key into environment inside of the build. But we don’t really like that fact that the data is being sent to AWS. So it works, but there just needs to be an external place.

We found another problem. We can’t use the “netlify” npm module in production. It’s pulling in typescript and @typescript-eslint because of @netlify-zip-it-and-ship-it. It should only do those require if I am actually doing a zip-it-and-ship it, but it’s included here:
Here is the stack

2020-09-23T15:57:39.757703Z Error: Cannot find module 'typescript' A 
2020-09-23T15:57:39.757751Z Require stack: A 
2020-09-23T15:57:39.757767Z - /root/run/node_modules/@typescript-eslint/typescript-estree/dist/parser.js A 
2020-09-23T15:57:39.757811Z - /root/run/node_modules/detective-typescript/index.js A 
2020-09-23T15:57:39.757821Z - /root/run/node_modules/precinct/index.js A 
2020-09-23T15:57:39.757832Z - /root/run/node_modules/@netlify/zip-it-and-ship-it/src/dependencies.js A 
2020-09-23T15:57:39.757840Z - /root/run/node_modules/@netlify/zip-it-and-ship-it/src/main.js A 
2020-09-23T15:57:39.757850Z - /root/run/node_modules/netlify/src/deploy/hash-fns.js A 
2020-09-23T15:57:39.757858Z - /root/run/node_modules/netlify/src/deploy/index.js A 
2020-09-23T15:57:39.757867Z - /root/run/node_modules/netlify/src/index.js A 
2020-09-23T15:57:39.757878Z - /root/run/server/run/status.js A 
2020-09-23T15:57:39.757886Z - /root/run/server/routes/index.js A 
2020-09-23T15:57:39.757895Z - /root/run/server/app.js A 
2020-09-23T15:57:39.757907Z     at Function.Module._resolveFilename (internal/modules/cjs/loader.js:965:15) A 
2020-09-23T15:57:39.757917Z     at Module.Hook._require.Module.require (/root/run/node_modules/require-in-the-middle/index.js:61:29) A 
2020-09-23T15:57:39.757926Z     at require (internal/modules/cjs/helpers.js:72:18) A 
2020-09-23T15:57:39.757935Z     at Object.<anonymous> (/root/run/node_modules/@typescript-eslint/typescript-estree/src/parser.ts:5:1) A 

We’ll need to develop a separate api client to talk to the api.
Thanks
Andy

Since I can’t promise any changes to that functionality, let’s try to get you something working: what happens if you import the typescript module explicitly?

We have it working, but we cannot use the netlify module. Typescript will not be available on the deployment system.

Hi @richtera, since the netlify module is open source, would you be willing to file an issue on its repository here: https://github.com/netlify/js-client/issues?

I’m facing the same issue and like the proposal. I also understand it might be quite a heavy feature to implement. @fool, you were mentioning that variables declared in netlify.toml are not sent to functions. Would a middle ground between what we have now and what @rob suggest be to let us provide environment variables for the build and the functions separately? If that was available in the UI, we could delay having to sign up for a secrets service.

Hey folks - just fyi that we aren’t ignoring you but having some in depth conversations around what we might build in case we can come to a conclusion to inform you. No promises - we could decide we can’t decide yet based on other circumstances (there are a lot of dependencies between our handling of this and the future of build plugins and functions as a whole) - but we’re trying to get you the best answer we can :slight_smile:

More to come!

1 Like

Great, thanks for the update @fool .

As info, I tried to solve my use case with the the inline-functions-env plugin and a local env file but ran into issues with it so it also didn’t solve it.

Just checking in to see if there has been any progress on it this year?

howdy rob, thanks for checking in. i am trying trying to get some updated information to share with you.

Hi rob,

Unfortunately, nothing has still changed in this area.