Feature request
Allow a Netlify function to have an override configuration that specifies which env vars to include in the lambda environment parameter when Netlify creates the Lambda on AWS. If there are >4096 env vars for the entire site but only 1KB needed for a function, the function configuration can select just those env vars that make up the 1KB and thus the Lambda deploy would not fail and thus the whole site deploy would not fail.
Original issue that prompted this feature idea
We recently ran into the 4KB limitation of environment variables in the Lambda functions. I’ve written about it in this support thread:
Why is this bad?
The entire site deployment was broken because of this issue and we had no warning of the impending problem. We had no idea how close we were to this limit before-hand to proactively manage it. Since the total size of ALL site-wide env vars is counted and we had exceeded the 4096 size, we were left with no option but to eliminate/reduce env vars despite that not being the best design decision for the app and despite the fact that many were not even referenced in the Lambda functions.
Caveat
We rely upon a /functions/lib folder for shared local util libraries that we write and use across many of our netlify functions. At this time, I believe this only works if the netlify functions live in the root of the functions folder. If we use the structure of putting a netlify function in it’s own directory inside of the functions folder then the the lib folder is no longer accessible. (I may be remembering this wrong). If that’s the case, then placement of a function’s env vars override could need to not rely upon the folder type configuration for a function. Instead it may be best to put this into the netlify.toml somewhere.
This is a great feature request. I think the minimum viable mitigation is to tell you the problem, since we can detect that your env vars are too big when the deploy fails - but as you point out, you might not be aware that there is such a limit, or that you are hitting it - at least we could inform you in the build logs!
Your suggestion is more elegant but also much more intensive to implement.
Nonetheless, I don’t write the code, so I will suggest both and we’ll follow up here if either is implemented. Thanks so much for taking the time to suggest this so thoughtfully!
It would be great if the input fields for environment variables would be bigger (or could expand), especially the value fields. We use the Netlify API to create them, but the way we use it in combination with GatsbyJS, we need to create single values that are JSON strings that can become quite long. Editing existing environment values for apps is something we do surprisingly often, and it would be very helpful if the web interface would allow us to do this more easily instead of us having to scroll inside a tiny input field or having to open up a separate text editor to copy paste it to and fro just to be able to see more text at once.
I am facing the same problem. How do I hide environment variables from the functions deployment step? I cannot deploy my functions, because the variables are too large, but need to be available during the build step in order to be serialized into the lambda function source.
That’s an idea except then both the encrypted file and the encryption key are in the same repository. Then we might as well store all of the secret values in the repo which is what we don’t want. So it gets around the one problem by creating another. Can’t think of a way to adjust this to make it work.
build two sites - one with the functions, one without. Maybe the functions don’t need so much to build on their own that they overrun the AWS limits?
build or pay for a secrets service that you can query during build. Then you need only one token set in our UI, to access that service.
We are talking a bit internally about the right way to productize a solution, and I’ve added this thread to the ones to notify in case we do come up with a better, native solution. But as a pragmatist, I’m not going to suggest that what we might build tomorrow is a solution to the problem you see today.
Right we have built a secret service and put the key into environment inside of the build. But we don’t really like that fact that the data is being sent to AWS. So it works, but there just needs to be an external place.
We found another problem. We can’t use the “netlify” npm module in production. It’s pulling in typescript and @typescript-eslint because of @netlify-zip-it-and-ship-it. It should only do those require if I am actually doing a zip-it-and-ship it, but it’s included here:
Here is the stack
2020-09-23T15:57:39.757703Z Error: Cannot find module 'typescript' A
2020-09-23T15:57:39.757751Z Require stack: A
2020-09-23T15:57:39.757767Z - /root/run/node_modules/@typescript-eslint/typescript-estree/dist/parser.js A
2020-09-23T15:57:39.757811Z - /root/run/node_modules/detective-typescript/index.js A
2020-09-23T15:57:39.757821Z - /root/run/node_modules/precinct/index.js A
2020-09-23T15:57:39.757832Z - /root/run/node_modules/@netlify/zip-it-and-ship-it/src/dependencies.js A
2020-09-23T15:57:39.757840Z - /root/run/node_modules/@netlify/zip-it-and-ship-it/src/main.js A
2020-09-23T15:57:39.757850Z - /root/run/node_modules/netlify/src/deploy/hash-fns.js A
2020-09-23T15:57:39.757858Z - /root/run/node_modules/netlify/src/deploy/index.js A
2020-09-23T15:57:39.757867Z - /root/run/node_modules/netlify/src/index.js A
2020-09-23T15:57:39.757878Z - /root/run/server/run/status.js A
2020-09-23T15:57:39.757886Z - /root/run/server/routes/index.js A
2020-09-23T15:57:39.757895Z - /root/run/server/app.js A
2020-09-23T15:57:39.757907Z at Function.Module._resolveFilename (internal/modules/cjs/loader.js:965:15) A
2020-09-23T15:57:39.757917Z at Module.Hook._require.Module.require (/root/run/node_modules/require-in-the-middle/index.js:61:29) A
2020-09-23T15:57:39.757926Z at require (internal/modules/cjs/helpers.js:72:18) A
2020-09-23T15:57:39.757935Z at Object.<anonymous> (/root/run/node_modules/@typescript-eslint/typescript-estree/src/parser.ts:5:1) A
We’ll need to develop a separate api client to talk to the api.
Thanks
Andy
Since I can’t promise any changes to that functionality, let’s try to get you something working: what happens if you import the typescript module explicitly?
I’m facing the same issue and like the proposal. I also understand it might be quite a heavy feature to implement. @fool, you were mentioning that variables declared in netlify.toml are not sent to functions. Would a middle ground between what we have now and what @rob suggest be to let us provide environment variables for the build and the functions separately? If that was available in the UI, we could delay having to sign up for a secrets service.
Hey folks - just fyi that we aren’t ignoring you but having some in depth conversations around what we might build in case we can come to a conclusion to inform you. No promises - we could decide we can’t decide yet based on other circumstances (there are a lot of dependencies between our handling of this and the future of build plugins and functions as a whole) - but we’re trying to get you the best answer we can
As info, I tried to solve my use case with the the inline-functions-env plugin and a local env file but ran into issues with it so it also didn’t solve it.
