I am trying to use Netlify lambda functions but I need access to certain environmental variables. It does not seem like I have access to those variables that I set in my Netlify.toml, but only have access to the environmental variables set up in the Netlify UI. Is there no way to access variables within the toml with a lambda function?
The function needs to know what type of build (branch-deploy, production, staging, develop, etc.), but because process.env doesn’t have access to the toml environment variables, is there a better way to know what type of build I have when the lambda function fires?
I was stuck in the same issue some time back and unfortunately at the moment there is no way for the lambda to have access to the .toml environment variables.
@perry This seems like a major shortcoming. I know I can use environment variables set in the UI – but that doesn’t answer my use case (or that of the OP) – because I can’t set the build context as an environment variable.
So, I have what seems like an extremely common use case: I have an ecommerce site, and the purchasing is handled via functions. What I want to do is use sandbox Stripe key/secrets for Pull Requests, and use the real, production key/secret in the production build.
Right now, the only way to do this is to base64-decode the context.clientContext.netlify. This feels really hacky, and I feel unsure if I can rely on it going forward.
At the very least the part of your documentation where you say “all these variables are available during your build” should have a BIG notice that says “but are NOT available to your functions”.
Is there an engineering challenge to injecting more contextual information either via process.env or the context.clientContext? Is this a feature that could be fast-tracked? Or at least could we get an assurance that if we start relying on base64-decoding the clientContext.netlify var that it will not change in the future?
Really appreciate any insight. I LOVE LOVE LOVE netlify, this just seems like a really common use case and a big shortcoming that might be easy to remedy. Thanks for listening!
I’d think that means that your deployed function could know it, at build time, and embed the string (or related settings set based on the string) in the code we create during build, and thus no dynamic, runtime determinations would need to be made - all is set before execution.
I can’t speak to engineering challenges on adding things to function environment in our deployment process code, since I don’t work on that codebase, but I can speak to the practical problem for other customers with doing it which is what makes this idea not an obvious slam dunk and why wouldn’t we do it already?! Folks frequently run into AWS’ own limit on environment size (mentioned in the yellow box in this section of the docs: Functions overview | Netlify Docs) - which boils down to “AWS only accepts function deployments with up to 4k worth of environment variables, when appended in a string like VAR1=val1,VAR2=val2, and base64 encoded - more will cause the deploy to fail!” So, there is a definite pressure to preserve space in the environment and that will factor into anything we add: if we start adding things that folks can’t opt out of, we create other problems!
I see you did understand that you can do this based on later statements in your thread, and we can absolutely get our documentation improved to be more explicit on this topic, so I’ll work with the docs team on that! Thanks again for talking through this with us!
@fool Thanks for the reply. Took me a minute to grok what you were saying, but I think I get it. In my function, I could have something like const context = '__CONTEXT__'; and then, I could run something as part of the build command like sed -i -E "s/__CONTEXT__/$CONTEXT/" ./functions/func.js. That would change the code of the function file before you guys ran your zip-it-and-ship-it during the build. Clever, thanks for the tip.
That said, 4 kilobytes seems like a LOT of space for environment variables, and I think if you added just one in there which was context: $CONTEXT I think it would definitely be the right thing to do.
It just seems so glaringly missing now – if someone asked me about Netlify, I would have to say: “everything they do is crazy good, but if you want your pull request deploy previews to use sandbox API keys for stuff like e-commerce, you’ll have to hack that in, it’s not supported.” I can almost hear the sad trombone.
I understand 4Kb is plenty for you. It is not enough for many people to fit an authentication token in there like an SSH key, and I’ve heard from dozens of them and only one of you, so we’ll probably not move on this in the immediate future since you have a good workaround
I’m hitting up against this as well. At least can use the sed approach for now but it does seem like a big oversight not to include CONTEXT in functions. Surely most people will need some way to switch api endpoints etc based on environment?
Also it’s worth pointing out that this is not a ‘good workaround’. It’s ugly. I use/need context in multiple different files within my multiple lambda folders and the sed mutation changes files in git when running build locally.
I could restructure everything so I only have to sed a single file but just feels like I shouldn’t need to do this
This has been insanely complicated and time consuming but I will share the solution I’ve reached for now. For me I have 4 lambda functions each in their own folders that need to make use of the CONTEXT to switch between prod and dev api endpoints - fairly standard use case.
Attempting to use sed to replace CONTEXT with $CONTEXT technically works but it’s messy, when running the script locally I end up with git changes that I need to handle. Also, of course, the linter complains about the constant condition and you have to comment everywhere why this weird string is there so really not a fan of that.
So I looked for better options. One simple approach I tried was using dotenv. In my prebuild script I run printenv > .env to put everything in a file that dotenv can pick up (obviously I could just put the stuff I need but I was trying to keep things simple!). Unfortunately this doesn’t work due to dotenv failing to find the .env file. I guess this is down to zip it and ship it ignoring it as it’s not explicitly imported?
So what I’m doing now is writing the build env vars to a json file in my prebuild script with:
Whoa. Serious work-around. I think I like it though, because that should actually also work for running ntl dev locally if I’m presuming correctly - dev injects .env file vars into the context then runs scripts… so if this works for both local dev based on .env file vars and also on branch / preview deploys, that would be incredible. Appreciate your time and efforts @bristoljon!! Going to poke at writing that little one-liner in js just so it fits the node workflow but that’s awesome!
No worries, yeah it wasn’t what I planned to be doing yesterday! Also to clarify the .env approach failed in the lambdas but would work for the front end code I believe and I did see that they got imported on running netlify dev.
Would prefer dotenv so all the vars can be accessed in process.env as now they are split between there and the json file import but it’s good enough for me for now. I just hope Netlify get round to passing CONTEXT through automatically as would be a lot nicer
Just want to add for others who may read this thread - having separate ENV vars running, for both the site builder (injecting client vars into front-end scripts, e.g.) and for the functions works perfectly well if you just need to split local dev from running-on-Netlify-in-the-cloud “prod”. In my case, I work on a lot of stuff with Stripe and test heavily locally then ship up to the prod branch and everything’s perfect. This works well because I do all the dev myself and doing it locally is fine. I don’t have test branches that are also cloud-deployed etc. So if that matches you and having local vs. prod works, you’re good to go. This conversation / discussion is more rooted around having multiple environments hosted on the cloud running different ENV vars.
–
Jon
EDIT: tl;dr: this whole conversation is for if you’re running deploy-previews or branch-previews and want different ENV vars for those builds. If you just run locally and on ‘production’ / your main site build, ENV vars should be straightforward (Netlify UI for prod env vars, .env file and netlify dev for local )
It’s now been a year since trying Netlify Dev and the almost perfect experience it has given. I want to second everyone’s callout here and point out that this is an experience killer in what Netlify & Netlify Dev attempts to accomplish and from my experience stands for - Make it is easy to deploy static sites and provide awesome deploy previews. The flexibility through netlify.toml and branch deploys is awesome and no other solution like it out there.
When Netlify Dev was announced, it did everything perfectly but read from the TOML. Given Netlify Dev is to mimic what goes on at Netlify, it would be great if the .toml is also read and ENVs also provided from it.
More related to this topic, the same must be applied to the Functions. It seems there are limitations in AWS Lambdas and the reason for not transfering them over is due to it’s 4k limit. I understand some people need more, but not enabling this for the few when the mass can have an awesome experience seems illogical.
Furthermore, their are other alternatives like the Netlify Dev and Netlify Cloud generating a .env file based on the TOML. I believe @ bristoljon attempt to do this on his own and failed as functions don’t read from the same build workspace.
All in all, I do love netlify and use it every day and promoting it all the time. However, being so close to having production-grade experience for functions is a shame. I really hope we get a better answer than there is a 4K limit and here’s a potential hack that’ll work around this. I think we’re all using netlify because we don’t want to hack our own solutions.
Hi, @uptownhr, the workaround that first comes to mind is writing the environment variables into the Node/Go files before the functions are bundled and deployed.
The environment variables are constants known at the time of build and deploy. This means you can write the literal strings into the files which will end up becoming the deployed function(s) during the site build.
You also mentioned not being able to access the environment variables from netlify.toml during local builds. For there there is a workaround as well. These environment variables are processed and available using the CLI command netlify build to build the site locally:
The netlify build command locally is a good way of testing methods to embed the environment variables in the node/Go function source files during the site build process.
I think this thread gets a lot of views without many responses so I just want to add that the concept @luke was describing is now fully available as a Netlify Build Plugin
This thread continues to get plenty of clicks even though it doesn’t get many replies. Configuring ENV vars correctly for multiple environments seems to be something folks continue to want to do.
I took some time to write up two methods for having separate ENV vars per environment (local, branch-deploy, deploy-preview, production). Both work great and allow your Functions and SSG to get the right values for the environment they’re running in.
Hey Joe, I also tried to use the environment variables from [context.branch] in the netlify.toml file. But, netlify only accepts environmental variables from the dashboard settings and not from the netlify.toml file. So, I have figured out a way to overcome this. I hope this helps.
There are more people who have trouble with this.
)
This also a deal-breaker for me and hopefully it will be addressed soon by the otherwise excellent netlify pipeline team.