Netlify Functions and Env Variables from netlify.toml

Hi Jared,

In theory, you know the context at build time - it is $CONTEXT as shown here: Build environment variables | Netlify Docs

I’d think that means that your deployed function could know it, at build time, and embed the string (or related settings set based on the string) in the code we create during build, and thus no dynamic, runtime determinations would need to be made - all is set before execution.

I can’t speak to engineering challenges on adding things to function environment in our deployment process code, since I don’t work on that codebase, but I can speak to the practical problem for other customers with doing it which is what makes this idea not an obvious slam dunk and why wouldn’t we do it already?! Folks frequently run into AWS’ own limit on environment size (mentioned in the yellow box in this section of the docs: Functions overview | Netlify Docs) - which boils down to “AWS only accepts function deployments with up to 4k worth of environment variables, when appended in a string like VAR1=val1,VAR2=val2, and base64 encoded - more will cause the deploy to fail!” So, there is a definite pressure to preserve space in the environment and that will factor into anything we add: if we start adding things that folks can’t opt out of, we create other problems!

I see you did understand that you can do this based on later statements in your thread, and we can absolutely get our documentation improved to be more explicit on this topic, so I’ll work with the docs team on that! Thanks again for talking through this with us!

2 Likes

@fool Thanks for the reply. Took me a minute to grok what you were saying, but I think I get it. In my function, I could have something like const context = '__CONTEXT__'; and then, I could run something as part of the build command like sed -i -E "s/__CONTEXT__/$CONTEXT/" ./functions/func.js. That would change the code of the function file before you guys ran your zip-it-and-ship-it during the build. Clever, thanks for the tip.

That said, 4 kilobytes seems like a LOT of space for environment variables, and I think if you added just one in there which was context: $CONTEXT I think it would definitely be the right thing to do.

It just seems so glaringly missing now – if someone asked me about Netlify, I would have to say: “everything they do is crazy good, but if you want your pull request deploy previews to use sandbox API keys for stuff like e-commerce, you’ll have to hack that in, it’s not supported.” I can almost hear the sad trombone.

Anyway, thanks again!

2 Likes

I understand 4Kb is plenty for you. It is not enough for many people to fit an authentication token in there like an SSH key, and I’ve heard from dozens of them and only one of you, so we’ll probably not move on this in the immediate future since you have a good workaround :slight_smile:

I’m hitting up against this as well. At least can use the sed approach for now but it does seem like a big oversight not to include CONTEXT in functions. Surely most people will need some way to switch api endpoints etc based on environment?

1 Like

Also it’s worth pointing out that this is not a ‘good workaround’. It’s ugly. I use/need context in multiple different files within my multiple lambda folders and the sed mutation changes files in git when running build locally.

I could restructure everything so I only have to sed a single file but just feels like I shouldn’t need to do this

“and I’ve heard from dozens of them and only one of you”

This is… so not true… :grinning: There are more people who have trouble with this.

Sorry, the proposed workaround is quite vague… Can you please explain the code a bit better?

// package.json
"scripts": {
  "build": "gatsby build -i -E "s/__CONTEXT__/$CONTEXT/",
}

I am using Gatsby for my projects. Is this the correct way to configure my build command?

What does the “-i -E” mean?

// ./functions/func.js
const context = '__CONTEXT__';

So, if I understand this correctly the context should output:

  • production
  • deploy-preview
  • branch-deploy

Is it possible to set up a custom context?

Peace and Love, :v:
Anton

This has been insanely complicated and time consuming but I will share the solution I’ve reached for now. For me I have 4 lambda functions each in their own folders that need to make use of the CONTEXT to switch between prod and dev api endpoints - fairly standard use case.

Attempting to use sed to replace CONTEXT with $CONTEXT technically works but it’s messy, when running the script locally I end up with git changes that I need to handle. Also, of course, the linter complains about the constant condition and you have to comment everywhere why this weird string is there so really not a fan of that.

So I looked for better options. One simple approach I tried was using dotenv. In my prebuild script I run printenv > .env to put everything in a file that dotenv can pick up (obviously I could just put the stuff I need but I was trying to keep things simple!). Unfortunately this doesn’t work due to dotenv failing to find the .env file. I guess this is down to zip it and ship it ignoring it as it’s not explicitly imported?

So what I’m doing now is writing the build env vars to a json file in my prebuild script with:

python -c 'import json, os;print(json.dumps(dict(os.environ)))' > env.json

Then in my lambdas I can import them with const env = require('./env.json');

Finally I can use env.CONTEXT === 'production' to switch the api tokens I use per environment.

Please sort this out Netlify, it’s really at odds with how effortless and straightforward everything else is when using the service :slight_smile:

2 Likes

Whoa. Serious work-around. I think I like it though, because that should actually also work for running ntl dev locally if I’m presuming correctly - dev injects .env file vars into the context then runs scripts… so if this works for both local dev based on .env file vars and also on branch / preview deploys, that would be incredible. Appreciate your time and efforts @bristoljon!! Going to poke at writing that little one-liner in js just so it fits the node workflow but that’s awesome!

No worries, yeah it wasn’t what I planned to be doing yesterday! Also to clarify the .env approach failed in the lambdas but would work for the front end code I believe and I did see that they got imported on running netlify dev.

Would prefer dotenv so all the vars can be accessed in process.env as now they are split between there and the json file import but it’s good enough for me for now. I just hope Netlify get round to passing CONTEXT through automatically as would be a lot nicer

Just want to add for others who may read this thread - having separate ENV vars running, for both the site builder (injecting client vars into front-end scripts, e.g.) and for the functions works perfectly well if you just need to split local dev from running-on-Netlify-in-the-cloud “prod”. In my case, I work on a lot of stuff with Stripe and test heavily locally then ship up to the prod branch and everything’s perfect. This works well because I do all the dev myself and doing it locally is fine. I don’t have test branches that are also cloud-deployed etc. So if that matches you and having local vs. prod works, you’re good to go. This conversation / discussion is more rooted around having multiple environments hosted on the cloud running different ENV vars.


Jon

EDIT: tl;dr: this whole conversation is for if you’re running deploy-previews or branch-previews and want different ENV vars for those builds. If you just run locally and on ‘production’ / your main site build, ENV vars should be straightforward (Netlify UI for prod env vars, .env file and netlify dev for local :+1:t2:)

It’s now been a year since trying Netlify Dev and the almost perfect experience it has given. I want to second everyone’s callout here and point out that this is an experience killer in what Netlify & Netlify Dev attempts to accomplish and from my experience stands for - Make it is easy to deploy static sites and provide awesome deploy previews. The flexibility through netlify.toml and branch deploys is awesome and no other solution like it out there.

When Netlify Dev was announced, it did everything perfectly but read from the TOML. Given Netlify Dev is to mimic what goes on at Netlify, it would be great if the .toml is also read and ENVs also provided from it.

More related to this topic, the same must be applied to the Functions. It seems there are limitations in AWS Lambdas and the reason for not transfering them over is due to it’s 4k limit. I understand some people need more, but not enabling this for the few when the mass can have an awesome experience seems illogical.

Furthermore, their are other alternatives like the Netlify Dev and Netlify Cloud generating a .env file based on the TOML. I believe @ bristoljon attempt to do this on his own and failed as functions don’t read from the same build workspace.

All in all, I do love netlify and use it every day and promoting it all the time. However, being so close to having production-grade experience for functions is a shame. I really hope we get a better answer than there is a 4K limit and here’s a potential hack that’ll work around this. I think we’re all using netlify because we don’t want to hack our own solutions.

3 Likes

Hi, @uptownhr, the workaround that first comes to mind is writing the environment variables into the Node/Go files before the functions are bundled and deployed.

The environment variables are constants known at the time of build and deploy. This means you can write the literal strings into the files which will end up becoming the deployed function(s) during the site build.

You also mentioned not being able to access the environment variables from netlify.toml during local builds. For there there is a workaround as well. These environment variables are processed and available using the CLI command netlify build to build the site locally:

https://cli.netlify.com/commands/build/

The netlify build command locally is a good way of testing methods to embed the environment variables in the node/Go function source files during the site build process.

1 Like

I think this thread gets a lot of views without many responses so I just want to add that the concept @luke was describing is now fully available as a Netlify Build Plugin

5 Likes

Hey again all :wave:t2:

This thread continues to get plenty of clicks even though it doesn’t get many replies. Configuring ENV vars correctly for multiple environments seems to be something folks continue to want to do.

I took some time to write up two methods for having separate ENV vars per environment (local, branch-deploy, deploy-preview, production). Both work great and allow your Functions and SSG to get the right values for the environment they’re running in.

Hope it helps :slight_smile:


Jon again

5 Likes

:+1: This also a deal-breaker for me and hopefully it will be addressed soon by the otherwise excellent netlify pipeline team.

3 Likes

Hey Joe, I also tried to use the environment variables from [context.branch] in the netlify.toml file. But, netlify only accepts environmental variables from the dashboard settings and not from the netlify.toml file. So, I have figured out a way to overcome this. I hope this helps.

2 Likes

Thanks for sharing, @Matheswaaran. For other finding this guide, please note the 4K environment variable limit is part of the underlying infrastructure Functions uses (which is AWS Lambda):

https://docs.aws.amazon.com/lambda/latest/dg/gettingstarted-limits.html

The limitation is listed on the page above, which is quoted below:

The following quotas apply to function configuration, deployments, and execution. They cannot be changed.

Resource Quota
Function memory allocation 128 MB to 10,240 MB, in 1-MB increments.
Function timeout 900 seconds (15 minutes)
Function environment variables 4 KB

Note, Netlify’s limitations can be more restrictive than those above. For example, the maximum function timeout is 28 seconds with a default of 10 seconds (with background functions being an exception).

However, we cannot offer a larger space for environment variables because of the limitation above.

1 Like

Posting here as well in case some wonder: netlify-plugin-inline-functions-env-typescript - npm inlineAll param, it might be what you need.

Tried Netlify functions again after three years and it’s a huge bummer this is something given no priority. I love Netlify for so many reasons, but this makes the functions feature feel not ready for the big time. I hit this issue (Figuring out if netlify-function is in a main env or production deploy) years go and ended up moving to Vercel.

That’s an unfortunate stance to take for something like this because it’s something that could easily be documented. People with more than 4kb can opt into to using the UI or resorting to sed if needed. Moreover, even for that use case they’d still like to know what environment they’re targeting I’m sure. I use Netlify because of the branch deploys and branch based environments which is all but broken without nasty hacks.

Voicing this as one of the “few” to even the scales compared to the “many.” :grinning:

hi there mwood23,

we do appreciate your feedback on this, and we hope to get things fixed up better when we can. I know there is an update to our doc re: env vars planned, and generally of course it would be better to have more than a 4kb limit, but i do believe that is still out of our hands.

if anything changes here, we will absolutely let you know.