Hopefully this saves someone some time (might be good to add the official documentation because I fell for this once in January and again today - although I realized faster why things weren’t working).
Question:
I overwrite some of my environment variables in my netlify.toml file, for my staging environment deploys. However, I’ve noticed that these overwrites only apply to the site, they don’t apply to my staging branch lambda functions. Is this a bug?
Support Answer (see update below, couldn’t get this to work):
It’s an open feature request to support environment variables from the toml file in functions.
In the meantime you may be able to leverage different build commands for deploy contexts per the TOML file to achieve your goal:
then create conditional build commands for each branch like this:
[context.production]
command = "export API_BACKEND=$API_BACKEND_PROD && npm run build"
[context.stage]
command = "export API_BACKEND=$API_BACKEND_STAGE && npm run build"
and use only plain $API_BACKEND in your build
Update: I tried the above, where API_BACKEND_STAGE and API_BACKEND_PROD are both set in the Netlify UI, but API_BACKEND is not available as an environment variable when my function executes. So it does not work. Will be looking for alternative solutions.
Update: I tried eval instead of export in the example above, and API_BACKEND is still not available in process.env when my function executes. I also tried both export and evalAPI_BACKEND=Test just to see if I could hard code something, but that also failed, when the staging function executes it does not have API_BACKEND in process.env
This is far from perfect and it might still need some clever solutions, like adding the functions endpoint to the process.env as well (e.g. process.env.ENDPOINT) so if your are hosting your website in the same netlify app, you can easily know which is your target domain.
It is a known shortcoming that the netlify.toml file environment variables are not usable in functions. You must set them in our UI in the Build & Deploy settings page, and then to use them, you must ALSO change your function in some way to redeploy it. We snapshot your function by checksum and won’t redeploy it in case you only change environment variables…which means those new variables won’t be used since they weren’t there in the old snapshot - you have to change its checksum in some way to trigger this.
We’re working on an article about some of those limitations of functions so you’ll be able to find advice like that here in the near future!
It is a known shortcoming that the netlify.toml file environment variables are not usable in functions. You must set them in our UI in the Build & Deploy settings page, and then to use them, you must ALSO change your function in some way to redeploy it.
Just to be clear, this means that there is no way to have specific Env Var values for your branch deploy functions. Because all functions (regardless of what branch they are associated with) will use the Env Vars that were set in the UI. There is no workaround that solves this issue (aside from hosting a separate site and using it purely as a staging environment as suggested by @pgarciacamou)
There is a workaround It’s to have a more intelligent build script that selects the appropriate variables to use during build. One can use contexts to choose a different build command, so something like this (untested) is what I might use:
in the UI, set $STAGING_ENDPOINT and $PRODUCTION_ENDPOINT to different values
in netlify.toml use build commands like this to incorporate them:
It’s to have a more intelligent build script that selects the appropriate variables to use during build.
So both me and @pgarciacamou have tried the intelligent build script approach (which we mentioned above) and haven’t gotten it to work. It seems that setting environment variables within a custom build command does not actually pass those variables to the env where the Function executes.
The issue you’re seeing is that exporting an env variable while executing a process makes that variable available to that process. However, when your function is deployed, the environment variables we pass into it are sent by our API, which is why you are limited to just those in the UI at the moment (until we get that bug fixed).
The only way to get environment variables from another source in to your functions without adding them via our UI is to embed them in to the file, which you can do manually or using a plugin like webpack-plugin-replace - npm and then you can do something like:
[context.production]
command = "export API_BACKEND='This is a test' && node replace.js"
As was mentioned, we do have an open feature request for this. In the meantime I recommend specifying the environment variables in the UI, which has the added benefit of not exposing them to anyone that has access to your repo.
@nerimplo I don’t know if your issue is related. You would have to provide more details, like what’s in that script that you’re running.
That doesn’t solve the problem @teakwood mentioned: because env variables in the UI apply to all deploys, there’s no way to use value A in production and value B in branch deploys.
The solution I’ll probably go with is to parse the host out of the request URL and determine the deploy context from that. It feels hacky, but it seems to be the simplest way to do this without adding a messy search-and-replace script to my functions build step (which currently just uses tsc).
Ah but that’s where you’re wrong You can choose which variables to use if you use this pattern:
set $API_PRODUCTION and $API_STAGING separately, on that same site
in your build script, look at $CONTEXT and choose which to use. Here’s one example of doing it via npm:
So, following from that example, your build command would be a script that looks something like this:
#!/bin/sh
if [$CONTEXT == "staging"] ;
then npm run build -p $STAGING_API
else if [$CONTEXT == "production"] ;
then npm run build -p $PRODUCTION_API
else
# we didn't anticipate any other values. Could use a fallback if we have one, otherwise...bail!
exit 1
fi
You could also choose to instead of passing that variable, setting it conditionally, and running the same build command:
#!/bin/sh
if [$CONTEXT == "staging"] ;
then export MY_CONTEXT_VAR=$STAGING_API
else if [$CONTEXT == "production"] ;
then export MY_CONTEXT_VAR=$PRODUCTION_API
fi
# npm is setup to always use $MY_CONTEXT_VAR, and assumes you have it set before you start!
npm run build
That works for build scripts, but is the CONTEXT variable available in functions at runtime? I believe I tried that and it didn’t work. I’d rather not complicate my functions build command (which currently just calls the TypeScript compiler) with some kind of find-and-replace step.
No, it is not available in functions at runtime. For those you’d have to specify the context during build (it’s available at build time, so you can add the value directly to your function code while building, to use the “correct” variable that you set in the UI, at runtime:))
I want to be able to add the below line to my netlify.toml, and then SECRET_KEY would be set accordingly in my staging lambda functions at runtime. That would be really slick.
I’ve come up against this issues - sad to see it’s still there after 2 years. Could you please add me to the feature request so I can see when it’s finally solved?
Hi there, just wanted to share that the Netlify UI, CLI, and API now supports context-based environment variable values. You can use one of these methods to apply contextual values, and they’ll reach all scopes (builds, functions, runtime, post-processing) unless you scope the variables otherwise. This post shares more details. Hope that helps!
It’s to have a more intelligent build script that selects the appropriate variables to use during build. One can use contexts to choose a different build command, so something like this (untested) is what I might use: