Netlify.toml context env variables do not apply to functions

Support
, , ,
#1

Hopefully this saves someone some time (might be good to add the official documentation because I fell for this once in January and again today - although I realized faster why things weren’t working).

Question:

I overwrite some of my environment variables in my netlify.toml file, for my staging environment deploys. However, I’ve noticed that these overwrites only apply to the site, they don’t apply to my staging branch lambda functions. Is this a bug?

Support Answer (see update below, couldn’t get this to work):

It’s an open feature request to support environment variables from the toml file in functions.

In the meantime you may be able to leverage different build commands for deploy contexts per the TOML file to achieve your goal:

  1. set both variables in our UI, say “API_BACKEND_STAGE = https://stage.com” and “API_BACKEND_PROD = https://prod.com”
  2. then create conditional build commands for each branch like this:
[context.production] 
    command = "export API_BACKEND=$API_BACKEND_PROD && npm run build"
[context.stage]
    command = "export API_BACKEND=$API_BACKEND_STAGE && npm run build"

and use only plain $API_BACKEND in your build

Update: I tried the above, where API_BACKEND_STAGE and API_BACKEND_PROD are both set in the Netlify UI, but API_BACKEND is not available as an environment variable when my function executes. So it does not work. Will be looking for alternative solutions.

This is related to the discussion here: https://github.com/netlify/netlify-lambda/issues/59#issuecomment-477383840 They’ve created an example to reproduce the behavior.

Update: I tried eval instead of export in the example above, and API_BACKEND is still not available in process.env when my function executes. I also tried both export and eval API_BACKEND=Test just to see if I could hard code something, but that also failed, when the staging function executes it does not have API_BACKEND in process.env

3 Likes
#3

I tried adding env variables in the build command, it didn’t work for me either.

In the same github issue mentioned above, I added a temporary workaround by just creating another netlify app that points to the same github repo but to the branch “staging” (or something), you can see it here: https://github.com/netlify/netlify-lambda/issues/59#issuecomment-482454231.

This is far from perfect and it might still need some clever solutions, like adding the functions endpoint to the process.env as well (e.g. process.env.ENDPOINT) so if your are hosting your website in the same netlify app, you can easily know which is your target domain.

1 Like
#4

Hey folks!

It is a known shortcoming that the netlify.toml file environment variables are not usable in functions. You must set them in our UI in the Build & Deploy settings page, and then to use them, you must ALSO change your function in some way to redeploy it. We snapshot your function by checksum and won’t redeploy it in case you only change environment variables…which means those new variables won’t be used since they weren’t there in the old snapshot - you have to change its checksum in some way to trigger this.

We’re working on an article about some of those limitations of functions so you’ll be able to find advice like that here in the near future!

#7

thanks!

It is a known shortcoming that the netlify.toml file environment variables are not usable in functions. You must set them in our UI in the Build & Deploy settings page, and then to use them, you must ALSO change your function in some way to redeploy it.

Just to be clear, this means that there is no way to have specific Env Var values for your branch deploy functions. Because all functions (regardless of what branch they are associated with) will use the Env Vars that were set in the UI. There is no workaround that solves this issue (aside from hosting a separate site and using it purely as a staging environment as suggested by @pgarciacamou)

2 Likes
Cannot access $DEPLOY_URL with netlify build
#8

There is a workaround :slight_smile: It’s to have a more intelligent build script that selects the appropriate variables to use during build. One can use contexts to choose a different build command, so something like this (untested) is what I might use:

  • in the UI, set $STAGING_ENDPOINT and $PRODUCTION_ENDPOINT to different values
  • in netlify.toml use build commands like this to incorporate them:
[context.staging]
  command = "./my-production-build-script.sh"
[context.production]
  command = "./my-staging-build-script.sh"

and then have those two scripts use separate environment variables. Another pattern using a makefile as shown here would work too: https://www.netlify.com/blog/2017/04/07/selective-password-protection/

#9

It’s to have a more intelligent build script that selects the appropriate variables to use during build.

So both me and @pgarciacamou have tried the intelligent build script approach (which we mentioned above) and haven’t gotten it to work. It seems that setting environment variables within a custom build command does not actually pass those variables to the env where the Function executes.

For example, if you try

[context.production]
     command = "export API_BACKEND=TEST && gatsby build"

and run console.log(process.env.API_BACKEND) within your lambda function, API_BACKEND won’t exist.

Have also tried eval instead of export.

1 Like
#10

Hey Chris (alias @fool),

Here goes nothing.

I tried again without eval/export, by directly assigning the env variable to the command, see: https://github.com/pgarciacamou/netlify-context-in-previews/pull/3 and same issue.

[context.deploy-preview]
  command = "WHO_AM_I=DeployPreview npm run build"

The Deploy Preview is generated/built correctly but the env variable is the value from the envs set in the UI:

https://deploy-preview-3–netlify-context-in-previews.netlify.com/.netlify/functions/hello

Basically, it should read “Hello DeployPreview” but instead returns “Hello Build”.

2 Likes
#11

I’m seeing exactly the same issue as what @pgarciacamou mentioned. I’ve just tried the approach @fool suggests but got Permission denied.

8:29:23 PM: Executing user command: ./setenv.sh; npm run generate; netlify-lambda build lambda
8:29:23 PM: /usr/local/bin/build: line 32: ./setenv.sh: Permission denied
#12

Hi @teakwood and @pgarciacamou,

The issue you’re seeing is that exporting an env variable while executing a process makes that variable available to that process. However, when your function is deployed, the environment variables we pass into it are sent by our API, which is why you are limited to just those in the UI at the moment (until we get that bug fixed).

The only way to get environment variables from another source in to your functions without adding them via our UI is to embed them in to the file, which you can do manually or using a plugin like https://www.npmjs.com/package/webpack-plugin-replace and then you can do something like:

[context.production]
 command = "export API_BACKEND='This is a test' && node replace.js"

As was mentioned, we do have an open feature request for this. In the meantime I recommend specifying the environment variables in the UI, which has the added benefit of not exposing them to anyone that has access to your repo.

@nerimplo I don’t know if your issue is related. You would have to provide more details, like what’s in that script that you’re running.

4 Likes
#15

That doesn’t solve the problem @teakwood mentioned: because env variables in the UI apply to all deploys, there’s no way to use value A in production and value B in branch deploys.

The solution I’ll probably go with is to parse the host out of the request URL and determine the deploy context from that. It feels hacky, but it seems to be the simplest way to do this without adding a messy search-and-replace script to my functions build step (which currently just uses tsc).

#16

Ah but that’s where you’re wrong :slight_smile: You can choose which variables to use if you use this pattern:

  1. set $API_PRODUCTION and $API_STAGING separately, on that same site
  2. in your build script, look at $CONTEXT and choose which to use. Here’s one example of doing it via npm:

So, following from that example, your build command would be a script that looks something like this:

#!/bin/sh
if [$CONTEXT == "staging"] ;
  then npm run build -p $STAGING_API
else if [$CONTEXT == "production"] ; 
  then npm run build -p $PRODUCTION_API
else
  # we didn't anticipate any other values.  Could use a fallback if we have one, otherwise...bail!
  exit 1
fi

You could also choose to instead of passing that variable, setting it conditionally, and running the same build command:

#!/bin/sh
if [$CONTEXT == "staging"] ;
  then export MY_CONTEXT_VAR=$STAGING_API
else if [$CONTEXT == "production"] ; 
  then export MY_CONTEXT_VAR=$PRODUCTION_API
fi
# npm is setup to always use $MY_CONTEXT_VAR, and assumes you have it set before you start!
npm run build
#17

That works for build scripts, but is the CONTEXT variable available in functions at runtime? I believe I tried that and it didn’t work. I’d rather not complicate my functions build command (which currently just calls the TypeScript compiler) with some kind of find-and-replace step.

1 Like
#18

No, it is not available in functions at runtime. For those you’d have to specify the context during build (it’s available at build time, so you can add the value directly to your function code while building, to use the “correct” variable that you set in the UI, at runtime:))

#19

I want to be able to add the below line to my netlify.toml, and then SECRET_KEY would be set accordingly in my staging lambda functions at runtime. That would be really slick.

[context.staging.environment]
SECRET_KEY=“INSERT_STAGING_SECRET_KEY”

I’m also using the URL parse solution suggested by @drmercer.

#21

hey @teakwood - thanks for your feedback.

I’ve added your voice to a feature request we have open to revisit this!