Missing environment variables using yarn 3?

Hey all,

I seem to be having an issue getting my Netlify environment variables working – set directly in the Environment site settings. They aren’t there when i console.log(process.env) in my app (in production). The only thing I can think of that’s different is i upgraded to Yarn 3.2.0, but I could just be missing something basic…

The console output returns the standard environment variables sans the ones I expect to see:

{NODE_ENV: 'production', PUBLIC_URL: '', WDS_SOCKET_HOST: undefined, WDS_SOCKET_PATH: undefined, WDS_SOCKET_PORT: undefined, …}
FAST_REFRESH: true
NODE_ENV: "production"
PUBLIC_URL: ""
WDS_SOCKET_HOST: undefined
WDS_SOCKET_PATH: undefined
WDS_SOCKET_PORT: undefined
[[Prototype]]: Object

Here is a manual deployment from Netlify that fails due to Yarn, if I use netlify-cli for deployment it deploys and publishes fine, but the env. var issue above happens:

3:50:26 PM: Build ready to start
3:50:27 PM: build-image version: 122b31996ccaffd45d820a452d6227f8312110cc (focal)
3:50:27 PM: build-image tag: v4.5.3
3:50:27 PM: buildbot version: 7eafb394e33f42f945c880ce4ac17c149867813a
3:50:28 PM: Building without cache
3:50:28 PM: Starting to prepare the repo for build
3:50:28 PM: No cached dependencies found. Cloning fresh repo
3:50:28 PM: git clone https://github.com/MikelMNJ/app-template
3:50:28 PM: Preparing Git Reference refs/heads/main
3:50:28 PM: Parsing package.json dependencies
3:50:29 PM: Starting build script
3:50:29 PM: Installing dependencies
3:50:29 PM: Python version set to 2.7
3:50:30 PM: Downloading and installing node v16.14.0...
3:50:30 PM: Downloading https://nodejs.org/dist/v16.14.0/node-v16.14.0-linux-x64.tar.xz...
3:50:30 PM: Computing checksum with sha256sum
3:50:31 PM: Checksums matched!
3:50:33 PM: Now using node v16.14.0 (npm v8.3.1)
3:50:33 PM: Started restoring cached build plugins
3:50:33 PM: Finished restoring cached build plugins
3:50:33 PM: Attempting ruby version 2.7.2, read from environment
3:50:34 PM: Using ruby version 2.7.2
3:50:34 PM: Using PHP version 8.0
3:50:34 PM: Started restoring cached yarn cache
3:50:34 PM: Finished restoring cached yarn cache
3:50:35 PM: Found yarn version (1.22.10) that doesn't match expected (3.2.0)
3:50:35 PM: up to date, audited 1 package in 214ms
3:50:35 PM: found 0 vulnerabilities
3:50:35 PM: Installing yarn at version 3.2.0
3:50:35 PM: Installing Yarn!
3:50:35 PM: > Downloading tarball...
3:50:35 PM: [1/2]: https://yarnpkg.com/downloads/3.2.0/yarn-v3.2.0.tar.gz --> /tmp/yarn.tar.gz.ON1Oz3b7OA
3:50:35 PM:   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
3:50:35 PM:                                  Dload  Upload   Total   Spent    Left  Speed
3:50:35 PM:   0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
3:50:35 PM: 100    77  100    77    0     0    647      0 --:--:-- --:--:-- --:--:--   647
3:50:35 PM: 100    90  100    90    0     0    426      0 --:--:-- --:--:-- --:--:--   426
3:50:35 PM:   0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
3:50:35 PM: curl: (22) The requested URL returned error: 404
3:50:35 PM: [2/2]: https://yarnpkg.com/downloads/3.2.0/yarn-v3.2.0.tar.gz.asc --> /tmp/yarn.tar.gz.ON1Oz3b7OA.asc
3:50:35 PM:   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
3:50:35 PM:                                  Dload  Upload   Total   Spent    Left  Speed
3:50:35 PM:   0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
3:50:35 PM: 100    81  100    81    0     0   1350      0 --:--:-- --:--:-- --:--:--  1350
3:50:35 PM: 100    94  100    94    0     0   1236      0 --:--:-- --:--:-- --:--:--  1236
3:50:35 PM:   0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
3:50:35 PM: curl: (22) The requested URL returned error: 404
3:50:35 PM: > Failed to download https://yarnpkg.com/downloads/3.2.0/yarn-v3.2.0.tar.gz.
3:50:36 PM: Creating deploy upload records
3:50:35 PM: mv: cannot stat '/opt/buildhome/.yarn': No such file or directory
3:50:35 PM: No yarn workspaces detected
3:50:35 PM: Started restoring cached node modules
3:50:35 PM: Finished restoring cached node modules
3:50:35 PM: /opt/build-bin/run-build-functions.sh: line 141: yarn: command not found
3:50:35 PM: Installing NPM modules using Yarn version
3:50:36 PM: Failed during stage 'building site': Build script returned non-zero exit code: 1 (https://ntl.fyi/exit-code-1)
3:50:36 PM: /opt/build-bin/run-build-functions.sh: line 152: yarn: command not found
3:50:36 PM: Error during Yarn install
3:50:36 PM: Build was terminated: Build script returned non-zero exit code: 1
3:50:36 PM: Failing build: Failed to build site
3:50:36 PM: Finished processing build request in 8.763477796s

Hey there, @MikelMNJ :wave:

Thanks for your patience here, and sorry to be slow to respond! Can you please share a link to your site with us? Additionally, have you taken any additional debugging steps in the past five days? Let us know!

Hi,

Sorry for the late reply myself. I switched to using a .env file and netlify-cli for deployment. Manually deploying from the site fails with the above error.

Edit: I should also note I turned off the auto deployment in the interface by setting it to a production branch that doesn’t exist called “disabled”. Got tired of deploying on CLI and having the working version overridden when I merged a branch to main.

Also, I changed the app name from app-template to “starterapp” since the first post.

https://starterapp.netlify.com

Sorry for double reply – didn’t want things to get too confusing.

Update 2: I re-linked my repo, removed the YARN_VERSION override I had in the initial post’s screenshot and let the deploy (through the GUI) fall back to yarn 1.22.10 and it built successfully. While i’m happy that worked, it means there’s no Yarn 3 support on Netlify.

Still sticking with netlify-cli for now since that does build the total package with the functions folder, yarn 3 and all env variables. GUI only seems to do yarn 1.22.10, all env variables (yay), but ignores or can’t find the lambda functions folder. Here’s the log from the latest build:

9:04:17 AM: Build ready to start
9:04:19 AM: build-image version: 195fbe127e5c374d9c4758652cb62e3b8936a395 (focal)
9:04:19 AM: build-image tag: v4.6.0
9:04:19 AM: buildbot version: 4ac77ee15240cafa5a14a04b6ca18d8959fa98e6
9:04:19 AM: Fetching cached dependencies
9:04:20 AM: Starting to download cache of 237.6MB
9:04:22 AM: Finished downloading cache in 2.518936947s
9:04:22 AM: Starting to extract cache
9:04:31 AM: Finished extracting cache in 8.854974541s
9:04:31 AM: Finished fetching cache in 11.40983015s
9:04:31 AM: Starting to prepare the repo for build
9:04:31 AM: Preparing Git Reference refs/heads/main
9:04:31 AM: Parsing package.json dependencies
9:04:33 AM: Starting build script
9:04:33 AM: Installing dependencies
9:04:33 AM: Python version set to 2.7
9:04:33 AM: Started restoring cached node version
9:04:35 AM: Finished restoring cached node version
9:04:35 AM: v16.14.2 is already installed.
9:04:36 AM: Now using node v16.14.2 (npm v8.5.0)
9:04:36 AM: Started restoring cached build plugins
9:04:36 AM: Finished restoring cached build plugins
9:04:36 AM: Attempting ruby version 2.7.2, read from environment
9:04:38 AM: Using ruby version 2.7.2
9:04:38 AM: Using PHP version 8.0
9:04:38 AM: Started restoring cached yarn cache
9:04:43 AM: Finished restoring cached yarn cache
9:04:43 AM: No yarn workspaces detected
9:04:43 AM: Started restoring cached node modules
9:04:43 AM: Finished restoring cached node modules
9:04:44 AM: Installing NPM modules using Yarn version 1.22.10
9:04:44 AM: yarn install v1.22.10
9:04:44 AM: [1/4] Resolving packages...
9:04:50 AM: warning react-scripts > @svgr/webpack > @svgr/plugin-svgo > svgo@1.3.2: This SVGO version is no longer supported. Upgrade to v2.x.x.
9:04:53 AM: warning @testing-library/jest-dom > css > source-map-resolve@0.6.0: See https://github.com/lydell/source-map-resolve#deprecated
9:04:55 AM: warning netlify-lambda > webpack > micromatch > snapdragon > source-map-resolve@0.5.3: See https://github.com/lydell/source-map-resolve#deprecated
9:04:55 AM: warning netlify-lambda > webpack > watchpack > watchpack-chokidar2 > chokidar@2.1.8: Chokidar 2 does not receive security updates since 2019. Upgrade to chokidar 3 with 15x fewer dependencies
9:04:55 AM: warning netlify-lambda > webpack > watchpack > watchpack-chokidar2 > chokidar > fsevents@1.2.13: fsevents 1 will break on node v14+ and could be using insecure binaries. Upgrade to fsevents 2.
9:04:55 AM: warning netlify-lambda > webpack > micromatch > snapdragon > source-map-resolve > resolve-url@0.2.1: https://github.com/lydell/resolve-url#deprecated
9:04:55 AM: warning netlify-lambda > webpack > micromatch > snapdragon > source-map-resolve > source-map-url@0.4.1: See https://github.com/lydell/source-map-url#deprecated
9:04:55 AM: warning netlify-lambda > webpack > micromatch > snapdragon > source-map-resolve > urix@0.1.0: Please see https://github.com/lydell/urix#deprecated
9:04:55 AM: warning netlify-lambda > webpack > node-libs-browser > url > querystring@0.2.0: The querystring API is considered Legacy. new code should use the URLSearchParams API instead.
9:04:56 AM: [2/4] Fetching packages...
9:04:57 AM: info fsevents@2.3.2: The platform "linux" is incompatible with this module.
9:04:57 AM: info "fsevents@2.3.2" is an optional dependency and failed compatibility check. Excluding it from installation.
9:04:57 AM: info fsevents@1.2.13: The platform "linux" is incompatible with this module.
9:04:57 AM: info "fsevents@1.2.13" is an optional dependency and failed compatibility check. Excluding it from installation.
9:04:57 AM: [3/4] Linking dependencies...
9:04:58 AM: warning "react-scripts > tailwindcss@3.0.23" has unmet peer dependency "autoprefixer@^10.0.2".
9:04:58 AM: warning "react-scripts > eslint-config-react-app > eslint-plugin-flowtype@8.0.3" has unmet peer dependency "@babel/plugin-syntax-flow@^7.14.5".
9:04:58 AM: warning "react-scripts > eslint-config-react-app > eslint-plugin-flowtype@8.0.3" has unmet peer dependency "@babel/plugin-transform-react-jsx@^7.14.9".
9:04:58 AM: warning "react-scripts > react-dev-utils > fork-ts-checker-webpack-plugin@6.5.0" has unmet peer dependency "typescript@>= 2.7".
9:04:58 AM: warning "react-scripts > eslint-config-react-app > @typescript-eslint/eslint-plugin > tsutils@3.21.0" has unmet peer dependency "typescript@>=2.8.0 || >= 3.2.0-dev || >= 3.3.0-dev || >= 3.4.0-dev || >= 3.5.0-dev || >= 3.6.0-dev || >= 3.6.0-beta || >= 3.7.0-dev || >= 3.7.0-beta".
9:04:58 AM: warning " > @testing-library/user-event@13.5.0" has unmet peer dependency "@testing-library/dom@>=7.21.4".
9:05:02 AM: [4/4] Building fresh packages...
9:05:04 AM: success Saved lockfile.
9:05:04 AM: Done in 19.68s.
9:05:04 AM: NPM modules installed using Yarn
9:05:04 AM: Started restoring cached go cache
9:05:04 AM: Finished restoring cached go cache
9:05:04 AM: go version go1.16.5 linux/amd64
9:05:04 AM: go version go1.16.5 linux/amd64
9:05:04 AM: Installing missing commands
9:05:04 AM: Verify run directory
9:05:06 AM: ​
9:05:06 AM: ────────────────────────────────────────────────────────────────
9:05:06 AM:   Netlify Build                                                 
9:05:06 AM: ────────────────────────────────────────────────────────────────
9:05:06 AM: ​
9:05:06 AM: ❯ Version
9:05:06 AM:   @netlify/build 26.5.0
9:05:06 AM: ​
9:05:06 AM: ❯ Flags
9:05:06 AM:   baseRelDir: true
9:05:06 AM:   buildId: 623877d1713345000900b34a
9:05:06 AM:   deployId: 623877d1713345000900b34c
9:05:06 AM: ​
9:05:06 AM: ❯ Current directory
9:05:06 AM:   /opt/build/repo
9:05:06 AM: ​
9:05:06 AM: ❯ Config file
9:05:06 AM:   /opt/build/repo/netlify.toml
9:05:06 AM: ​
9:05:06 AM: ❯ Context
9:05:06 AM:   production
9:05:06 AM: ​
9:05:06 AM: ────────────────────────────────────────────────────────────────
9:05:06 AM:   1. build.command from netlify.toml                            
9:05:06 AM: ────────────────────────────────────────────────────────────────
9:05:06 AM: ​
9:05:06 AM: $ yarn build
9:05:06 AM: yarn run v1.22.10
9:05:06 AM: $ react-scripts build
9:05:08 AM: Creating an optimized production build...
9:05:26 AM: Compiled successfully.
9:05:26 AM: 
9:05:26 AM: File sizes after gzip:
9:05:26 AM: Creating deploy upload records
9:05:26 AM:   111.35 kB  build/static/js/main.2e946a04.js
9:05:26 AM:   22.76 kB   build/static/css/main.9f462bb8.css
9:05:26 AM: The project was built assuming it is hosted at /.
9:05:26 AM: You can control this with the homepage field in your package.json.
9:05:27 AM: Starting post processing
9:05:26 AM: The build folder is ready to be deployed.
9:05:26 AM: You may serve it with a static server:
9:05:26 AM:   yarn global add serve
9:05:26 AM:   serve -s build
9:05:26 AM: Find out more about deployment here:
9:05:27 AM: Post processing - HTML
9:05:26 AM:   https://cra.link/deployment
9:05:26 AM: Done in 20.11s.
9:05:26 AM: ​
9:05:27 AM: Post processing - header rules
9:05:26 AM: (build.command completed in 20.3s)
9:05:26 AM: ​
9:05:26 AM: ────────────────────────────────────────────────────────────────
9:05:26 AM:   2. Functions bundling                                         
9:05:27 AM: Post processing - redirect rules
9:05:26 AM: ────────────────────────────────────────────────────────────────
9:05:26 AM: ​
9:05:26 AM: The Netlify Functions setting targets a non-existing directory: lambda
9:05:26 AM: ​
9:05:26 AM: (Functions bundling completed in 2ms)
9:05:27 AM: Post processing done
9:05:26 AM: ​
9:05:26 AM: ────────────────────────────────────────────────────────────────
9:05:26 AM:   3. Deploy site                                                
9:05:26 AM: ────────────────────────────────────────────────────────────────
9:05:26 AM: ​
9:05:26 AM: Starting to deploy site from 'build'
9:05:26 AM: Creating deploy tree 
9:05:27 AM: 0 new files to upload
9:05:27 AM: 0 new functions to upload
9:05:27 AM: Site deploy was successfully initiated
9:05:27 AM: ​
9:05:27 AM: (Deploy site completed in 238ms)
9:05:27 AM: ​
9:05:27 AM: ────────────────────────────────────────────────────────────────
9:05:27 AM:   Netlify Build Complete                                        
9:05:27 AM: ────────────────────────────────────────────────────────────────
9:05:27 AM: ​
9:05:27 AM: (Netlify Build completed in 20.6s)
9:05:27 AM: Caching artifacts
9:05:27 AM: Started saving node modules
9:05:27 AM: Finished saving node modules
9:05:27 AM: Started saving build plugins
9:05:27 AM: Finished saving build plugins
9:05:27 AM: Started saving yarn cache
9:05:30 AM: Site is live ✨
9:05:31 AM: Finished saving yarn cache
9:05:31 AM: Started saving pip cache
9:05:31 AM: Finished saving pip cache
9:05:31 AM: Started saving emacs cask dependencies
9:05:31 AM: Finished saving emacs cask dependencies
9:05:31 AM: Started saving maven dependencies
9:05:31 AM: Finished saving maven dependencies
9:05:31 AM: Started saving boot dependencies
9:05:31 AM: Finished saving boot dependencies
9:05:31 AM: Started saving rust rustup cache
9:05:31 AM: Finished saving rust rustup cache
9:05:31 AM: Started saving go dependencies
9:05:31 AM: Finished saving go dependencies
9:05:31 AM: Build script success
9:06:08 AM: Finished processing build request in 1m48.653410334s

Hey there, @MikelMNJ :wave:

Thanks so much for following up. Can you elaborate on how you are able to get this working with the CLI? Can you share your .toml file as well as your build log from your CLI deployment?

Additionally, I am wondering if changing "lambda" to "functions" in your .toml file would improve this?

Edit: I should also mention i’ve been saying “CLI” this whole time and referring to “netlify-lambda”, not “netlify-cli”. So perhaps that’s the root cause of the difference in workflows?

Hey again, my netlify.toml file is pretty simple:

[build]
command = "yarn build"
functions = "lambda"
publish = "build"

Here is the successful build log from CLI:

Creating an optimized production build...
Compiled successfully.

File sizes after gzip:

  112.37 kB  build/static/js/main.f55f0903.js
  22.84 kB   build/static/css/main.c8a02745.css

The project was built assuming it is hosted at /.
You can control this with the homepage field in your package.json.

The build folder is ready to be deployed.
You may serve it with a static server:

  yarn global add serve
  serve -s build

Find out more about deployment here:

  https://cra.link/deployment

netlify-lambda: Building functions
Hash: 670394284dbebcc76571
Version: webpack 4.46.0
Time: 2796ms
Built at: 04/01/2022 4:26:50 PM
    Asset      Size  Chunks             Chunk Names
server.js  2.31 MiB       0  [emitted]  server
Entrypoint server = server.js
[104] ../node_modules/express/index.js 224 bytes {0} [built]
[163] ../node_modules/dotenv/lib/main.js 2.63 KiB {0} [built]
[182] ../node_modules/mongoose/index.js 186 bytes {0} [built]
[291] ../node_modules/vary/index.js 2.86 KiB {0} [built]
[293] ./middleware/limitMiddleware.js 368 bytes {0} [built]
[471] ./server.js 703 bytes {0} [built]
[472] ./connectDB/db.js 390 bytes {0} [built]
[651] ../node_modules/express/lib/express.js 2.35 KiB {0} [built]
[725] ../node_modules/serverless-http/serverless-http.js 659 bytes {0} [built]
[726] ../node_modules/serverless-http/lib/finish.js 847 bytes {0} [built]
[727] ../node_modules/serverless-http/lib/framework/get-framework.js 1.49 KiB {0} [built]
[736] ../node_modules/cors/lib/index.js 6.47 KiB {0} [built]
[738] ./routes/sampleRoutes.js 832 bytes {0} [built]
[744] ./routes/emailRoutes.js 325 bytes {0} [built]
[887] ./routes/authRoutes.js 332 bytes {0} [built]
    + 894 hidden modules

Deploy path:        /home/mikel/Documents/Git/starter-app/build
Functions path:     /home/mikel/Documents/Git/starter-app/lambda
Configuration path: /home/mikel/Documents/Git/starter-app/netlify.toml
Deploying to main site URL...
✔ No cached functions were found
✔ Finished hashing 22 files and 1 functions
✔ CDN requesting 0 files and 0 functions
✔ Finished uploading 0 assets
✔ Deploy is live!

Logs:              https://app.netlify.com/sites/starterapp/deploys/6247600c1c9d3079969cb26d
Unique Deploy URL: https://6247600c1c9d3079969cb26d--starterapp.netlify.app
Website URL:       https://starterapp.netlify.app

If I try to change functions = "lambda" to functions = "functions" in netlify.toml I get a failed build through CLI with an error stating the functions folder should not be named functions, since that’s where it’s looking for all the back-end lambda stuff already. Here is the failed CLI build log for that scenario:

Creating an optimized production build...
Compiled successfully.

File sizes after gzip:

  112.37 kB  build/static/js/main.f55f0903.js
  22.84 kB   build/static/css/main.c8a02745.css

The project was built assuming it is hosted at /.
You can control this with the homepage field in your package.json.

The build folder is ready to be deployed.
You may serve it with a static server:

  yarn global add serve
  serve -s build

Find out more about deployment here:

  https://cra.link/deployment

netlify-lambda: Building functions
Error: 
      netlify-lambda Error: Function source folder (specified in netlify-lambda serve/build command) and publish folder (specified in netlify.toml)
      should be different. They are both set to /home/mikel/Documents/Git/starter-app/functions.

      This is a common mistake for people switching from Netlify Dev to netlify-lambda. For an easy fix, change your functions key inside netlify.toml to something else, like "functions-build".
      You will then need to build your functions to that directory before they will work locally and the built functions will also need to be pushed to your repo.
      For more info, check https://github.com/netlify/netlify-lambda#usage
      
    at webpackConfig (/home/mikel/Documents/Git/starter-app/node_modules/netlify-lambda/lib/build.js:111:11)
    at async Object.exports.run (/home/mikel/Documents/Git/starter-app/node_modules/netlify-lambda/lib/build.js:208:18)

My take-away from this so far is:

  1. Yarn 3 is not supported on deployment via Netlify GUI, only CLI with the version in your project on your box found in .yarn/releases/yarn-3.x.x.cjs.
  2. Only one deployment workflow should be adheared to: CLI or Netlify auto publish in site settings.
  3. If using CLI, auto publish should be disabled by setting it to a branch that doesn’t exist in settings > build and deploy > branches on the site, to prevent the auto publish from override the working CLI deployment when a merge occurs.
  4. Piggy backing off of point 3: Auto deployment doesn’t seem to recognize any other value for functions = "" in the .toml settings if building from the site, as told by the error message details for the failed build above.

I could also just be doing something horribly wrong, so please let me know :open_mouth:

Hey @MikelMNJ,

To start with, why are you using netlify-lambda? I don’t think you need that package anymore. CLI does the bundling of functions itself and probably in a much better way than netlify-lambda. With that configuration, if you use functions = "functions" I believe it should work.

Thanks @hrishikesh,

I will do some exploration with netlify-cli in regards to functions bundling. netlify-lambda was something I picked up years ago and never dropped, but it may be time :slight_smile:

1 Like