Yarn 2 needs proper support in builds

PizzaFox · May 7, 2021, 8:21pm

Currently Netlify does a few things that are bad for Yarn 2 users:

Caches the node_modules directory regardless of the configured Yarn cacheFolder
Uses the deprecated cache-folder option, which will throw a YN0050 error

The most annoying consequence of this is that every time one of my Yarn 2 projects is built it downloads 2 copies of the dependencies, every single build (one from Netlify cache, one from Yarn 2). This results in much longer builds as there is now effectively 0 caching of dependencies.

The Netlify build image should be updated to better handle projects that use Yarn 2.

jen · January 29, 2021, 8:59pm

Hey @PizzaFox,
Thanks for this feature request. We have several open issues about this on the build-image repo- I believe this comment links to all of them:

Please feel free to chime in there and/or follow along for updates.

jen · April 17, 2021, 6:22pm

Hi @PizzaFox,
Wanted to share an exciting update about Yarn workspaces: we’ve made improvements to how we cache dependencies for these projects. You can try out our new caching by creating an environment variable called NETLIFY_YARN_WORKSPACES and setting it to "true"

We want your feedback before we roll this out more widely- please let us know in this thread what your experience is like! More about the change in that thread as well.

PizzaFox · April 17, 2021, 9:44pm

I am not using Yarn workspaces so enabling the feature flag doesn’t work (expected?). Build log:

Build log

2:31:23 PM: Build ready to start
2:31:25 PM: build-image version: be42e453d6c8f171cc2f654acc29c0a8b60e6d93
2:31:25 PM: build-image tag: v3.7.1
2:31:25 PM: buildbot version: e8edb9f29a9e870ab9525bcf3d3ee805e7b84580
2:31:25 PM: Fetching cached dependencies
2:31:25 PM: Starting to download cache of 410.0MB
2:31:28 PM: Finished downloading cache in 2.872669341s
2:31:28 PM: Starting to extract cache
2:31:38 PM: Finished extracting cache in 10.518259401s
2:31:38 PM: Finished fetching cache in 13.586009601s
2:31:38 PM: Starting to prepare the repo for build
2:31:39 PM: Preparing Git Reference refs/heads/master
2:31:41 PM: Different publish path detected, going to use the one specified in the Netlify configuration file: 'out' versus '' in the Netlify UI
2:31:41 PM: Different build command detected, going to use the one specified in the Netlify configuration file: 'npm run build; npm run export' versus '' in the Netlify UI
2:31:41 PM: Starting build script
2:31:41 PM: Installing dependencies
2:31:41 PM: Python version set to 2.7
2:31:42 PM: Started restoring cached node version
2:31:46 PM: Finished restoring cached node version
2:31:46 PM: Downloading and installing node v15.14.0...
2:31:47 PM: Downloading https://nodejs.org/dist/v15.14.0/node-v15.14.0-linux-x64.tar.xz...
2:31:47 PM: Computing checksum with sha256sum
2:31:47 PM: Checksums matched!
2:31:50 PM: Now using node v15.14.0 (npm v7.7.6)
2:31:51 PM: Started restoring cached build plugins
2:31:51 PM: Finished restoring cached build plugins
2:31:51 PM: Attempting ruby version 2.6.2, read from environment
2:31:52 PM: Using ruby version 2.6.2
2:31:52 PM: Using PHP version 5.6
2:31:52 PM: Started restoring cached yarn cache
2:31:52 PM: Finished restoring cached yarn cache
2:31:53 PM: Found yarn version (1.17.0) that doesn't match expected (1.22.5)
2:31:53 PM: up to date, audited 1 package in 244ms
2:31:53 PM: found 0 vulnerabilities
2:31:53 PM: Installing yarn at version 1.22.5
2:31:53 PM: Installing Yarn!
2:31:53 PM: > Downloading tarball...
2:31:53 PM: [1/2]: https://yarnpkg.com/downloads/1.22.5/yarn-v1.22.5.tar.gz --> /tmp/yarn.tar.gz.tAIIXeh2zI
2:31:53 PM:   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
2:31:53 PM:                                  Dload  Upload   Total   Spent    Left  Speed
2:31:54 PM:   0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
2:31:54 PM:   0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
2:31:54 PM: 100    79  100    79    0     0    332      0 --:--:-- --:--:-- --:--:--   331
2:31:54 PM: 100    93  100    93    0     0    188      0 --:--:-- --:--:-- --:--:--   188
2:31:54 PM: 100   625  100   625    0     0    774      0 --:--:-- --:--:-- --:--:--   774
2:31:55 PM:  25 1215k   25  305k    0     0   281k      0  0:00:04  0:00:01  0:00:03  281k
2:31:55 PM: 100 1215k  100 1215k    0     0  1041k      0  0:00:01  0:00:01 --:--:-- 10.9M
2:31:55 PM: [2/2]: https://yarnpkg.com/downloads/1.22.5/yarn-v1.22.5.tar.gz.asc --> /tmp/yarn.tar.gz.tAIIXeh2zI.asc
2:31:55 PM: 100    83  100    83    0     0   1924      0 --:--:-- --:--:-- --:--:--  1924
2:31:55 PM: 100    97  100    97    0     0   1368      0 --:--:-- --:--:-- --:--:--  1368
2:31:55 PM: 100   629  100   629    0     0   3836      0 --:--:-- --:--:-- --:--:--  3836
2:31:55 PM: 100   832  100   832    0     0   3898      0 --:--:-- --:--:-- --:--:--  3898
2:31:55 PM: > Verifying integrity...
2:31:55 PM: gpg: Signature made Thu 27 Aug 2020 11:16:49 AM UTC using RSA key ID 69475BAA
2:31:55 PM: gpg: Good signature from "Yarn Packaging <yarn@dan.cx>"
2:31:55 PM: gpg: WARNING: This key is not certified with a trusted signature!
2:31:55 PM: gpg:          There is no indication that the signature belongs to the owner.
2:31:55 PM: Primary key fingerprint: 72EC F46A 56B4 AD39 C907  BBB7 1646 B01B 86E5 0310
2:31:55 PM:      Subkey fingerprint: 6D98 490C 6F1A CDDD 448E  4595 4F77 6793 6947 5BAA
2:31:55 PM: > GPG signature looks good
2:31:55 PM: > Extracting to ~/.yarn...
2:31:55 PM: > Adding to $PATH...
2:31:55 PM: > Successfully installed Yarn 1.22.5! Please open another terminal where the `yarn` command will now be available.
2:31:55 PM: NETLIFY_YARN_WORKSPACES feature flag set
2:31:56 PM: {"type":"error","data":"Cannot find the root of your workspace - are you sure you're currently in a workspace?"}
2:31:56 PM: No workspace detected
2:31:56 PM: Started restoring cached node modules
2:31:56 PM: Finished restoring cached node modules
2:31:56 PM: Installing NPM modules using Yarn version 2.4.1
2:31:57 PM: ➤ YN0050: The cache-folder option has been deprecated; use rc settings instead
2:31:57 PM: ➤ YN0000: ┌ Resolution step
2:31:58 PM: ➤ YN0002: │ jonah.pw@workspace:. doesn't provide eslint (pbcd11), requested by eslint-config-xo-react➤ YN0002: │ jonah.pw@workspace:. doesn't provide eslint (p8e163), requested by eslint-plugin-prettier➤ YN0002: │ jonah.pw@workspace:. doesn't provide eslint (pda118), requested by eslint-plugin-react-hooks➤ YN0002: │ jonah.pw@workspace:. doesn't provide eslint (pf0c3b), requested by eslint-plugin-react➤ YN0002: │ next-pwa@npm:5.2.9 [9e7cc] doesn't provide @babel/core (pca77b), requested by babel-loader➤ YN0002: │ next-pwa@npm:5.2.9 [9e7cc] doesn't provide webpack (p32c59), requested by babel-loader➤ YN0002: │ next-pwa@npm:5.2.9 [9e7cc] doesn't provide webpack (p35e26), requested by clean-webpack-plugin➤ YN0002: │ next-pwa@npm:5.2.9 [9e7cc] doesn't provide webpack (p7bb95), requested by terser-webpack-plugin➤ YN0002: │ next-pwa@npm:5.2.9 [9e7cc] doesn't provide webpack (p5a631), requested by workbox-webpack-plugin➤ YN0002: │ xo@npm:0.38.2 doesn't provide webpack (p69b0b), requested by eslint-import-resolver-webpack➤ YN0000: │ Some peer dependencies are incorrectly met; run yarn explain peer-requirements <hash> for details, where <hash> is the six-letter p-prefixed code
2:31:58 PM: ➤ YN0000: └ Completed in 0s 266ms
2:31:58 PM: ➤ YN0000: ┌ Fetch step
2:32:50 PM: ➤ YN0013: │ 5 packages were already cached, 1097 had to be fetched
2:32:50 PM: ➤ YN0000: └ Completed in 52s 324ms
2:32:50 PM: ➤ YN0000: ┌ Link step
2:32:50 PM: ➤ YN0062: │ fsevents@patch:fsevents@npm%3A2.3.2#builtin<compat/fsevents>::version=2.3.2&hash=11e9ea The platform linux is incompatible with this module, link skipped.
2:32:51 PM: ➤ YN0062: │ fsevents@patch:fsevents@npm%3A1.2.13#builtin<compat/fsevents>::version=1.2.13&hash=11e9ea The platform linux is incompatible with this module, link skipped.
2:33:01 PM: ➤ YN0007: │ ejs@npm:2.7.4 must be built because it never did before or the last one failed
2:33:01 PM: ➤ YN0000: └ Completed in 10s 680ms
2:33:01 PM: ➤ YN0000: Done with warnings in 1m 4s
2:33:01 PM: NPM modules installed using Yarn
2:33:02 PM: Started restoring cached go cache
2:33:04 PM: Finished restoring cached go cache
2:33:04 PM: Installing Go version 1.12
2:33:04 PM: unset GOOS;
2:33:04 PM: unset GOARCH;
2:33:04 PM: export GOROOT='/opt/buildhome/.gimme_cache/versions/go1.12.linux.amd64';
2:33:04 PM: export PATH="/opt/buildhome/.gimme_cache/versions/go1.12.linux.amd64/bin:${PATH}";
2:33:04 PM: go version >&2;
2:33:04 PM: export GIMME_ENV='/opt/buildhome/.gimme_cache/env/go1.12.linux.amd64.env';
2:33:04 PM: go version go1.12 linux/amd64
2:33:04 PM: Installing missing commands
2:33:04 PM: Verify run directory
2:33:05 PM: 
2:33:05 PM: ────────────────────────────────────────────────────────────────
2:33:05 PM:   Netlify Build                                                 
2:33:05 PM: ────────────────────────────────────────────────────────────────
2:33:05 PM: 
2:33:05 PM: ❯ Version
2:33:05 PM:   @netlify/build 11.1.0
2:33:05 PM: 
2:33:05 PM: ❯ Flags
2:33:05 PM:   deployId: 607b53abb222c200072f401f
2:33:05 PM: 
2:33:05 PM: ❯ Current directory
2:33:05 PM:   /opt/build/repo
2:33:05 PM: 
2:33:05 PM: ❯ Config file
2:33:05 PM:   /opt/build/repo/netlify.toml
2:33:05 PM: 
2:33:05 PM: ❯ Context
2:33:05 PM:   production
2:33:05 PM: 
2:33:05 PM: ❯ Loading plugins
2:33:05 PM:    - netlify-plugin-cache-nextjs@1.5.2 from netlify.toml and package.json
2:33:05 PM: 
2:33:05 PM: ────────────────────────────────────────────────────────────────
2:33:05 PM:   1. onPreBuild command from netlify-plugin-cache-nextjs        
2:33:05 PM: ────────────────────────────────────────────────────────────────
2:33:05 PM: 
2:33:07 PM: .next/cache exists on disk
2:33:07 PM: Restored the cached .next folder at the location `.next/cache`
2:33:07 PM: 
2:33:07 PM: (netlify-plugin-cache-nextjs onPreBuild completed in 1.1s)
2:33:07 PM: 
2:33:07 PM: ────────────────────────────────────────────────────────────────
2:33:07 PM:   2. build.command from netlify.toml                            
2:33:07 PM: ────────────────────────────────────────────────────────────────
2:33:07 PM: 
2:33:07 PM: $ npm run build; npm run export
2:33:07 PM: > jonah.pw@4.0.0 build
2:33:07 PM: > next build
2:33:09 PM: info  - Using webpack 5. Reason: future.webpack5 option enabled https://nextjs.org/docs/messages/webpack5
2:33:10 PM: warn  - You have enabled experimental feature(s).
2:33:10 PM: warn  - Experimental features are not covered by semver, and may cause unexpected or broken application behavior. Use them at your own risk.
2:33:10 PM: info  - Checking validity of types...
2:33:13 PM: info  - Creating an optimized production build...
2:33:13 PM: > [PWA] Compile server
2:33:13 PM: > [PWA] Compile client (static)
2:33:13 PM: > [PWA] Auto register service worker with: /opt/build/repo/node_modules/next-pwa/register.js
2:33:13 PM: > [PWA] Service worker: /opt/build/repo/public/sw.js
2:33:13 PM: > [PWA]   url: /sw.js
2:33:13 PM: > [PWA]   scope: /
2:33:39 PM: info  - Compiled successfully
2:33:39 PM: info  - Collecting page data...
2:33:40 PM: info  - Generating static pages (0/3)
2:33:40 PM: info  - Generating static pages (3/3)
2:33:40 PM: info  - Finalizing page optimization...
2:33:40 PM: Page                                Size     First Load JS
2:33:40 PM: ┌ ○ /                               3.23 kB        76.6 kB
2:33:40 PM: ├   └ css/94bdbace2b021c933bd5.css  1.09 kB
2:33:40 PM: ├   /_app                           0 B            73.4 kB
2:33:40 PM: └ ○ /404                            3.09 kB        76.5 kB
2:33:40 PM:     └ css/8d08df8c9ec9233fa72e.css  1.07 kB
2:33:40 PM: + First Load JS shared by all       73.4 kB
2:33:40 PM:   ├ chunks/288.5430e8.js            8.84 kB
2:33:40 PM:   ├ chunks/620.74b039.js            5.06 kB
2:33:40 PM:   ├ chunks/679.743b9b.js            13.5 kB
2:33:40 PM:   ├ chunks/framework.5e33f4.js      42 kB
2:33:40 PM:   ├ chunks/main.3909f9.js           168 B
2:33:40 PM:   ├ chunks/pages/_app.d25938.js     2.74 kB
2:33:40 PM:   ├ chunks/webpack.a6ad60.js        993 B
2:33:40 PM:   └ css/5629acc0aaa6d6359098.css    871 B
2:33:40 PM: λ  (Server)  server-side renders at runtime (uses getInitialProps or getServerSideProps)
2:33:40 PM: ○  (Static)  automatically rendered as static HTML (uses no initial props)
2:33:40 PM: ●  (SSG)     automatically generated as static HTML + JSON (uses getStaticProps)
2:33:40 PM:    (ISR)     incremental static regeneration (uses revalidate in getStaticProps)
2:33:41 PM: > jonah.pw@4.0.0 preexport
2:33:41 PM: > rm -rf out
2:33:41 PM: > jonah.pw@4.0.0 export
2:33:41 PM: > next export
2:33:42 PM: info  - Using webpack 5. Reason: future.webpack5 option enabled https://nextjs.org/docs/messages/webpack5
2:33:43 PM: warn  - You have enabled experimental feature(s).
2:33:43 PM: warn  - Experimental features are not covered by semver, and may cause unexpected or broken application behavior. Use them at your own risk.
2:33:43 PM: info  - using build directory: /opt/build/repo/.next
2:33:43 PM: info  - Copying "static build" directory
2:33:43 PM: info  - No "exportPathMap" found in "next.config.js". Generating map from "./pages"
2:33:43 PM: info  - Launching 7 workers
2:33:43 PM: info  - Exporting (0/2)
2:33:43 PM: info  - Copying "public" directory
2:33:44 PM: info  - Exporting (2/2)
2:33:44 PM: Export successful. Files written to /opt/build/repo/out
2:33:44 PM: 
2:33:44 PM: (build.command completed in 37.1s)
2:33:44 PM: 
2:33:44 PM: ────────────────────────────────────────────────────────────────
2:33:44 PM:   3. onPostBuild command from netlify-plugin-cache-nextjs       
2:33:44 PM: ────────────────────────────────────────────────────────────────
2:33:44 PM: 
2:33:44 PM: .next/cache exists on disk
2:33:45 PM: Cached the .next folder at the location `.next/cache`
2:33:45 PM: 
2:33:45 PM: (netlify-plugin-cache-nextjs onPostBuild completed in 1.2s)
2:33:45 PM: 
2:33:45 PM: ────────────────────────────────────────────────────────────────
2:33:45 PM:   4. Deploy site                                                
2:33:45 PM: ────────────────────────────────────────────────────────────────
2:33:45 PM: 
2:33:45 PM: Starting to deploy site from 'out'
2:33:45 PM: Creating deploy tree 
2:33:45 PM: Creating deploy upload records
2:33:45 PM: 5 new files to upload
2:33:45 PM: 0 new functions to upload
2:33:45 PM: Site deploy was successfully initiated
2:33:45 PM: 
2:33:45 PM: (Deploy site completed in 450ms)
2:33:45 PM: 
2:33:45 PM: ────────────────────────────────────────────────────────────────
2:33:45 PM:   Netlify Build Complete                                        
2:33:45 PM: Starting post processing
2:33:45 PM: ────────────────────────────────────────────────────────────────
2:33:45 PM: 
2:33:45 PM: (Netlify Build completed in 40.4s)
2:33:46 PM: Post processing - HTML
2:33:46 PM: Caching artifacts
2:33:46 PM: Started saving node modules
2:33:46 PM: Finished saving node modules
2:33:46 PM: Started saving build plugins
2:33:46 PM: Finished saving build plugins
2:33:46 PM: Started saving yarn cache
2:33:46 PM: Finished saving yarn cache
2:33:46 PM: Started saving pip cache
2:33:46 PM: Finished saving pip cache
2:33:46 PM: Started saving emacs cask dependencies
2:33:46 PM: Finished saving emacs cask dependencies
2:33:46 PM: Started saving maven dependencies
2:33:46 PM: Finished saving maven dependencies
2:33:46 PM: Started saving boot dependencies
2:33:46 PM: Finished saving boot dependencies
2:33:46 PM: Started saving rust rustup cache
2:33:46 PM: Finished saving rust rustup cache
2:33:46 PM: Started saving go dependencies
2:33:46 PM: Post processing - header rules
2:33:46 PM: Post processing - redirect rules
2:33:46 PM: Post processing done
2:33:46 PM: Site is live ✨
2:33:48 PM: Finished saving go dependencies
2:33:52 PM: Build script success
2:34:52 PM: Finished processing build request in 3m27.280148043s

Note the

2:31:55 PM: NETLIFY_YARN_WORKSPACES feature flag set
2:31:56 PM: {"type":"error","data":"Cannot find the root of your workspace - are you sure you're currently in a workspace?"}
2:31:56 PM: No workspace detected

jen · April 18, 2021, 3:23pm

Ah yeah, my mistake- conflated Yarn 2 and workspaces. Sorry for the false excitement alarm.

camjackson · April 20, 2021, 10:59am

Thanks for chiming in Jen. So if your earlier link was about yarn workspaces rather than yarn 2, I guess PizzaFox’s original question still stands!

I have a similar issue - when deploying with yarn 2, I see the error about using the deprecated cache-folder flag. The yarn docs led me to this issue, which includes a slightly dodgy workaround in the comments. There’s also this pull request which supposedly will fix it.

Hopefully the linked issue & PR can get some attention at some point

ridem · April 30, 2021, 9:59am

Even though the cache-folder option is deprecated, it’s also completely ignored by yarn 2/3, so anyone can set up the env variables that suits their needs. The PR you mentioned won’t fix the issue that you’d have to set an env in your netlify config to get it working.

We use the YARN_CACHE_FOLDER variable and it has been very useful. Installs the whole project in 15s instead of 5mn (when only the node_modules are cached).

All in all, these are the env variables that I find relevant for all things yarn2-related on Netlify:

[build.environment]
  # Yarn classic version used to bootstrap. Default for @netlify/build < 4 is 1.17, which isn't enough
  YARN_VERSION = "1.22.5"
  # Only needed in case Yarn can't be detected in your base folder
  NETLIFY_USE_YARN = "true"
  # Feature flag to cache all the node_modules folders of the workspaces detected by `YARN_IGNORE_PATH=1 yarn workspaces --json info`
  # Only use this if you have `workspaces` in your package.json
  NETLIFY_YARN_WORKSPACES = "true"
  # Equivalent to setting `cacheFolder` in your .yarnrc.yml
  YARN_CACHE_FOLDER = "/opt/buildhome/.yarn_cache"

Alternatively, as mentioned in the PR you linked to @camjackson, you can also set yarn’s global folder to the location cached by netlify and instruct yarn to use this as the cache, i.e. instead of YARN_CACHE_FOLDER = "/opt/buildhome/.yarn_cache", doing:

YARN_GLOBAL_FOLDER  = "/opt/buildhome/.yarn_cache"
YAN_ENABLE_GLOBAL_CACHE = 1

@PizzaFox does this help?

PizzaFox · April 30, 2021, 4:18pm

Neither solution worked for me with Yarn saying there were no packages cached. I’m not super concerned with that though as those should just be temporary fixes until Netlify can properly implement a solution.