No yarn workspaces detected despite using workspaces for build command

Support
, ,
1

Hello, netlify doesn’t appear to be caching my node_modules properly and takes 3mins before each build retrieving dependencies. build does not fail just the fetch step takes way longer than expected
what am I doing wrong? and how can I speed up the build process?
is using one netlify project that deploys to a domain and subdomain causing the issue?
this is possibly related to Improved caching for yarn workspaces or Gatsby.js, yarn workspaces and caching node_modules
YARN_VERSION: 3.1.1
NODE_VERSION: 17.3.1
YARN_WORKSPACES: true
sitename: lunarium
netlify config toml: netlify build settings for lunarium site · GitHub
root package.json: root yarn package config · GitHub
project setup:

/.yarn
--|cache/ (ignored)
--| releases/yarn-3.1.1.cjs
--| install-state.gz (ignored)
/frontend/ <--- builds when on main branch to lunariumspokane.com
  --|package.json
  --|node_modules (ignored)
/studio/ <-- builds to subdomain when build run on studio branch to studio.lunariumspokane.com
 --|package.json
 --| node_modules (ignored)
/node_,modules <--this is cached when built locally (ignored)
/package.json
/netlify.toml
/yarn.lock

notice the line that reads no yarn workspaces detected

8:44:22 PM: Build ready to start
8:44:24 PM: build-image version: 73def8bb10593b9b818f44989a75ea508018ccb7 (focal)
8:44:24 PM: build-image tag: v4.5.2
8:44:24 PM: buildbot version: 8cd320132979d1a1a502e85e77b47b97cc7d659f
8:44:24 PM: Fetching cached dependencies
8:44:24 PM: Starting to download cache of 596.1MB
8:44:27 PM: Finished downloading cache in 2.943623127s
8:44:27 PM: Starting to extract cache
8:44:45 PM: Finished extracting cache in 17.988958798s
8:44:45 PM: Finished fetching cache in 21.072139188s
8:44:45 PM: Starting to prepare the repo for build
8:44:46 PM: Preparing Git Reference pull/8/head
8:44:47 PM: Parsing package.json dependencies
8:44:47 PM: Starting build script
8:44:48 PM: Installing dependencies
8:44:48 PM: Python version set to 2.7
8:44:48 PM: Started restoring cached node version
8:44:50 PM: Finished restoring cached node version
8:44:51 PM: v17.3.1 is already installed.
8:44:51 PM: Now using node v17.3.1 (npm v8.3.0)
8:44:51 PM: Started restoring cached build plugins
8:44:51 PM: Finished restoring cached build plugins
8:44:52 PM: Attempting ruby version 2.7.2, read from environment
8:44:53 PM: Using ruby version 2.7.2
8:44:53 PM: Using PHP version 8.0
8:44:53 PM: Started restoring cached yarn cache
8:44:53 PM: Finished restoring cached yarn cache
8:44:55 PM: No yarn workspaces detected
8:44:55 PM: Started restoring cached node modules
8:44:55 PM: Finished restoring cached node modules
8:44:55 PM: Installing NPM modules using Yarn version 3.1.1
8:44:56 PM: ➤ YN0050: The cache-folder option has been deprecated; use rc settings instead
8:44:57 PM: ➤ YN0000: ┌ Resolution step
8:44:57 PM: ➤ YN0000: │ Some peer dependencies are incorrectly met; run yarn explain peer-requirements <hash> for details, where <hash> is the six-letter p-prefixed code
8:44:57 PM: ➤ YN0000: └ Completed in 0s 506ms
8:44:57 PM: ➤ YN0000: ┌ Fetch step
2

This might apply to your issue: Gatsby.js, yarn workspaces and caching node_modules - #2 by fool

Not sure it would cache. A Netlify employee would need to confirm.

Edit: I did find this though: Improved caching for yarn workspaces

3

I checked those posts prior to making this post.
I tried the suggested env variables such as NETLIFY_USE_YARN = true
YARN_WORKSPACES = true
neither of which affected the caching/workspace detection.
I’m using my project root as base url for both types of builds like in the improved caching post
unfortunately, nothing has worked so far. perhaps I’m missing something

4

Hi @AndreBClark

This variable is only needed if there is no yarn.lock in the repository.

As posted in Build fails when trying to use latest Yarn v 3.0.2 - #4 by hrishikesh there is an open issue with Yarn 3 workspaces

1 Like
5

Good Catch with that post. I’m reading through the post. and I’m looking to see if there’s any version that works. is any yarn v3 not caching? does v2 work fine?

6

Can neither confirm nor deny as I have not tried.

7

I’m investigating this

1 Like
8

Ahh, good catch @coelmay :grin:

@AndreBClark yeah, I’d do some test repo’s and see if you can get it to reproduce.

9

minimal reproduction repo with netlify site, uses yarn 3.2.0
yarn 2.4.3 fails to build packages with typescript 4.5.5
https://github.com/AndreBClark/yarn-2-netlify-cache
https://caching-issue-yarn-3-netlify.netlify.app/
running builds and will post times tomorrow

10

Hey @AndreBClark :wave:

Thanks for reaching out and for the details you just shared. Would you mind also sharing the site/build in question? That could help us get to the bottom of the problem here by checking the logs and config.

Now, based on what you shared, it is true that Yarn Berry ( yarn >=2) has some compatibility issues within our platform when it comes to caching - Improve Yarn berry support · Issue #612 · netlify/build-image · GitHub - however the logs you shared have caught my eye speficailly because your workspaces should be detected and your node_modules should be cached.

One thing I spotted is that your root package.json doesn’t have a "private": true entry. Albeit this is no longer required for the latest yarn versions, it is a requirment for yarn classic which we rely on for detecting your workspaces, as it is the only way to have a consistent output accross different yarn versions - build-image/run-build-functions.sh at d589800043f90181ea256c37be616bd873b03da0 · netlify/build-image · GitHub. Could you possibly test your site/builds with the "private": true field added to your root package.json and let us know how it goes?

3 Likes
11

apologies for slow reply these past couple days were my days off.

site: https://lunariumspokane.com
latest deploy discovered the workspaces with “private: set to true”,
so that works! will trigger another build to see if caching works properly
in terms of the code, should I set my github repo to public or is there a more private way to provide access?

my test repository I’m using three branches one for each version of yarn.
3.1.1 seems to be stuck and using up all build time.
2.4.3 appears to detect the workspaces with private set to true on the root package.json but fails because of a typescript compatibility issue.
v1 works

12

latest deploy:

caching appears to work, but the fetch step still appears to hang for several minutes. this could be expected behavior, because it had to fetch more packages

14

@jgantunes , your post appears to have resolved the “no workspaces detected issue” so I’m gonna mark this as the solution. is there any additional info regarding my build that would beneficial to you?

15

I don’t see the hanging behaviour. Could you point out what exactly do you mean?

1 Like
16

at the time i was referring to this

5:13:48 PM: ➤ YN0000: ┌ Fetch step
5:16:25 PM: ➤ YN0013: │ 56 packages were already cached, 2248 had to be fetched

I was referring to the fetch step taking 3 mins when it should have been cached, but this appears to be expected as it was fetching packages that were not cached yet. it no longer does that as all packages appear to be properly cached now, I think?
here is what the latest build looks like below. as you can see the workspaces are detected now, and appear to cache properly. I would say the issue at hand is resolved by adding private: true to the root package as suggested by @jgantunes .

9:15:21 PM: Waiting for other deploys from your team to complete. Check the queue: https://app.netlify.com/teams/cosmic-division/builds
9:15:29 PM: Build ready to start
9:15:31 PM: build-image version: 122b31996ccaffd45d820a452d6227f8312110cc (focal)
9:15:31 PM: build-image tag: v4.5.3
9:15:31 PM: buildbot version: 45faaf324677495f3400b572da79977ab3212265
9:15:31 PM: Fetching cached dependencies
9:15:31 PM: Starting to download cache of 839.7MB
9:15:34 PM: Finished downloading cache in 3.070760319s
9:15:34 PM: Starting to extract cache
9:16:03 PM: Finished extracting cache in 28.499352071s
9:16:03 PM: Finished fetching cache in 31.711668543s
9:16:03 PM: Starting to prepare the repo for build
9:16:03 PM: Preparing Git Reference refs/heads/main
9:16:04 PM: Parsing package.json dependencies
9:16:05 PM: Starting build script
9:16:05 PM: Installing dependencies
9:16:05 PM: Python version set to 2.7
9:16:06 PM: Started restoring cached node version
9:16:08 PM: Finished restoring cached node version
9:16:09 PM: v17.3.1 is already installed.
9:16:09 PM: Now using node v17.3.1 (npm v8.3.0)
9:16:10 PM: Started restoring cached build plugins
9:16:10 PM: Finished restoring cached build plugins
9:16:10 PM: Attempting ruby version 2.7.2, read from environment
9:16:11 PM: Using ruby version 2.7.2
9:16:11 PM: Using PHP version 8.0
9:16:11 PM: WARNING: The environment variable 'NODE_ENV' is set to 'production'. Any 'devDependencies' in package.json will not be installed
9:16:11 PM: Started restoring cached yarn cache
9:16:11 PM: Finished restoring cached yarn cache
9:16:13 PM: Yarn workspaces detected
9:16:13 PM: Started restoring workspace frontend node modules
9:16:13 PM: Finished restoring workspace frontend node modules
9:16:13 PM: Started restoring workspace root node modules
9:16:13 PM: Finished restoring workspace root node modules
9:16:13 PM: Installing NPM modules using Yarn version 3.1.1
9:16:14 PM: ➤ YN0050: The cache-folder option has been deprecated; use rc settings instead
9:16:15 PM: ➤ YN0000: ┌ Resolution step
9:16:15 PM: ➤ YN0000: └ Completed in 0s 477ms
9:16:15 PM: ➤ YN0000: ┌ Fetch step
9:18:19 PM: ➤ YN0000: └ Completed in 2m 5s
9:18:19 PM: ➤ YN0000: ┌ Link step
9:18:26 PM: ➤ YN0000: │ lunarium@workspace:. STDOUT husky - Git hooks installed
9:18:27 PM: ➤ YN0000: │ svelte-preprocess@npm:4.10.2 [0de79] STDOUT [svelte-preprocess] Don't forget to install the preprocessors packages that will be used: node-sass/sass, stylus, less, postcss & postcss-load-config, coffeescript, pug, etc...
➤ YN0000: └ Completed in 7s 350ms
9:18:27 PM: ➤ YN0000: Done in 2m 13s
9:18:28 PM: NPM modules installed using Yarn
9:18:29 PM: Started restoring cached go cache
9:18:29 PM: Finished restoring cached go cache
9:18:29 PM: go version go1.16.5 linux/amd64
9:18:29 PM: go version go1.16.5 linux/amd64
9:18:29 PM: Installing missing commands
9:18:29 PM: Verify run directory
9:18:31 PM: ​
9:18:31 PM: ────────────────────────────────────────────────────────────────
9:18:31 PM:   Netlify Build                                                 
9:18:31 PM: ────────────────────────────────────────────────────────────────
9:18:31 PM: ​
9:18:31 PM: ❯ Version
9:18:31 PM:   @netlify/build 26.2.2
9:18:31 PM: ​
9:18:31 PM: ❯ Flags
9:18:31 PM:   baseRelDir: true
9:18:31 PM:   buildId: 61f61ee93f70ca00085e003f
9:18:31 PM:   deployId: 61f61ee93f70ca00085e0041
9:18:31 PM: ​
9:18:31 PM: ❯ Current directory
9:18:31 PM:   /opt/build/repo
9:18:31 PM: ​
9:18:31 PM: ❯ Config file
9:18:31 PM:   /opt/build/repo/netlify.toml
9:18:31 PM: ​
9:18:31 PM: ❯ Context
9:18:31 PM:   production
9:18:31 PM: ​
9:18:31 PM: ❯ Loading plugins
9:18:31 PM:    - netlify-plugin-subfont@6.0.0 from netlify.toml and package.json
9:18:31 PM:    - netlify-plugin-submit-sitemap@0.2.3 from netlify.toml and package.json
9:18:35 PM: ​
9:18:35 PM: ────────────────────────────────────────────────────────────────
9:18:35 PM:   1. build.command from netlify.toml                            
9:18:35 PM: ────────────────────────────────────────────────────────────────
9:18:35 PM: ​
9:18:35 PM: $ yarn workspace frontend build
9:18:48 PM: 05:18 AM [build] 2 pages built in 8.46s (4231ms/page)
9:18:48 PM: 05:18 AM [build] 🚀 Done
9:18:51 PM: dist/index.html 248ms
9:18:51 PM: dist/menu/index.html 649ms
9:18:52 PM: ​
9:18:52 PM: (build.command completed in 16.7s)
9:18:52 PM: ​
9:18:52 PM: ────────────────────────────────────────────────────────────────
9:18:52 PM:   2. netlify-plugin-subfont (onPostBuild event)                 
9:18:52 PM: ────────────────────────────────────────────────────────────────
9:18:52 PM: ​
9:18:52 PM: Running subfont version 6.4.1
9:18:52 PM:  ✔ 0.001 secs: logEvents
9:18:53 PM:  ✔ 0.728 secs: loadAssets
9:18:54 PM:  ✔ 1.058 secs: populate
9:18:54 PM:  ✔ 0.006 secs: checkIncompatibleTypes
9:18:54 PM:  ✔ 0.001 secs: applySourceMaps
9:18:54 PM:  ✔ 0.003 secs: populate
9:18:58 PM:  ✔ 0.003 secs: populate
9:19:07 PM:  ✔ 0.003 secs: serializeSourceMaps
9:19:07 PM:  ✔ 0.041 secs: writeAssetsToDisc
9:19:07 PM: frontend/dist/index.html: 1 font (3 variants) in use, 987 kB total. Created subsets: 25.2 kB total
9:19:07 PM:   Fira Sans Extra Condensed:
9:19:07 PM:     400 : 71/1687 codepoints used (52 on this page), 304 kB (ttf) => 10.9 kB (woff2)
9:19:07 PM:     900 : 36/1687 codepoints used (16 on this page), 342 kB (ttf) => 6.16 kB (woff2)
9:19:07 PM:     700 : 34/1687 codepoints used (14 on this page), 341 kB (ttf) => 8.14 kB (woff2)
9:19:07 PM: frontend/dist/menu/index.html: 1 font (3 variants) in use, 987 kB total. Created subsets: 25.2 kB total
9:19:07 PM:   Fira Sans Extra Condensed:
9:19:07 PM:     400 : 71/1687 codepoints used (70 on this page), 304 kB (ttf) => 10.9 kB (woff2)
9:19:07 PM:     900 : 36/1687 codepoints used (30 on this page), 342 kB (ttf) => 6.16 kB (woff2)
9:19:07 PM:     700 : 34/1687 codepoints used (28 on this page), 341 kB (ttf) => 8.14 kB (woff2)
9:19:07 PM: HTML/SVG/JS/CSS size increase: 2.24 kB
9:19:07 PM: Total savings: 1.92 MB
9:19:07 PM: Output written to file:///opt/build/repo/frontend/dist/
9:19:07 PM: ​
9:19:07 PM: (netlify-plugin-subfont onPostBuild completed in 14.7s)
9:19:07 PM: ​
9:19:07 PM: ────────────────────────────────────────────────────────────────
9:19:07 PM:   3. Deploy site                                                
9:19:07 PM: ────────────────────────────────────────────────────────────────
9:19:07 PM: ​
9:19:07 PM: Starting to deploy site from 'frontend/dist'
9:19:07 PM: Creating deploy tree 
9:19:07 PM: Creating deploy upload records
9:19:07 PM: 5 new files to upload
9:19:07 PM: 0 new functions to upload
9:19:08 PM: Starting post processing
9:19:08 PM: Post processing - HTML
9:19:08 PM: Post processing - header rules
9:19:08 PM: Post processing - redirect rules
9:19:08 PM: Post processing done
9:19:10 PM: Site is live ✨
9:19:12 PM: Finished waiting for live deploy in 4.106200333s
9:19:12 PM: Site deploy was successfully initiated
9:19:12 PM: ​
9:19:12 PM: (Deploy site completed in 4.6s)
9:19:12 PM: ​
9:19:12 PM: ────────────────────────────────────────────────────────────────
9:19:12 PM:   4. netlify-plugin-submit-sitemap (onSuccess event)            
9:19:12 PM: ────────────────────────────────────────────────────────────────
9:19:12 PM: ​
9:19:12 PM: Going to submit sitemap to google
9:19:12 PM:  --> URL: https://www.google.com/ping?sitemap=https://lunariumspokane.com/sitemap.xml
9:19:12 PM: Going to submit sitemap to bing
9:19:12 PM:  --> URL: https://www.bing.com/ping?sitemap=https://lunariumspokane.com/sitemap.xml
9:19:12 PM: Going to submit sitemap to yandex
9:19:12 PM:  --> URL: https://webmaster.yandex.ru/ping?sitemap=https://lunariumspokane.com/sitemap.xml
9:19:12 PM:  ✓  DONE! Sitemap submitted succesfully to google 
9:19:12 PM:  ✓  DONE! Sitemap submitted succesfully to bing 
9:19:12 PM:  ✓  DONE! Sitemap submitted succesfully to yandex 
9:19:12 PM: ​
9:19:12 PM: (netlify-plugin-submit-sitemap onSuccess completed in 849ms)
9:19:13 PM: ​
9:19:13 PM: ────────────────────────────────────────────────────────────────
9:19:13 PM:   Netlify Build Complete                                        
9:19:13 PM: ────────────────────────────────────────────────────────────────
9:19:13 PM: ​
9:19:13 PM: (Netlify Build completed in 41.7s)
9:19:13 PM: Caching artifacts
9:19:13 PM: Started saving workspace frontend node modules
9:19:13 PM: Finished saving workspace frontend node modules
9:19:13 PM: Started saving workspace root node modules
9:19:13 PM: Finished saving workspace root node modules
9:19:13 PM: Started saving build plugins
9:19:13 PM: Finished saving build plugins
9:19:13 PM: Started saving yarn cache
9:19:13 PM: Finished saving yarn cache
9:19:13 PM: Started saving pip cache
9:19:13 PM: Finished saving pip cache
9:19:13 PM: Started saving emacs cask dependencies
9:19:13 PM: Finished saving emacs cask dependencies
9:19:13 PM: Started saving maven dependencies
9:19:13 PM: Finished saving maven dependencies
9:19:13 PM: Started saving boot dependencies
9:19:13 PM: Finished saving boot dependencies
9:19:13 PM: Started saving rust rustup cache
9:19:13 PM: Finished saving rust rustup cache
9:19:13 PM: Started saving go dependencies
9:19:13 PM: Finished saving go dependencies
9:19:13 PM: Build script success
9:21:28 PM: Finished processing build request in 5m57.329705863s
17

Hey @AndreBClark, sorry for taking so long to get back to you :pray: I’m glad to see the private: true option worked. Skimming through your settings I couldn’t spot anything out of the ordinary besides that. We have a series of issues open around improving Yarn berry support so hopefully we’ll be able to improve that in the future - Improve Yarn berry support · Issue #612 · netlify/build-image · GitHub.

1 Like