Improved caching for yarn workspaces

Hello folks :wave:

As part of our effort to improve users’ build experience for projects relying on JavaScript monorepos, we’re releasing an improvement to the way we cache dependency installs for projects using yarn workspaces.

Monorepo projects depend a lot on the base directory build setting. However, this option makes our build system change to this directory and run all the relevant dependency install commands there. For yarn this is a particular problem because when yarn detects that it’s part of a workspace, it installs and hoists (moves to the root of the repo) all the package dependencies. Until recently, in these cases, we cached the node_modules dependencies in the site base directory, but not the ones installed in the root node_modules directory or any other package node_modules directories. Because these other directories weren’t cached, they would have to be re-installed on every build, which can significantly increase build time.

We are changing that. :smiley: We are introducing a change that will cache all the relevant node_modules directories for yarn workspace projects. We’ll be rolling it out slowly to all of our builds but we’re super interested in collecting feedback and hearing from you folks directly. So, if you want to try this out, you can use the following environment variable NETLIFY_YARN_WORKSPACES=true which will enable this feature. (EDIT: This has now been rolled out to all of our builds so no action is required to start taking advantage of our caching improvements :tada:)

On a first build execution you should see something similar to this :point_down: in your logs:

Followed by this :point_down: at the end of your build logs (notice we’re now caching multiple workspace directories)

On a follow up build execution the cache should start to take effect! :tada:

So, what do you think? We are looking forward to hearing back from you, so leave a comment below with your feedback!

2 Likes

Hi @jgantunes !

I’m trying to use NETLIFY_YARN_WORKSPACES but its not showing up in the build logs even when I try the clear cache & deploy option. The build fails because I’m using a sibling package as a dependency, but netlify is trying to find it on NPM. :thinking:

Hello @Optimism!

Could you share a link to a build of your project that we can look through? Is your project open source and if so are you able to share a link to it? Else could you provide some info as to how your project is structured in terms of workspaces and sites you’re trying to deploy (e.g. if you any netlify.toml config and where it’s located, which folders are you trying to deploy, do you have any base directory configured)?

A couple of things that could be related to it. This caching feature is only working for projects using yarn, we currently detect this on our end by spotting the presence of a yarn.lock in the base directory of the project we’re building. If this base directory is pointing to a child directory (such as the case when we’re building a child workspace), there won’t be a yarn.lock file present so you’ll need to add an extra environment variable NETLIFY_USE_YARN=true to force our build system to use yarn for your build.

Looking forward to earing back from you!

Adding NETLIFY_USE_YARN=true did the trick. Huge thanks!!

2 Likes

We’ve decided to give yarn workspaces a go on Netlify for one of our major projects. So far everything has been running smoothly except for one thing.

Our folder structure is as follows:

  • root
    – workspaces
    — common
    — a
    — b
    — c
    — d

We’ve added “common”: “1.0.0” to the package.json file inside a, b, c, and d.
Now whenever we make changes within common and push to github, Netlify begins the build process and cancels it automatically with the reason that it did not detect any changes.

I’m trying to wrap my head around what we need to do to get this working.
Would we need to update common to 1.0.1 and update a, b, c, d to also use “common”: “1.0.1” every time we make a change? It would be great if we could at least use “common”: “latest” but then it tries to search the online registry for “common”.

This may be something to do with how workspaces work or my lack of understanding on the topic.
Could you offer some insight?

Thank you.

Hey @leo-stefan :wave:

If I’m reading you correctly you wish to also build your sites whenever changes to common take place, right?

By default, when relying in the base directory config, Netlify will ignore any changes that happen outside of the configured directory (e.g. for your website a, assuming you’re relying on the base directory config, new builds will only be triggered when changes to any files in the directory workspaces/a/ are detected). To override this behaviour you can use the ignore configuration field, so that it not only tracks changes within your site but also the common directory. A potential example for your website a:

(Edited to add an actually working example below :point_down:)

[build]
  # Build when differences in the current dir `a` and `../workspace/common/ ` are detected
  ignore = "git diff --quiet HEAD^ HEAD . ../../workspaces/common/"

Here’s a working example for a project with the following structure:

root/
├─ package.json
├─ packages/
│  ├─ blog-1/
│  │  ├─ package.json
│  │  ├─ netlify.toml
│  ├─ blog-2/
│  │  ├─ package.json
│  ├─ dir with spaces/
│  │  ├─ package.json

netlify.toml has the following section:

[build]
  ignore = "git diff --quiet HEAD^ HEAD . ../../packages/blog-2/"

Changes to dir with spaces render the following cancelled build (no changes detected) while changes to blog-2 actually trigger a build.

Let us know if this addresses your probelms :+1:

Okay thanks! This seems to work. I am getting this in the console however:

Detected ignore command in Netlify configuration file. Proceeding with the specified command: 'git diff --quiet HEAD^ HEAD workspaces/a/ workspaces/common/'
3:07:50 PM: fatal: ambiguous argument 'workspaces/a/‘: unknown revision or path not in the working tree.
3:07:50 PM: Use '--' to separate paths from revisions, like this:
3:07:50 PM: 'git <command> [<revision>...] -- [<file>...]'
3:07:50 PM: Starting build script

Should I be doing something different?

Thank you!

Hey @leo-stefan, apologies for not having provided a clearer example. I’ve edited the comment above to add something that will most likely work in your case (the problem is that your current base directory is actually ./workspaces/a so we need to account for that in our ignore command). This should work:

[build]
  # Build when differences in the current dir `a` and `../workspace/common/ ` are detected
  ignore = "git diff --quiet HEAD^ HEAD . ../../workspaces/common/"

Let us know if it did the trick :+1:

1 Like