Gatsby.js, yarn workspaces and caching node_modules

Support
, , , ,
#1

Hello.
It seems that deployment bot doesn’t cache project’s node_modules.
I use yarn’s workspaces feature with ‘lerna’.
My directories structure is:

...
package.json
yarn.lock
lerna.json
packages/
    components/
        package.json
    project/
        {other package files including src}
        package.json
        yarn.lock

Base directory is ‘/’ (root of repo)
Build command is cd packages/modern-agency && gatsby build && cd ../../

Possibly, node_modules of package aren’t cached but root node_modules should be cached, shouldn’t it? If it’s true, yarn install shouldn’t take a lot of time but it did. On local machine yarn install takes less than 10s if only root node_modules exists (node_modules of every packages doesn’t exist) and takes around minute if no any node_modules exists. When it’s building on Netlify yarn install always takes around minute but I expected less than 10s. What did I do wrong?

No yarn workspaces detected despite using workspaces for build command
#2

We do not have full lerna/monorepo support. We will only cache the node_modules that we install automatically before your build command runs, which would not include any not included in the root node_modules file.

You might want to explore using our monorepo support with the base directory set to packages/modern-agency to see if it works better:

But otherwise, yes, we don’t intend to cache those modules with your current configuration; only the ones we find in the /yarn.lock

No yarn workspaces detected despite using workspaces for build command
#3

Thanks a lot.
I’m facing a strange problem. I specify base directory as a root.
I found out yarn installs almost all modules into root node_modules folder.

I did do experiment on my local machine:

  1. I removed all node_modules folders including in all sub packages.
  2. I executed yarn install in the root folder. It took around one minute.
  3. I removed non-root node_modules (as if they weren’t cached).
  4. I executed yarn install in the root folder one more time. It took around 10 second.

So, on my local machine subsequent executing yarn install is much more faster than the first time, but on Netlify’s system it always takes the same time (about one minute).
It seems root node_modules isn’t being cached.

#4

You could try debugging by running locally: https://github.com/netlify/build-image#running-locally , and file a bug if you can understand more or create a reproduction case: https://github.com/netlify/build-image/issues

Not sure that it’s a bug, may be intended behavior, but that’s the best place to file those kinds of requests, particularly if you can easily reproduce :slight_smile:

#5

I will try, thank you :slight_smile:

#6

Hey did you get anywhere with this? @DalerAkhmetov

#7

Our build spends 2 mins in yarn install, about 45 sec to a min of which seems to spent on getting dependencies as shown below:

2:55:16 PM: yarn install v1.22.4
2:55:16 PM: [1/4] Resolving packages...
2:55:17 PM: [2/4] Fetching packages...
2:55:30 PM: warning url-loader@1.1.2: Invalid bin field for "url-loader".
2:55:52 PM: info fsevents@2.1.2: The platform "linux" is incompatible with this module.
2:55:52 PM: info "fsevents@2.1.2" is an optional dependency and failed compatibility check. Excluding it from installation.
2:55:53 PM: info fsevents@1.2.11: The platform "linux" is incompatible with this module.
2:55:53 PM: info "fsevents@1.2.11" is an optional dependency and failed compatibility check. Excluding it from installation.
2:55:53 PM: [3/4] Linking dependencies...

This, when we have specified a local package cache folder in .yarnrc

workspaces-experimental true
yarn-offline-mirror "./packages-cache”
yarn-offline-mirror-pruning true
child-concurrency 20

All packages are present and checked in with the code, in the ./package-cache folder.

Env

YARN_VERSION=“1.22.4”
NODE_VERSION=“12.16.1”
NPM_VERSION=“6.14.2”

What are we doing wrong? This works locally but seems to take a long time to get dependencies in Netlify.

1 Like
#9

Hi @pdeka - not sure what is causing that from the small log snippet you share, but I’d debug in our build image running locally:

We definitely may not be handling cache in the way that you/yarn expect though, as you can see we have some fairly specific cache handling in our build logic:

1 Like
#10

Hi folks,
Wanted to share an exciting update about Yarn workspaces: we’ve made improvements to how we cache dependencies for these projects. You can try out our new caching by creating an environment variable called NETLIFY_YARN_WORKSPACES and setting it to "true"

We want your feedback before we roll this out more widely- please let us know in this thread what your experience is like! More about the change in that thread as well.