My site deploy fails unless Netlify’s build cache is cleared (Microsoft Rush monorepo)

Support
,
1

I read the following article, and point #4 seems to pertain to me:

You manually manipulate your node_modules directory in some way during install - and that can corrupt the node module cache which we store and attempt to reuse. This is not generally regarded as a good practice and should be avoided for optimal results in our continuous deployment environment.

Our project is a monorepo built with Rush: https://rushjs.io/

As described in their documentation, they have the following feature:

A single NPM install: In one step, Rush installs all the dependencies for all your projects into a common folder. This is not just a “package.json” file at the root of your repo (which might set you up to accidentally require() a sibling’s dependencies). Instead, Rush uses symlinks to reconstruct an accurate “node_modules” folder for each project, without any of the limitations or glitches that seem to plague other approaches.

Furthermore, the root folder that contains all of the dependencies is not node_modules in the project root, but it is /common/temp/node_modules.

This appears to be causing problems with our builds. Our initial build succeeded, while subsequent builds failed. However, subsequent builds will succeed after clearing the cache.

Our build runs a command to build a Storybook application, and errors with Error: Cannot find module '/opt/build/repo/node_modules/@storybook/react/bin/build.js, leading me to believe dependencies have not properly been installed (or reinstated from cache).

Are there any suggestions on how to fix this issue?

Netlify Instance
https://yolkai.netlify.com/

Did you use the search before posting?
Yes, read the linked article above and both linked articles within.
Also did a search of the forum for issues related to Rush.

1 Like
2

Sounds like our caching may not be a good match for your setup. I wasn’t quite able to tell from your description if there is a good way to make things work using the cache, so this isn’t a “we can’t accommodate you” so much as a “that is more detail than we can likely dive into”, but you have a few options to work around it which I will enumerate here:

  1. “repair” things during your build. Not sure what that looks like (maybe it’s more explicit npm installs of missing packages, moving files around, or something else entirely), but you can test out using our build-image locally as specified here: GitHub - netlify/build-image: This is the build image used for running automated builds for a faster debug cycle.
  2. remove the cache before you run. You might avoid us automatically creating, then deleting, by using something like a $NPM_FLAGS of --dry-run to prevent our automagic npm install from “doing any real work”, and then you could wipe the cache as your first build step so you’re starting with a clean slate.
3

@fool Thank you for the follow up. Having the ability to clear the cache at the beginning of each build sounds like it could fix our issue.

and then you could wipe the cache as your first build step so you’re starting with a clean slate.

I’m not clear on how to approach this within the Netlify build. Would you mind elaborating?

Thanks!

4

Sure, check out our build scripts here:

You can see what directory to rm -rf /path/to/cache && run-my-build on based on that :slight_smile: