Busting the `node_modules` cache when certain files change?

bennypowers · October 11, 2021, 10:53am

Hello!

For my netlify site apolloelements.dev (apollo-elements), I use patch-package to patch files in node_modules. I run patch-package in postinstall. This works great when the node_modules cache is clear, or if the cache hits but there are no changes in patches/*.diff

Netlify already automatically busts the node_modules cache whenever package.json (or package-lock.json?) changes. I’d like to add patches/*.diff to that list so that changes there also invalidate the cache.

Can I do that?

Thanks

fool · October 12, 2021, 8:37pm

Heya @bennypowers and welcome to our community!

I think you’ve correctly characterized the behavior shown here in our build-image:

github.com

netlify/build-image/blob/focal/run-build-functions.sh#L187


      
            fi
          else
            # if corepack is installed use it for changing the yarn version
            if [ "$(yarn --version)" != "$yarn_version" ]; then
              echo "Installing Yarn version $yarn_version"
              corepack prepare yarn@$yarn_version --activate
            fi
          fi
          
          
restore_node_modules "yarn"
          
          
echo "Installing npm packages using Yarn version $(yarn --version)"
          
          
# Remove the cache-folder flag if the user set any.
          # We want to control where to put the cache
          # to be able to store it internally after the build.
          local yarn_local="${YARN_FLAGS/--cache-folder * /}"
          # The previous pattern doesn't match the end of the string.
          # This removes the flag from the end of the string.
          yarn_local="${yarn_local%--cache-folder *}"

which does this comparison:

github.com

netlify/build-image/blob/focal/run-build-functions.sh#L75-L87


      
          
          
: ${YARN_FLAGS=""}
          : ${NPM_FLAGS=""}
          : ${PNPM_FLAGS=""}
          : ${BUNDLER_FLAGS=""}
          
          
# Feature flags are a comma-separated list.
          # The following logic relies on the fact that feature flags cannot currently
          # have escaped commas in their value. Otherwise, parsing the list as an array,
          # e.g. using `IFS="," read -ra <<<"$1"` would be needed.
          has_feature_flag() {
            if [[ "${1}," == *"${2},"* ]]
            then

…which has the basic effect you describe.

I don’t know of a way for you to accomplish that goal directly, but perhaps you could wrap the postinstall step in a conditional clause, since the error status returned to the shell is what causes the build to be marked as failed:

I don’t know what you can do in a postinstall script, but I’d guess it can be a shell script that you could catch that error on and self-clear the cache if you want to, or carry on without the patch (return 0 from the process even though patching failed), if you don’t need to do anything?

If you need to “do something” to fix things, such as say clearing your cache, you could from inside the script I describe, when you detect the patch system failing, choose to make an API call (you’ll want to set your API token as an environment variable in the UI to keep it safe and out of your public codebase: Netlify App) to the URL we use to trigger a build with cache cleared, before exiting the script with a non-zero status to terminate that build and allow the new one to start up.

What’s that API call?

POST https://api.netlify.com/api/v1/sites/SITE-ID-HERE/builds

with a body of {clear_cache: true}

How’d I find it? Using this process: [Support Guide] Understanding and using Netlify's API

A better fix might be in the build image code; if you’d like to try a PR or file an issue, that repo is open and welcomes contributions and suggestions: GitHub - netlify/build-image: This is the build image used for running automated builds

If you do plan a PR, I’d still start with an issue filed there (feel free to ping back here with a link and we’ll make sure the team reviews ASAP!) and explain your plans so the team can review and ensure they think you’re working in a direction they can support, since we can’t merge a PR that breaks things for other customers and there may even be some works in progress that I don’t know about to support this already

bennypowers · December 20, 2021, 9:10am

Thanks for the reply, Chris

A few notes:

The problem isn’t that the build fails necessarily, the build might even pass, but the cached node_modules mean I don’t receive the changes I made in the patch. This might have the effect of breaking a page in my app because it’s dependencies aren’t patched before the build.

This sounds like a way forward, however I’d much prefer configuration here to imperative calls, I’m sure you’d agree that’d be better. I opened an issue at Allow cache file glob customization · Issue #714 · netlify/build-image · GitHub as you suggested, let’s continue the conversation there

luke · December 21, 2021, 6:43am

Hi, @bennypowers. It took me a minute to understand the feature request and I think grok it now. I’ve made a comment in the issue filed to clarify exactly which code needs to be changed.

If you do have a PR which fixes this, please to feel free to share it via that repo.

janpio · January 25, 2023, 9:23pm

(The repo with these issues and the code has been archived today)

Topic		Replies	Views
How to ignore package.json and yarn.lock? Support building	6	1847	January 5, 2024
Build fails on netlify from caching (but builds perfectly locally) Support building , deployment	6	356	April 14, 2024
[Support Guide] My site deploy fails unless Netlify's build cache is cleared Support Guides building , deployment , support-guide	2	6961	May 21, 2021
Yarn 3.0 native caching Features building	1	511	August 17, 2021
How to enforce package-lock.json Support building	26	11993	October 26, 2023

Busting the `node_modules` cache when certain files change?

Related topics