NPM cache not cleared in a commit hook context after clearing it manually first

Support
, ,
#1

We have this problem where apparently an older NPM cache is used when doing Git commit hook deploys.

Clicking into the failed deployed and clicking “Clear Cache and deploy site” will deploy the same commit it without any problems. But when adding a new commit afterwards, and the site is redeployed automatically, it will fail again, just as if it is still using the old cache somehow.

Any ideas on what is going on? This is happening when a project npm package name is renamed. I have tried to regenerate package-lock files, but that doesn’t help either.

2 Likes
#2

Hi @skogsmaskin, can you link to one of these deploys that’s failing? Also can you tell us how you are determining that the problem is with an older npm module and not somewhere else?

When you build successfully we create new build cache, which uses the updated NPM modules rather than any old ones. We would need to dig in to your build to see what’s happening.

#3

Hi @futuregerald, here’s one of the deploys that isnt working https://app.netlify.com/sites/andrews-powdercoating/deploys/5d493da080a3153cd437122e

I’m not quite sure how he determined what the issue was exactly but he went through my git commit history for the project and that’s what he came up with. I can build locally with no problems, and when I click ‘clear cache and deploy site’ from your netlify admin, it builds fine. however when i commit to master and the webhook triggers the build, it fails. same thing happens when clicking ‘deploy’ from within the sanity dashboard and that uses a webhook as well I believe. If you need access to the repository I can give it to you

Thanks,
Travis Ballard

#4

@futuregerald i haven’t touched the site since this was posted a couple days ago, but i just committed some changes and it triggered the build again. this time they were successful. both the sanity studio build and the website build.

however when triggering the build with a webhook on content update still seems to have this issue. here’s the build log for that: https://app.netlify.com/sites/andrews-powdercoating/deploys/5d4c4aa575d0a53229d4570c

#5

I think that the first build you linked does not have any dependency installation, because we’d say so in the build logs if we did it (example from another site):

9:14:46 AM: Started restoring cached node modules
9:14:46 AM: Finished restoring cached node modules
9:14:46 AM: Installing NPM modules using NPM version 6.9.0

We only do npm dependency installation when you have a package.json or yarn.lock in the root of your repo (or in the base directory if you have one set which you don’t seem to for your site) and I can see your build pattern is to cd somewhere and run npm, which probably means you have a package.json there but not in the main directory? In that case we don’t cache dependencies for you since we didn’t install them, so your build should be installing them every time (inefficient, I knpow, but we don’t today have a good implementation of handling sites laid out as yours appears to be in several directories that all have package.json files in them).

#6

@fool there is a package.json in the project root and @sanity/cli is one of the dependencies inside of it. I actually created the project using Sanity’s create tool and selecting the blog template here https://www.sanity.io/create?template=sanity-io%2Fsanity-template-gatsby-blog

I made the repository to be public so that you can access it and see for yourself and see if maybe that will help diagnose it further. https://github.com/TravisBallard/andrews-powdercoating is the url to it

Thanks,
Travis Ballard

#7

Hi @futuregerald, thanks for the reply, and sorry about my late follow up. So this is the thing, we have a monorepo using lerna containing two NPM packages/projects, one for the Sanity Studio and one for the web frontend in the same repository. When I rename these packages these problems occur. Here’s a link to a failing build: https://app.netlify.com/sites/sanity-gatsby-blog-studio-fazquans/deploys/5d49495ae50d6400083a7635

Here’s a build which succeeds with the same commit hash, but is deployed manually by “Clear Cache and deploy site” in the Netlify interface: https://app.netlify.com/sites/sanity-gatsby-blog-studio-fazquans/deploys/5d494a0f2cc8567dd54c16c5

Then when using the commit hook again (just update readme or whatever, no real code changes), I would expect it to deploy properly again as cache is cleared, but it doesn’t and I can’t figure out why.

1 Like
#8

Hi folks - Y’all are working together, right @TravisBallardand @skogsmaskin?

Anyway, sorry Travis, I do see a package.json there and I do see it being used in builds on the site such as https://app.netlify.com/sites/andrews-powdercoating/deploys/5d4f7847132462d7b74f61a0 - not sure what I was looking at before, perhaps a wrong deploy in another site!

@skogsmaskin that’s pretty expected behavior - not that the cache wouldn’t work, but that any use of a build hook would use a cache, since the only way to clear the cache is via our UI and thus you clear it there and then that build finishes and we save the cache and it is there for your hook-triggered build (unless you can get that hook triggered build to start before the other one finishes maybe?)

Anyhow it is not particularly surprising that lerna & monorepo projects with package.json’s in other directories aren’t building right since our support for them is pretty much nonexistent: https://github.com/netlify/build-image/issues/196 , but we are working on the next iteration of our build network which will handle them better.

For today, I think clearing the cache manually is going to be your best path for deploying such a site here, but you may be able to figure out some better workaround by using our build image locally to try different settings for the base directory and tweaking package.json layout: https://github.com/netlify/build-image#running-locally .

You’ll probably be most interested in tracking that issue in the build-image repo so you can try the new system once it is available (it’s getting close to testable now :))