Java dependencies gets downloaded on every build

It seems the $HOME folder of the builtbot doesn’t include a link to .m2 folder from the cache causing dependencies to be downloaded (unnecessarily) on every build.

226  10:57:38 PM: shadow-cljs - updating dependencies
227  10:57:39 PM: Retrieving thheller/shadow-cljs/2.19.6/shadow-cljs-2.19.6.pom from https://repo.clojars.org/
228  10:57:39 PM: Retrieving org/clojure/clojure/1.11.1/clojure-1.11.1.pom from https://repo1.maven.org/maven2/
...
526  10:57:45 PM: shadow-cljs - dependencies updated

This one time I ran the following as part of my build script:

rm -rf $HOME/.m2
ln -s $NETLIFY_BUILD_BASE/cache/.m2 $HOME/.m2

Which would skip all downloads of dependencies:

 85  11:28:14 PM: shadow-cljs - updating dependencies
 86  11:28:15 PM: shadow-cljs - dependencies updated

But that doesn’t seem to be the “right” way to solve it :wink:

Is this a bug in the build image?

Hi @jacobemcken :wave:t6: ,

Welcome to the forums and thanks for reaching out. It’s not possible at the moment to skip the download. Netlify will install all dependencies, languages and frameworks that your project might need and only then proceed to the build command.

This works fine for a lot of people, could you please explain what your use case is that won’t work fine with this workflow?

Sorry for not being clear.

During build when a dependency isn’t found in the cache, it should of course be downloaded.

But on subsequent builds, I would expect Java dependencies to be cached (in the .m2 directory), in a similar manner to how it is done for Node dependencies (in the node_modules directory).
Node doesn’t need to re-download dependencies already available in the cache (node_modules directory) while Java does.

The thing I don’t understand is, that when I manually inspect the contents of $NETLIFY_BUILD_BASE/cache/.m2, I can see cached Java dependencies from the last successful build. But they aren’t leveraged because the $HOME/.m2/ (where Java expects to be able to read and write its dependencies) is nearly empty.

My builds work (as in my sites are put live correctly), but I suspect that a bit of unnecessary build minutes is wasted due to the cache not being leveraged.

I am almost certain caching of Java dependencies used to work.

Did my elaboration paint a clearer picture of my issue?

Hi,

Could try modifying your build process to copy the contents of the $NETLIFY_BUILD_BASE/cache/.m2 directory to the $HOME/.m2 directory before running the build? If the subsequent build completes faster, then it’s likely that Netlify is using a separate cache for each build.

I’ve added the following to my build script:

rm -rf $HOME/.m2/
cp -a $NETLIFY_BUILD_BASE/cache/.m2/ $HOME/.m2/

Before the above change builds would take somewhere around: 1m 24s to 1m 30s

With the above, a build now takes around 1m 15s

Any reason why Netlify doesn’t populate the “Java cache” automatically?

From my understanding it’s a risk to include cache files from a previous build, so we build in a new environment every time. You can configure your build though to cache certain files every time but that’s up to you.

Awesome, thanks for elaborating! :heart:

1 Like