Permission denied: Cannot modify node_modules folder for monorepo caching

Support Build Plugins
, ,
#1

Site Name: air-dev

Since Netlify doesn’t seem to support node_modules caching for monorepos out of the box, we are trying to take the clever approach over at https://github.com/slawekkolodziej/netlify-lerna-caching-demo to make this work.

The approach is to save the sub-packages node_modules in the root node_modules after the build has completed so you can read from them on the next build (via preinstall) and copy them into their respective packages.

However, we are running into a permission issue when we try to mv the sub-package node_modules to their respective packages. Why are permissions disabled for moving, copying, editing the node_modules folder?

Here is our preinstall script:

# https://github.com/slawekkolodziej/netlify-lerna-caching-demo
#!/bin/sh
NODE_MODULES_CACHE=node_modules
PACKAGES_CACHE="$NODE_MODULES_CACHE/packages-cache"

mkdir -p "$PACKAGES_CACHE"

restore_deps() {
    PACKAGES=$(ls -1 $1)

    for PKG in $PACKAGES
    do
        PKG_CACHE="$PACKAGES_CACHE/$PKG"
        ls $PKG_NODE_MODULES
        if [ -d $PKG_CACHE ];
        then
            mv $PKG_CACHE $1/$PKG/node_modules
            echo "Restored node modules for $PKG"
        else
            echo "Unable to restore cache for $PKG"
        fi
    done
}

restore_deps packages
#2

Hey @dwilt4rville,
Thanks for your patience on this! We’d suggest, rather than relying on a specific implementation of the buildbot, checking out the cache utility of our Build Plugins:

You might want to check out this recently contributed plugin:

and other cache-related plugins:
https://app.netlify.com/teams/YOUR-TEAM-NAME/plugins?search=cache

Do you want to investigate some of those and let us know if you have follow-up questions?

#3

Hey Jen,
Build plugins don’t get initialized until after the yarn install step that you guys run on builds automatically completes. And since we’re trying to cache the node_modules for all of the subpackages in our monorepo, this doesn’t work. Let me try and reframe the question:

Is there a folder that we can put files/folders into after our build has completed that will be brought back/cached on the next build that we can read those files from?

#4

Hey @dwilt4rville, you can actually run build plugins at a multitude of different phases, including onPreInstall :+1:

#5

I don’t see a onPreInstall hook anywhere in the documentation for the build plugins

#6

Also, came across this: Plugin pre-install steps

#7

There also seems to be an open issue on GH: https://github.com/netlify/build/issues/1149

#8

hey @dwilt4rville - i moved this post to the build plugins area as this seems more appropriate.

#9

Apologies, I was crossing my wires. I did mean onPreBuild though I think you’re right; this may be a little too late.