Using pnpm and pnpm workspaces

[stefanmaric]

Just as an informative note for future users, here some info on how to workaround Netlify’s CI in order to use pnpm instead of npm or yarn and pnpm workspaces.

1. Remove your package-lock.json or yarn.lock file and add it to your .gitignore, if applicable. Be sure to track pnpm-lock.yaml instead.

2. If you’re using workspaces: add pnpm itself as a dev dependency on your top-level project, this so it is possible to use pnpm recursive commands inside your build scripts

   pnpm install --save-dev pnpm@3
   

3. Add the NPM_FLAGS="--prefix=/" env var to your netlify.toml or set it on Netlify’s web dashboard

   [build.environment]
       NPM_FLAGS="--prefix=/"
   

   This makes netlify skip their automatic npm install, to let pnpm take care of it.

4. Add pnpm install as a build step in your build process, only for Netlify, most probably before anything else. Assuming your build command is npm run build , you might add the following script into your top-level package.json:

   "prebuild": "[[ $CI = true ]] && npx pnpm@3 install -r --store=node_modules/.pnpm-store || echo skiping pnpm install"
   

   Use the -r option only if you’re using workspaces. The --store option tells pnpm where to save its “global” cache. I initially tried to save it to the netlify cache dir, but netlify is picky on what saves in cache for the next build. Even when I was able to make netlify save the pnpm cache, the install was not very efficient because pnpm had to resort to the copy strategy because netlify mounts the cache folder in a different drive, so symlinking is not possible. Saving the store into node_modules makes the install quite fast.

We have been using this setup in prod without issues for a while now. And unlike with yarn, we haven’t experienced issues with missing node_modules inside workspaces, since pnpm stores every dep inside the top-level node_modules (which is the only Netlify saves for cache) and pnpm install simply creates symlinks.

[perry]

thank you for sharing this!! this is very useful for people!

[TomPichaud]

Hello,

Thank you for sharing this but it seems to be a bit outdated now.

It seems like Netlify should handle pnpm automatically now

Add support for `pnpm` + Fix issue with `gridsome`detector

But I can’t manage to get it to work, do you?

[stefanmaric]

Hey @TomPichaud

Looking at the linked PR, it seems the author pulled out the pnpm bits from the PR but didn’t change the title/commit message.

Also, proper support for pnpm should be added in the build image repo instead of the cli one. I’ve looked up and it seems there’s still no support for pnpm.

To be fair, adding direct support for pnpm workspaces caching, specifically, in the way npm and yarn are supported, is a bit complicated. I tried such route myself in the past, before settling on this solution.

I haven’t looked into it, but I guess it is now possible to add pnpm support via the new, beta API for build plugins.

I’ve actually migrated my projects using pnpm workspaces out of netlify about a week ago, but the workaround I initially posted was still in use, and it worked perfectly: ~200 builds a month for almost a year without issues. So I would recommend you going with it.

A clean pnpm install (no cache at all) on netlify’s build servers for a project with 9 workspaces and ~250 direct dependencies takes about 50 seconds and re-linking those dependencies from cache on subsequent builds takes just 1 second.

I have to say pnpm is pretty neat; I haven’t found a better alternative for dependency management on monorepos.

Hope that helps you.

[stefanmaric]

I cannot edit my initial post, so pushing an update as a comment:

Remove the @3 parts from the commands, so 1) you get the latest pnpm version and 2) it actually saves you a few seconds when building from cache because if you specify the version, npx always fetches such version from the registry even if the package is available locally in node_modules/.

2nd update: you can skip the -r flag (--recursive) as well since, starting with pnpm@4, recursive is the default behavior for workspaces.

[jcayzac]

One word of caution: Netlify only restores node_modules if there’s a root-level package.json (they do persist the folder to the cache either way), so you may have to restore the folder yourself if using pnpm workspaces and don’t have a root-level package.json, otherwise pnpm will download all modules every build.

[jcayzac]

FWIW, I sent a PR.

[sklar]

It seems like disabling the npm install by setting the NPM_FLAGS="--prefix=/" doesn’t work anymore
The operation was rejected by your operating system

[stefanmaric]

@sklar

It seems like disabling the npm install by setting the NPM_FLAGS="--prefix=/" doesn’t work anymore

@sklar it seems that either npm v6 added some permission checks or the Netlify build image changed the build user’s groups.

I tried with /dev/null and it seems to be working:

[build.environment]
    NPM_FLAGS="--prefix=/dev/null"

I made a test repo for it since I’m not really hosting those projects I was using pnpm with at Netlify anymore. You might see it here: https://github.com/stefanmaric/test-pnpm-on-netlify

Another thing I noticed, is that either pnpm or Netlify changed the default POSIX shell the package.json scripts uses, therefore the test command was failing on me. So update:

"prebuild": "[[ $CI = true ]] && npx pnpm install --store=node_modules/.pnpm-store || echo skiping pnpm install"

With:

"prebuild": "test \"$CI\" = true && npx pnpm install -r --store=node_modules/.pnpm-store || echo skiping pnpm install",

Hope that helps.

[perry]

thank you for sharing this!

[mudlabs]

It’s not working

Netlify log errors from:

$ pnpm run build
bash: pnpm: command not found
 ​
┌─────────────────────────────┐
│   "build.command" failed    │
└─────────────────────────────┘
​
Error message
Command failed with exit code 127: pnpm run build
​
Error location
In build.command from netlify.toml:
pnpm run build

Resolved config
build:
command: pnpm run build
commandOrigin: config
environment:
- NPM_FLAGS
publish: /opt/build/repo/.dist

My netlify.toml is:

[build]
    publish = ".dist"
    command = "pnpm run build"

[build.environment]
    NPM_FLAGS="--prefix=/dev/null"

My package.json scripts are:

"scripts": {
    "prebuild": "test \"$CI\" = true && npx pnpm install -r --store=node_modules/.pnpm-store || echo skipping pnpm install",
    "build": "gulp build --dir .dist",
    "dev": "gulp dev --dir .dev"
  }

[mudlabs]

I had to update my build command in netlify.toml

[build]
    publish = ".dist"
    command = "pnpm run build || ( npm install pnpm && pnpm run build )"

[build.environment]
    NPM_FLAGS = "--prefix=/dev/null"

Of course that created a package-lock.json file, which we don’t want or need, because pnpm will create pnpm-lock.yaml file. To fix that, create an .npmrc file in the repo root, and add the line package-lock=false. Then add package-lock.json to .gitignore.

I do get a warning after the prebuild cammand is called. But everything seems to be working so… whatever.

$ pnpm run build || npm install pnpm && pnpm run build
> mudlify@1.0.0 prebuild /opt/build/repo
> test "$CI" = true && npx pnpm install -r --store=node_modules/.pnpm-store || echo skiping pnpm install
|  WARN  A pnpm-lock.yaml file exists. The current configuration prohibits to read or write a lockfile

[stefanmaric]

@mudlabs I believe you don’t really need to use pnpm as the build command, you can keep npm run build and use any CLI tools you need inside the package.json scripts, provided they were installed in your workspace, as seen here in the test repo I provided: https://github.com/stefanmaric/test-pnpm-on-netlify/blob/main/package.json