[Support Guide] Using private NPM modules on Netlify

Last reviewed by Netlify Support: December 2022

You want to build your site on Netlify but you have some private npm modules served from npmjs.org, and now you’re wondering how you can access them during build?

The most straightforward way to do this is two steps:

  1. Find your token inside the ~/.npmrc file in your home folder and set it as an NPM_TOKEN environment variable through Netlify’s admin UI. This will be on the Build & Deploy settings page of your site configuration, in the section called "Environment Variables”:

  1. If you don’t have an existing /.npmrc file for your project, our system will automatically generate one with the necessary line. If you already have an /.npmrc file for your project, add the following line manually (and commit it to Git):

//registry.npmjs.org/:_authToken=${NPM_TOKEN}

That should let us automatically pull the module from npm using your token, and your token does NOT need to be committed to your repo. If you’d rather have it there than in Netlify for some reason, that’s also functional :slight_smile: If you store the token at Netlify, the only people who can see it are people who you’ve invited to your Netlify account. Alternately, if you store it in your git repo, it will be visible to everyone who can see your repo.

Some other related information on the topic can be found in the following threads:

  1. This thread from StackOverflow talks about how you can set the environment variable so npmrc can use it. This might explain why your .env file won’t “just work” in this case.
  2. If you need to set up access to something OTHER than a module to fetch from npm, you might instead find this post in our support forums to be useful. It describes accessing secondary repositories during build.
7 Likes

Unfortunately, I couldn’t get this to work with a local project .npmrc and an environment variable. It did, however work, if I inline the environment variable into the committed .npmrc file.

I was also trying to get it to work by creating a npmrc on the fly during build, but that also wasn’t working. Would love to figure out a way to get this working without committing creds to source control.

Hmm, what happened when you tried locally? Perhaps there was a typo in my spec. I don’t have a private npm module to test with, but the workflow has been handed out to a few dozen customers in the time I’ve worked here and none have reported trouble with it, so I think once you get it working locally it should work here too :slight_smile: This is the (local) test I’d do ($ is the prompt on my mac, the other commands can be cut and pasted into the SAME shell window as long as you run a BASH-like shell to use as is, with the exception of replacing NPM_TOKEN with your own token):

$ export NPM_TOKEN=00000-000-000-0000000
$ cd project/
$ echo "//registry.npmjs.org/:_authToken=${NPM_TOKEN}" >> .npmrc # don't clobber other existing settings
$ npm i

Apologies on the slow reply – missed the notification.

I don’t think it matters, but I’m actually trying to do this with bit.dev, which works the same way as any private npm registry. Here’s their docs on how to set this up, which is very similar to what you’re recommending here.

Locally, on my dev machine, this all works fine. I’m inlining the auth token in my machine-wide ~/.npmrc and the project local .npmrc, b/c the npm environment variable syntax is weird and breaks some stuff in my local project config.

Currently on netlify i’m running into some weird behavior where two identical environment configurations fail on one and succeed on another. Same codebase (staging/production for the same codebase). On one netlify environment yarn install succeeds, on the other it fails with a 401 for the private repo.

If there was a way to ssh into netlify I’d be able to diagnose the issue better, but afaik this is not possible.

I’m realizing now that the build is not even getting to the user build script – it’s failing when installing yarn:

10:14:17 AM: Build ready to start
10:14:20 AM: build-image version: 9e0f207a27642d0115b1ca97cd5e8cebbe492f63
10:14:20 AM: build-image tag: v3.3.2
10:14:20 AM: buildbot version: 75cd99f62ada9e21edea53208e8baf0eab85a045
10:14:20 AM: Fetching cached dependencies
10:14:21 AM: Starting to download cache of 255.1KB
10:14:21 AM: Finished downloading cache in 104.993978ms
10:14:21 AM: Starting to extract cache
10:14:21 AM: Failed to fetch cache, continuing with build
10:14:21 AM: Starting to prepare the repo for build
10:14:21 AM: No cached dependencies found. Cloning fresh repo
10:14:21 AM: git clone git@gitlab.com:[username]/[repo]
10:14:23 AM: Preparing Git Reference refs/heads/master
10:14:24 AM: Starting build script
10:14:24 AM: Installing dependencies
10:14:26 AM: Downloading and installing node v8.16.0...
10:14:26 AM: Downloading https://nodejs.org/dist/v8.16.0/node-v8.16.0-linux-x64.tar.xz...
10:14:26 AM: 
#
10:14:26 AM:   1.7%
10:14:26 AM: 
#####################
10:14:26 AM:  29.9%
10:14:26 AM: 
################
10:14:26 AM: #################################################         91.0%
10:14:26 AM: 
########################################################################
10:14:26 AM: 100.0%
10:14:27 AM: Computing checksum with sha256sum
10:14:27 AM: Checksums matched!
10:14:29 AM: Now using node v8.16.0 (npm v6.4.1)
10:14:29 AM: Attempting ruby version 2.3.6, read from environment
10:14:30 AM: 
10:14:30 AM: ** WARNING **
10:14:30 AM: Using custom ruby version 2.3.6, this will slow down the build.
10:14:30 AM: To ensure fast builds, set the RUBY_VERSION environment variable, or .ruby-version file, to an included ruby version.
10:14:30 AM: Included versions: 2.6.2
10:14:30 AM: 
10:14:30 AM: Required ruby-2.3.6 is not installed - installing.
10:14:31 AM: Searching for binary rubies, this might take some time.
10:14:31 AM: Found remote file https://rvm_io.global.ssl.fastly.net/binaries/ubuntu/16.04/x86_64/ruby-2.3.6.tar.bz2
10:14:31 AM: Checking requirements for ubuntu.
10:14:32 AM: Requirements installation successful.
10:14:32 AM: ruby-2.3.6 - #configure
10:14:32 AM: ruby-2.3.6 - #download
10:14:33 AM: ruby-2.3.6 - #validate archive
10:14:41 AM: ruby-2.3.6 - #extract
10:14:45 AM: ruby-2.3.6 - #validate binary
10:14:46 AM: ruby-2.3.6 - #setup
10:14:47 AM: ruby-2.3.6 - #gemset created /opt/buildhome/.rvm/gems/ruby-2.3.6@global
10:14:47 AM: ruby-2.3.6 - #importing gemset /opt/buildhome/.rvm/gemsets/global.gems
10:14:49 AM: ............................................................
10:14:49 AM: ruby-2.3.6 - #generating global wrappers
10:14:49 AM: .......
10:14:49 AM: ruby-2.3.6 - #gemset created /opt/buildhome/.rvm/gems/ruby-2.3.6
10:14:49 AM: ruby-2.3.6 - #importing gemsetfile /opt/buildhome/.rvm/gemsets/default.gems evaluated to empty gem list
10:14:49 AM: ruby-2.3.6 - #generating default wrappers
10:14:49 AM: .......
10:14:50 AM: Using /opt/buildhome/.rvm/gems/ruby-2.3.6
10:14:50 AM: Using ruby version 2.3.6
10:14:50 AM: Using PHP version 5.6
10:14:50 AM: Started restoring cached node modules
10:14:50 AM: Finished restoring cached node modules
10:14:50 AM: Started restoring cached yarn cache
10:14:50 AM: Finished restoring cached yarn cache
10:14:50 AM: Installing yarn at version 1.3.2
10:14:50 AM: Installing Yarn!
10:14:50 AM: > Downloading tarball...
10:14:50 AM: [1/2]: https://yarnpkg.com/downloads/1.3.2/yar
10:14:50 AM: n-v1.3.2.tar.gz --> /tmp/yarn.tar.gz.3XCWlG4Gdu
10:14:50 AM:   % Total    % Received % Xferd  Average
10:14:50 AM: Speed   Time    Time     Time  Current
10:14:50 AM:                       Dload  Upload   T
10:14:50 AM: otal   Spent    Left  Speed
10:14:50 AM: 
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:-
10:14:50 AM: -:--     0
10:14:51 AM: 
100    91  100    91    0     0    320      0 --:--:-- --:--:-- --:--:--
10:14:51 AM: 321
10:14:51 AM: 
100   608    0   608    0     0    955
10:14:51 AM:   0 --:--:-- --:--:-- --:--:--   955
10:14:51 AM: 
  0  865k    0     0    0     0      0      0 --:--:-- --:
10:14:51 AM: --:-- --:--:--     0
10:14:51 AM: 
100  865k  100  865k    0     0   796k      0  0:00:01  0:00:01
10:14:51 AM: --:--:-- 5549k
10:14:51 AM: [2/2]: https://yarnpkg
10:14:51 AM: .com/downloads/1.3.2/yarn-v1.3.2.tar.gz.asc --> /tmp/yarn.tar.gz.3XCWlG4Gdu.asc
10:14:51 AM: 
100    95  100    95    0     0   2247      0 --:--:-- --:--
10:14:51 AM: :-- --:--:--  2247
10:14:52 AM: 
100   612    0   612    0     0   3991      0 --:--:-- --:--:
10:14:52 AM: -- --:--:--  3991
10:14:52 AM: 
100  1027  100  1027    0     0   5327      0 --:--:-- --:
10:14:52 AM: --:-- --:--:--  5327
10:14:52 AM: > Verifying integrity...
10:14:52 AM: gpg: Signature made Thu 02 Nov 2017 04:44:10 PM UTC using RSA key ID FD2497F5
10:14:52 AM: gpg: Good signature from "Yarn Packaging <yarn@dan.cx>"
10:14:52 AM: gpg: Note: This key has expired!
10:14:52 AM: Primary key fingerprint: 72EC F46A 56B4 AD39 C907  BBB7 1646 B01B 86E5 0310
10:14:52 AM:      Subkey fingerprint: 6A01 0C51 6600 6599 AA17  F081 46C2 130D
10:14:52 AM:  FD24 97F5
10:14:52 AM: > GPG signature looks good
10:14:52 AM: > Extracting to ~/.yarn...
10:14:52 AM: > Adding to $PATH...
10:14:52 AM: > We've added the following to your /opt/buildhome/.profile
10:14:52 AM: > If this isn't the profile of your current shell then please add the following to your correct profile:
10:14:52 AM: export PATH="$HOME/.yarn/bin:$HOME/.config/yarn/global/node_modules/.bin:$PATH"
10:14:52 AM: 
10:14:52 AM: > Successfully installed Yarn 1.3.2! Please open another terminal where the `yarn` command will now be available.
10:14:53 AM: Installing NPM modules using Yarn version 1.3.2
10:14:53 AM: yarn install v1.3.2
10:14:54 AM: [1/4] Resolving packages...
10:14:54 AM: [2/4] Fetching packages...
10:14:56 AM: error An unexpected error occurred: "https://node.bit.dev/ajsharp.vinvin.booking-time-field/-/ajsharp.vinvin.booking-time-field-1.0.0.tgz: Request failed \"401 Unauthorized\"".
10:14:56 AM: info If you think this is a bug, please open a bug report with the information provided in "/opt/build/repo/booking-plugin/yarn-error.log".
10:14:56 AM: info Visit https://yarnpkg.com/en/docs/cli/install for documentation about this command.
10:15:07 AM: Error during Yarn install
10:15:07 AM: failed during stage 'building site': Build script returned non-zero exit code: 1
10:15:07 AM: Error running command: Build script returned non-zero exit code: 1
10:15:08 AM: Shutting down logging, 2 messages pending

Also, it’s probably worth mentioning that the failing environment is attempt to install and use yarn version 1.3.2, and one working is using yarn 1.13.0.

UPDATE:

Good news: I was able to resolve this by manually setting the yarn version to 1.13.0 by setting the YARN_VERSION environment variable to 1.13.0.

Not clear if this is a yarn bug or a netlify bug, but private npm registries seem broken with newer versions of yarn.

Hi @fool,

Thank you for your great explanation.
Unfortunately it does not work for my project which contains an Angular (v8) App with AOT compilation.
I created an npmrc file and added my npm token as a netlify environment variable but when deploying the app, it seems like the private packages can not be found which is why the deployment fails:

2:50:10 PM: Build ready to start
2:50:15 PM: build-image version: 9e0f207a27642d0115b1ca97cd5e8cebbe492f63
2:50:15 PM: build-image tag: v3.3.2
2:50:15 PM: buildbot version: ef8d0929ed0baabafd8bbb7d0b021e1fc24180c0
2:50:15 PM: Fetching cached dependencies
2:50:15 PM: Starting to download cache of 162.5MB
2:50:16 PM: Finished downloading cache in 1.247109651s
2:50:16 PM: Starting to extract cache
2:50:24 PM: Finished extracting cache in 7.700806118s
2:50:24 PM: Finished fetching cache in 8.991370823s
2:50:24 PM: Starting to prepare the repo for build
2:50:25 PM: Preparing Git Reference refs/heads/master
2:50:25 PM: Starting build script
2:50:25 PM: Installing dependencies
2:50:26 PM: Started restoring cached node version
2:50:28 PM: Finished restoring cached node version
2:50:29 PM: v10.16.0 is already installed.
2:50:30 PM: Now using node v10.16.0 (npm v6.9.0)
2:50:30 PM: Attempting ruby version 2.6.2, read from environment
2:50:31 PM: Using ruby version 2.6.2
2:50:32 PM: Using PHP version 5.6
2:50:32 PM: Started restoring cached node modules
2:50:32 PM: Finished restoring cached node modules
2:50:32 PM: Installing NPM modules using NPM version 6.9.0
2:51:20 PM: > core-js@2.6.9 postinstall /opt/build/repo/node_modules/babel-runtime/node_modules/core-js
2:51:20 PM: > node scripts/postinstall || echo "ignore"
2:51:20 PM: Thank you for using core-js ( https://github.com/zloirock/core-js ) for polyfilling JavaScript standard library!
2:51:20 PM: The project needs your help! Please consider supporting of core-js on Open Collective or Patreon: 
2:51:20 PM: > https://opencollective.com/core-js 
2:51:20 PM: > https://www.patreon.com/zloirock 
2:51:20 PM: Also, the author of core-js ( https://github.com/zloirock ) is looking for a good job -)
2:51:20 PM: > core-js@2.6.9 postinstall /opt/build/repo/node_modules/karma/node_modules/core-js
2:51:20 PM: > node scripts/postinstall || echo "ignore"
2:51:20 PM: Thank you for using core-js ( https://github.com/zloirock/core-js ) for polyfilling JavaScript standard library!
2:51:20 PM: The project needs your help! Please consider supporting of core-js on Open Collective or Patreon: 
2:51:20 PM: > https://opencollective.com/core-js 
2:51:20 PM: > https://www.patreon.com/zloirock 
2:51:20 PM: Also, the author of core-js ( https://github.com/zloirock ) is looking for a good job -)
2:51:20 PM: > @angular/cli@8.0.6 postinstall /opt/build/repo/node_modules/@angular/cli
2:51:20 PM: > node ./bin/postinstall/script.js
2:51:21 PM: > core-js@3.1.4 postinstall /opt/build/repo/node_modules/core-js
2:51:21 PM: > node scripts/postinstall || echo "ignore"
2:51:21 PM: Thank you for using core-js ( https://github.com/zloirock/core-js ) for polyfilling JavaScript standard library!
2:51:21 PM: The project needs your help! Please consider supporting of core-js on Open Collective or Patreon: 
2:51:21 PM: > https://opencollective.com/core-js 
2:51:21 PM: > https://www.patreon.com/zloirock 
2:51:21 PM: Also, the author of core-js ( https://github.com/zloirock ) is looking for a good job -)
2:51:22 PM: npm notice created a lockfile as package-lock.json. You should commit this file.
2:51:22 PM: npm
2:51:22 PM: WARN optional SKIPPING OPTIONAL DEPENDENCY: fsevents@1.2.9 (node_modules/fsevents):
2:51:22 PM: npm WARN notsup SKIPPING OPTIONAL DEPENDENCY: Unsupported platform for fsevents@1.2.9: wanted {"os":"darwin","arch":"any"} (current: {"os":"linux","arch":"x64"})
2:51:22 PM: added 101 packages from 65 contributors, removed 171 packages, updated 236 packages, moved 7 packages and audited 19366 packages in 48.976s
2:51:22 PM: found 0 vulnerabilities
2:51:22 PM: NPM modules installed
2:51:22 PM: Started restoring cached go cache
2:51:22 PM: Finished restoring cached go cache
2:51:22 PM: unset GOOS;
2:51:22 PM: unset GOARCH;
2:51:22 PM: export GOROOT='/opt/buildhome/.gimme/versions/go1.12.linux.amd64';
2:51:22 PM: export PATH="/opt/buildhome/.gimme/versions/go1.12.linux.amd64/bin:${PATH}";
2:51:22 PM: go version >&2;
2:51:22 PM: export GIMME_ENV='/opt/buildhome/.gimme/env/go1.12.linux.amd64.env';
2:51:22 PM: go version go1.12 linux/amd64
2:51:22 PM: Installing missing commands
2:51:22 PM: Verify run directory
2:51:22 PM: Executing user command: npm run build-staging
2:51:23 PM: > done-webapp@0.0.0 build-staging /opt/build/repo
2:51:23 PM: > ng build --prod --aot --configuration=staging
2:51:51 PM: Date: 2019-07-02T12:51:51.687Z
2:51:51 PM: Hash: e1fd16cadfe7aff533f4
2:51:51 PM: Time: 23973ms
2:51:51 PM: chunk {0} runtime-es5.741402d1d47331ce975c.js (runtime) 1.41 kB [entry] [rendered]
2:51:51 PM: chunk {1} main-es5.4af9b61479361f268d39.js (main) 128 bytes [initial] [rendered]
2:51:52 PM: failed during stage 'building site': Build script returned non-zero exit code: 1
2:51:51 PM: chunk {2} polyfills-es5.5639a156d6529acb291d.js (polyfills) 68.1 kB [initial] [rendered]
2:51:51 PM: ERROR in src/app/platform/shared/pressure-canvas/pressure-canvas.component.ts(9,33): error TS2307: Cannot find module '@done/sensor-client'.
2:51:51 PM: src/app/platform/shared/pressure-canvas/pressure-canvas.component.ts(10,38): error TS2307: Cannot find module '@done/sensor-client-player'.
2:51:51 PM: src/app/platform/shared/pressure-canvas/pressure-canvas.component.ts(12,46): error TS2307: Cannot find module '@done/sensor-client-edit'.
2:51:51 PM: src/app/platform/shared/pressure-canvas/pressure-canvas.component.ts(13,24): error TS2307: Cannot find module '@done/sensor-client-viewer'.
2:51:51 PM: npm
2:51:51 PM:  ERR! code ELIFECYCLE
2:51:51 PM: npm
2:51:51 PM: ERR! errno 1
2:51:51 PM: npm
2:51:51 PM: ERR!
2:51:51 PM:  done-webapp@0.0.0 build-staging: `ng build --prod --aot --configuration=staging`
2:51:51 PM: npm
2:51:51 PM: ERR! Exit status 1
2:51:51 PM: npm
2:51:51 PM:  ERR!
2:51:51 PM: npm
2:51:51 PM: ERR! Failed at the done-webapp@0.0.0 build-staging script.
2:51:51 PM: npm
2:51:51 PM: ERR! This is probably not a problem with npm. There is likely additional logging output above.
2:51:52 PM: npm
2:51:52 PM:  ERR! A complete log of this run can be found in:
2:51:52 PM: npm ERR!     /opt/buildhome/.npm/_logs/2019-07-02T12_51_51_766Z-debug.log
2:51:52 PM: Skipping functions preparation step: no functions directory set
2:51:52 PM: Caching artifacts
2:51:52 PM: Started saving node modules
2:51:52 PM: Finished saving node modules

The four modules which can not be found are the private repos.
When I start the build on my personal machine, everything works as expected.

Thank you for your help.
Stefan

I don’t see the attempt to install those modules in your logs? They would need to be in package.json, and a failure to install them should produce an error BEFORE your build starts. Perhaps you have them set as devDependencies and you are using NODE_ENV=production or something? Also this workflow is for installing private NPM modules - not code from private repos, and you may have been using the term interchangeably but I wanted to make sure you were talking about the same thing as me :slight_smile:

1 Like

Hi @fool

Thanks for your response.
This is exactly what makes me wondering as well.
I do have the packages set in the dependencies and not in the devDependencies. And yes I’m talking about private npm modules and not private repos :slight_smile:

There is indeed an error if I remove the .npmrc which says that the four private npm modules can not be found on npm (because they’re private). That’s actually the reason why I got here and followed your advice :slight_smile:
So I guess there is an attempt to install those modules but it’s not shown? I’m not sure and stuck a little that’s why I’m reaching out to you.
Another thought that came in my mind is that the build has some issues with the Angular AOT compilation? So maybe it checks for the private modules before they have been installed?

Thanks a lot!

I just disabled the AOT compilation and tried the deployment again. But same error. So that’s not the reason I guess.

Ah, well, we won’t re-install the modules if they are already there in the site’s dependency cache from some previous build, and perhaps they are some of the sort that can be installed quietly and wouldn’t say anything. That seems like it was a wrong guess, sorry!

Next step for debugging would be to try to debug in our build environment run locally so you can look around in the filesystem in the build container. This shows how to do that:

1 Like

It’s probably worth mentioning that adding a .npmrc file to your repository breaks nvm locally as it can’t find the NPM_TOKEN environment variable.

It would be great if netlify had a way to either run a script prior to the npm install or just a way to add some repository authorization.

2 Likes

It works fine, if you define $NPM_TOKEN locally which we presumed would be your use case since you’d want the same access locally, right? Even if you don’t otherwise use it for some reason, you could still set it to allow one build script to rule them all :slight_smile:

We do have an open feature request around “do something before netlify installs deps” to which I’ve added this thread so we can notify you and others who might find it if that becomes a reality!

Usually the NPM_TOKEN or any other token on the local machine is configured on the machine .npmrc (~/.npmrc file).
Writing .npmrc locally in the project overrides the global definition.

The problem with setting tokens globally is that they might be prone to changes, which can take a while to figure out.

A local script before install solves this by creating the .npmrc file dynamically:
echo >>.npmrc

On a side issue - it is customary for CI machines to have an environment variable CI set to true, so actions can be executed based on CI. e.g. in the script above.

This is a big issue for us.

I understand you are concerned about two things, and while I can’t offer any relief today around npmrc except making sure that the person working on the retooling of our build environment sees this thread, there are at least many environment variables you can key off of to see that you are in the netlify CI:

Hey! I’m trying to use a package from github package registry. I’ve followed the instructions in this post, but I’m getting this error:

7:48:46 PM: ERR! code
7:48:46 PM:  E401
7:48:46 PM: npm
7:48:46 PM:  ERR! Unable to authenticate, need: Basic realm="GitHub Package Registry"

I’ve made sure that my token has read and write permissions for packages and also repo permissions.

My .npmrc looks like this:

//npm.pkg.github.com:_authToken=${GITHUB_PACKAGES_TOKEN}
@guayom:registry=https://npm.pkg.github.com

And GITHUB_PACKAGES_TOKEN is defined in my env vars

Actually, the repo is public. I just created it for learning purposes

I’ve been trying to publish for a couple hours now, but no luck.
Any ideas?

It seems that the token doesn’t have enough permissions: Basic realm="GitHub Package Registry". Can you share the netlify site that you are trying to deploy that repo on?

Yeah, it is this one: naughty-kare-e205e8
It hasn’t been deployed yet

I can get this working with the NPM_TOKEN env variable and the .npmrc file, but it’s not ideal, and here’s why:

Sometimes I don’t want to create a .npmrc file and push it to git - in this case I’m out of options.
Now.sh implemented this by also exposing a NPM_RC env variable, where you can put the whole contents of the file. This is nice because I don’t want devs to have .npmrc in their repo when they check out the repo - I just want the build server to have a special npm configuration via the NPM_RC variable.

Any chance we could see this implemented? Thanks a lot!

1 Like

You can certainly create that workflow today:

  • wire things so that our automatic installation of things doesn’t happen (maybe add an NPM_FLAGS of --dry-run ?)
  • have your build check for a variable you set only at netlify (or you can just check for $NETLIFY or other variables we set at build time: Build environment variables | Netlify Docs
  • use that to “gate” your build on first constructing .npmrc correctly based on variables in the account, or use the default settings instead if it is a local build.

But to your point of making the build more configurable, that is the goal of our new build plugins system: Netlify Build Plugins: add powerful capabilities to every build where you can more easily make your dreams a reality and I think that is the best path to get things working to match your precise needs, since we won’t make very big changes to our existing production (soon to be legacy) less configurable build system.

1 Like