You want to build your site on Netlify but you have some private npm modules served from npmjs.org, and now you’re wondering how you can access them during build?
The most straightforward way to do this is two steps:
Find your token inside the ~/.npmrc file in your home folder and set it as an NPM_TOKEN environment variable through Netlify’s admin UI. This will be on the Build & Deploy settings page of your site configuration, in the section called "Environment Variables”:
If you don’t have an existing /.npmrc file for your project, our system will automatically generate one with the necessary line. If you already have an /.npmrc file for your project, add the following line manually (and commit it to Git):
//registry.npmjs.org/:_authToken=${NPM_TOKEN}
That should let us automatically pull the module from npm using your token, and your token does NOT need to be committed to your repo. If you’d rather have it there than in Netlify for some reason, that’s also functional If you store the token at Netlify, the only people who can see it are people who you’ve invited to your Netlify account. Alternately, if you store it in your git repo, it will be visible to everyone who can see your repo.
Some other related information on the topic can be found in the following threads:
If you need to set up access to something OTHER than a module to fetch from npm, you might instead find this post from our forums to be useful. It describes accessing secondary repositories during build.
Unfortunately, I couldn’t get this to work with a local project .npmrc and an environment variable. It did, however work, if I inline the environment variable into the committed .npmrc file.
I was also trying to get it to work by creating a npmrc on the fly during build, but that also wasn’t working. Would love to figure out a way to get this working without committing creds to source control.
Hmm, what happened when you tried locally? Perhaps there was a typo in my spec. I don’t have a private npm module to test with, but the workflow has been handed out to a few dozen customers in the time I’ve worked here and none have reported trouble with it, so I think once you get it working locally it should work here too This is the (local) test I’d do ($ is the prompt on my mac, the other commands can be cut and pasted into the SAME shell window as long as you run a BASH-like shell to use as is, with the exception of replacing NPM_TOKEN with your own token):
$ export NPM_TOKEN=00000-000-000-0000000
$ cd project/
$ echo "//registry.npmjs.org/:_authToken=${NPM_TOKEN}" >> .npmrc # don't clobber other existing settings
$ npm i
Apologies on the slow reply – missed the notification.
I don’t think it matters, but I’m actually trying to do this with bit.dev, which works the same way as any private npm registry. Here’s their docs on how to set this up, which is very similar to what you’re recommending here.
Locally, on my dev machine, this all works fine. I’m inlining the auth token in my machine-wide ~/.npmrc and the project local .npmrc, b/c the npm environment variable syntax is weird and breaks some stuff in my local project config.
Currently on netlify i’m running into some weird behavior where two identical environment configurations fail on one and succeed on another. Same codebase (staging/production for the same codebase). On one netlify environment yarn install succeeds, on the other it fails with a 401 for the private repo.
If there was a way to ssh into netlify I’d be able to diagnose the issue better, but afaik this is not possible.
Also, it’s probably worth mentioning that the failing environment is attempt to install and use yarn version 1.3.2, and one working is using yarn 1.13.0.
UPDATE:
Good news: I was able to resolve this by manually setting the yarn version to 1.13.0 by setting the YARN_VERSION environment variable to 1.13.0.
Not clear if this is a yarn bug or a netlify bug, but private npm registries seem broken with newer versions of yarn.
Thank you for your great explanation.
Unfortunately it does not work for my project which contains an Angular (v8) App with AOT compilation.
I created an npmrc file and added my npm token as a netlify environment variable but when deploying the app, it seems like the private packages can not be found which is why the deployment fails:
2:50:10 PM: Build ready to start
2:50:15 PM: build-image version: 9e0f207a27642d0115b1ca97cd5e8cebbe492f63
2:50:15 PM: build-image tag: v3.3.2
2:50:15 PM: buildbot version: ef8d0929ed0baabafd8bbb7d0b021e1fc24180c0
2:50:15 PM: Fetching cached dependencies
2:50:15 PM: Starting to download cache of 162.5MB
2:50:16 PM: Finished downloading cache in 1.247109651s
2:50:16 PM: Starting to extract cache
2:50:24 PM: Finished extracting cache in 7.700806118s
2:50:24 PM: Finished fetching cache in 8.991370823s
2:50:24 PM: Starting to prepare the repo for build
2:50:25 PM: Preparing Git Reference refs/heads/master
2:50:25 PM: Starting build script
2:50:25 PM: Installing dependencies
2:50:26 PM: Started restoring cached node version
2:50:28 PM: Finished restoring cached node version
2:50:29 PM: v10.16.0 is already installed.
2:50:30 PM: Now using node v10.16.0 (npm v6.9.0)
2:50:30 PM: Attempting ruby version 2.6.2, read from environment
2:50:31 PM: Using ruby version 2.6.2
2:50:32 PM: Using PHP version 5.6
2:50:32 PM: Started restoring cached node modules
2:50:32 PM: Finished restoring cached node modules
2:50:32 PM: Installing NPM modules using NPM version 6.9.0
2:51:20 PM: > core-js@2.6.9 postinstall /opt/build/repo/node_modules/babel-runtime/node_modules/core-js
2:51:20 PM: > node scripts/postinstall || echo "ignore"
2:51:20 PM: Thank you for using core-js ( https://github.com/zloirock/core-js ) for polyfilling JavaScript standard library!
2:51:20 PM: The project needs your help! Please consider supporting of core-js on Open Collective or Patreon:
2:51:20 PM: > https://opencollective.com/core-js
2:51:20 PM: > https://www.patreon.com/zloirock
2:51:20 PM: Also, the author of core-js ( https://github.com/zloirock ) is looking for a good job -)
2:51:20 PM: > core-js@2.6.9 postinstall /opt/build/repo/node_modules/karma/node_modules/core-js
2:51:20 PM: > node scripts/postinstall || echo "ignore"
2:51:20 PM: Thank you for using core-js ( https://github.com/zloirock/core-js ) for polyfilling JavaScript standard library!
2:51:20 PM: The project needs your help! Please consider supporting of core-js on Open Collective or Patreon:
2:51:20 PM: > https://opencollective.com/core-js
2:51:20 PM: > https://www.patreon.com/zloirock
2:51:20 PM: Also, the author of core-js ( https://github.com/zloirock ) is looking for a good job -)
2:51:20 PM: > @angular/cli@8.0.6 postinstall /opt/build/repo/node_modules/@angular/cli
2:51:20 PM: > node ./bin/postinstall/script.js
2:51:21 PM: > core-js@3.1.4 postinstall /opt/build/repo/node_modules/core-js
2:51:21 PM: > node scripts/postinstall || echo "ignore"
2:51:21 PM: Thank you for using core-js ( https://github.com/zloirock/core-js ) for polyfilling JavaScript standard library!
2:51:21 PM: The project needs your help! Please consider supporting of core-js on Open Collective or Patreon:
2:51:21 PM: > https://opencollective.com/core-js
2:51:21 PM: > https://www.patreon.com/zloirock
2:51:21 PM: Also, the author of core-js ( https://github.com/zloirock ) is looking for a good job -)
2:51:22 PM: npm notice created a lockfile as package-lock.json. You should commit this file.
2:51:22 PM: npm
2:51:22 PM: WARN optional SKIPPING OPTIONAL DEPENDENCY: fsevents@1.2.9 (node_modules/fsevents):
2:51:22 PM: npm WARN notsup SKIPPING OPTIONAL DEPENDENCY: Unsupported platform for fsevents@1.2.9: wanted {"os":"darwin","arch":"any"} (current: {"os":"linux","arch":"x64"})
2:51:22 PM: added 101 packages from 65 contributors, removed 171 packages, updated 236 packages, moved 7 packages and audited 19366 packages in 48.976s
2:51:22 PM: found 0 vulnerabilities
2:51:22 PM: NPM modules installed
2:51:22 PM: Started restoring cached go cache
2:51:22 PM: Finished restoring cached go cache
2:51:22 PM: unset GOOS;
2:51:22 PM: unset GOARCH;
2:51:22 PM: export GOROOT='/opt/buildhome/.gimme/versions/go1.12.linux.amd64';
2:51:22 PM: export PATH="/opt/buildhome/.gimme/versions/go1.12.linux.amd64/bin:${PATH}";
2:51:22 PM: go version >&2;
2:51:22 PM: export GIMME_ENV='/opt/buildhome/.gimme/env/go1.12.linux.amd64.env';
2:51:22 PM: go version go1.12 linux/amd64
2:51:22 PM: Installing missing commands
2:51:22 PM: Verify run directory
2:51:22 PM: Executing user command: npm run build-staging
2:51:23 PM: > done-webapp@0.0.0 build-staging /opt/build/repo
2:51:23 PM: > ng build --prod --aot --configuration=staging
2:51:51 PM: Date: 2019-07-02T12:51:51.687Z
2:51:51 PM: Hash: e1fd16cadfe7aff533f4
2:51:51 PM: Time: 23973ms
2:51:51 PM: chunk {0} runtime-es5.741402d1d47331ce975c.js (runtime) 1.41 kB [entry] [rendered]
2:51:51 PM: chunk {1} main-es5.4af9b61479361f268d39.js (main) 128 bytes [initial] [rendered]
2:51:52 PM: failed during stage 'building site': Build script returned non-zero exit code: 1
2:51:51 PM: chunk {2} polyfills-es5.5639a156d6529acb291d.js (polyfills) 68.1 kB [initial] [rendered]
2:51:51 PM: ERROR in src/app/platform/shared/pressure-canvas/pressure-canvas.component.ts(9,33): error TS2307: Cannot find module '@done/sensor-client'.
2:51:51 PM: src/app/platform/shared/pressure-canvas/pressure-canvas.component.ts(10,38): error TS2307: Cannot find module '@done/sensor-client-player'.
2:51:51 PM: src/app/platform/shared/pressure-canvas/pressure-canvas.component.ts(12,46): error TS2307: Cannot find module '@done/sensor-client-edit'.
2:51:51 PM: src/app/platform/shared/pressure-canvas/pressure-canvas.component.ts(13,24): error TS2307: Cannot find module '@done/sensor-client-viewer'.
2:51:51 PM: npm
2:51:51 PM: ERR! code ELIFECYCLE
2:51:51 PM: npm
2:51:51 PM: ERR! errno 1
2:51:51 PM: npm
2:51:51 PM: ERR!
2:51:51 PM: done-webapp@0.0.0 build-staging: `ng build --prod --aot --configuration=staging`
2:51:51 PM: npm
2:51:51 PM: ERR! Exit status 1
2:51:51 PM: npm
2:51:51 PM: ERR!
2:51:51 PM: npm
2:51:51 PM: ERR! Failed at the done-webapp@0.0.0 build-staging script.
2:51:51 PM: npm
2:51:51 PM: ERR! This is probably not a problem with npm. There is likely additional logging output above.
2:51:52 PM: npm
2:51:52 PM: ERR! A complete log of this run can be found in:
2:51:52 PM: npm ERR! /opt/buildhome/.npm/_logs/2019-07-02T12_51_51_766Z-debug.log
2:51:52 PM: Skipping functions preparation step: no functions directory set
2:51:52 PM: Caching artifacts
2:51:52 PM: Started saving node modules
2:51:52 PM: Finished saving node modules
The four modules which can not be found are the private repos.
When I start the build on my personal machine, everything works as expected.
I don’t see the attempt to install those modules in your logs? They would need to be in package.json, and a failure to install them should produce an error BEFORE your build starts. Perhaps you have them set as devDependencies and you are using NODE_ENV=production or something? Also this workflow is for installing private NPM modules - not code from private repos, and you may have been using the term interchangeably but I wanted to make sure you were talking about the same thing as me
Thanks for your response.
This is exactly what makes me wondering as well.
I do have the packages set in the dependencies and not in the devDependencies. And yes I’m talking about private npm modules and not private repos
There is indeed an error if I remove the .npmrc which says that the four private npm modules can not be found on npm (because they’re private). That’s actually the reason why I got here and followed your advice
So I guess there is an attempt to install those modules but it’s not shown? I’m not sure and stuck a little that’s why I’m reaching out to you.
Another thought that came in my mind is that the build has some issues with the Angular AOT compilation? So maybe it checks for the private modules before they have been installed?
Ah, well, we won’t re-install the modules if they are already there in the site’s dependency cache from some previous build, and perhaps they are some of the sort that can be installed quietly and wouldn’t say anything. That seems like it was a wrong guess, sorry!
Next step for debugging would be to try to debug in our build environment run locally so you can look around in the filesystem in the build container. This shows how to do that:
It works fine, if you define $NPM_TOKEN locally which we presumed would be your use case since you’d want the same access locally, right? Even if you don’t otherwise use it for some reason, you could still set it to allow one build script to rule them all
We do have an open feature request around “do something before netlify installs deps” to which I’ve added this thread so we can notify you and others who might find it if that becomes a reality!
Usually the NPM_TOKEN or any other token on the local machine is configured on the machine .npmrc (~/.npmrc file).
Writing .npmrc locally in the project overrides the global definition.
The problem with setting tokens globally is that they might be prone to changes, which can take a while to figure out.
A local script before install solves this by creating the .npmrc file dynamically:
echo >>.npmrc
On a side issue - it is customary for CI machines to have an environment variable CI set to true, so actions can be executed based on CI. e.g. in the script above.
I understand you are concerned about two things, and while I can’t offer any relief today around npmrc except making sure that the person working on the retooling of our build environment sees this thread, there are at least many environment variables you can key off of to see that you are in the netlify CI:
It seems that the token doesn’t have enough permissions: Basic realm="GitHub Package Registry". Can you share the netlify site that you are trying to deploy that repo on?
I can get this working with the NPM_TOKEN env variable and the .npmrc file, but it’s not ideal, and here’s why:
Sometimes I don’t want to create a .npmrc file and push it to git - in this case I’m out of options.
Now.sh implemented this by also exposing a NPM_RC env variable, where you can put the whole contents of the file. This is nice because I don’t want devs to have .npmrc in their repo when they check out the repo - I just want the build server to have a special npm configuration via the NPM_RC variable.
Any chance we could see this implemented? Thanks a lot!
wire things so that our automatic installation of things doesn’t happen (maybe add an NPM_FLAGS of --dry-run ?)
have your build check for a variable you set only at netlify (or you can just check for $NETLIFY or other variables we set at build time: Build environment variables | Netlify Docs
use that to “gate” your build on first constructing .npmrc correctly based on variables in the account, or use the default settings instead if it is a local build.
But to your point of making the build more configurable, that is the goal of our new build plugins system: Netlify Build Plugins: add powerful capabilities to every build where you can more easily make your dreams a reality and I think that is the best path to get things working to match your precise needs, since we won’t make very big changes to our existing production (soon to be legacy) less configurable build system.