Modify package.json before npm install is run

Support
1

My site name is playbook-cobalt.jeffmcaleer.com

I need to modify package.json before npm install gets run. My package.json references a private repo that netlify doesn’t have access to, and I need to modify it to point to a local file version of that repo. I have written some javascript that will import package.json and package-lock.json and modify both files, but I can’t for the life of me figure out how to get Netlify to actually run my javascript before npm install. I tried a preinstall script, which doesn’t seem to run. I tried a custom Netlify plugin, but onPreBuild happens well after install. I don’t see any hooks that happen before installation.

I found this thread (Override npm install or execute a command before it) that suggests preinstall, but that doesn’t seem to be working for me. The preinstall script seems to run after install, and not before.

Am I going about this the entirely wrong way?

2

Duplication of Package.json preinstall not running on build

3

It was my intention not to create a duplicate but to ask a more open ended question. My initial question (which was receiving no responses), was very narrowly focused on a singular problem. I have re-asked the question in a way that clarifies my actual end goal and doesn’t focus on a single solution. It’s entirely possible preinstall isn’t the path at all for solving the problem, and that other Netlify functions are the proper way to handle this, which is why I reframed the question.

4

I’m staying out of this.

5

The community forum isn’t particularly active and isn’t a priority support channel for Netlify’s staff.

If your question cannot be answered by active community members you just have to wait.

Both posts seek to solve the same issue for you, so this thread could have been added as additional information/clarification to the original one.

We only point out that it’s effectively a “duplicate” so that nobody spends any time answering each question independently, and that those seeking an answer in the future can crawl the various links until they find one.

1 Like
6

Both posts seek to solve the same issue for you, so this thread could have been added as additional information/clarification to the original one.

That would not have solved the fact that the initial post asked the wrong question in the first place. It was a bad question, which I learned as I kept trying to solve the problem on my own instead of just sitting around waiting for a response. If people read the question title, decided they didn’t know the answer to that, and moved on, clarifications that it was the wrong question further down the page would not have helped.

I should have just deleted the original bad question. I apologize.

I don’t suppose you can actually help with the question?

7

You’ve made a bunch of assumptions here, many of which I know not to be true.

I read your original thread yesterday, I knew that it could be answered by “anything that would let you execute your own code prior to the npm install”), however I did not know a way to do that off the top of my head, so I did not respond.

Clarifying in the very next post would absolutely be read, in particular by the Netlify support staff member that eventually answers.

Creating new threads just causes what few people are here to need to answer multiple threads.

I can not, hence why your original thread was inactive, I guarantee you both I and @dig read it yesterday, and neither of us knew (or could quickly find) a confirmed answer, so left it for Netlify to answer.

1 Like
8

Well then I apologize for letting my experiences on other online help boards color my expectations here. Hopefully someone who can help will come along soon enough.

9

Note that, preinstall should work. That’s not a Netlify thing to change. It’s handled by npm - the same npm that you run locally. If your local npm runs preinstall, the same should happen on Netlify.

Except in the situation where we restore your node modules from our build cache. In that case, since no npm install is run (we run it if it’s needed, we skip it otherwise), preinstall would not run either.

However, it sounds like you’re reinventing the wheel, because the problem that you’re actually trying to solve already has a solution: [Support Guide] Using private NPM modules on Netlify

10

However, it sounds like you’re reinventing the wheel, because the problem that you’re actually trying to solve already has a solution: [Support Guide] Using private NPM modules on Netlify

Unfortunately It’s not a private NPM module that is accessible on the internet with a key, it’s literally a private repo that exists only on a VPN.

Note that, preinstall should work. That’s not a Netlify thing to change. It’s handled by npm - the same npm that you run locally. If your local npm runs preinstall, the same should happen on Netlify.

Except in the situation where we restore your node modules from our build cache. In that case, since no npm install is run (we run it if it’s needed, we skip it otherwise), preinstall would not run either.

This is the output I get when I submit my package json that references my private repo.

5:21:43 PM: Build ready to start
5:21:57 PM: build-image version: fcb0c1b3ada6d25c1cb58e8bc514f5f23cc14f15 (focal)
5:21:57 PM: buildbot version: ee2ef905a4bdb64f7733dd60a718aa00319e6f82
5:21:57 PM: Fetching cached dependencies
5:21:57 PM: Starting to download cache of 127.1MB
5:21:58 PM: Finished downloading cache in 1.054s
5:21:58 PM: Starting to extract cache
5:22:00 PM: Finished extracting cache in 1.904s
5:22:00 PM: Finished fetching cache in 3.013s
5:22:00 PM: Starting to prepare the repo for build
5:22:00 PM: Preparing Git Reference refs/heads/master
5:22:02 PM: Custom build path detected. Proceeding with the specified path: 'UI'
5:22:02 PM: Custom publish path detected. Proceeding with the specified path: 'UI/dist'
5:22:02 PM: manpath: warning: $PATH not set
5:22:03 PM: Starting to install dependencies
5:22:03 PM: Python version set to 3.8
5:22:04 PM: Attempting Ruby version 2.7.2, read from environment
5:22:04 PM: Using Ruby version 2.7.2
5:22:05 PM: Started restoring cached go cache
5:22:05 PM: Finished restoring cached go cache
5:22:06 PM: go version go1.19.13 linux/amd64
5:22:07 PM: Using PHP version 8.0
5:22:09 PM: Started restoring cached Node.js version
5:22:10 PM: Finished restoring cached Node.js version
5:22:10 PM: v20.12.2 is already installed.
5:22:11 PM: Now using node v20.12.2 (npm v10.5.0)
5:22:11 PM: Enabling Node.js Corepack
5:22:11 PM: Started restoring cached build plugins
5:22:11 PM: Finished restoring cached build plugins
5:22:11 PM: Started restoring cached corepack dependencies
5:22:11 PM: Finished restoring cached corepack dependencies
5:22:11 PM: No npm workspaces detected
5:22:11 PM: Started restoring cached node modules
5:22:11 PM: Finished restoring cached node modules
5:22:11 PM: Installing npm packages using npm version 10.5.0
5:23:22 PM: Failed during stage 'Install dependencies': dependency_installation script returned non-zero exit code: 1
5:23:22 PM: npm ERR! code ERR_SSL_SSLV3_ALERT_HANDSHAKE_FAILURE
5:23:22 PM: npm ERR! errno ERR_SSL_SSLV3_ALERT_HANDSHAKE_FAILURE
5:23:22 PM: npm ERR! request to https://gitlab.avispl.com/api/v4/projects/1423/packages/npm/@avispl/hermes-endpoint/-/@avispl/hermes-endpoint-2.4.0.tgz failed, reason: 4068042DCE7F0000:error:0A000410:SSL routines:ssl3_read_bytes:sslv3 alert handshake failure:../deps/openssl/openssl/ssl/record/rec_layer_s3.c:1590:SSL alert number 40
5:23:22 PM: npm ERR!
5:23:22 PM: npm ERR! A complete log of this run can be found in: /opt/buildhome/.npm/_logs/2024-04-16T21_22_11_702Z-debug-0.log
5:23:22 PM: Error during npm install
5:23:22 PM: Failing build: Failed to install dependencies

And this is the output I get when I manually patch it and check in a version that references local files instead of the private repo

2:39:02 PM: build-image version: 3d3c7e8b4321e2c1a54a2c4584fb46ba742b1630 (focal)
2:39:02 PM: buildbot version: 3d3c7e8b4321e2c1a54a2c4584fb46ba742b1630
2:39:02 PM: Fetching cached dependencies
2:39:02 PM: Starting to download cache of 127.3MB
2:39:03 PM: Finished downloading cache in 1.444s
2:39:03 PM: Starting to extract cache
2:39:05 PM: Finished extracting cache in 1.81s
2:39:05 PM: Finished fetching cache in 3.308s
2:39:05 PM: Starting to prepare the repo for build
2:39:06 PM: Preparing Git Reference refs/heads/master
2:39:07 PM: Custom build path detected. Proceeding with the specified path: 'UI'
2:39:07 PM: Custom publish path detected. Proceeding with the specified path: 'UI/dist'
2:39:08 PM: manpath: warning: $PATH not set
2:39:08 PM: Starting to install dependencies
2:39:09 PM: Python version set to 3.8
2:39:09 PM: Attempting Ruby version 2.7.2, read from environment
2:39:10 PM: Using Ruby version 2.7.2
2:39:10 PM: Started restoring cached go cache
2:39:10 PM: Finished restoring cached go cache
2:39:10 PM: go version go1.19.13 linux/amd64
2:39:11 PM: Using PHP version 8.0
2:39:13 PM: Started restoring cached Node.js version
2:39:14 PM: Finished restoring cached Node.js version
2:39:14 PM: v20.12.2 is already installed.
2:39:14 PM: Now using node v20.12.2 (npm v10.5.0)
2:39:14 PM: Enabling Node.js Corepack
2:39:15 PM: Started restoring cached build plugins
2:39:15 PM: Finished restoring cached build plugins
2:39:15 PM: Started restoring cached corepack dependencies
2:39:15 PM: Finished restoring cached corepack dependencies
2:39:15 PM: No npm workspaces detected
2:39:15 PM: Started restoring cached node modules
2:39:15 PM: Finished restoring cached node modules
2:39:15 PM: Installing npm packages using npm version 10.5.0
2:39:18 PM: > playbook-cobalt@1.0.0 preinstall
2:39:18 PM: > node setupEmulation.js
2:39:18 PM: Checking to see if any patching is needed for emulation mode
2:39:18 PM: Patching package.json
2:39:18 PM: package.json patching complete
2:39:18 PM: Patching package-lock.json
2:39:18 PM: package-lock.json patching complete
2:39:18 PM: up to date, audited 348 packages in 3s
2:39:18 PM: 56 packages are looking for funding
2:39:18 PM:   run `npm fund` for details
2:39:18 PM: 8 vulnerabilities (6 high, 2 critical)
2:39:18 PM: To address issues that do not require attention, run:
2:39:18 PM:   npm audit fix
2:39:18 PM: To address all issues possible (including breaking changes), run:
2:39:18 PM:   npm audit fix --force
2:39:18 PM: Some issues need review, and may require choosing
2:39:18 PM: a different dependency.
2:39:18 PM: Run `npm audit` for details.
2:39:18 PM: npm packages installed
2:39:19 PM: Successfully installed dependencies
2:39:19 PM: Starting build script
2:39:20 PM: Detected 1 framework(s)
2:39:20 PM: "vite" at version "4.5.3"
2:39:20 PM: Section completed: initializing

Note how in the second one the install happens first, and then the preinstall script runs (you can see it’s console logs). “Install” happens and takes 3 seconds, then it runs the preinstall script and says all packages are installed in the same timestamp.

Weirdly enough if I commit with the modified package.json and package-lock.json that reference the local file, it creates an accurate node_modules folder, and everything works, but if I revert that commit the issue returns. I had hoped that creating a functioning cached node_modules would help, but it doesn’t seem to have.

11

Oh okay, that’s yeah not something that could be easily possible to handle otherwise.

Could you see if the behaviour is same with clear cache and deploy? I would think not. What I think is happening currently that:

  • We restore node_modules from cache
  • See that package.json has changed and try installing the modules
  • Thus, your preinstall script might be running when the modules are already there

Clearing cache should ensure that there are no node_modules to restore so npm install would be run from scratch.

12 
9:17:32 AM: build-image version: 3d3c7e8b4321e2c1a54a2c4584fb46ba742b1630 (focal)
9:17:32 AM: buildbot version: 3d3c7e8b4321e2c1a54a2c4584fb46ba742b1630
9:17:32 AM: Building without cache
9:17:32 AM: Starting to prepare the repo for build
9:17:33 AM: No cached dependencies found. Cloning fresh repo
9:17:33 AM: git clone --filter=blob:none https://github.com/jeffderek/playbook-cobalt
9:17:33 AM: Preparing Git Reference refs/heads/master
9:17:35 AM: Custom build path detected. Proceeding with the specified path: 'UI'
9:17:35 AM: Custom publish path detected. Proceeding with the specified path: 'UI/dist'
9:17:35 AM: Starting to install dependencies
9:17:36 AM: Python version set to 3.8
9:17:36 AM: Attempting Ruby version 2.7.2, read from environment
9:17:36 AM: Using Ruby version 2.7.2
9:17:37 AM: Started restoring cached go cache
9:17:37 AM: Finished restoring cached go cache
9:17:37 AM: go version go1.19.13 linux/amd64
9:17:38 AM: Using PHP version 8.0
9:17:39 AM: Downloading and installing node v20.12.2...
9:17:39 AM: Downloading https://nodejs.org/dist/v20.12.2/node-v20.12.2-linux-x64.tar.xz...
9:17:40 AM: Computing checksum with sha256sum
9:17:40 AM: Checksums matched!
9:17:42 AM: Now using node v20.12.2 (npm v10.5.0)
9:17:42 AM: Enabling Node.js Corepack
9:17:42 AM: Started restoring cached build plugins
9:17:42 AM: Finished restoring cached build plugins
9:17:42 AM: Started restoring cached corepack dependencies
9:17:42 AM: Finished restoring cached corepack dependencies
9:17:42 AM: No npm workspaces detected
9:17:42 AM: Started restoring cached node modules
9:17:42 AM: Finished restoring cached node modules
9:17:42 AM: Installing npm packages using npm version 10.5.0
9:17:45 AM: npm WARN deprecated axios@0.19.2: Critical security vulnerability fixed in v0.21.1. For more information, see https://github.com/axios/axios/pull/3410
9:18:54 AM: Failed during stage 'Install dependencies': dependency_installation script returned non-zero exit code: 1
9:18:54 AM: npm ERR! code ERR_SSL_SSLV3_ALERT_HANDSHAKE_FAILURE
9:18:54 AM: npm ERR! errno ERR_SSL_SSLV3_ALERT_HANDSHAKE_FAILURE
9:18:54 AM: npm ERR! request to https://gitlab.avispl.com/api/v4/projects/1423/packages/npm/@avispl/hermes-endpoint/-/@avispl/hermes-endpoint-2.4.0.tgz failed, reason: 40C864B8D17F0000:error:0A000410:SSL routines:ssl3_read_bytes:sslv3 alert handshake failure:../deps/openssl/openssl/ssl/record/rec_layer_s3.c:1590:SSL alert number 40
9:18:54 AM: npm ERR!
9:18:54 AM: npm ERR! A complete log of this run can be found in: /opt/buildhome/.npm/_logs/2024-04-18T13_17_42_633Z-debug-0.log
9:18:54 AM: Error during npm install
9:18:54 AM: Failing build: Failed to install dependencies

Still doesn’t work. Cleared the cache, you can see that it tries to install packages and never prints any of the console log statements from my preinstall script, which leads me to believe it’s not being run yet.

13

This looks like an npm issue: [BUG] Preinstall script runs after installing dependencies · Issue #2660 · npm/cli (github.com). There are more people with a very similar use case as mentioned in this comment: [BUG] Preinstall script runs after installing dependencies · Issue #2660 · npm/cli · GitHub, so sadly this won’t work currently (unless, you use npm < 7).

14

Excellent find, thank you! That explains what is going on at least.

Is it possible to use npm < 7 on Netlify? I took a quick glance at the build config and didn’t see an option for npm version. Or is this something that might conceivably be fixed by using yarn? I’m relatively new to web development, coming from the back end, and I’ve never actually used a package manager other than npm.

Alternately, backing up a bit, is there any way to do anything after importing the repo from GitHub and before calling npm install?

15

You can set NPM_VERSION environment variable to any valid version.

As for this:

Unfortunately no. What you can do is:

  • Mark your dependency as optional, so npm install won’t fail even if that dependency fails to install.
  • Create a plugin to run before your build so that it can download the code and save it where it needs to be.
16

OK yes that sounds like it would work.

I’m reasonably sure all I need to do is copy the contents of \src\assets\emulation\hermes-endpoint\ to \node_modules\@avispl\hermes-endpoint\

I’ve got the installation script running properly with the optional dependency, and I have a hello world plugin running, but it’s not obvious to me from the plugin documentation how to copy/paste or just move files. I’ve searched here as well but am not coming up with much. Can you point me in the right direction for that?

Thanks for your help.

17

I have created a plugin that copies the files to node_modules where I expect them to be, and it seems to be succeeding, but the build is still failing. Investigating further.

EDIT: It was just bad javascript in my plugin, the core concept described above works

18

That worked! Thanks a ton. I still have some cleanup to do in my repo but the proof of concept is functioning. I appreciate you thinking outside the box and helping me get this working.

19

glad to hear it. thanks for writing back in. (: