NPM 7 and Individual Netlify TOML Files

Support
1

I was excited when I saw the new netlify mono repo support and the improved caching for yarn workspaces and decided to try it out with our large Lerna based mono repo that we have upgraded to NPM 7.

However there are quite a few things I haven’t been able to get working properly.

Here is a small example repo that I have been using to test

This repo consists of two react apps, one called @zgriesinger/react-app which is camelify.netlify.app in Netlify and one called @zgriesinger/react-app-two which is camelify-two.netlify.app in Netlify.

Each of these is an application that has config in a sub folder netlify.toml included specific environment variables for the version and a NPM_VERSION environment variable to set the proper npm version in Netlify.

To build one of these UIs it is just 2 simple commands

# npm i -g npm@7 if npm 7 not installed
npm i
npm run build -w @zgriesinger/react-app

Here are my issues:

camelify-two.netlify.app

Staring with the second one, this one is set up in Netlify with no base path meaning the sub folder apps/react-app-two/netlify.toml is not being respected (as showing an error when you load the page).
However once I updated the build config for this application to match the netlify.toml the build works as expected with great caching and quick build times. The problem here is that in our mono repo of 10+ apps and 10+ packages for those apps it gets very cumbersome to not manage the netlify config in source control. Not to mention the benefit of ignoring builds outside of the base path.

camelify.netlify.app

This is where I was trying to get it working while respecting the base path option and the sub folder netlify.toml.

First attempt was to follow the Netlify doc recommendations and set the build command to cd up to the proper directory.

build = "cd ../../../ && npm run build -w @zgriesinger/react-app"

This however did not work, since npm workspaces are not context aware (i.e. npm i from a subfolder does not work the same as from a root folder) the install step failed before even making it to my build script as it was unable to find the unpublished version of the internal package in the mono repo. (links removed from logs because I can only post 6)

12:41:46 PM: Installing NPM modules using NPM version 7.20.1
12:42:14 PM: npm ERR! code E404
12:42:14 PM: npm ERR! 404 Not Found - GET @zgriesinger%2fcamelify - Not found
12:42:14 PM: npm ERR! 404
12:42:14 PM: npm ERR! 404  '@zgriesinger/camelify@1.0.0' is not in the npm registry.
12:42:14 PM: npm ERR! 404 You should bug the author to publish it (or use the name yourself!

Second attempt was now adding in NPM_CONFIG_ONLY=development to the build environment and changing the build command to:

  command = "cd ../../ && npm -v && npm i && npm run build -w @zgriesinger/react-app"

This is where things started getting weird. Netlify’s install step finished properly and moved onto the build, however it seems like the NPM_VERSION parameter was not respected.

12:46:33 PM: Found npm version (6.14.13) that doesn't match expected (7.20.1)
Installing npm at version 7.20.1
12:46:38 PM: up to date in 4.081s
12:46:38 PM: NPM installed successfully
12:46:38 PM: Installing NPM modules using NPM version 6.14.13

After installing npm 7 it instantly goes back to npm 6 for the install step. After that the resulting npm version printed out by the build command is 6 and the build fails.

Finally, after setting the build command to install npm 7 manually, the build works.

  command = "cd ../../ && NPM_CONFIG_ONLY=null npm i -g npm@7 && npm -v  && NPM_CONFIG_ONLY=null npm i && npm run build -w @zgriesinger/react-app"

However, this incurs a significantly longer build time, as I believe it is breaking the cache, as well as duplicated the install and installing npm multiple times. It is almost triple that of camelify-two.

Questions

After going through this I have a few questions.

  1. Does (can?) Netlify support a INSTALL_DIRECTORY? That would solve almost every problem, we would be able to check in Netlify.toml files to source control, handle base directory build ignores properly and leverage Netlify’s caching for workspaces. It would cleanup and speed up builds significantly. Even an option to disable any kind of npm/yarn install would work.
  2. If not, how should npm workspaces be used with Netlify? The install step fails right away in the case where you have a version not published to an npm registry. This comes up for us quite frequently as we have mono repos that don’t publish modules at all, and deploy previews that update an application and the package in tandem.
  3. Is NPM_VERSION working properly? The second build above seems like something is off. I’m guessing it has to do with me adding in NPM_CONFIG_ONLY, but without that workaround it doesn’t seem like Netlify supports npm workspaces.
3

Hey @zackery, appreciate the thorough post. I’m by no means a workspaces expert though I’ll answer your Qs to the best of my knowledge.

I think Qs 1 and 2 can be covered by this portion of our docs, where we describe that npm workspaces aren’t context-aware. It’s very much a WIP so any insight, suggestions or feedback is greatly appreciated!

As for Q3 – have you considered setting a NODE_VERSION env var to, say, 17? NPM v7 is bundled at this version.

4

Thank you for the response!

As for Node 16/17 I don’t think we are ready to upgrade these applications to that version, but that would assuredly fix the issue with having to npm install a specific version of npm.

As for questions 1 and 2, would Netlify be open to supporting an option to either skip the install step or specify a directory for install? I’m not sure if npm workspaces will ever be context aware, and since yarn v1 is not under active development anymore I think it would be beneficial so npm workspaces can work well with netlify.

5

hey @zackery , scott has been out for a day or two, but i will make sure he sees this when he gets back!

7

Hey there, thanks for your patience.

There’s no harm in asking the question! It would definitely be interesting to create a post over on our features forum, in order to gauge how many other folks may be struggling with npm workspaces!