How to include dependencies in Netlify Lambda Functions?

cfjedimaster · July 25, 2019, 2:13pm

I’m testing locally with netlify dev and my function works perfectly. When I commit to Github, my deploy fails because it can’t find a required dependency under functions/myfunc. Obviously I’ve got node_modules in .gitignore and I assumed that Netlify would run npm i automatically on build for functions. Since it isn’t, how do I correct this?

futuregerald · July 25, 2019, 2:20pm

Hi @cfjedimaster,

One option is to include your dependencies in your root package.json. If you have a second package.json in your functions directory then you can chain their installation as part of your build command using &&.

cfjedimaster · July 25, 2019, 2:22pm

Ok… so… are you saying use a package.json at project root with a script that runs npm i for the function subdirectory?

Also - I tried adding netlify-lambda to my project. In my root package.json, I then tried netlify-lambda build functions’. This is documented here: Functions overview | Netlify Docs. Doing so gives me a new error:

netlify-lambda Error: Function source folder (specified in netlify-lambda serve/build command)
9:18:39 AM:       and publish folder (specified in netlify.toml)
9:18:39 AM:       should be different.

cfjedimaster · July 25, 2019, 3:25pm

As an update - I’ve now tried specifying ./functions in my site settings. I have that same value in netlify.toml, but maybe it will help? Currently my deploy is enqueued so I’m waiting.

swyx · July 25, 2019, 3:29pm

hi raymond!

Ok… so… are you saying use a package.json at project root with a script that runs npm i for the function subdirectory?

yes. infact i like to do this as a prebuild or preinstall hook:

{
  "scripts": {
    "prebuild" : "cd functions/myfunction && npm i",
   "build": "gatsby build"
  }
}

you can also stick it inside the netlify.toml command field, but thats really up to personal preference. bottom line is, you’re responsible for writing the bash commands to install any dependencies that aren’t at the root.

i’m well aware this isn’t ideal. this may change in future iterations of the build bot that we are currently prototyping. but its what we have for now.

Also - I tried adding netlify-lambda to my project. In my root package.json, I then tried netlify-lambda build functions '. This is documented here: Functions overview | Netlify Docs. Doing so gives me a new error:

this is almost a different topic altogether. to use netlify-lambda, be sure to read the docs: “At a high level, netlify-lambda takes a source folder (e.g. src/lambda , specified in your command) and outputs it to a built folder, (e.g. built-lambda , specified in your netlify.toml file).”

given this it would evidently be a bad idea to have your source and dist folders be the same folder. this is what the error is warning you about. you’re coming from a netlify dev workflow, which has no build step, so you don’t have a source and dist folder. For an easy fix, change your functions key inside netlify.toml to something else.

netlify-lambda serves slightly different and overlapping needs than netlify dev. I did my best to explain in the docs: GitHub - netlify/netlify-lambda: Helps building and serving lambda functions locally and in CI environments “Netlify Dev vs Netlify-Lambda”

DavidWells · July 25, 2019, 3:48pm

This will be updated with a future version of our build system.

In the meantime you can use this script that will automatically loop through your functions directories and ensure that the dependancies are installed.

github.com

netlify-labs/all-the-functions/blob/master/scripts/installer.js

const path = require('path')
const fs = require('fs')
const globby = require('globby')
const cp = require("child_process")

function installDeps(functionDir, cb) {
  cp.exec("npm i", { cwd: functionDir }, cb)
}

(async () => {
  const findJSFiles = ['*/package.json', '!node_modules', '!**/node_modules']
  const directory = path.join(__dirname, '..', 'functions')
	const foldersWithDeps = await globby(findJSFiles, { cwd: directory })

  const folders = foldersWithDeps.map(fnFolder => {
    return fnFolder.substring(0, fnFolder.indexOf("package.json"))
  }).map((folder) => {
    installDeps(path.join(__dirname, '..', 'functions', folder), () => {
      console.log(`${folder} dependencies installed`)
    })

This file has been truncated. show original

and run this in the prebuild scrip of your package.json like so:

  "scripts": {
    "prebuild": "node ./scripts/installer.js",
    "build": "react-scripts build"
  },

cfjedimaster · July 25, 2019, 4:16pm

Thank you both. I was in a meeting the last 40 minutes or so and was able to quickly test.

So first - yeah, netlify-lamda was a mistake on my part. I was just throwing crap at the wall to see what would stick. I’m going to read your link, but it seems that for how I use Netlify Functions, I can probably avoid it.

Running npm i on the subdirectory worked great, and David, your script to do this dynamically is cool too.

I’m going to write this up in a blog post to specifically cover the use case of:

using netlify dev locally to test
using the CLI to scaffold an app
using code that requires node modules (which will be most functions)
and using Git to deploy where you are not committing node_modules

It sounds like yall may have a way to make this easier in the future too, which is good. Honestly this feels like something I’d have trouble documenting, but until you do get the fix working, maybe a quick reminder at the end of Build functions | Netlify Docs? Before you get into “Tools for Building JavaScript Functions”. And actually, isn’t that section a bit out of date now with the way the netlify function can scaffold an application and netlify dev can run it for you?

Finally - thank you all!

David_Davis · October 3, 2019, 8:43pm

Hi @DavidWells, is this installer.js necessary, or is it now included in the build system?

fool · October 11, 2019, 6:04pm

It’s not available yet, and won’t be for you for at least a little while. We’ll post more widely about the new build system as things develop

Kristen · October 18, 2019, 8:59pm

I just wanted to share here that the new and improved Netlify docs include a couple improvements relevant to this discussion.

The section about tools for building serverless functions with JavaScript now includes Netlify Dev
There’s a new dedicated page about managing dependencies

cfjedimaster · October 22, 2019, 1:47pm

So is it fair to say, given serverless functions A, B, C:

A needs foo, goo npm dependencies
B needs zoo
C needs foo, roo

If I use a root package.json using foo, goo, zoo, roo, it will then handle installing what I need?

But wait - that would install to my site root node_modules, not relative to my functions.

How would you handle this?

Dennis · October 29, 2019, 8:59am

Hi @cfjedimaster, as long as you require the dependencies in your lambda functions, what you describe should be handled correctly. Specifically, a file structure like:

/functions/foo.js
/package.json
/node_modules/

Our buildbot uses zip-it-and-ship-it to handle your dependencies, which is flexible enough to handle dependencies in either your root package.json and/or a package.json in your functions folder.

Let me know if that answers our question. Thanks!

cfjedimaster · October 29, 2019, 3:23pm

Well, this part: “and/or a package.json in your functions folder” to me means you made it work as easy as possible. So given functions/foo/node_modules and functions/goo/node_modules, both functions requiring what they need and such, Netlify will run npm i for both folders correctly?

Dennis · October 29, 2019, 5:24pm

The buildbot won’t actually do the npm i except for the root package.json, based on the code as far as I can tell. You will need to do the install in each of those function subfolders individually (as part of your build command). I think maybe it makes more sense to have one package.json in the function folder that will have the dependencies collectively so you only need to run the npm i once.

cfjedimaster · October 29, 2019, 5:51pm

So… how does that work then? If I need node-fetch in /functions/goo, do I installed it in /functions/ and in goo’s code require(’…/node-fetch’) or some such? Sorry if I’m missing the obvious here.

Dennis · November 4, 2019, 5:55pm

Yup, you’ll make sure the node-fetch package is in the package.json inside your functions/ folder. During your build command, you’ll need to make sure to run npm i in that folder as well (e.g. cd functions/ && npm i && <rest of your build command>).

Alternatively, you can have node-fetch in your main package.json file so you don’t need to add those commands in your build command.

Lastly, you’re correct that you’ll need to have the require statement as well.

Hopefully that filled in what you were missing. Let me know if it doesn’t.

cfjedimaster · November 4, 2019, 8:49pm

Hmm. So probably the most ideal solution is, at site root, npm i my needs, and in my functions/X folders, use require(’…/…/SOMETHING’), right?

Dennis · November 6, 2019, 5:50pm

Yea, that sounds about right. You might not even need to use the absolute path as our buildbot should traverse your repo for package.json files to resolve that module.

Dergarcon · November 7, 2019, 8:15am

I can’t get node modules to work in the functions. Help would be realy appreciated.

These are the steps I have taken so far:

Folder structure

/functions/submission-created.js
/package.json
/node_modules/

I do not have the installer.js nor package.json in functions folder. Also I do not run the prescript.

My goal is to make it work like @cfjedimaster using require(’…/…/SOMETHING’)

I want to use the googleapis. googleapis is in dependencies of package.json in root folder

 "dependencies": {
    "googleapis": "^45.0.0",    
  },

I am failing to require the googleapis from root out of my functions/submission-created.js. The function gets called but fails. I get one of those following errors depending on what I am trying.

 Unable to import module 'submission-created': Error
    at Function.Module._resolveFilename (module.js:547:15)
    at Function.Module._load (module.js:474:25)
    at Module.require (module.js:596:17)
    at require (internal/module.js:11:18)
    at Object.<anonymous> (/var/task/submission-created.js:1:78)
    at Module._compile (module.js:652:30)
    at Object.Module._extensions..js (module.js:663:10)
    at Module.load (module.js:565:32)
    at tryModuleLoad (module.js:505:12)
    at Function.Module._load (module.js:497:3)

the other error:

{"errorMessage":"Cannot find module 'googleapis'","errorType":"Error","stackTrace":["Function.Module._resolveFilename (module.js:547:15)","Function.Module._load (module.js:474:25)","Module.require (module.js:596:17)","require (internal/module.js:11:18)","exports.handler (/var/task/submission-created.js:6:18)"]}

I tried inside of sumbission-created.js :

var {google} = require('.../googleapis');
var {google} = require('../googleapis');
var {google} = require('/googleapis');
var {google} = require('googleapis');

Looking forward to your answers.

Thank you guys.

futuregerald · November 12, 2019, 3:37pm

@Dergarcon can you share the code that’s in your function? If your repo is public, it would be useful to see it. Thanks.

Topic		Replies	Views
Install Netlify functions dependencies Support building , deployment , lambda-functions	1	1471	April 7, 2020
Function compiled with netlify-lambda throwing a dependency error once deployed Support building , deployment , lambda-functions	4	1986	May 10, 2020
My lambda function's `package.json` file is causing mystery failures Support building , lambda-functions	4	672	January 4, 2021
Using require to include a relative module in Netlify Functions on Node Support deployment , lambda-functions	21	9404	April 20, 2021
Function cannot find modules installed in root package.json when deployed via GitHub Support lambda-functions	4	1180	October 30, 2020

How to include dependencies in Netlify Lambda Functions?

Related topics