How to include dependencies in Netlify Lambda Functions?

Support
22

Do you have a working example of how functions can access modules from root package.json?

I have so far had no luck accessing anything that isn’t preinstalled in the functions/myFunction folder.

23

hi @jakobrosenberg - thank you for your patience. We are still investigating and talking internally about this. We’ll respond here as soon as we have a definitive response!

24

Hi @jakobrosenberg

I have an example in one of my personal repositories:

You’ll see that I have the @google/maps required by my function and it is installed as a dependency in my main package.json. Those are the only two requirements for zip it and ship it to do its thing.

Let me know if that helps.

25

Ok, I’m back. I worked on a site locally with netlify dev. I’ve got a root package.json. Everything worked until I deployed to production and my functions don’t have their dependencies. I thought Netlify would run npm i automatically? Or is the issue that I’m not doing npm i in my function subdirectories?

Also - I really, really, really wish netlify dev would match what netlify does. Having it not act the same way is very frustrating in what is - outside of that - a wonderful tool. I don’t expect things to break when going from my machine to production.

2 Likes
26

So FYI, a quick “cd X && npm i && cd …/Y && npm i” fixed it for me.

27

I just want to circle back to this. Should I have used a root package.json and then require(…/…/) in my functions? If so, it would have worked in dev and production, right?

29

Hey Raymond!

The require calls definitely don’t have to change. Node.js will always go up the tree until it finds a node_modules folder.

The problem here is really: what gets bundled into your function deployment?

  • If you use netlify-lambda your functions (including any node modules) will be compiled into a single file.

  • Otherwise, functions will be shipped by just making a zip file out of the single function file or folder. functions/my-function.js or functions/my-function/handler.js .
    In the second case with the subdirectory it’s possible to put a package.json into it and run cd functions/my-function && npm i at build time. (Sadly this is manual right now, but we are working on improvements)

However, recommendation from our staff: if you are having more than one function and sharing dependencies heavily it might make sense to use netlify-lambda !

Let me know if that helps or no.

1 Like
30

THis feels like a VERY good summary, but should it be spelled out in the docs? I don’t think it is right now, at least in one spot.

2 Likes
32

You’re right, that is a good summary. We’ll think on it! :smiley:

1 Like
34

Does this actually work right now?

My folder structure is as follows (simplified for example):

ProjectName/
  app/
  functions/
    fooFunction/
      node_modules/
      .gitignore
      fooFunction.js
      package.json  
  node_modules/
  .gitignore
  package.json

Where each .gitignore file includes the relevant node_modules/ dir, keeping third party node modules out of version control / GitHub, as one would expect.

Due to many reasons but not least of which the ongoing recommendation from Netlify, Ive migrated to using the CLI, or “Netlify dev”, but wish to continue using the continuous integration capabilities provided by Netlify, where my code is simply deployed when merged into my main (master for me) branch.

The way I understand things working is that when an “unbundled” function directory is merged or updated in my main branch, that Netlify will automatically run a compression process (Zip) on it, and then deploy the zipped file to Lambda. Given the advice above, it seems that the general method we are supposed to use to get our node modules included into this process without having to commit them outright, is to have a script prepended to our main build script, that first installs node modules in our functions. To that end my main root-level (app) package.json has the step, buildFunctions prepended to it (build is the script Ive told Netlify to run during deployment):

  "scripts": {
    "buildFunctions": "cd functions/fooFunction && npm install && cd ../../",
    "build": "npm run buildFunctions && webpack --config webpack.app.production.js",
    ...
  }

During deployment, I can actually see this being ran in the Netlify console. First it changes directories into my fooFunction folder, then it installs modules - which I can see the output for -, then it goes back to root and runs the rest of the build, deploying my site.

Here is a slightly edited bit of my deploy output showing this stuff working:

... stuff ...
2:23:09 AM: Executing user command: npm run build
2:23:10 AM: > ProjectName@0.0.1 build /opt/build/repo
2:23:10 AM: > npm run buildFunctions && webpack --config webpack.app.production.js
2:23:10 AM: > ProjectName@0.0.1 buildFunctions /opt/build/repo
2:23:10 AM: > cd functions/fooFunction && npm install && cd ../../
2:23:14 AM: added 37 packages from 53 contributors and audited 56 packages in 2.902s
2:24:34 AM: Hash: abcdefghijklmnop
2:24:34 AM: Version: webpack 4.42.1
2:24:34 AM: Time: 78343ms
2:24:34 AM: Built at: 04/20/2020 6:24:34 AM
... a little later ...
2:24:34 AM: Function Dir: /opt/build/repo/functions
2:24:34 AM: TempDir: /tmp/zisi-abcdefghijklmnop
2:24:35 AM: Prepping functions with zip-it-and-ship-it 0.3.1
2:24:36 AM: [ { path: '/tmp/zisi-abcdefghijklmnop/fooFunction.zip',
2:24:36 AM:     runtime: 'js' } ]
2:24:36 AM: Prepping functions complete
... etc ...

However, after deployment, if I try to run my function it errors with a 502 (aside, can someone point me to documentation about why Netlify uses 502 status for these errors? Id like to catch in frontend and be confident that this is always going to be the status of a function error). If I inspect the output of the console for the function, I see this:

10:58:05 AM: 2020-04-20T14:58:05.146Z undefined ERROR Uncaught Exception  {"errorType":"Runtime.ImportModuleError","errorMessage":"Error: Cannot find module 'googleapis'\nRequire stack:\n- /var/task/lib/services/GoogleSheets/SheetService.js\n- /var/task/fooFunction.js\n- /var/runtime/UserFunction.js\n- /var/runtime/index.js","stack":["Runtime.ImportModuleError: Error: Cannot find module 'googleapis'","Require stack:","- /var/task/lib/services/GoogleSheets/SheetService.js","- /var/task/fooFunction.js","- /var/runtime/UserFunction.js","- /var/runtime/index.js","    at _loadUserApp (/var/runtime/UserFunction.js:100:13)","    at Object.module.exports.load (/var/runtime/UserFunction.js:140:17)","    at Object.<anonymous> (/var/runtime/index.js:43:30)","    at Module._compile (internal/modules/cjs/loader.js:1158:30)","    at Object.Module._extensions..js (internal/modules/cjs/loader.js:1178:10)","    at Module.load (internal/modules/cjs/loader.js:1002:32)","    at Function.Module._load (internal/modules/cjs/loader.js:901:14)","    at Function.executeUserEntryPoint [as runMain] (internal/modules/run_main.js:74:12)","    at internal/main/run_main_module.js:18:47"]}

Which appears to imply that the node_modules were never zipped into the function. That is, why cant it find the module, googleapis (Ive noticed many others having issues with googleapis… is there a problem with this package specifically?).

Here is what’s in my function’s package.json:

  "dependencies": {
    "googleapis": "^25.0.0"
  },


**** Also want to add: I know there is other advice to just add the function dependencies to the root-level package.json. Id very much like not to do this. Im trying to build a long-term maintainable system, and anticipate the possibilities of growing function needs (requiring more and more additions) AND/OR of wanting to untie my AWS Lambda function usage from being managed by Netlify (I already have a secondary functions folder for this because I have non-HTTP “server” specific Lambda needs that Netlify isnt built to help with), and the idea of complicating my app’s package.json dependencies for this setup doesnt feel like a wise decision.

35

Hi @bradleygriffith, I actually have an example where I do something very similar here: function-deploy-test/package.json at master · netlify/function-deploy-test · GitHub and it works without issue (though I do zip the function myself in this example). Have you tried deploying a simplified version just to test if the issue is with the buildFunctions script at all?

Alternatively, you could try zipping your function manually like in my example. Just make sure the unzipped function isn’t in the folder you’ve configured as your functions folder so our system doesn’t try to deploy both.

Regarding the 502, since we host all our lambda functions on AWS lambda, we just pass their HTTP code back to you. Basically, a 502 means the server (your function) couldn’t handle the request, which is appropriate in this case since it error-ed out.

Hope that helps.

1 Like
36

So I actually did find a solution last night, and maybe a bug (or at least something worth documenting or sharing here).

Main Setup:

  1. Install netlify-lambda. (Note: you’ll still be using netlify dev, this will just help with building your functions).
  2. I found the instructions in netlify-lambda to be a little confusing for this, so just in case, ensure that your functions = setting in your netlify.toml file is just pointing at the same directory holding your unbundled functions, as you probably already had it doing if you’re reading this.
  3. Rather than the awkward addition to the build command (see above) that cd’s around directories during your build step, we are going to use netlify-lambda to bundle functions. If you followed my format above with the additional buildFunctions script, remove that and add one called prebuild that runs netlify-lambda install path/to/functions where path/to/functions is the same setting you have for functions in netlify.toml. Also remove the call to buildFunctions from the build command.
  4. Netlify will now automatically bundle packages from the package.json files in your individual function folders.

Here is a more fleshed out layout of the above…

Folder structure:

ProjectName/
  app/
  functions/
    fooFunction/
      node_modules/
      .gitignore
      fooFunction.js
      package.json  
  node_modules/
  .gitignore
  package.json

Scripts in root-level package.json of project:

"scripts": {
    "prebuild": "netlify-lambda install path/to/functions",
    "build": "webpack --config webpack.app.production.js",
    ...
  }

Ok so, that “fixes it”, in that it is a somewhat more elegant solution than having to add code to manually walk around directories during the build step. That said, Im not actually 100% positive that it does anything different than the setup before. The reason why is that I think I found a bug, or at least something that should be known by people experiencing this issue.

“Bug” and Solution:

  1. As stated earlier, the error I was seeing in my console on Netlify was that a component of my function was unable to find a node module that it should have been able to find.
  2. My setup is a little more complicated than a simple monolithic function.js file that does everything. Instead I have subdirectories within my function folder holding various utility or service scripts that the main entrypoint, function.js imports for use.
  3. Specifically, my main function.js file was importing these additional scripts using path.resolve (e.g.; const myUtil = require(path.resolve(__dirname, "path/to/utils/myUtil.js");).
  4. While on Netlify the function didnt throw any errors around being able to locate these files, it appears that if imported this way, imported scripts will not be able to locate the node modules they rely on. Once I removed path and path.resolve, everything began working fine (e.g.; const myUtil = require("./path/to/myUtil.js");).
  5. Want to add that the path.resolve method works fine on functions I deploy to Lambda myself (those not managed by Netlify at all).

Hope this helps someone.

37

Hi,

So, netlify-lambda will bundle your function into a single js file and can make things easier to deploy but will not work for dependencies that have binary components (like node-sharp). For functions like those, I would recomment that you use the built-in zip-it-and-ship-it (or zisi) method.

The reason zisi isn’t working for you is that this method checks your functions ‘require’ statements that requires node modules directly to know which dependencies to include. It doesn’t actually include your whole node_modules folder, only those it referenced in your main function file. Since you aren’t doing that but instead using path.resolve, our buildbot doesn’t know about your dependencies. In cases like these, I recommend zipping up your function manually as I previously mentioned, which gives you a lot more freedom with how you want to structure your function.

For more information on zisi that our buildbot uses, you can go here: GitHub - netlify/zip-it-and-ship-it: Intelligently prepare Node.js Lambda functions for deployment. It mentions: Zip It and Ship It will only include dependencies that’s been required from the relevant handler file.

38

Adding my findings to this thread in case it helps someone. This is relevant to netlify-lambda only.

So I forked netlify-express for my project, and added node-fetch as a dependency using require. While the app worked fine on my local system, it threw errors on Netlify servers.

After some debugging, I realized the problem was that the ES Module of node-fetch was being bundled (instead of CommonJS) by the webpack of netlify-lambda. And so the export of node-fetch was of type Object[Module] on production, whereas on my local machine it was of type function (which happens to be the function ‘fetch’).

I found two fixes for this problem.

The first was to remove mjs extension in netlify-lambda webpack configuration. This would force the CommonJS version of node-fetch to get bundled, but would also affect any other libraries that bundled ES Modules instead of Common JS.

In my case, it was just node-fetch behaving this way so I looked for an alternate solution localized to node-fetch only.

I found that replacing require(‘node-fetch’) with require(‘node-fetch’).default in my project worked, both locally and on Netlify servers. This is a known workaround for dealing with this webpack issue.

Additionally, I also had to add the NPM package encoding to my package.json dependencies for Netlify to build my function. This was being imported conditionally by node-fetch, and it was optional as well (node-fetch would still work if it was not found). Netlify is probably doing a static check on their side for all dependencies to be available at function build time?

1 Like