Function size > 50 Mb... but can't figure out why?

Support
,
1

Hello team,

I’m having issues with this deploy.

It’s saying that one of my functions is > 50 Mb, however the grand total of the files is < 1 Mb, and the dependencies (unzipped!) are < 10 Mb.

I’ve seen elsewhere on the forum that the dependencies are built platform-specific, but since the build fails I can’t download the built files and pull it apart to see what is happening.

Can anyone here give me the next steps for troubleshooting this?

Lastly, I’m getting ~67 notifications by email for each failed build… Please fix :slight_smile:

Screen Shot 2022-09-19 at 5.18.40 PM
Screen Shot 2022-09-19 at 5.18.40 PM220×1224 6.85 KB

2

Hi Daniel,

This doc explains how to work on that function size problem: next-runtime/large-functions.md at main · netlify/next-runtime · GitHub

Could you give that workflow a try and let us know what you find?

Re: zillions of emails, that’s quite odd - our system doesn’t show sending that many. Mind fishing me out the full headers of a couple of them (a couple of identical ones, I mean), and sending via DM to me?

3

Hi @fool,

Yes, I saw that document and unfortunately, I don’t see any dev dependencies that are being mistakenly included.

I think it’s due to a single depency, “chartjs-node-canvas”, which is including both chartjs and a canvas-for-node implementation, neither of which are lightweight.

When I build locally, it comes to < ~24Mb. I tried building using the build-image instructions here but I could get the build to proceed properly.

My js files are less than 500 Kb.

Failed deploy is here

You can see the multiple failures (which seem weird to me) and the multiple error messages for the same function.

5

@fool I’ve figured out that one of my dependencies includes ~20Mb (unzipped) of documentation, and when I delete that locally and deploy via CLI, I don’t get any errors.

So, now I’m trying to avoid bundling it.
I’ve tried :

  1. excluding it via included_files, and that didn’t work:
included_files = ["!src/api/GenerateSendReport/node_modules/chart.js/dist/docs/*","!node_modules/chart.js/dist/docs/*"]
// Above didn't seem to work, that folder is still bundled into the function when building locally and via Netlify's deployments

  1. adding a command to delete that subfolder from node_modules before bundling, while it seems to delete, it doesn’t seem to do so in a way that prevents it from being bundled:
    build: "rm -rf ./node_modules/chart.js/dist/docs && rest-of-build-commands-here

  2. adding the dependency locally after removing the offending folder… Same issue.

Interestingly enough with #3, when I build locally, the zipped function is only 6.6 Mb, unzipped 29.6 Mb… so it seems like I should be well below the limits. What gives?

So, my questions is: How can I exclude that folder (chartjs/dist/docs/*) from being bundled with the function? What is the right filepath to either exclude or delete at build time?

Thank you

8

Hi @danielthedifficult

It looks like you’re on the right track as all those approaches should have worked. Maybe at build-time it does not recognize the path?

Can you please try step 2 and let’s try to ls instead of removing just to see that we’re on the right path:

ls ./node_modules/chart.js/dist/docs && rest-of-build-commands-here

Also for step 1, try to add a relative path:

"!./src/api/GenerateSendReport/node_modules/chart.js/dist/docs/*","!./node_modules/chart.js/dist/docs/*"

Let me know how that goes!

9

Thanks @gualter

Here are my updated build commands:

    "prebuild": "ls ./node_modules/chart.js/dist/docs && find . -name 'chart.js' -print -exec rm -rfv '{}/docs' \\; && rm -rf ./dist && mkdir ./dist",
    "build": "apidoc -i src/api/ -o ./dist/docs && cp -r ./src/* ./dist",
    "postbuild": "cd ./dist && rm -rf api Client-Email-Templates CloudBackend espace_clients",

And here is the resulting build

Relevant log entries:

11:33:14 AM: $ yarn build
11:33:14 AM: yarn run v1.22.10
11:33:14 AM: $ ls ./node_modules/chart.js/dist/docs && find . -name 'chart.js' -print -exec rm -rfv '{}/docs' \; && rm -rf ./dist && mkdir ./dist
11:33:14 AM: 404.html
11:33:14 AM: api
11:33:14 AM: assets
11:33:14 AM: axes
11:33:14 AM: charts
11:33:14 AM: configuration
11:33:14 AM: developers
11:33:14 AM: favicon.ico
11:33:14 AM: general
11:33:14 AM: getting-started
11:33:14 AM: index.html
11:33:14 AM: logo.png
11:33:14 AM: logo.svg
11:33:14 AM: samples
11:33:14 AM: ./src/api/libraries/chart.js
11:33:14 AM: ./node_modules/chart.js
11:33:14 AM: ./node_modules/chart.js/dist/chart.js
11:33:14 AM: $ apidoc -i src/api/ -o ./dist/docs && cp -r ./src/* ./dist

And another attempt with a find / rm after the build script

10

Looks like a dead-end :sweat_smile: . Mind providing a reproduction repo?

11

Done!

For what it’s worth, building + deploying locally works.

Also, I did manage to get the test-build docker image working, and THAT works too.

PS I don’t know if it would help, but I could make this a background function if needed…

13

Hello @hrishikesh @fool @gualter :slight_smile:

Do you have what you need for the repro case ? Any ideas?

As I mentioned before, this is preventing us from shipping a major new feature (activity reports for our customers via email) and any information about your progress would help me decide how to manage things over here!

Thank you

14

I’d be surprised if it did because I just checked this and it looks like you’re facing this exact problem:

I tested on maOS, the Function zip was about 16 MB. I then tested on an Ubuntu machine, the zip turned out to be 55 MB. I then added:

[functions]
    included_files = ["!./node_modules/chart.js/dist/docs/**"]

inside the netlify.toml and that did bring the size down to 52.4 MB, but that’s still over the limit.

Apparently, platform-specific dependencies are taking up space in this function.

Here’s the screenshot from the Ubuntu VM:

image
image1812×1154 124 KB

and here’s the screenshot that shows that the docs directory is missing in the zip:

image
image3116×1018 213 KB

Note that, I’m not even using the Netlify build image at this point, testing in a vanilla Ubuntu 22.04 installation (with Git and Node installed). Since this is happening outside of Netlify, there’s not a lot we can do about it.

15

Boom, okay, thank you so much for this clear info. I should have thought of using an Ubuntu VM, but I was encouraged by the test-build docker image working.

I’ll pick apart the dependencies when building in Ubuntu and see if I can fix it another way…

Question: Is there any way to increase that function size limit for my account?

16

Not at the moment, unfortunately. This limit is imposed by AWS which sets it at 50 MB per zip (technically about 70, but that 20 MB is some overhead that AWS adds or needs).