Dynamic require statements

Hey there,
My site name is bus-fyi.netlify.app
And I’m trying to deploy a project with alot(tens of thousands) of json files, and a single on demand builder that dynamically calls one based on the path.

Here’s my deploy log

3:44:03 PM: build-image version: cd0a67ec27b8bcd87e2a257fe5ebcf1900a8021d (focal)
3:44:03 PM: buildbot version: f3871d4c438bbd4f48a8f44a50bec7489430735e
3:44:03 PM: Fetching cached dependencies
3:44:03 PM: Failed to fetch cache, continuing with build
3:44:03 PM: Starting to prepare the repo for build
3:44:03 PM: No cached dependencies found. Cloning fresh repo
3:44:03 PM: git clone --filter=blob:none https://github.com/ericmikkelsen/bus-fyi-netlify
3:44:04 PM: Preparing Git Reference refs/heads/main
3:44:18 PM: Parsing package.json dependencies
3:44:19 PM: Different build dir detected, going to use the one specified in the Netlify configuration file: '' versus '/' in the Netlify UI
3:44:20 PM: Starting to install dependencies
3:44:20 PM: Python version set to 3.8
3:44:20 PM: Attempting Ruby version 2.7.2, read from environment
3:44:21 PM: Using Ruby version 2.7.2
3:44:22 PM: Started restoring cached go cache
3:44:22 PM: Finished restoring cached go cache
3:44:22 PM: Installing Go version 1.19.5 (requested 1.19.5)
3:44:27 PM: go version go1.19.5 linux/amd64
3:44:28 PM: Using PHP version 8.0
3:44:30 PM: v16.20.0 is already installed.
3:44:30 PM: Now using node v16.20.0 (npm v8.19.4)
3:44:30 PM: Enabling Node.js Corepack
3:44:30 PM: Started restoring cached build plugins
3:44:30 PM: Finished restoring cached build plugins
3:44:30 PM: Started restoring cached corepack dependencies
3:44:30 PM: Finished restoring cached corepack dependencies
3:44:30 PM: No npm workspaces detected
3:44:30 PM: Started restoring cached node modules
3:44:30 PM: Finished restoring cached node modules
3:44:31 PM: Installing npm packages using npm version 8.19.4
3:44:37 PM: npm WARN deprecated source-map-url@0.4.1: See https://github.com/lydell/source-map-url#deprecated
3:44:37 PM: npm WARN deprecated urix@0.1.0: Please see https://github.com/lydell/urix#deprecated
3:44:37 PM: npm WARN deprecated resolve-url@0.2.1: https://github.com/lydell/resolve-url#deprecated
3:44:37 PM: npm WARN deprecated source-map-resolve@0.5.3: See https://github.com/lydell/source-map-resolve#deprecated
3:44:41 PM: npm WARN deprecated querystring@0.2.0: The querystring API is considered Legacy. new code should use the URLSearchParams API instead.
3:44:48 PM: added 1433 packages, and audited 1434 packages in 17s
3:44:48 PM: 186 packages are looking for funding
3:44:48 PM:   run `npm fund` for details
3:44:49 PM: 15 vulnerabilities (2 moderate, 13 high)
3:44:49 PM: To address issues that do not require attention, run:
3:44:49 PM:   npm audit fix
3:44:49 PM: To address all issues (including breaking changes), run:
3:44:49 PM:   npm audit fix --force
3:44:49 PM: Run `npm audit` for details.
3:44:49 PM: npm packages installed
3:44:49 PM: Install dependencies script success
3:44:49 PM: Starting build script
3:44:50 PM: Detected 0 framework(s)
3:44:50 PM: Section completed: initializing
3:44:52 PM: ​
3:44:52 PM:   Netlify Build                                                 
3:44:52 PM: ────────────────────────────────────────────────────────────────
3:44:52 PM: ​
3:44:52 PM: ❯ Version
3:44:52 PM:   @netlify/build 29.10.1
3:44:52 PM: ​
3:44:52 PM: ❯ Flags
3:44:52 PM:   baseRelDir: true
3:44:52 PM:   buildId: 645829ad0625d500084557c2
3:44:52 PM:   deployId: 645829ad0625d500084557c4
3:44:52 PM: ​
3:44:52 PM: ❯ Current directory
3:44:52 PM:   /opt/build/repo
3:44:52 PM: ​
3:44:52 PM: ❯ Config file
3:44:52 PM:   /opt/build/repo/netlify.toml
3:44:52 PM: ​
3:44:52 PM: ❯ Context
3:44:52 PM:   production
3:44:52 PM: ​
3:44:52 PM:   1. Functions bundling                                         
3:44:52 PM: ────────────────────────────────────────────────────────────────
3:44:52 PM: ​
3:44:52 PM: Packaging Functions from netlify/functions directory:
3:44:52 PM:  - stops.js
3:44:52 PM: ​
3:45:19 PM: ​
3:45:19 PM: (Functions bundling completed in 27s)
3:45:19 PM: ​
3:45:19 PM:   2. Deploy site                                                
3:45:19 PM: ────────────────────────────────────────────────────────────────
3:45:19 PM: ​
3:45:19 PM: Starting to deploy site from '/'
3:45:21 PM: Calculating files to upload
3:45:34 PM: 1 new files to upload
3:45:34 PM: 1 new functions to upload
3:45:41 PM: Unzipped size must be smaller than 262144000 bytes
3:45:42 PM: Unzipped size must be smaller than 262144000 bytes
3:45:44 PM: Unzipped size must be smaller than 262144000 bytes
3:45:47 PM: Unzipped size must be smaller than 262144000 bytes
3:45:49 PM: Unzipped size must be smaller than 262144000 bytes
3:45:52 PM: Unzipped size must be smaller than 262144000 bytes
3:45:56 PM: Unzipped size must be smaller than 262144000 bytes
3:46:03 PM: Unzipped size must be smaller than 262144000 bytes
3:46:17 PM: Unzipped size must be smaller than 262144000 bytes
3:46:43 PM: Unzipped size must be smaller than 262144000 bytes
3:47:04 PM: Unzipped size must be smaller than 262144000 bytes
3:47:04 PM: Failed to upload file: stops
3:47:04 PM: Section completed: deploying
3:47:04 PM: Site deploy was successfully initiated
3:47:05 PM: Execution cancelled
3:47:05 PM: Error running command: Command was cancelled
3:47:05 PM: Shutting down logging, 2 messages pending

Is this happening because I’m just sending too much over the wire? Or is it because the netlify function is getting compiled with like 400mb of json? Or some weird third thing?

The maximum size a of a bundled and zipped function is 50mb, unzipped ~250mb *(see here and here.)

This is telling you the same: 262144000 bytes = 250mb.

Perhaps better to store these in another location (AWS, for example) and fetch them from the function.

Hey there, thank you!
I think the first option I’d to rule out first is whether or not there’s a way to keep the function from compiling with the json files inlined or not. Is that a thing?

If you want the json data accessible in the function (without locating them elsewhere) there is not other option (that I am aware of) than to have them bundled with the function.