Hi
I’m trying to deploy a site of over 50,000 pages using the Netlify CLI but I’m running into an error.
RangeError: Maximum call stack size exceeded
If I restrict the number of pages generated then it is successful.
Does anyone know how how to resolve this issue?
Thanks in advance for your assistance.
Hey there, @ThomCM 
Can you share a link to your site or your netlify site name, as well as a full copy of last deploy log? This will help us look into this further.
Thanks so much!
Hi @hillary
Thanks for your quick response.
The site name is ‘borgs’. Since I last attempted this I’ve made some changes so that the majority of the page is NOT generated: I only really need the <head> to be generated for the sake of sharing on social media (twitter cards, OG tags, etc). This has reduced the local build time from approximately 15 minutes to approximately 11 minutes.
This change has meant that deploy triggers via Netlify are now completing the build phase (previously they weren’t) but now running into ‘Build exceeded maximum allowed runtime’.
Do you think if I upgraded from Pro to Enterprise that would make a difference?
Why do build times take so much longer on Netlify?
Thanks very much for your time.
__
1:21:38 PM: Build ready to start
1:21:41 PM: build-image version: 3fa6e2800752667ac56af498951f923e208f7404 (xenial)
1:21:41 PM: build-image tag: v3.10.3
1:21:41 PM: buildbot version: e07f38cea73012d29c839c78dec87335277fe704
1:21:41 PM: Fetching cached dependencies
1:21:41 PM: Starting to download cache of 144.8MB
1:21:42 PM: Finished downloading cache in 1.260873746s
1:21:42 PM: Starting to extract cache
1:21:47 PM: Finished extracting cache in 5.210792647s
1:21:47 PM: Finished fetching cache in 6.496925606s
1:21:47 PM: Starting to prepare the repo for build
1:21:48 PM: Preparing Git Reference refs/heads/master
1:21:49 PM: Parsing package.json dependencies
1:21:49 PM: Starting build script
1:21:49 PM: Installing dependencies
1:21:49 PM: Python version set to 2.7
1:21:50 PM: Started restoring cached node version
1:21:52 PM: Finished restoring cached node version
1:21:53 PM: v12.18.0 is already installed.
1:21:54 PM: Now using node v12.18.0 (npm v6.14.4)
1:21:54 PM: Started restoring cached build plugins
1:21:54 PM: Finished restoring cached build plugins
1:21:54 PM: Attempting ruby version 2.7.2, read from environment
1:21:55 PM: Using ruby version 2.7.2
1:21:56 PM: Using PHP version 5.6
1:21:56 PM: Started restoring cached node modules
1:21:56 PM: Finished restoring cached node modules
1:21:56 PM: Started restoring cached go cache
1:21:56 PM: Finished restoring cached go cache
1:21:56 PM: go version go1.14.4 linux/amd64
1:21:56 PM: go version go1.14.4 linux/amd64
1:21:56 PM: Installing missing commands
1:21:56 PM: Verify run directory
1:21:57 PM:
1:21:57 PM: ────────────────────────────────────────────────────────────────
1:21:57 PM: Netlify Build
1:21:57 PM: ────────────────────────────────────────────────────────────────
1:21:57 PM:
1:21:57 PM: ❯ Version
1:21:57 PM: @netlify/build 26.0.2
1:21:57 PM:
1:21:57 PM: ❯ Flags
1:21:58 PM: baseRelDir: true
1:21:58 PM: buildId: 61c477e2c6c2d9000897b4d5
1:21:58 PM: deployId: 61c477e2c6c2d9000897b4d7
1:21:58 PM:
1:21:58 PM: ❯ Current directory
1:21:58 PM: /opt/build/repo
1:21:58 PM:
1:21:58 PM: ❯ Config file
1:21:58 PM: No config file was defined: using default values.
1:21:58 PM:
1:21:58 PM: ❯ Context
1:21:58 PM: production
1:21:58 PM:
1:21:58 PM: ────────────────────────────────────────────────────────────────
1:21:58 PM: 1. Build command from Netlify app
1:21:58 PM: ────────────────────────────────────────────────────────────────
1:21:58 PM:
1:21:58 PM: $ npm run generate
1:21:58 PM: > borgs-v2@1.0.0 generate /opt/build/repo
1:21:58 PM: > nuxt generate
1:42:33 PM:
1:42:33 PM: (build.command completed in 20m 35.7s)
1:42:33 PM:
1:42:33 PM: ────────────────────────────────────────────────────────────────
1:42:33 PM: 2. Deploy site
1:42:33 PM: ────────────────────────────────────────────────────────────────
1:42:33 PM:
1:42:33 PM: Starting to deploy site from ‘dist’
1:42:44 PM: Creating deploy tree asynchronously
1:42:46 PM: Creating deploy upload records
1:43:50 PM: 150914 new files to upload
1:43:50 PM: 0 new functions to upload
1:51:39 PM: Build exceeded maximum allowed runtime
Hiya @ThomCM . Thanks for that detail about the sitename! I’ve adjusted your build timelimit to an hour and am hopeful that your build in our CI will work better now. Could you give another try and link us to the deploy logs in our UI either way so we can see how it went?
Thanks in advance for your help in troubleshooting.
Hi @fool thanks for your assistance. Unfortuantely the deploy failed. Please see deploy log below.
8:19:32 PM: Build ready to start
8:19:34 PM: build-image version: 3fa6e2800752667ac56af498951f923e208f7404 (xenial)
8:19:34 PM: build-image tag: v3.10.3
8:19:34 PM: buildbot version: e07f38cea73012d29c839c78dec87335277fe704
8:19:34 PM: Fetching cached dependencies
8:19:35 PM: Starting to download cache of 144.8MB
8:19:35 PM: Finished downloading cache in 795.677718ms
8:19:35 PM: Starting to extract cache
8:19:43 PM: Finished extracting cache in 7.593366007s
8:19:43 PM: Finished fetching cache in 8.419147397s
8:19:43 PM: Starting to prepare the repo for build
8:19:43 PM: Preparing Git Reference refs/heads/master
8:19:44 PM: Parsing package.json dependencies
8:19:45 PM: Starting build script
8:19:45 PM: Installing dependencies
8:19:45 PM: Python version set to 2.7
8:19:46 PM: Started restoring cached node version
8:19:49 PM: Finished restoring cached node version
8:19:50 PM: v12.18.0 is already installed.
8:19:51 PM: Now using node v12.18.0 (npm v6.14.4)
8:19:51 PM: Started restoring cached build plugins
8:19:51 PM: Finished restoring cached build plugins
8:19:51 PM: Attempting ruby version 2.7.2, read from environment
8:19:53 PM: Using ruby version 2.7.2
8:19:53 PM: Using PHP version 5.6
8:19:53 PM: Started restoring cached node modules
8:19:53 PM: Finished restoring cached node modules
8:19:53 PM: Started restoring cached go cache
8:19:53 PM: Finished restoring cached go cache
8:19:54 PM: go version go1.14.4 linux/amd64
8:19:54 PM: go version go1.14.4 linux/amd64
8:19:54 PM: Installing missing commands
8:19:54 PM: Verify run directory
8:19:56 PM:
8:19:56 PM: ────────────────────────────────────────────────────────────────
8:19:56 PM: Netlify Build
8:19:56 PM: ────────────────────────────────────────────────────────────────
8:19:56 PM:
8:19:56 PM: ❯ Version
8:19:56 PM: @netlify/build 26.0.2
8:19:56 PM:
8:19:56 PM: ❯ Flags
8:19:56 PM: baseRelDir: true
8:19:56 PM: buildId: 61c4d9d4e6cf76000771ebd0
8:19:56 PM: deployId: 61c4d9d4e6cf76000771ebd2
8:19:56 PM:
8:19:56 PM: ❯ Current directory
8:19:56 PM: /opt/build/repo
8:19:56 PM:
8:19:56 PM: ❯ Config file
8:19:56 PM: No config file was defined: using default values.
8:19:56 PM:
8:19:56 PM: ❯ Context
8:19:56 PM: production
8:19:56 PM:
8:19:56 PM: ────────────────────────────────────────────────────────────────
8:19:56 PM: 1. Build command from Netlify app
8:19:56 PM: ────────────────────────────────────────────────────────────────
8:19:56 PM:
8:19:56 PM: $ npm run generate
8:19:56 PM: > borgs-v2@1.0.0 generate /opt/build/repo
8:19:56 PM: > nuxt generate
8:42:15 PM:
8:42:15 PM: (build.command completed in 22m 19.4s)
8:42:15 PM:
8:42:15 PM: ────────────────────────────────────────────────────────────────
8:42:15 PM: 2. Deploy site
8:42:15 PM: ────────────────────────────────────────────────────────────────
8:42:15 PM:
8:42:15 PM: Starting to deploy site from ‘dist’
8:42:28 PM: Creating deploy tree asynchronously
8:42:31 PM: Creating deploy upload records
8:43:29 PM: 143052 new files to upload
8:43:29 PM: 0 new functions to upload
9:32:40 PM: Site deploy was successfully initiated
9:32:40 PM:
9:32:40 PM: (Deploy site completed in 50m 24.9s)
9:32:40 PM:
9:32:40 PM: ────────────────────────────────────────────────────────────────
9:32:40 PM: Netlify Build Complete
9:32:40 PM: ────────────────────────────────────────────────────────────────
9:32:40 PM:
9:32:40 PM: (Netlify Build completed in 1h 12m 44.4s)
9:32:40 PM: Execution timed out after 1h12m55.155866894s
9:32:40 PM: Error running command: Command did not finish within the time limit
9:32:41 PM: Creating deploy upload records
9:32:41 PM: Failing build: Failed to build site
9:32:41 PM: Failed during stage ‘building site’: Command did not finish within the time limit
9:32:41 PM: Finished processing build request in 1h13m6.393087668s
Update
The previous attempt ended up being successful. Not sure what happened as it was saying that it had failed at the time of my previous post. Please see updated log below.
Just a note to say that I anticipate the site getting larger than it is currently (approximately 75,000 pages) over the next 6 months or so. Ideally I would be able to trigger deploys via build hook and it wouldn’t take quite so long.
8:19:32 PM: Build ready to start
8:19:34 PM: build-image version: 3fa6e2800752667ac56af498951f923e208f7404 (xenial)
8:19:34 PM: build-image tag: v3.10.3
8:19:34 PM: buildbot version: e07f38cea73012d29c839c78dec87335277fe704
8:19:34 PM: Fetching cached dependencies
8:19:35 PM: Starting to download cache of 144.8MB
8:19:35 PM: Finished downloading cache in 795.677718ms
8:19:35 PM: Starting to extract cache
8:19:43 PM: Finished extracting cache in 7.593366007s
8:19:43 PM: Finished fetching cache in 8.419147397s
8:19:43 PM: Starting to prepare the repo for build
8:19:43 PM: Preparing Git Reference refs/heads/master
8:19:44 PM: Parsing package.json dependencies
8:19:45 PM: Starting build script
8:19:45 PM: Installing dependencies
8:19:45 PM: Python version set to 2.7
8:19:46 PM: Started restoring cached node version
8:19:49 PM: Finished restoring cached node version
8:19:50 PM: v12.18.0 is already installed.
8:19:51 PM: Now using node v12.18.0 (npm v6.14.4)
8:19:51 PM: Started restoring cached build plugins
8:19:51 PM: Finished restoring cached build plugins
8:19:51 PM: Attempting ruby version 2.7.2, read from environment
8:19:53 PM: Using ruby version 2.7.2
8:19:53 PM: Using PHP version 5.6
8:19:53 PM: Started restoring cached node modules
8:19:53 PM: Finished restoring cached node modules
8:19:53 PM: Started restoring cached go cache
8:19:53 PM: Finished restoring cached go cache
8:19:54 PM: go version go1.14.4 linux/amd64
8:19:54 PM: go version go1.14.4 linux/amd64
8:19:54 PM: Installing missing commands
8:19:54 PM: Verify run directory
8:19:56 PM:
8:19:56 PM: ────────────────────────────────────────────────────────────────
8:19:56 PM: Netlify Build
8:19:56 PM: ────────────────────────────────────────────────────────────────
8:19:56 PM:
8:19:56 PM: ❯ Version
8:19:56 PM: @netlify/build 26.0.2
8:19:56 PM:
8:19:56 PM: ❯ Flags
8:19:56 PM: baseRelDir: true
8:19:56 PM: buildId: 61c4d9d4e6cf76000771ebd0
8:19:56 PM: deployId: 61c4d9d4e6cf76000771ebd2
8:19:56 PM:
8:19:56 PM: ❯ Current directory
8:19:56 PM: /opt/build/repo
8:19:56 PM:
8:19:56 PM: ❯ Config file
8:19:56 PM: No config file was defined: using default values.
8:19:56 PM:
8:19:56 PM: ❯ Context
8:19:56 PM: production
8:19:56 PM:
8:19:56 PM: ────────────────────────────────────────────────────────────────
8:19:56 PM: 1. Build command from Netlify app
8:19:56 PM: ────────────────────────────────────────────────────────────────
8:19:56 PM:
8:19:56 PM: $ npm run generate
8:19:56 PM: > borgs-v2@1.0.0 generate /opt/build/repo
8:19:56 PM: > nuxt generate
8:42:15 PM:
8:42:15 PM: (build.command completed in 22m 19.4s)
8:42:15 PM:
8:42:15 PM: ────────────────────────────────────────────────────────────────
8:42:15 PM: 2. Deploy site
8:42:15 PM: ────────────────────────────────────────────────────────────────
8:42:15 PM:
8:42:15 PM: Starting to deploy site from ‘dist’
8:42:28 PM: Creating deploy tree asynchronously
8:42:31 PM: Creating deploy upload records
8:43:29 PM: 143052 new files to upload
8:43:29 PM: 0 new functions to upload
9:32:40 PM: Site deploy was successfully initiated
9:32:40 PM:
9:32:40 PM: (Deploy site completed in 50m 24.9s)
9:32:40 PM:
9:32:40 PM: ────────────────────────────────────────────────────────────────
9:32:40 PM: Netlify Build Complete
9:32:40 PM: ────────────────────────────────────────────────────────────────
9:32:40 PM:
9:32:40 PM: (Netlify Build completed in 1h 12m 44.4s)
9:32:40 PM: Execution timed out after 1h12m55.155866894s
9:32:40 PM: Error running command: Command did not finish within the time limit
9:32:41 PM: Creating deploy upload records
9:32:41 PM: Failing build: Failed to build site
9:32:41 PM: Failed during stage ‘building site’: Command did not finish within the time limit
9:32:41 PM: Finished processing build request in 1h13m6.393087668s
9:43:46 PM: Starting post processing
9:43:46 PM: Post processing - HTML
10:55:05 PM: Post processing - header rules
10:55:05 PM: Post processing - redirect rules
10:55:05 PM: Post processing done
10:55:06 PM: Site is live 
netlify build && netlify deploy --dir=dist --prod yielded a very similar result. It threw a timeout error but continued to show the Waiting for deploy to go live... immediately after. I let it continue for another 30/40 minutes but nothing happened. The process of uploading took approximately 140 mins. Please see below:
Deploying to main site URL...
✔ Finished hashing 150928 files
✔ CDN requesting 100634 files
✔ Finished uploading 100634 assets
⠏ Waiting for deploy to go live... › Warning:
{
"name": "TimeoutError"
}
⠏ Waiting for deploy to go live...
Hi Thom,
Thanks for those additional details. At a high level, that site is just massive - these lines from your logs are going to give you big problems:
✔ Finished hashing 150928 files
✔ CDN requesting 100634 files
✔ Finished uploading 100634 assets
and
8:43:29 PM: 143052 new files to upload
As you’ve been seeing, deploying >100k files on a site on our service is a challenge. Most of our sites are much smaller and those that aren’t have to be very intentional about how they build and deploy their site. So, let’s talk about what could work and what would be a good sign that you should stop trying to make it work here 
First off, I applied some yet higher timeouts for your site that should hopefully make deploys from our CI environment (deploys built by us and uploaded from our service to our service) work a bit better.
However, you could (should, if you intend to continue NOT to use our form handling service on this site) make another one, by turning off form processing here: Netlify App as that is a big part of what takes a long time in our scanning each newly uploaded file.
To be clear, having a site this large isn’t a problem; uploading that many new files is a problem. Check out this article about how we handle uploading and caching files:
…and from it your takeaway should be "I’ll want to make sure that subsequent builds of my site change as few files as possible - hundreds, not hundreds of thousands should be your steady state target - and that will help you once you get a single successful deploy since we’ll have a copy of all the files in our database.
From your last CI deploy logs I do see that you managed to upload all files:
8:43:29 PM: 0 new functions to upload
9:32:40 PM: Site deploy was successfully initiated
9:32:40 PM:
9:32:40 PM: (Deploy site completed in 50m 24.9s)
(here they were uploaded)
9:32:41 PM: Finished processing build request in 1h13m6.393087668s
9:43:46 PM: Starting post processing
9:43:46 PM: Post processing - HTML
10:55:05 PM: Post processing - header rules
10:55:05 PM: Post processing - redirect rules
10:55:05 PM: Post processing done
10:55:06 PM: Site is live :sparkles:
and here they were preserved after upload since the deploy succeeded (despite the error about the build failing, which was in this case a red herring).
However - it seems that the CLI went to upload 100k different files after that, so I suspect your build is not creating identical outputs when rerun, at least for that 100k files.
You could also consider a change to site layout; for instance you could shard your site into multiple sites using a workflow like this: [Support Guide] Can I deploy multiple repositories in a single site? (in your case, one repo, but maybe stitching together several subdirectories? This would allow your deploys to be reasonably sized and not take forever to upload, since we only upload changed files.
So - I think you’ll want to focus on ensuring that your builds don’t change so many files and then our CI should work better; our CLI may not since ever for a site that size, as it doesn’t upload as fast as our CI environment does.
Hi @fool
Thanks for your continued assistance. I did as you suggested and disabled form detection as-well as reading through those articles…
My site was uploading thousands of new files due to the way Nuxt versions static assets in the distribution folder. At least I believe this to be the case since it was only the unix-timestamped directory that changed when comparing the same file from 2 different - but largely identical - deploys. I found this comment that recommends adding the following snippet to the nuxt.config.js generate property to circumvent this issue:
staticAssets: {
version: 'prod' // overwrites timestamp used in static asset versioning
}
The results from the first test deploy which took about 70 minutes:
✔ Finished hashing 68389 files
✔ CDN requesting 61726 files
✔ Finished uploading 61726 assets
✔ Deploy is live!
The result from the second deploy - where one additional page had been added - which took about 5 minutes:
✔ Finished hashing 68392 files
✔ CDN requesting 7 files
✔ Finished uploading 7 assets
✔ Deploy is live!
To note: there was a decrease in the number of files on the subsequent deploy without this addition to the config, but insignificant on a site of this size.
I did have a few more questions:
Is the aforementioned snippet something you’ve come across before and would recommend? I know Netlify & Nuxt are distinct entities but I’ve not been able to find the above in any of their documentation and am wondering whether this may actually be causing issues for me that I’ve just not yet detected.
Would upgrading my account decrease deployment times? Obviously the above is only useful when the code hasn’t changed.
Will I need to contact Netlify should I need to redeploy the whole site? Todays tests were of about 20,000 pages compared to the 50,000 from before which I got the impression Netlify staff were having to manually push over the line. If all goes well the site will have just over 100,000 pages.
Hi, @ThomCM
I’ve not see it before but I could recommend using the solution you have used.
We have a support guide which explains this which @fool linked to above called “Making the most of Netlify’s CDN cache”.
To summarize what it says, you don’t need to use “tricks” like hashes or timestamps to invalidate the cache at Netlify. Nuxt is using this “trick” but it should be disabled. It also appears you were successful in disabling the cache invalidation trick. We have a blog post with more information about the caching at Netlify here.
At the end of the day, yes, I would recommend the change you have made. We will automatically invalidate the cache if any files change, even without the “trick”.
There is a high-performance builds (HP Builds) package available on the Enterprise plan. This can provide more CPUs and memory for your build but I don’t know if that will improve the build time or not. That woudl depend on where the bottlenecks are in the site build and if the build can be parallelize to take advance or more CPUs or not.
The Enterprise plan’s pricing varies it starts at around $2000 monthly without the HP Builds and would be more with it. If you are interested in upgrading to Enterprise, we can have our sales team contact you.
Note, you might also build the site locally and use manual deploys to deploy at Netlify. If doing manual deploys locally, I could recommend using the Netlify CLI tool to make those deploys. You might test this with a new site to see if it works better for you.
@fool increase the post processing time out for that site. If you have other sites with these long deploy times, our support team can increase the post processing for those other sites as well.
The root cause of the failures is the timeout. If you don’t change tens of thousands of files each deploy, I don’t think you will have any problems. If you do change many tens of thousands of files, we recommend makes those changes in batches so that no individual deploy updates more than 10-20k files.
If there are other questions, please reply anytime.