Failing build upload

[Voorhoede]

Hi there!

I have a static website of around 80000 pages. I know that’s a lot. My build is succeeding, but uploading the distribution fails. The first time I deployed, it failed on me after some time. See the screenshot below:

12:44:04 PM: Build script success
12:44:04 PM: Starting to deploy site from 'dist'
12:44:08 PM: Creating deploy tree asynchronously
12:44:11 PM: Creating deploy upload records
12:44:56 PM: 76173 new files to upload
12:44:56 PM: 0 new functions to upload
1:10:09 PM: Failed to upload file: posts/47480/index.html
1:10:10 PM: Failed to upload file: posts/47477/index.html
1:10:12 PM: Failed to upload file: posts/47479/index.html
1:10:17 PM: Failed to upload file: posts/47475/index.html
1:10:18 PM: Failed to upload file: posts/47476/index.html
1:10:19 PM: Failed to upload file: posts/47478/index.html
1:10:22 PM: Failed to upload file: posts/47474/index.html
1:10:51 PM: Failed to upload file: posts/47473/index.html
1:11:02 PM: Failed to upload file: posts/47481/index.html
1:11:12 PM: failed during stage 'deploying site': Failed to execute deploy: Upload cancelled: posts/47482/index.html
1:11:12 PM: Failed to upload file: posts/47482/index.html
1:11:12 PM: Failing build: Failed to deploy site

After that I manually triggered the build again and it succeeded. It looks like most of the uploading was done in the previous build and the rest was done in the second build. See the image below:

8:45:10 PM: Build script success
8:45:10 PM: Starting to deploy site from 'dist'
8:45:14 PM: Creating deploy tree asynchronously
8:45:17 PM: Creating deploy upload records
8:46:06 PM: 13704 new files to upload
8:46:06 PM: 0 new functions to upload
8:50:35 PM: Finished processing build request in 11m59.689325838s
8:54:10 PM: Starting post processing
11:47:49 PM: Post processing done
11:47:49 PM: Site is live

My question is: What can I do to have more success in my upload when building the complete website?

I know there are related issues and also some documentation around this. E.g.:

• Build troubleshooting tips | Netlify Docs
• [Support Guide] Making the most of Netlify's CDN cache

All of these don’t really give me any option, but build a less large website.
I did a lot of stuff to speed up the build already:

• I created this intra-build cache that I use to speed up the build.
• I make heavily use of webhooks to just update pages that have changed.

But sometimes I just have to build the whole site.

Help would be much appreciated!

[fool]

The main thing you can do is never upload so many files at once. This article describes how to avoid changing every file, and thus needing to reupload it, in the future:

[Support Guide] Making the most of Netlify's CDN cache

Last reviewed by Netlify Support - August, 2023 Our CDN derives much of its efficiency from files whose checksums don’t change between deploys. For instance, in a blog, you usually write or update one article in one file, while the rest of the content remains unchanged. In this typical use case, we don’t need to re-upload and re-process these files, since our CDN already has them stored - no checksums have changed except the one file. This also means that your visitors’ browsers don’t need to …

Is that a possibility in your workflow?

[Voorhoede]

Thanks for your response! I do a lot of those things already, but I do have around 80000 detail pages. The build where it failed on me actually was the first time I build the complete website so there was no cache yet. And once in a while I just have to invalidate all of those detail pages. So I don’t think that would always work with my workflow. I already:

• created an intra-build cache that I use to speed up the build.
• make heavily use of webhooks to just update pages that have changed.

Do you have any other suggestions? Any advice or direction will do!

[luke]

Are you using any of our post processing features or forms for this site?

If not, our support team can disable those entirely for this site and that is known to improve post processing time for deploys.

We can also extend the build timeout for the site to see if that allows the deploy to complete.

If you want us to do so please send us a link to the site in the Netlify UI or send us the API ID (not the key, just the ID) for the site. We will be happy to make both changes to see if this helps.