Netlify build and deploy of large scale websites (70,000+ pages)

Support
1

Hi,
I currently run several websites on the Netlify and I have a client who I am picking up 7 websites for.

I want to develop the new versions as static sites using HUGO and NetlifyCMS.

I have some concerns about deploy times, one of the sites is 70,000+ pages and the others range from 10,000 to 40,000 pages.

One of my current websites is about 200 pages and the build and deploy time looks to be around 1-2mins, which gives me concern for larger websites.

Does anyone have any stats about build and deploy times for large websites, of the same or larger scale than to the ones I mentioned above on your platform?

Thanks

2

Hi James,

I think the sales team reached out to you around custom settings that are available to enable large sites.

However for the edification of everyone else:

  1. building the sites using a system like hugo is totally doable - we have folks building 40k+ page sites on hugo in a few minutes.
  2. the uploading is the problem. It takes a fraction of a second to upload each changed file to our CDN and at 70k pages, we’re likely talking about near 100k total assets considering I assume you have graphics and JS and CSS files and directories and whatnot. So multiply even 5 per second which is close to the speed we see, and you get 20,000 seconds which is…well, a lot of hours. While we can tune our system to handle this load, and we do for customers in that situation, it is not enabled by default.

However, the good news is that most sites do NOT need to change every file with every deploy, and when you just add some content or change a few pages, the upload will be much faster as we only need to upload changed files - we don’t re-upload files we’ve seen before (so we don’t deploy eleventy million copies of jquery.js, for instance).

This article talks more about the pattern of not changing all files and how it can be beneficial for upload times, but also for site speed: [Support Guide] Making the most of Netlify's CDN cache

Serving the files is no problem - getting them onto the CDN is the time-consuming part and hopefully if you optimize things a bit (some builds are fine by default; some are not and e.g. contain a timestamp or a CSS/JS filename with an ever-changing hash that is used in every single html file) our CDN is definitely up to BUILDING & SERVING sites at that size, but uploading them may need a bit of help from us :slight_smile:

1 Like
3

Thanks for the response, I did have a chat with the sales team last night.
I had wondered about a partial release, and I’d assumed that everytime that hugo built the hash of files would be different.
I’ll have a look around that issue and see what options there are.

The sales team were supposed to be sending over some documentation and alternative solutions for me, but I’ve not seen any emails yet.

4

I pinged them to remind them to send :slight_smile:

2 Likes
5

thanks for that, hopefull there is something we can sort out.

1 Like