Partial (re)Build - Adding files to netlify folder instead of regenerating everything

Partial rebuild of a static application

For ease of explaining I will simplify the problem a bit.

We have 200 blogs that are statically generated by a tool (CRA, next.js,… Could be anything really).

If we add 2 blogs, the entire application will build to generate a new output.

That is 200 pages (in multiple languages that are exactly the same as before) build unnecessary.

How could we just add these 2 pages to the output folder, without building the entire application again?

Kind regards

Bram

2 Likes

At this time, there is no way to update only parts of a site. Behind the scenes we do detect files which haven’t changed and we do not modify those. Cached version of pages, for example, will still load from the browser cache if they don’t change between deploys.

But as far as uploading a partial list of files and adding those over or to the existing deploy (similar to an overlay filesystem) is not possible at this time.

If there are other questions about this, please let us know and we’ll be happy to answer.

Is this a feature that is on the roadmap of Netlify?
I experience the same kind of “problems” and the process is slow because of no partial updates.

hi @samvloeberghs, i don’t believe this is on the roadmap at this time, but we can definitely update you here in case that changes. Sorry, i wish i had better news for you!

What I’m not really getting is we can update the files via the Netlify API? Why not just allow to add as well? It’s probably something technical that is holding you back, but at least then we would be able to somehow manage that ourselves, no?

Or what is the real purpose of this: https://open-api.netlify.com/#operation/getSiteFileByPathName

Because I’m still trying to set something like this up…

The problem I’m having is that the build time of the application is around 2-3 minutes, but the upload time takes almost 7-8. Which is really unacceptable for us.

Any ideas on how to work through this would be welcome @perry

The only other option I’m thinking about trying if this doesn’t work out, is to try the ZIP-file upload via the api.

Hi, @BramDecuypere, you can send us only the file contents for the change file. But each deploy must state all files and checksums included. If this isn’t done, the API wouldn’t know what files were included in that specific deploy.

Our service uses the concept of “atomic deploys”. Each deploy contains all files needed. If you want to roll back to a previous version of the site you can re-publish an old deploy at ay time as we know all the files in that version of the previous deploy.

If you could modify an existing deploy, then the deploys would no longer be atomic and there would be no way to roll back to a previous deploy.

Now, once the checksums and full file list are sent, our API will only ask you to upload new or changed files. If files didn’t change or if they exist in any previous deploys of the site, the API will not request you send them again.

If fact, if a new deploy happens to only include files previously included in other deploys (maybe deploy 4 is a mix of files already included in deploys 1 to 3), after sending the file list and checksums, our API won’t ask you to upload any files because it has them all already from previous deploys.

To clarify:

  • by definition all deploys must define all the paths for the files included and the checksums for those files
  • deploys are atomic and cannot be changed after deployment
  • if only one file changes or is added in a deploy only that one file needs to be sent to the API (but it still needs all checksums even for unchanged files)

So, yes, there is a design decision of “atomic deploys”. It enables being able to publish complete deploys atomically including rollbacks. The same design choice also means that it isn’t possible to modify an existing deploy.

If there are other questions, ask away! :+1:

The best solution I found is to use other service such as Azure Blob Storage or AWS S3 as a cache (crazy, I know), downloading all files from this cache first, then making partial update of these files, then uploading everything back to the cache and only then uploading to Netlify. It is a bit wacky, but it is way faster than re-generating thousands of pages every time.

Additional option here is to download https://yourwebsites.netlify.app/__all_contents.zip, unpack the contents, make partial update of that, then archive everything to __all_contents.zip and then upload everything to netlify.

Hi AlenPelin!

I’ve tried the zipping and unzipping, but it takes even more time! Haven’t found a great solution yet.
For now I’m just keeping it as is. It’s not perfect, but we can live with a 10 minute build for now and we just build hourly instead of on updates.

Thanks for jumping in!

1 Like

I guess instead of zipping we can create a file with checksums and use it instead as suggested by Luke