Skipping CDN file diff

Support
8

Thanks @luke

I’ve gone back to using a small test data set during development, so my immediate issue of using up all my build minutes is manageable again.

Just want to make sure I understand what you are saying, in the “a manual deploy” case you list, which is what I am doing via netlify-cli, there is no way to trigger a Netlify build after file upload but before deploy to the CDN, is that correct?

9

Hi, @mjgs. Correct, you cannot trigger builds within a manual deploy.

You can do a manual deploy and then, after it completes, trigger a new deploy what is a continuous deployment deploy with a build & deploy at Netlify. However, they are two separate deploys and the files must be checksummed in both. There is no escaping checking all file for all deploys.

I want to help you understand why this is required.

The API needs to know which files are the same and which files are different. It also must ensure that the uploaded files matches the local files. This is why all files are checksummed both before and after uploading.

Let’s say you site has 100 files. When you first deploy using a manual deploy this is the workflow:

  1. You build outside of Netlify.
  2. You checksum all the files to be deployed.
  3. You send the manifest of all filenames and checksums to the API.
  4. The API says “all files are new so send upload them all to the API”.
  5. You upload all the files to API.
  6. The API checksums the files again.
  7. If all checksums match (and there are no other errors) the deploy is successful.

Now, let’s say that you change five files and add five new files. The site now has 105 files and 10 of those have never been seen on this site before (10 because 5 are new and 5 are changed).

This is the workflow for the second deploy:

  1. You build outside of Netlify.
  2. You checksum all the files to be deployed.
  3. You send the manifest of all filenames and checksums to the API.
  4. The API says “only 10 files are new so only these 10 files to the API”.
  5. You upload the 5 new files and the 5 changed files the files to API.
  6. The API checksums the files upload again.
  7. If all checksums match (and there are no other errors) the deploy is successful.

This is why the checksums are required for every deploy. ​Please let us know if there are any other questions about why this requirement exists.

10

Thanks for the info @luke

I totally understand the need to checksum all the files during CDN upload. Makes sense.

I was hoping it would be possible to trigger a Netlify build after the file transfer, but before the CDN upload. That would have enabled the following workflow:

  1. Build in Github actions, outputting a gzipped tarball of the website
  2. Use Netlify-cli to upload the gzipped tarball to Netlify
  3. Trigger a small build script on Netlify that untars the website tarball
  4. Netlify diffs and uploads files to CDN

That would make the Github to Netlify deploy very quick because instead of transferring hundreds of Megabytes it would be just a few Megabytes.

That would ensure I don’t use up all my Github actions minutes.

Is anything like that possible?

Are there any plans to add compression to the Netlify-cli upload?

Seems like it would be an obvious way to speed up deploys for everyone.

11

Not a tarball, but we do support a plain zip for uploads using API. That probably won’t cut down on the transfer size, but it will send the entire zip to Netlify and then Netlify would have to diff it on its servers. I believe you can still save your GitHub Actions time in this manner.

Compression on the local device would still take time, so it’s a plus, minus situation.

12

Where would I find example code for an api deploy?

13

Hi, @mjgs. If you want to work with undocumented parts of the API, this support guide is the starting place:

https://answers.netlify.com/t/support-guide-understanding-and-using-netlifys-api/160

Please read that support guide above. If there are any questions after that, please let us know what steps you have tried so far and what the results were.

15

Thanks for sending the link.

I wasn’t aware that this part if the API was undocumented. Is deploy via api something other people are successfully doing currently?

16

Also from one of the docs you sent links to:

https://github.com/netlify/js-client#site-deployment

Support for site deployment has been removed from this package in version 7.0.0. You should consider using the deploy command of Netlify CLI.

Is site deployment via API still supported?

17

Hi, @mjgs. Yes, it is still supported. Every time someone does a drag and drop deploy, it is using this API.

Drag and drop is still a supported deployment method and it will continue to be. There are no plans at all to deprecate that feature or API endpoint.

18

A quick additional question about the diff.

So I was thinking maybe I’ll just upload the files that changed in the build, but it looks to me that no matter how I deploy all the existing files get deployed over the previous deploy, so what’s the point in the diff?

Is there a way to deploy just a few files?

19

I’m not sure how you got that @mjgs. In the entire conversation we’re having here, we’ve been telling you only the changed files get uploaded.

You need to send a list of all the files that you wish to upload, not the files itself. Once you send the list of files with their SHAs, we will request only the files that are missing on our CDN (as compared using SHAs).

Instead of going back-and-forth, it would be worth if you try it out yourself and experience it, so you can see it happen live.

20

Thanks for the reply. Sorry I wasn’t very clear in my last post.

Where I’m at with this is that I’m almost certainly not going to have any time to build any API stuff. The other thing that happened in the past couple of days is that the deploys jumped from 30-40mins to 3-4 mins, and I didn’t make any changes. It’s like everything was slowed down by a factor of 10 for about a week, then it’s ‘magically’ fine again. So perhaps it’s all moot, if the deploys continue to be in the 3-4 mins range.

In my previous post I was just saying that when I deploy using the cli, though I understand on your end you do a diff, from the end user perspective all existing files get replaced by the new set specified. So it’s like there is no point in a diff. I could not see a way, using the cli, to just deploy a few files, rather than the whole set.

If there is some way to configure things so I can upload just changed files via the cli, please let me know.

Hopefully I’m not coming across as argumentative, that’s definitely not intentional, it’s been quite a frustrating week, and just trying to describe the situation.

I’ve run out of build minutes now, so I’ll need to wait a few days to try anything new.

21

Hi, @mjgs. There is a false assumption being made here:

In my previous post I was just saying that when I deploy using the cli, though I understand on your end you do a diff, from the end user perspective all existing files get replaced by the new set specified. So it’s like there is no point in a diff. I could not see a way, using the cli, to just deploy a few files, rather than the whole set.

The CLI is already only sending a few changed files. This is already what happens. It is happening automatically and invisibly to you. However, that is what is happening.

  • The CLI sends the deployment API endpoint a list of all file paths and the SHA1 checksums for all files.
  • The API looks at all files in the deploy history for the site. It doesn’t just check the previous deploy. It checks all files in every deploy for this individual site.
  • The API then sends the CLI a list of the files where the SHA1 checksum has never been seen before.
  • The CLI then uploads only the files that have not been seen before for this site.

This means if a site has 1000 files but only 5 files are new or have changed, the API only requests those five files. The checksums prevent wasting time. They prevent the sending of the other 995 files which have been uploaded already.

However, this can only be done if you checksum all files for every deploy. This is why the checksumming is alway required. It is the only way for the API to know which files are new.

Without the checksums, then you would be forced to upload every file with every deploy. With the checksums, now only changed or new files are sent.

23

Thanks for the explanation.

The deploy times have jumped back up from a 3-4 mins to over 15-20 mins again, and I haven’t changed the size of the build.

I’m still confused about how this works. Can you help clarify this:

If I deploy 1000 files, then 500 files, the site ends up with 500 files.

If I deploy 1000 files, then just 1 file, the site will only contain 1 file.

How do I deploy 1000 files, then just one file, but end up with all original 999 files and the 1 new file?

25

@mjgs,

Sorry to say, but it looks like you should re-read this thread from the top. The questions you’re asking in your latest comments have all been answered already, and at this point, it simply appears we’re going in circles.

Here was a short version of the answer you’re now are looking for:

But, to break it down one more time (assuming you’ve made a deploy with 1000 files already):

  1. Send a API request containing the list of all files that you want the deploy to contain. So in your case, you need to send a request with the name and SHA1 of all 1000 files you want in the new deploy.
  2. Netlify will process the request, and send a response in which it will mention the required files for that deploy to complete. In your example, the required_files array in the response will only have 1 file.
  3. Send the file to Netlify API and the deploy would be completed.

In other words, that files that you expect the deploy to have should all be listed in the API request that you’d make. Only then if you upload the changed files, you don’t have to upload the remaining 999.

26

Thanks for your reply. Let me stop you right here:

Send a API request

As I said in my previous post, I won’t be able to use the API, I’m only able to use the CLI. I think maybe that’s where there’s a confusion on your side.

It would be really great if you could re-read my previous post with this in mind, and the example given and offer any comments.

27

I thought you’re talking about the CLI too, but when you said this:

I don’t see how you can do this with CLI, unless you’re manually selecting the files that you wish to deploy.

29

You just specify the folder to deploy that contains the files of your site. Does that make sense?

30

Hi, @mjgs. About this:

How do I deploy 1000 files, then just one file, but end up with all original 999 files and the 1 new file?

I have answered this same question several times in this thread already. The answer is:

  • There is no way to do this.

I do understand what it is that you want to do and why you want to do it. It would make things simpler for your workflow if the API did have an endpoint that allowed you to update an existing deploy by sending a list of files to add. However, our API has no such endpoint.

The deploy API always requires a full list of all files with their complete paths and all checksums for those files for every deploy. There is no escaping that requirement.

There is an open feature request for what you are asking for here:

There are currently no plans to complete that feature request. However, if many people were to add comments to that issue above saying this would help them as well, this might be reconsidered. Please do add a comment there if you want to see this changed.

31

@luke Thanks for your reply, it’s crystal clear now that deploying just one file into an existing site can’t be done via the CLI.

I’m still not clear on why the deploy times are jumping around from 3 mins to 60 mins with the same set of files. If a deploy went through in 3 mins via the CLI, isn’t it the case that subsequent deploys of the same set of files via the CLI should take a similar amount of time?

I’ve read through the feature request link you sent. Incremental deploys certainly sounds like what I’m looking to do, except I’d like to do it via the CLI.

Something like a flag that merged the deploy dir files into the existing files rather than replace them entirely.

Should I be making a feature request on the CLI project instead?