Skipping CDN file diff

Thanks for the explanation.

The deploy times have jumped back up from a 3-4 mins to over 15-20 mins again, and I haven’t changed the size of the build.

I’m still confused about how this works. Can you help clarify this:

If I deploy 1000 files, then 500 files, the site ends up with 500 files.

If I deploy 1000 files, then just 1 file, the site will only contain 1 file.

How do I deploy 1000 files, then just one file, but end up with all original 999 files and the 1 new file?

@mjgs,

Sorry to say, but it looks like you should re-read this thread from the top. The questions you’re asking in your latest comments have all been answered already, and at this point, it simply appears we’re going in circles.

Here was a short version of the answer you’re now are looking for:

But, to break it down one more time (assuming you’ve made a deploy with 1000 files already):

  1. Send a API request containing the list of all files that you want the deploy to contain. So in your case, you need to send a request with the name and SHA1 of all 1000 files you want in the new deploy.
  2. Netlify will process the request, and send a response in which it will mention the required files for that deploy to complete. In your example, the required_files array in the response will only have 1 file.
  3. Send the file to Netlify API and the deploy would be completed.

In other words, that files that you expect the deploy to have should all be listed in the API request that you’d make. Only then if you upload the changed files, you don’t have to upload the remaining 999.

Thanks for your reply. Let me stop you right here:

Send a API request

As I said in my previous post, I won’t be able to use the API, I’m only able to use the CLI. I think maybe that’s where there’s a confusion on your side.

It would be really great if you could re-read my previous post with this in mind, and the example given and offer any comments.

I thought you’re talking about the CLI too, but when you said this:

I don’t see how you can do this with CLI, unless you’re manually selecting the files that you wish to deploy.

You just specify the folder to deploy that contains the files of your site. Does that make sense?

Hi, @mjgs. About this:

How do I deploy 1000 files, then just one file, but end up with all original 999 files and the 1 new file?

I have answered this same question several times in this thread already. The answer is:

  • There is no way to do this.

I do understand what it is that you want to do and why you want to do it. It would make things simpler for your workflow if the API did have an endpoint that allowed you to update an existing deploy by sending a list of files to add. However, our API has no such endpoint.

The deploy API always requires a full list of all files with their complete paths and all checksums for those files for every deploy. There is no escaping that requirement.

There is an open feature request for what you are asking for here:

There are currently no plans to complete that feature request. However, if many people were to add comments to that issue above saying this would help them as well, this might be reconsidered. Please do add a comment there if you want to see this changed.

@luke Thanks for your reply, it’s crystal clear now that deploying just one file into an existing site can’t be done via the CLI.

I’m still not clear on why the deploy times are jumping around from 3 mins to 60 mins with the same set of files. If a deploy went through in 3 mins via the CLI, isn’t it the case that subsequent deploys of the same set of files via the CLI should take a similar amount of time?

I’ve read through the feature request link you sent. Incremental deploys certainly sounds like what I’m looking to do, except I’d like to do it via the CLI.

Something like a flag that merged the deploy dir files into the existing files rather than replace them entirely.

Should I be making a feature request on the CLI project instead?

Hi, @mjgs. The feature request already exists. I’ve linked you to it above.

About the deploy times, I believe you said this was happening in a build and deploy at GitHub. Is that correct?

If so, could you share the complete deploy logs with us here?

If the build is happening at Netlify, would you please link us to the deploy logs (or share the deploy id where you see the slow deploy occurring)?

Thanks Luke

The feature request already exists

It only exists on the api, as I’ve said several times already, I’m using the cli

If so, could you share the complete deploy logs with us here?

I’ll have a look and see if that’s available. With all the issues that happened, I’ve hit package storage limits and have had to delete logs and packages (currently metered together) in order to continue being able to deploy.

I’ll also have a look in the Netlify logs, though the build is on GitHub, you will probably be able to see the long deploy times in your logs.

Hi @luke

I’ve had a look on Github and it’s as I expected, all jobs are currently showing

The logs for this run have expired and are no longer available.

And there’s a similar message on Netlify:

[ERROR] Deploy logs are currently unavailable. We are working on resolving the issue.

I just noticed that although there is an error message in the Netlify console, the deploy still happened. Here are two examples that show a slow deploy, and a fast deploy of the same site:

Slow (30+ mins)

Quick (2mins)

About this:

It only exists on the api, as I’ve said several times already, I’m using the cli

The CLI is a wrapper around the API. Yes, the CLI does other things beyond just allowing API access but at its core - it’s just an API wrapper. The API is where the support must be added. Once that API endpoint exists the CLI can be extended to use it.

No changes can happen for the CLI until the API endpoint is created. This is why the feature request is on the API. That is the starting place for adding this support.

About this:

Slow (30+ mins)

Netlify App

Quick (2mins)

Netlify App

If examine the messages at the very top of both deploys, the slower one says this:

  • 17028 new files uploaded
    17022 generated pages and 6 assets changed.

While the faster one says this:

  • 7 new files uploaded
    1 generated page and 6 assets changed.

The slower deploy had 17251 files and 17028 of those were new. All 17028 files needed to be uploaded to Netlify. All of those files also needed to be analyzed for post-processing (as post-processing is enabled for this site).

The faster deploy actually had more files than the slower deploy. (The faster one had 17278 which is 27 more than the 17251 in the slower deploy.) However, despite having more files, it was much faster. This is the behavior I’ve been talking about.

This second deploy is faster because the API only needed the one new file and six changed files to be uploaded. As only seven files were new or change, only seven file needed to be checked for post-processing. The other 17271 files did not need to be uploaded because all have already be uploaded during previous deploys. The other 17271 files didn’t need to be post-processed for this deploy because it was also done during previous deploys. The post-processing is only needed for the new or changed files.

To summarize, the slower deploy uploaded and did post-processing for 17028 files. The faster deploy uploaded and did post-processing for only 7 files.

Also, your site might not even need the post-processing. By this I mean it is enabled but it might not be providing much benefit to the site. For example, most modern frameworks do the same code minifying we do automatically. If your framework does so already, there is no point in doing it a second time at Netlify. You could test disabling that feature here and making a new deploy:

https://app.netlify.com/sites/gmzh/settings/deploys#post-processing

That won’t change the file upload times. However, it will completely remove the time spent on post-processing. For the slower deploy, the post-processing was about half of the deploy time. Removing the post-processing would reduce the deploy time significantly for your deploys with many new or changed files.

​Please let us know if there are remaining questions.

Thanks for the explanation.

I’ll look into the post processing setting you mentioned.

So far today all the deploys have been very quick so perhaps it was just that all the files were new.

Hi @mjgs :wave:t6: ! Thanks for letting us know and please don’t hesitate to reach out if you have any other questions. :smiling_face_with_three_hearts: