Hello,
First, let me thank for brilliant API for uploading!
But I found a gap that forces API client stateful behaviour: “required” (files) json property is an empty list for deploys that are async (tested at least for async) and in ‘uploading’ state.
There is no way to resume upload process without storing current upload state (list of upload files) on Client side.
2nd inconvenient part is that while you can assign name “title” json property, there is no way to filter through deployment list do grab all resumable uploads (for now it is only one in ‘prepared’ state) and with the title. In other words there is no way to grab deploys that can be resumed with for automated API client.
Thank you for reaching out! I am sorry for the delayed response here.
I am going to bring your feedback to our team to get some additional insights for you. If you have any specific questions in the meantime, please include them in this thread.
We have shared your first point with our team. More to come in the future!
Regarding your second point, this sounds like a potential feature request. Could you share a bit more with me about your use case for this? What scenarios would this be beneficial in? The additional context will be useful for our team!
Unfortunately I can’t agree with you, as per documentation it should return list of files that should be uploaded.
So its is subject to update doc or implement as per documentation.
The case is simple:
Imagine that upload is failed due to networking, so there is no way to resume upload - the only thing I can do is or upload everything I get (~4Gb) or start new deploy with new required list.
Hi, @paw. Do you happen to have any links to the actual code you are using so we can test it to see what you are seeing? Are you preparing a deploy file manifest when this happens?
I ask because I believe there could be a misunderstanding about what the required property means and how it functions. That property never contains a list of files missing from in progress deploys.
Instead, when you send a deploy manifest (the files property) to the API containing a list of all files in a deploy and all checksums for those files, the required property contains a list of files which do not exist in any previous deploys. It doesn’t pertain to the current deploy. It pertains only to previous completed and successful deploys as they compare to the files property used when creating the current deploy.
For example, if the files list contains 100 files and 99 of those files exist in any other previous deploy of the same site, the required property will include only the name of the one file we haven’t seen before. You won’t need to upload the other 99 because they are already in the deploy history of the site.
If you are not preparing a manifest with a list of files and checksums, then all files are required because there is no way to verify what is new and what was previously seen for this site.
To summarize, the required property is not (and never has been) used to track the files already uploaded in the current deploy. The required property contains a list of files which are not included in any previous deploys for the same site and it is only used when a files property is sent when creating the deploy.
If you have other questions about this, would you please answer the questions above so we can better understand how the API is being used?
Thanks for explanation, I think I get the paradigm (btw, it is really great!).
And behaviour of “required” in that paradigm for “uploading” deploys is confusing.
I have simple state machine that determines behaviour based on deploy status, so nothing interesting. Instead I just share scenario:
Assume that we have a site prepared
Start async deployment with manifest POST: {{baseUrl}}/sites/:site_id/deploys?title=test_deploy
When deployment status is “prepared”
Grab “required” file list
Upload “required”
Deployment changes its status to “uploading”
If failed and has correct status retry from step 4
And here the problem:
There is no way to resume upload for deployment in “uploading” status - step 4 if status is “uploading” always return empty “required”.
Desired solution: Return required files for deployments in “uploading” status.
Possible solution: Update documentation describing the case.
Sorry - I think I must be missing something. Are you saying you can’t cache the return from the initial POST call? I think that should have the required files list, which you can then use later in your process, without making any additional queries - right?
There is an empty list as a value of “required” json property if a deployment
I’m using getDeploy Operation for checking status of deployment and grabbing “required” json property.
Can you please confirm you can reproduce the bug.
If yes then we have 2 solutions:
Fix it to keep REST API stateless
-Or update documentation
Hey @paw,
Are you calling the API with our js-client? We’ve implemented retries on various kinds of failures there, like in this issue: Retry request on ETIMEDOUT · Issue #166 · netlify/js-client · GitHub If you’re seeing a different timeout error, that repository would be the place to file the bug.
You may also want to check out Netlify CLI- it also handles retries in many cases, or you can set your own timeout. You can read more about the netlify deploy command here:
We do have an open feature request for more filterability on the getDeploy call- I’ve added this thread to that conversation so we can be sure to follow up here if we ship any updates.
Thanks a lot, but I have my dedicated java application.
The bug is also reproducible in curl and postman, simply because it is a gap in REST API.
And yes I can implement different kinds of retries and save state of upload locally, but we losing here the idea of clean API.
Hey @paw,
I discussed this with our backend team and they have opened an issue to improve the documentation for the getDeploy endpoint to indicate that the required field will only be populated if the deploy is prepared and not in any other state like uploading. Thanks for flagging this.
