Deploying a site using the file digest API

I’m working on a project that will allow users to deploy sites to their Netlify accounts.

After digging through the documentation, the open-api docs, the requests made on the Netlify dashboard, code for the netlify-api and netlify-cli packages, asking some friends who work at Netlify and trying a lot of things out, I’ve finally managed to get an initial version of my integration working (a deployment that contains a single html file), but I’m struggling to get past that. Here are the steps I’m currently taking (I’m coding in PHP in case that makes a difference, and would be willing to share all the code that powers this process):

  • First, I get the sha1 hashes for the files I want to upload. I’ve been testing with sites that have around 300 files (including images), but I expect way bigger sites once the project is launched, in case that makes a difference in any way.

  • Then, I try to create a deployment by POSTing to the /deploys endpoint and specifying the hashes. I was initially adding the async flag (as I plan to be handling large sites) but that caused the API request to return no content (not even a deploy id I could poll for status), so I temporarily removed the flag (I’d also be interested in help to make that work).

  • Then, I start PUTing the required files using the /deploy/:deployId/files/:path endpoint. For future reference, the path needs to be urlencoded (I think this is mentioned in the documentation but it was still confusing for me, as it looks like it’s part of the path). You should also make sure to get rid of trailing slashes at the start of the path, as it makes Netlify not recognize the file you’re uploading.

That’s where I’m stuck now. Netlify randomly throws empty 401 responses when uploading the files (I’m sending the token via the header). I also got some errors where Netlify complained that the file I was uploading was different to the one before, but I’m only getting the 401s now.

As I mentioned, I’m building a product where the core functionality relies on this API, so I’d love to get some help on this, ideally getting a contact of someone who works at this part of the platform and can guide me on properly implementing this.

Finally, thank you for making an awesome platform (and making it free, as a 17-year-old who started coding with HTML I can tell you it makes a difference). The developer experience at Netlify is awesome at almost all parts of the platform (except the API but I’m guessing that’s not where the focus is at the moment, and that’s understandable :D), and you’ve done an amazing job at making the web truly open for everyone. Thanks! :smiley:

1 Like

Hey @m1guelpf

Check out how deployments are handled via the our javascript SDK: https://github.com/netlify/js-client/tree/master/src/deploy

https://github.com/netlify/js-client/blob/master/src/deploy/index.js#L47-L50

Hey @DavidWells!

As I described I have already source-dived the implementation on the Netlify js client (that’s where I got my deploy steps from) and I’m still facing the issues described above :wink:

Hey @m1guelpf! I’m having very similar issues. Can you share all of your code? Maybe we can figure this out together. I’ve tried uploading zip files and the file digest way, still having issues.

Figured this out for anyone who’s wondering. I’ve outlined the basic steps to update a single, index.html file and the process. I am using Postman on an iMac, incase thats relevant. Not going through authorization setup either.

Create new deploy for your site. You will need to generate the sha1 of the file. On my mac the command is

shasum index.html

Once you have your sha1, you will pass that into your POST to create the new deploy in the files object as application/json. Example below of what I put in the body

{
	"files":{
       "index.html":"fc4c36f94cc2582625cd08c5c2cbf8dfce7a2baa"
    }
}

Now just send a POST request to this endpoint with above in the body

https://api.netlify.com/api/v1/sites/:site_id/deploys

You will receive an id of the deploy in the return object. Grab the id. I refreshed the UI just to make sure it said ‘uploading’ in the new deploy. The next step will be to run a PUT request to that file name, match the SHA1, and then upload the file. The SHA’s are important to make sure the integrity of the file is there.

Netlify is atomic and likes to re-upload the entire site if anything is changed. In Postman, you can view the files of that deploy to make sure its working as a GET request.

https://api.netlify.com/api/v1/deploys/:deploy_id/files

You should get an array back with the files it expects with the sha1’s. Now time to upload your new index.html file. Send a new PUT request to the following with the final parameter being the file name. In our case, since the file name is ‘index.html’, the :file_name is ‘index.html’. It also expects the request to be type “Content-Type application/octet-stream”, and the body to be the file contents. In postman you can go into the Body tab, select binary, and locate your file. If you are using cURL, I believe you can use --data-binary “@index.html” or something like that. Not 100% on using cURL.

https://api.netlify.com/api/v1/deploys/:deploy_id/files/index.html

If there are spaces, make sure you encode. If you want to test before hand to make sure this is working, send a GET request to the same endpoint. It should return an object if successful with the SHA1. For example if you named a file “/”. To get this file, the final parameter will be “%20/” (without quotes)

If successful, it should return the same object. Check out the UI or your site to see the new changes.

Happy coding! If you have questions, please ping me. A lot of the api documentation is still being developed.

3 Likes

Hey @garrettbland, thanks for sharing! :bowing_man::tada:

Would really love to see how to authenticate via POSTMAN:

This is about as far as I can get :frowning:

Not sure what the redirect is supposed to be. Any tips?

I’m not sure either; I don’t ever use that flow. I use the Auth token that you can make here:

And then I set an HTTP header of Authorization: Bearer MY-TOKEN-HERE

That should work in postman, too, I think?

1 Like

Thanks @fool for that suggestion - did the trick!

Now another speed bump. I don’t see a deploy_id key in the JSON response after initially sending my SHA1s. Would anyone know why? I also don’t see a single item listed in the required array. The documentation clearly says that I should see both of these things :frowning:

… which made me wonder if the deploy_id key is really just the id key from the response above. To test that assumption out, I went ahead an used id to PUT my index.html into that deploy. Unfortunately, I got the following error :confused:

You’re absolutely right that the id in the response you screenshotted is the deploy id you’re looking for- I’ve made a note to file that docs change request.

However, the PUT request should work if you follow the API call structure and use that id, which it looks like you are.

One reason why that required array may be returning empty is because you’ve already PUT the file and checksum to our CDN- if the checksum is cached on our CDN, we no longer need you to upload the file. The empty array will probably happen if you’re using the checksum in the docs example, since our CDN nodes likely have that checksum cached. One thing I might try is creating a different index.html file and running through this process with that file- new checksum, new file, nothing already cached, etc.

The other thing you’ll want to check on your PUT request is that you’re using the Content-Type: application/octet-stream header.

Want to give that a shot and let us know how it goes or if we can help further?

Hi, I tried the solution in python but i get the response
{“code”:422,“message”:“Uploaded file … did not match deploy”}
I double checked multiple times and the mac shasum matches the required SHA1 generated and returned.
Python Code for SHA1:
page_digest = hashlib.sha1(data).hexdigest()

Python Code for put request:
files = {‘file’: open(path, ‘rb’)}
r = requests.post(url, headers=headers, files=files)
What am I missing?

Hey @intellemo,
Did you get the 422 after the POST or the PUT? To recap @garrettbland’s solution in case that’s helpful:

Step 1
POST to here:
https://api.netlify.com/api/v1/sites/:site_id/deploys

with this:

{
	"files":{
       "index.html":"fc4c36f94cc2582625cd08c5c2cbf8dfce7a2baa"
    }
}

Step 2
PUT to here:
https://api.netlify.com/api/v1/deploys/:deploy_id/files/index.html

with index.html file contents and “Content-Type: application/octet-stream” header

It was after the PUT request.
I solved it.
The error was due to the Python request method that I was using.
I changed the method to pass the bytes as data and it worked.
Thanks for your help. :slight_smile:

1 Like