Infrastructure as code options?

We are trying to manage all of our servers/services via an infrastructure as code model, where we have a separate code repo (in terraform/terragrunt) that specifies what versions of other repos should be deployed/live for different environments.

I’m struggling to find a way to work with Netlify in this manner. There are terraform providers for setting up a site or creating new build hooks, but what we’re trying to do is along the lines of updating a SHA that’s checked in to the terraform repo and having that trigger a deploy to Netlify (typically, a terraform module would check to see what’s CURRENTLY deployed and whether it’s up to date and then if not, use the new value to trigger the deployment).

One idea I had was to stop auto-publishing and use terraform to change the git tag or SHA that is deployed on Netlify, but as far as I can tell, Netlify ONLY allows you to configure a branch and always sets the HEAD of the branch. I do understand that we can control builds and deploys without using Netlify’s build but I see no way to control this in a configuration-based manner.

I’m wondering if anyone out there is using Netlify with terraform or other methodology for this kind of workflow? (Please don’t just forward URLs to terraform providers without explaining something about them, I’ve been combing through docs and don’t see a straightforward path)!

Thanks for any ideas/help…

Hi @uooq,

You’re right about this:

Netlify will always build for the latest commit on a branch - even if you try to retry a specific deploy, it’s always the HEAD that’s fetched and re-built.

We currently don’t have a way to specify which commit to build for, so chances are you would not be able to make use of Netlify’s build system.

The only way that might make this remotely possible is that you can use [skip ci] or [skip netlify] anywhere in the commit message which could prevent any builds for that commit. Then, you can perform any kind of checks that you need and trigger a build by sending in another commit without those flags in the commit message and sending the content of that particular commit as the head.

So, the workflow could be something like:

Push to your repo with one of the flags in the commit message → do your stuff → using Netlify API, change the published branch of your website to a new branch name → push the required commit’s content to that new branch → Netlify will build for that commit → Delete the branch.

Not a perfect workflow, but might work.

1 Like

To add to Hrishikesh’s comments, @uooq - yup, there is no maintained terraform provider for Netlify for you to use. The archived one: https://github.com/hashicorp/terraform-provider-netlify worked well in the past but Hashicorp was not interested in maintaining it and offered to transfer it to us, but never did, instead archiving it. You could definitely fork and extend for your use if you insist on managing things using exactly that software. I don’t think that has been maintained for some time so may need some updates to get working, and you’ll see how to figure those out below.

My advice for finding a solution is to take a step back to get more clarity about what you are trying to accomplish. What you have asked for is help with a solution which you are proposing - we do better when you bring us a problem: “I want to manage my netlify config in an automated fashion” - great, we have a REST API (https://open-api.netlify.com/) and a JS library (https://github.com/netlify/js-client) to do so! Want to use terraform? OK, there is a starting place in the repo I linked + the API docs I also linked.

Hope something in there can help you accomplish your goals!

1 Like

Thanks for the thoughtful reply-- I will gladly take you up on your suggestion to describe the end result we want!

Background

  • We already use terraform to manage our non-Netlify backend deployments-- servers, databases, dns configurations, etc. Terraform gives us two things that we really like/want to keep & extend
  • Benefit 1: For each tier of our environment (e.g. dev, staging, pre-production, production) we know exactly what is deployed where because that configuration is checked in (i.e. the terraform repo is a documentation of our whole system).
  • Benefit 2: Terraform itself keeps track of what needs to change if we edit that repo. So, if I change the SHA of the API server for the staging environment, terraform looks at our current staging environment and knows that it needs to be updated to that SHA but recognizes that our job server is current/does not need to be refreshed.

So-- what we’re looking for is SOME way to extend this type of system to our static webapp-- a complex but purely browser-based application that is frequently updated and has close connections to our api server.

We’d like not just a means of triggering new deployments via api calls or by pushing to a specific branch, but a terraform module that is smart enough to know when the current deployment is current or needs something new to happen. That smarts of detecting the difference between the code description of what’s deployed and the actual version of what’s deployed is fundamental to our workflow and our infrastructure as code goals.

Now, I’m 100% sure that given a little time and practice, we could ourselves write a new terraform module that would wrap the full-featured Netlify apis to do just this for us! That being said, we’re not a devops shop and we’ve got a long laundry list of “to dos” for our own application that mean that we’re not going to tackle this kind of project in the foreseeable future.

if we can’t find something out there that fits into our workflow, we’ll do one of two things: create some CI automation to do netlify deployments or start using some designated branches (e.g. having a branch for “pre-production” and one for “staging” etc) and just leave our front-end out of our terraform/infrastructure as code system altogether. But in either case, we’ll be losing some aspects of the workflow and inherent documentation we get by having a full-featured terraform provider that doesn’t just provision sites, but can also manage which version of our code is on the site.

I know I’ve been a little specific about HOW we use terraform here, and we’re more than willing to use another system— the key parts are that we have a way to record in code (separate from the code being deployed) what version of our apps should be currently running in each environment and that we can manage changes to that versioning via pull requests and merges to the infrastructure codebase.

Regards,
Boon

Hi, would still love to hear your thoughts-- I think I did what you asked and explained the scenario rather than the proposed solution.

@uooq, the specifications as you described them do not fit our current system. It would take significant changes to our services to create the terraform provider as you have described the requirements (tracking deployment by SHA or git tag). To be clear, a terraform provider itself is possible - just not with the requirement as you have defined them.

If you would like to speak to our sales team about a custom solution, please let us know and we will have them contact you. Again, though, the SHA/tag level tracking is not supported at this time.

Wait, what?! It seems like @fool was suggesting that I explain the scenario more generally so you could suggest a solution. I’m NOT saying it needs to be terraform-- I explained what we DO with terraform but I’m happy to use a different system.

I’m not going to hire you guys to create a custom solution. My original question stands: do you have any suggestions for infrastructure as code type workflow. It sounds like you’re saying “No” but it also seems like you made me jump through a lot of hoops to get that No answer from you.

Hi again Boon,

We asked you to explain your use case in detail, to see if we could come up with an idea about how to implement it using today’s features on our platform. That’s our Support team’s primary goal: enable your success with our product. When you bring a solution like “I want terraform to manage X”, we can’t do that as well. However, after reading your details - with or without terraform, today, we don’t store data about git tags, nor can we do anything useful with commit refs unless you have successfully (or tried unsuccessfully) deployed from that ref as notified by your git provider - in which case you can find and activate or rollback that deploy via our API (albeit slowly, since there is not a query by sha, but a loop over all deploys).

Can you still reason about or be sure about what’s deployed using what we have already? I think so. You use locked deploys and only promote to production commits that you trust; you can see the commit it came from when you are doing automated operations.

We aren’t experts in automation and so can’t speak to the available tools outside our platform, so our experience and advice comes through that filter.

But it is my read that you want to control which deploys are made using our API without having git notify us about each commit, and to do that you will in fact have to do one of these things:

we could ourselves write a new terraform module that would wrap the full-featured Netlify apis to do just this for us! That being said, we’re not a devops shop and we’ve got a long laundry list of “to dos” for our own application that mean that we’re not going to tackle this kind of project in the foreseeable future.

or

if we can’t find something out there that fits into our workflow, we’ll do one of two things: create some CI automation to do netlify deployments or start using some designated branches (e.g. having a branch for “pre-production” and one for “staging” etc)

I think that “create some CI automation to do Netlify deployments” may actually be the best path, since then you know exactly what is being deployed before you send it. You don’t have to use our build system; you don’t even have to deploy from it (you could build, save to git, do QA on the output, then manually deploy once QA is complete, using those files and no build command).

Regardless, I hope you don’t think you wasted your time in describing your use case; it is indeed not something our system enables in the way you want, after several fairly smart product experts thought about it :slight_smile:

PS: Mind being a bit kinder to us in your responses? You come across as really angry and insulting some of the time and it’s pretty offputting; we’re humans and we are trying to help.

1 Like

I apologize for sounding that way-- I felt like I was VERY pleased by your initial response and wanted to answer it thoughtfully and spent a long time writing it. I was surprised not to hear back and felt myself rather put out by hearing back after a long delay by someone else altogether. On my end it felt like my own time wasn’t really being respected. A simple: “sorry, we don’t support that kind of scenario” would have been preferable, but I now acknowledge that I didn’t take into account the possibility that this wasn’t clear to you at the outset.

I do appreciate your attempts to help and I apologize for any disrespect in my tone. I find myself frustrated with your product and platform and that frustration shows, but you are entirely in the right that I should not express that frustration towards the people supporting the product and platform.

The frustration itself stems from Netlify being SO CLOSE to a great solution for a sophisticated SPA web app, but not quite there. So many things are very easy/solved out-of-box with your platform and so it’s quite a blow to discover that the last bit that would make it a great solution instead of rolling-our-own is missing… I understand also that perhaps our use case does not represent that of your target customer or the meatiest business opportunity.

Thanks for the time and apologies again

1 Like

Thanks for the help with the kindness, and sorry we don’t have a better story for this today.

Just as an FYI, we have passed this thread over to our Product Management team who construct our roadmap as some valuable input into what we could build (without you paying us to :). You are exactly the kind of customer they like to talk with about how to plan the future of our product. While I cannot promise they’ll be in touch, it is a distinct possibility (and you can ignore them or tell them you don’t want to talk if you like). I can however promise that they will read and consider all of your points here, which are totally valuable, and well presented, feedback - so thank you very much for that!

1 Like