Terraform Cloud costs are an absolute joke. I was actually the decision maker a couple of years ago when we chose our IAC stack and decided not to go with Terraform Cloud despite thinking the product is strong, and it was entirely due to the business model and cost.
For us (and I'm guessing for many VC funded companies), SAAS managed IAC is a nice to have, but certainly not a MUST-have and certainly not something in the same "willing to spend money range" as your monitoring or your main cloud hosting costs.
I see these kind of tools as a tier below your monitoring tools like DataDog/Splunk etc. And these tools are a tier below your AWS/GCP costs. If your IAC or CI costs are approaching or overtaking your monitoring costs, something weird is afoot. Likewise, if your Datadog costs are approaching your AWS bill, this is obviously wrong.
Hashicorp, in my opinion, thinks their tool is more mission-critical and more of a value-add than in reality it actually is, and I think they also don't understand that in the current high interest rate environment, companies are FAR more willing to put in engineering time to do migrations or money saving projects. My own company put in 100s of hours of Engineering time to reduce the Datadog bill by roughly 40-50%.
Honest question. If you need to pump data into datadog from your cloud, why not just stop doing that, and use the cloud tools? Datadog isnt' creating metrics. The metrics are already on your cloud provider (gcp,aws,etc) -- AND all these providers have ways to view metrics. Why do people pay 20k per month for data dog? They dont' want to use GCP metrics explorer? Same with TF cloud. You can just setup a jenkins job and do all the features TF cloud has in a few hours. Why pay TF cloud? Are people confused? I must be missing something. I worked at startups that thought all metrics need to be in datadog, but they didnt' understand they can just use metrics explorer. You can't even use sql to create graphs in datadog, it's awful.
IMHO If you are serious, dump the metrics into bigquery/redshift and start doing sql like an adult.
If you start pulling all the metrics we currently use in DataDog out of something like BQ/RedShift you're going to spend WAY more in terms of engineering hours and infrastructure than we're currently paying DataDog.
I keep hearing something about a company that switched their monitoring from DataDog to DataBricks and I can see that yes you probably could go build a monitoring solution on top of a datastore like that. But I certainly wouldn't want to.
For viewing, cloudwatch is simply nowhere as convenient, writing whole custom queries takes a lot of time, and setting up altering for just the right thing sometimes requires you to copy the complex result as another metric. Basically I can create a time-shifted difference of two metrics reporting at different intervals, smooth it and make it alert the right teams through slack and pagerduty in a minute or so (while getting a visual feedback on the query the whole time) - this would take a significant amount of time to do through the plain AWS process.
It's a bit like "you can write everything you want in assembler without the overhead of extra layers of X". Yeah, sure, I can. But I appreciate the extra layers of X. I'll take the right cost/convenience balance, like an adult, thank you.
I would not consider metrics explorer to be a particularly good product. For queries where the query builder isn't good enough and you write PromQL instead, they don't even let you alias the legend; instead you must see the entire query as the label for that line.
A pretty minor nitpick, but indicative of the level of attention they give the product.
We use Atlantis [0] for CI/CD automation of Terraform pull requests to a centralized repository. It's pretty good too, especially for a self-hosted solution. I can't see how Terraform Cloud's costs would be justifiable for us without a custom contract.
At my last gig we built a bespoke version of this on top of our CI provider. Now i'm a technical co-founder in a startup and this is something I haven't quite solved yet, and this looks like exactly what we're after. Thanks!
How does this work? I thought the terraform state file was the single source of truth - if people are applying terraform 'manually' I assume that means on their local device? Are people sharing around the state file but don't have a central location for a lock file? Apologises if this seems obvious...
My assumption is that they're still using a remote backend for state, but they haven't set it up to use the locking features of the backend.
For example, I've use S3 as my Terraform backend for years, but I've never bothered to set up the locking feature, which uses DynamoDB.
In a small team that deploys Terraform changes rarely, you may never encounter the problems solved by using locking. Maybe good communication and a Slack channel works well enough for you.
I think Terraform Cloud (and most Hashi's enterprise offerings) aim at absolutel behemoths of deployments with many, many infra teams where the complexity comes from scale and the companies are not Google or Facebook and therefore this problem can't be solved through talent. For many such enterprises it's easier to throw money at the problem.
Except those same companies then run into hidden limits that exist on Terraform Cloud and hadn't been previously mentioned. Honestly, even bigger companies are better off going with Spacelift as it can scale extremely well, has reasonable support, and is much more feature rich.
As a disclaiming I'm currently writing a book on Terraform, and I've been interacting with a lot of the people in the space. Up until a few years ago I was a huge fan of Hashicorp, but their price changes and lack of support were what made it so I couldn't recommend Terraform Cloud anymore. The license changes they made were the icing on that cake.
You would think that, but having just went through trying to implement Terraform Cloud at work, its tools for doing anything that approaches behemoth deployment are abysmal. Permissions management is a nightmare and any sort of orchestration between different layers is barely there.
Although it's certainly not always the right strategy, it's certainly a valid strategy to serve a smaller number of relatively price-insensitive customers (and provide them with ancillary services/high-touch sales/etc. that they want). The fact that you may not personally be on the market for that sort of thing doesn't mean there isn't a market.
In particular think major regional non-technical companies, like grocery store chains or logistics companies or hospital networks. Beyond basic help desk support none of their IT is in-house and they don't want it to be in-house any more than an IT company wants to run an MRI machine for its employees, so they have dedicated a block of money to making that problem go away and don't really care how much it all costs as long as it comes in under that line.
I personally like what render.com is doing with IaC through their blueprints format. It's definitely a kid's toy version of IaC, but it's a really nice step up from Heroku.
What financial problems? They were bought out by another company and tanked. This is the exact reasons why you don't want to tie yourself to proprietary solutions.
Sure, the concept of buildpacks were copied to other platforms (I personally worked on building one of those platforms, Cloud Foundry), but they were a proprietary solution that others adopted for ease of transition off of Heroku, not because it was some great solution to IaC.
Why would I care about proprietary/opensource if it's affordable and saves time as a small dev shop? I just wanna ship and the company might be dead before any of this matters.
For us (and I'm guessing for many VC funded companies), SAAS managed IAC is a nice to have, but certainly not a MUST-have and certainly not something in the same "willing to spend money range" as your monitoring or your main cloud hosting costs.
I see these kind of tools as a tier below your monitoring tools like DataDog/Splunk etc. And these tools are a tier below your AWS/GCP costs. If your IAC or CI costs are approaching or overtaking your monitoring costs, something weird is afoot. Likewise, if your Datadog costs are approaching your AWS bill, this is obviously wrong.
Hashicorp, in my opinion, thinks their tool is more mission-critical and more of a value-add than in reality it actually is, and I think they also don't understand that in the current high interest rate environment, companies are FAR more willing to put in engineering time to do migrations or money saving projects. My own company put in 100s of hours of Engineering time to reduce the Datadog bill by roughly 40-50%.