You have to understand that when IaC was new, the marketing was “it’s so simple you can just write YAML/JSON/etc” because frankly the industry was too dumb to understand that “using a real programming language to generate a description of the desired resource state” and “using a real programming language to imperatively reconcile the current and desired states oneself” are different things. So Terraform began with something that resembled YAML in its static-ness, and over time, more power was required so they would bolt on a dynamic feature but were reluctant to give the impression that they were building a programming language so the feature would be as obscure as possible. But that wouldn’t be enough either so they would add still more dynamic features, each comparably obscure until in time they’d built a complete, obscure programming language.

But this wasn’t just Terraform! The entire industry did this too. CloudFormation began as simple JSON, but over time they allowed you to encode the abstract syntax tree of a shitty programming language in your YAML, and CloudFormation would interpret it. However stupid that may sound, in the Kubernetes world, we have Helm which lets you generate YAML with text templates which is honestly the dumbest idea in the world (imagine a compiler that generates syntactically invalid machine code if the input program has an extra white space character).

Of course in all of these cases the answer is staring us in the face: use a static language (YAML, JSON, etc) to describe the desired state, and use a higher level language (like Python or Starlark or Dhall or etc) to generate that static desired state description. The only thing Terraform (or any IaC tool) should care about is the YAML description. That it is generated from Starlark or TypeScript is just an implementation detail.

Instead of that, though, we get CDKs which are so close, but admittedly I haven’t used them in anger yet.

One of the best parts of CloudFormation was their introduction of Macros. You can take either your whole template or just a snippet, and perform dynamic transformations by calling a lambda. I'll admit I've gone so far as being able to embed ERB (Ruby) into my templates in order to more dynamically define some resources based on stack parameters. I can also create N resources with common configuration based on the values of a CommaDelimitedList.

I think the idea here is that macros are neat in any language, but in CloudFormation they can help automate stuff that is only difficult because of CloudFormation, and the macros themselves are harder to use than those in a normal programming language. In all cases, I think it’s strictly less nice than generating your CloudFormation YAML with Python or similar.

The problem of generating your own YAML is you end up having to maintain multiple copies of nearly identical templates and keeping them in S3. I have done a bit of that and maintaining those as build-time assets rather than run-time assets is less appealing. Granted, that's required whenever you get into the realm of dynamically-determined sets of parameters.

Yes, this is another problem with cloudformation, although IIRC I just used “aws cloudformation deploy” and let it manage my s3 resources.

I think this is less that it is opinionated but more that the HCL evaluation feels like a pile of hacks. There are unclear rules on what can be evaluated when and what dependencies are possible. Part of this is so that `plan` can work as it does, but it seems like there are just major gaps in general. For example providers can't depend on resources. This makes it very difficult to for example set up EKS then use the kubernetes provider to manage the resources in the cluster. The solution is obviously separate stacks but that brings in a whole bunch of other problems.

I think Terraform is quite possibly the best tool available, but there are clear flaws with both the model and the implementation. I think if I were to make a Terraform v2 I would make `plan` completely pure. This would avoid the provider issues, make validation and testing in CI easier and a whole bunch of other benefits. Of course there are downsides. For example EC2 instance IDs are random so you can't just include them in your pure plan. You would need some type of placeholder that is used for evaluation. This does cause some issues as it limits the operations that you can do with that value (so you can't pick the instance size based on the random instance ID) but overall I don't think it would be a major issue if the final substitution was handled well by the framework.