In my opinion, databases are not cattle, and don't need to be automatically created (and destroyed!) in your main Terraform plan.
It's perfectly OK to have a completely separate Terraform project that just configures the DB initially (or even manually, I see lots of places running DB's that predate Terraform with immutable infrastructure for everything else), and applies minor non-destructive changes in the future. This way you get the benefits of IaC, but the DB plan doesn't participate with the rest of your infrastructure that IS ok to blow away and re-create at will.
BTW, Amazon RDS backups work the exact same way: Destroy the database and the backups are also destroyed. Therefore, same region automated RDS backups are fine for day-to-day, but in a true "DB goes poof" disaster you should expect that you WILL lose them too! You need cross-region, or even better, cross-account DB replication or snapshots to survive this.
> You need cross-region, or even better, cross-account DB replication or snapshots to survive this.
Excellent point.
AWS provide a couple of decent options for cross account backups, the Aurora Snapshot Tool [1] and the AWS Backup service. I've used both successfully.
It's worth checking out "terragrunt" it helps one breakdown larger terraform sites into re-usable modules which can be ran independently without having to redefine common configuration such as backends.
It allows for running a plan or apply on the whole site, or each individual module in the site.
So in this case it's easy to just apply the database changes, and then run the whole site plan to make sure everything is indeed in the right state.
Databases (or cattle, how or what ever they may be) are always created as a separate stack in your IaC of choice. The values you need for this stack are always outputted for consumption into other pipelines.
The benefit being you can make lots of small changes to your upstream stacks without breaking your database. And you can place additional controls around the database stack to prevent instance replacement from occurring.
> Databases (or cattle, how or what ever they may be)
It's a reference to the idea that you should treat your servers as "cattle, not pets". In other words, they're disposable and you should be fine with destroying one and creating a new one to take its place.
The parent is saying that it's fine to treat databases more like pets, where they're pampered and looked after carefully, and fed treats - much like DBAs are, I believe.
That's chock full of bugs unfortunately. For example, if you lock an Azure DNS Zone to prevent it being deleted, you then cannot delete any DNS record under it! It's a strict hierarchy, there's no way to turn off the inheritance.
(It is possible to create a custom RBAC role that excludes zone deletion only, but this is very fiddly and not-quite-the-same in complex ways.)
One of the benefits of creating databases in terraform or some other IaaS tool is having the DBA teams pre-define files to create databases along with the standard set of monitoring, logging, and other devops goodness. That way everyone gets the same set of “golden signals” monitoring and dash boarding for free.
It's perfectly OK to have a completely separate Terraform project that just configures the DB initially (or even manually, I see lots of places running DB's that predate Terraform with immutable infrastructure for everything else), and applies minor non-destructive changes in the future. This way you get the benefits of IaC, but the DB plan doesn't participate with the rest of your infrastructure that IS ok to blow away and re-create at will.
BTW, Amazon RDS backups work the exact same way: Destroy the database and the backups are also destroyed. Therefore, same region automated RDS backups are fine for day-to-day, but in a true "DB goes poof" disaster you should expect that you WILL lose them too! You need cross-region, or even better, cross-account DB replication or snapshots to survive this.