I'm afraid you're trying to make system administration look much harder than it actually is. As an example, adding good defaults to your nginx config is automated by certbot, or you could use caddy. You could run your apps statelessly by containerizing your applications or by simply writing Ansible playbooks and then not have to worry about upgrades - you simply deploy the application on the new server and spin down the old one.
As someone who has been doing this for some years now, it is not simple.
To be fair, you can write a simple Ansible playbook if Ansible isn't doing much for you. But if you're using Ansible to manage things which are themselves not simple (like "just install a Node runtime please") you are at the mercy of whatever shell script Ansible eventually ends up calling.
I've been through Ubuntu version updates and Ansible really didn't help with issues like "that package is now no longer in the PPA you got it from".
Administration doesn't take a lot of my time, but when I do need to do it, it can take a solid day of focus to make sure I know what I'm doing, make the changes, recover from the pitfalls I always forget (e.g. "this Ansible step fails in --check mode because it depends on a file that is created in an earlier step, which doesn't happen in check mode") then work through the inevitable issues. I wouldn't want to do it without Ansible, but it's not "simply" anything.
If you stick to shipping containers it gets rid of 99% of these problems at the cost of some extra storage for the N copies of runtimes. Then your base infrastructure is reduced to “something that runs containers” which can be anything from vanilla docker, to docker-compose, to one of the many diy-PaaS platforms, to a full blown k8s cluster.
I've maintained large systems and small systems. FreeBSD and Linux systems. I helped build and maintain the serverless platform that hosts the Netflix API. I managed build systems and CI/CD for docker images deployed to k8s and built a package manager to address the instability that is inherent with rebuilding artifacts the way tools like Docker do (which are already a marked improvement over Ansible).
Ansible is essentially logging in and running a series of shell scripts. This works great in isolation, but do it long enough and you'll realize a lot of things you thought were idempotent, atomic, and infallible are not. Most package managers are glorified tarballs with shell scripts wired up to lifecycle hooks during install. You YOLO unpack them into a global namespace and hope for the best. With any luck, when something surprising happens, you can just rerun your script to bring the server back into a good state. But often times the server just ends up borked and you have to throw it away and start over.
K8S somewhat addresses this by maintaining the desired state in a declarative format and comparing the actual state against the declared state in an eval loop. But K8S is absolutely massive and unbelievably complicated. Most declarative systems are non-trivial. The closest I've seen our industry get to this ideal is Nix.
Linux itself is a beast, a reliable beast, but it's a chunk of software I don't think you can just wave your hand at and say "this is easy!" It's easy because it works. When it doesn't work, it's absolutely not trivial.
And this is the core of it: everything you just listed off that makes server administration easy has no delegation of responsibility. They are abstractions that you ultimately own. When they stop working, that's your problem. The Ansible project has no vested interest in the health of your server or the success of your CI/CD pipeline. They have no engineers standing by to help you bring your site back up. That's all 100% you even if you've pushed it down under the covers.
Compare that to my serverless deployments. I pay a vendor to be responsible for everything I possibly can, and everything I end up being responsible for I keep as minimal as possible. These deployments aren't mine, they are my customers'. My customers are small to medium sized businesses (for my fortune 500 contracts, I build the systems you're talking about and a whole lot more). A small to medium sized business can not maintain Ansible. They are mechanics, plumbers, drywallers, etc. They are not Linux System Administrators. And I'm not here to milk them for money, I want to get in, get done, and leave them with a stable system that requires minimal maintenance. I do that by having vendors lined up that are responsible for the system running below my software and those vendor's support contracts are a lot cheaper than my weekly rate.
