What I'd like to see is something for which adding capacity is plug&play: connect your machine to some network, boot it and automagically it will appear in your farm's UI to become a node. Like how pluging a USB stick in your PC gives you some GB of storage.
Quite doable with kubernetes - honestly, the main issue there is storage, because distributed storage is inherently complex topic that you need to tackle separately, even if you run on top of k8s.
However, if you can simplify your setup by allowing for example a performant NFS filer as storage provider (can be random machine with FreeNAS on top, if you have small needs even a random NAS home box), one server as "control plane", the rest can be plugged in automatically (with nfs-provisioner handling simple storage management for you)
