>2. Pay a cloud vendor to supply a working, managed Kubernetes cluster.
If one is at that level already, I don't think there's anything better than AWS ECS out there. It just works. Just works. Yes sure, it does not offer stateless workloads for example among other things but it works for 90% of the cases.
> 3. Don't get clever with Kubernetes networking. In fact, touch it as little as possible and hope really hard it continues to work.
Pretty much... Each CNI generates the SDN its own way slightly differing then the others. It is like you can write the program to print a chessboard on terminal in ten different ways.
Unfortunately, these implementation details aren't written or documented anywhere and they of course would keep changing from release to release anyway. Your only way out if you have production workloads that you can't afford going down without missing revenue? Just pay for the support for respective CNI as only they would know what the voodoo magic is under the hood.
Sure you can see the source code and all of them are open source but that's not your main business or the main day job and of course, the solutions aren't 100 line trivial implementations either.
> If one is at that level already, I don't think there's anything better than AWS ECS out there.
100%. To answer the OP's question: my nightmare is having to use it at all. I work with small, very early-stage companies whose applications by and large are not complicated. Perhaps at some level of scale and/or complexity, k8s makes sense. For the vast majority of the cases I see, something like ECS does everything they need, while being significantly more simple to understand, develop for, and debug.
Do you still recommend they host their applications in containers (e.g. Docker)? I feel like it's fairly low effort to start out that way, but can be a pain to add later.
> I don't think there's anything better than AWS ECS out there
Do you have any experience with Kubernetes on GCP being less good than AWS ECS? I'd expect them to be the gold-standard when it's a project coming from Google originally and we haven't had any Kubernetes problems that were related to GCP.
I have experience with Google's managed Kubernetes service (GKE).
It's basically great. Solid, few surprises, no compatibility issues with third party software packaged for Kubernetes. Autopilot looks even better – billing you only for the resources you allocate rather than for the full nodes, basically removing the bin-packing problem. Very little about our config was Google specific or would cause issues porting to another provider. It was up-to-date enough for us to use relatively new features, while lagging enough that everything felt pretty stable.
The only issue we had was wanting to use a somewhat obscure configuration for the Google Cloud load balancer instance that was underlying the Kubernetes ingress. This was possible, we just had to configure it manually in Terraform and point it at the cluster rather than being able to treat it as a cluster resource if I remember correctly. This was only a temporary solution while they were in the process of adding more custom control via K8s.
As far as I can tell it is considered to be the gold-standard.
Disclaimer: I now work for Google on non-cloud stuff, but this was my experience doing a port from bare-metal to GKE.
>2. Pay a cloud vendor to supply a working, managed Kubernetes cluster.
If one is at that level already, I don't think there's anything better than AWS ECS out there. It just works. Just works. Yes sure, it does not offer stateless workloads for example among other things but it works for 90% of the cases.
> 3. Don't get clever with Kubernetes networking. In fact, touch it as little as possible and hope really hard it continues to work.
Pretty much... Each CNI generates the SDN its own way slightly differing then the others. It is like you can write the program to print a chessboard on terminal in ten different ways.
Unfortunately, these implementation details aren't written or documented anywhere and they of course would keep changing from release to release anyway. Your only way out if you have production workloads that you can't afford going down without missing revenue? Just pay for the support for respective CNI as only they would know what the voodoo magic is under the hood.
Sure you can see the source code and all of them are open source but that's not your main business or the main day job and of course, the solutions aren't 100 line trivial implementations either.