It is a requirement because you can't find GPUs in a single region reliably and Kubernetes doesn't run on multiple regions.
>> * Tailnet as the default network stack
> That would probably be the first thing I look to rip out if I ever was to use that.
This is fair, we find it very useful because it easily scales cross clouds and even bridges them locally. It was the simplest solution we could implement to get those properties, but in no way would we need to be married to it.
>> * Multi-tenant from the ground up
> How would this be any different from kubernetes?
Kuberentes is deeply not multi-tenant, anyone who has tried to make a multi-tenant solution over kube has dealt with this. I've done it at multiple companies now, its a mess.
>> * Bittorrent as the default storage stack
> Might be interesting, unless you also mean seeding public container images. Egress traffic is crazy expensive.
Yeah egress cost is a concern here, but its lazy so you don't pay for it unless you need it. This seemed like the lightest solution to sync data when you do live migrations cross cloud. For instance, I need to move my dataset and ML model to another cloud, or just replicate it there.
The user you're responding to probably doesn't really understand how kubernetes works, and the trade-offs that the etcd has taken to allow in order to be "distributed".
But you're right. You launch node pretty much anywhere, as long as you have network connectivity (and you don't even need full network connectivity, a couple of tcp ports open are enough).
It's not really recommended (due to latency), but you can also run the control-plane across different regions.
It's not really built to be globally scalable, Mesos was built to be globally scalable. I don't know of any major hosted kubernetes provider that does this.
It is a requirement because you can't find GPUs in a single region reliably and Kubernetes doesn't run on multiple regions.
>> * Tailnet as the default network stack
> That would probably be the first thing I look to rip out if I ever was to use that.
This is fair, we find it very useful because it easily scales cross clouds and even bridges them locally. It was the simplest solution we could implement to get those properties, but in no way would we need to be married to it.
>> * Multi-tenant from the ground up
> How would this be any different from kubernetes?
Kuberentes is deeply not multi-tenant, anyone who has tried to make a multi-tenant solution over kube has dealt with this. I've done it at multiple companies now, its a mess.
>> * Bittorrent as the default storage stack
> Might be interesting, unless you also mean seeding public container images. Egress traffic is crazy expensive.
Yeah egress cost is a concern here, but its lazy so you don't pay for it unless you need it. This seemed like the lightest solution to sync data when you do live migrations cross cloud. For instance, I need to move my dataset and ML model to another cloud, or just replicate it there.