"Lightweight Kubernetes" and then a graph involving 2 different nodes with 10+ services running on them.
Nomad seems to be a truly "lightweight Kubernetes" and for the small amount of time I've been using it, it seems to do it's job really well and it's easy to understand all the moving pieces without spending countless of hours reading docs and source code.
Although, it's hard to recommend Nomad for future uses as it sadly stopped being FOSS :/
Nomad seems to be a truly "lightweight Kubernetes"
k3s requirements: 1 node, 512MB RAM, 1 CPU core
Nomad requirements: "Nomad servers may need to be run on large machine instances. We suggest having between 4-8+ cores, 16-32 GB+ of memory, 40-80 GB+ of fast disk and significant network bandwidth."
That can be said of many systems, does hashicorp document it though? I’m guessing they must, otherwise that would be the first thing you hear on the other side of a support call.
That's a bad take and you know it. HashiCorp couldn't make money because their enterprise offering was slim on features and overpriced. Rather than compete, they're playing protect-the-castle with some aggressive plays to keep their company afloat. They're bad stewards of open source (always have been) and are now getting their just desserts.
This is for large deployments in prod.
Nomad and Consul servers at home are running on an old Rpi3 with 2G of ram managing several devices and about 10 vm's.
Larger deployment of ~30vm's was managed with a nomad/consul cluster of 3x2cpu4ram with no issues.
I've commented on this before in a different k8s thread (one about a k8s outage) but something that bears repeating is that the entire job market is Kubernetes.
My personal experience is that it is very, very hard to find a job right now if your professional experience is primarily non-k8s orchestration systems. Most job positions out there require deep Kubernetes knowledge as well as hands-on experience with different supporting products like ArgoCD and Flux.
I chose Nomad for a large initiative at my current employer and it is honestly pretty amazing given the use case but I regret choosing it because I feel like I'm unhirable now given that every devops/SRE/platform engineering position open on the market (each one with hundreds of applicants) is heavily Kubernetes focused.
A lot of smaller SaaS companies avoid k8s like the plague and only start using it once they scale to a certain size. K8s might be ubiquitous past a certain scale, but there are many jobs operating below the threshold for which the complexity of your system with k8s outweighs the complexity of your system without k8s.
If you find yourself on the Google cloud for some reason, GKE Autopilot is such a great hands-off solution that most alternative solutions will probably be more complex.
I only recently had to use AWS EKS, and am pretty sure that many people dislike k8s because that's the main incarnation of it they experience (how is there no pre-installed ingress, and even the setup for their own load balancer is a 10+ step manual process?).
It is really weird that the AWS Load Balancer Controller isn't an EKS Addon, and the permissions requirements being in the base requirements for EKS. Also, the only officially supported CNI is the VPC CNI.
And if you want to use EKS-A (EKS-Anywhere) it's even more effort to do the AWS IRSA setup.
Installing an ingress controller that will do SSL is a steep cliff preventing use with so many cloud Kubernetes offerings. I don't know why it needs to have dozens of steps, roles, and resource definitions.
Yeah, though I have to admit that even though you get SSL certificates "out of the box" with a GKE + GCP load balancer setup, their provisioning has always been really slow for me (usually 1-2h, sometimes half a day). In comparison self-installed nginx-ingress + Let's Encrypt provisions in minutes.
Cloud Run is just Knative which is also Kubernetes, so in the end it comes out to about the same. I used to use Cloud Run, but in the end there were usually always some things that ended up making GKE the more convenient choice (e.g. access to Helm charts).
If that is so then the industry has a lot to recon with soon because that would mean most at chasing a shiny new thing for absolutely all the wrong reasons.
K8s is complete overkill at best for the majority of companies/workloads and introduces lots of other dependencies on teams in terms of workflows and architectures to make things not be a dumpster fire for all but very mature teams.
I keep hearing this viewpoint, but I really don’t understand it. I use k3s for all sorts of personal projects, specifically because it makes things easier for me. I don’t have to worry about the underlying OS. I can focus on the application that’s actually providing value for me.
On the professional side, k8s handles orchestration issues far better than anything else I’ve worked with. Using autoscaling instances is a nightmare by comparison, and requires most of the same initial effort to do it right.
There is probably a middle ground where it requires a certain amount of complexity in the K8s configuration that isn’t worth it compared to other platforms (especially on bare metal), but I haven’t found it, yet.
I think it’s easy to forget how much goes into running even a simple application with high uptime requirements. Kelsey Hightower makes some great points about this here:
Nomad also doesn't have nearly half of the features that Kubernetes does. Need service discovery? Set up a Consul cluster. Need secret management? Install vault. Need vault enterprise? Install a separate Consul cluster! This was a few years ago, maybe it's changed? I dunno.
Anyway, lightweight here means that whole bunch of external dependencies have been ripped out. E.g. AWS/GCP/Azure and I believe other things too.
You can have sd and secrets with just nomad now, it is perfect for small projects in that way.
K8s also can do secrets! base64 encoded secrets! But in case you need a more secure solution. Please pull in a Vault helm chart or mess around with sops or whatever. Want to scale out? See how long it will take you until you will have to move your etcd to dedicated hw/vm's. And let me tell you - setting up external etcd in prod sucks. Also maintaining mission control, because there is a bunch of ways to do things and devs love using things that they are not supposed to touch.
Nomad approach is modular this is why it works. Need proper multi dc SD? please run a consul cluster (single binary + single config file)x3. Need mesh networking? same. Need secrets? Vault (single binary + single config file)x3. Pki? same. Or maybe you don't need it? Then you have an option to just not use it.
One thing that i really don't like about it that there is not a lot of docs describing all the ways these tools can interact with each other.
K8s supports encryption at rest, ACLs and audit logs.
What it actually lacks is versioning, and a dynamic secrets engine, though you could build that with an operator, and things like the postgres-operator do.
You also don't get the whole sealing thing, but I'd argue that's more annoying than useful.
Unsealing is the process of obtaining the plaintext root key necessary to read the decryption key to decrypt the data, allowing access to the Vault.
Prior to unsealing, almost no operations are possible with Vault. For example authentication, managing the mount tables, etc. are all not possible. The only possible operations are to unseal the Vault and check the status of the seal.
You can run k3s with a single node. In that case it uses sqlite instead of etcd which is great for a smaller resource footprint. If you're comparing k8s distros, you'll be hard pressed to find a setup that uses fewer system resources.
Is it actually smaller though? For those non etcd data stores, Kine is often the component doing an etcd to database translation. Im my experience using it generated a load that was greater than a one node etcd instance.
> Nomad seems to be a truly "lightweight Kubernetes" and for the small amount of time I've been using it, it seems to do it's job really well and it's easy to understand all the moving pieces without spending countless of hours reading docs and source code.
Does Nomad expose an API that you can extend with controllers running inside the cluster? Because Kubernetes without operators is not Kubernetes.
> Because Kubernetes without operators is not Kubernetes.
Alright, so, what is it, then? I've been unfortunate to work at firms that generally have a minimal level of competency with Kubernetes, but across several billion dollars worth of firms, not a one has used operators in any capacity, but they leverage Kubernetes substantially. Help me understand the gap, would you?
> While Nomad doesn’t support the idea of Custom Resource Definitions, we can achieve an operator by utilising a regular Nomad job and the nomad HTTP API.
Kubernetes operator is a combination of a CRD reconciled by a controller. Without the CRD it's simply a controller.
I do like the linked post, it's well written and a valiant effort, but not close to comparable.
> "Lightweight Kubernetes" and then a graph involving 2 different nodes with 10+ services running on them.
Among its many strengths, core services for running Kubernetes running inside Kubernetes itself is one of its greatest.
> it's easy to understand all the moving pieces
It's legit to leverage your pre-existing knowledge. Nomad + Linux Pets + Other Hashichorp Pets works well.
There are many ways to run an application. Kubernetes + Machine Cattle, in my experience having started from zero, is superior to any other application deployment paradigm I've ever used.
Treat your machines like cattle, not pets. When I used to run kubernetes in the pre-'kube as a service' days. We had two autoscaling groups, one which had 3 or 5 nodes in it to run the control plane nodes and an open eneded one for workers (which were managed by the cluster autoscaler). Each ASG was set up in such a way that the nodes would come up and join the cluster fully automated and they never needed to be shelled into for anything but troubleshooting the weirdest failures (it was early days, the kubelet and docker had some extra fun bugs).
Ok sure, I'm very familiar w the cattle vs pets analogy / meme -- but the (capitalized!) "Machine Cattle" made it seem like a reference to a particular software tool or product.
Nomad is an alternative to k8s, but it is not really a "lightweight k8s". Nomad and k8s are way to different to call them versions of eachother.
And the whole license thing. Nothing changed for end users.
I think the tagline can be a bit misleading, you can run k3s on devices traditionally considered IoT (like the raspberry pi) but it will run on big heavy x86 servers too.
The amount of services running is not an indication of how lightweight the underlying tech is. K3S is for orchestration across nodes, and by design you can't run services on the master for security reasons unless you manually change it.
Are you trying to say that 10 services and 2 nodes would be less on Nomad?
> K3S is for orchestration across nodes, and by design you can't run services on the master for security reasons unless you manually change it.
That has not been my experience with k3s. Unlike normal distributions, k3s servers do allow workload to run on them unless you manually disable. From https://docs.k3s.io/datastore/ha-embedded :
"
An HA K3s cluster with embedded etcd is composed of:
- Three or more server nodes that will serve the Kubernetes API and run other control plane services, as well as host the embedded etcd datastore.
- Optional: Zero or more agent nodes that are designated to run your apps and services
"
So you can have a full HA k3s cluster with only 3 server nodes and 0 agent(worker) nodes.
This is not true of k8s in general, maybe certain on prem 'distributions' enforce this, i'm not sure. Sometimes kubernetes admins/operators who are running their own control planes and etcd will run etcd on non k8s controlled nodes, or they will run it on the same nodes that run their control plane services (scheduler, apiserver, controller manager, etc). Often those nodes will be tainted in such a way that they don't get overloaded by a noisy neighbor 'customer pod', but that's in no way 'by design'.. perhaps by convention or best practice.
Nomad is a lightweight container orchestrator, not a lightweight Kubernetes. The whole point is to keep syntax, terminology and config options etc as similar as possible between k8s and k3s.
If Nomad provides workload scheduling, a runtime, workload networking, service networking, and some sort of overall orchestrator/controller plus an API, then it has these same services. Running them in one process instead of ten doesn't make it more lightweight.
You're free to run either on one node, but other than for toy purposes like demos and learning, why use a cluster compute system when you're not actually running a cluster?
Nomad seems to be a truly "lightweight Kubernetes" and for the small amount of time I've been using it, it seems to do it's job really well and it's easy to understand all the moving pieces without spending countless of hours reading docs and source code.
Although, it's hard to recommend Nomad for future uses as it sadly stopped being FOSS :/