crun-vm (https://github.com/containers/crun-vm) is similar to RunCVM in that it can launch container images (or VM disk images) in VMs. It's an OCI runtime so it fits into the podman, Docker, or Kubernetes model.
krunvm (https://github.com/containers/krunvm) is a standalone tool with a similar workflow where you can launch a VM from an OCI image. It predates Bootable Container Images, so I think it injects its own kernel.
Although crun-vm states it let's you "run VMs as easily as you run containers" AFAIK it requires specialised VM images or bootable containers images be created first. By contrast, RunCVM aims to boot any standard OCI container image (as well as specialised container images) and this may make the learning curve and migration path to RunCVM easier for some applications.
=== krunvm ===
krunvm appears much more similar to RunCVM in its approach. However, RunCVM deploys each VM inside a standard runc container, providing an additional layer of isolation and encapsulation on the host (and I'm not sure krunvm does this). Remove the standard container using `docker rm` and you tear down the VM too.
And while krunvm states it "supports mounting additional local directories into the microVM and exposing ports from the guest to the host (and the
networks connected to it)" RunCVM does this too, but also makes it trivial to transparently create, prepopulate and mount virtual VM disks within the VM where additional read/write performance is needed (compared to virtiofs that both RunCVM and krunvm use for booting the VM's rootfs).
Both RunCVM and krunvm appear to provide seamless networking experience.
RunCVM gives you full control over the choice of kernel the VM boots, so it can boot what are essentially stock alpine, ubuntu, debian, or oracle kernels. It can even boot an OpenWRT kernel. Or any custom kernel (with minimal config requirements). AFAIK kruncvm does not offer this.
Very cool! It seems like this could solve docker-in-docker use cases for things like CI, perhaps? I would love to see benchmarks of mem/cpu footprint vs. the same workload in raw docker.
Yes there are definite CI applications. It's possible to run docker-in-docker with a privileged docker container or using the sysbox runtime, but it can be run using RunCVM inside a VM too.
But for a CI application that requires a running kernel, for example integration testing of an actual kernel module, or an application that uses kernel modules (like a VPN or firewall application) RunCVM could be very useful.
Think also about integration testing that requires hardware, such as a display.
In these cases, RunCVM can provide a way to encapsulate that VM and make it trivial to run the integration tests.
I don't have benchmarks to hand yet, but it's true that most applications will run more slowly (and require overall more host memory) in a VM compared to a plain container.
Although it's currently Linux-only, it seems this would be useful on Mac/Windows, as they already jury-rig containers within a VM and this would override that by default.
RunCVM (Run Container Virtual Machine) is an experimental open-source Docker container runtime for Linux, that makes launching containerised workloads in virtual machines (VMs) as easy as launching them in containers, using docker run, e.g.:
Launch nginx VM publishing VM port 80 on host port 8080:
`docker run --runtime=runcvm --name nginx1 --rm -p 8080:80 nginx`
Launch an interactive terminal on an Ubuntu VM:
`docker run --runtime=runcvm --name ubuntu1 --rm -it ubuntu`
RunCVM runs standard container workloads (like nginx or mariadb) as well as system workloads (like Systemd, Docker, stock or bespoke Linux kernels, even OpenWrt).
RunCVM:
- Provides stronger workload isolation than standard containers.
- Allows running and testing applications like Systemd, Docker, and Kubernetes that won't easily run (or run securely) in standard containers.
- Supports tools and apps like iptables, ipvsadm or openvpn, or Docker Swarm ingress routing, that require a running kernel (or a kernel version or modules not available on the host).
- Supports both stock kernels of major Linux distributions and custom kernels.
- Makes it easy to create arrays of networked VMs for testing complex multi-machine setups like Docker Swarms.
- Supports many standard `docker run` options including custom docker networks with docker internal DNS (`--network`), multiple network interfaces (with `docker network connect`), published ports (`-p`), plus volume, tmpfs and bind mounts
- Uses virtiofs for fast booting and supports prepopulated KVM virtual disks on almost any regular disk filepath (except /) for improved I/O performance.
- Can be easily customised to emulate specific hardware e.g. disks, network cards, and graphics displays.
- Useful as a playground for some bare-metal training and testing use-cases.
BACKGROUND:
RunCVM was born out of difficulties experienced using the Docker and Podman CLIs to launch Kata Containers v2, and a belief that launching containerised workloads in VMs using Docker needn't be so complicated.
Like Kata, RunCVM aims to be a secure container runtime with lightweight virtual machines that feel and perform like containers, but provide stronger workload isolation using hardware virtualisation technology.
However, while Kata aims to launch standard container images inside a restricted-privileges namespace inside a VM running a single fixed and heavily customised kernel and Linux distribution optimised for this purpose, RunCVM intentionally aims to launch container or VM images as the VM's root filesystem using stock or bespoke Linux kernels, the upshot being RunCVM's can run VM workloads that Kata's security and kernel model would explicitly prevent.
FURTHER DETAILS:
- Uses a lightweight 'wrapper-runtime' technology that subverts the behaviour of the standard container runtime runc to cause a VM to be launched within the container (making its code footprint and external dependencies extremely small, and its internals extremely simple and easy to understand and tailor for specific purposes).
- Highly portable among Linux distributions and development platforms providing KVM. Can be installed on Google Cloud or on GitHub Codespaces.
- Experimental support for podman run.
- RunCVM can even be used to launch VMs nested inside a RunCVM VM - i.e. an 'inner' RunCVM Container/VM guest can be launched by Docker running within an 'outer' RunCVM Container/VM guest (assuming the host supports nested VMs) - in this sense, RunCVM is 'reentrant'.
RunCVM was first released under the Apache license in April 2023 and its latest release, v1.4.0, is the culmination of an extensive amount of R&D over roughly two years.
Questions, suggestions and feedback are most welcome.
Looks cool and I understand what it does. Can you explain the production use case for this solution? It's not clear to me how this would be used in real life.
Not OP and not at all related to the project, but I imagine in production you would just replace the container runtime of your Kubernetes/Nomad/Swarm/homemade orchestrator, and run Docker container as usual (but each one is a separate VM). The advantages would be that you have actual isolation between the containers, so it's great for higher security contexts (e.g. banks) or multi tenant setups.
That's about right. RunCVM should be compatible with orchestrators today that use the Docker command line or API. As an experimental project, Kubernetes compatibility is beyond our current scope but it would be an interesting exercise.
Just add "--runtime=krun" to your podman command line along the other arguments and you'll get the container running inside a VM powered by libkrun.