I don't think it's the virtualization API -- if you run docker in a Linux VM under macOS, it's vastly faster than running it via Docker Desktop. I use a Parallels VM for it. I honestly don't know what Docker Desktop is doing that makes it so much slower, even when you're not mounting anything from the host.
No, but I wasn't doing it in Docker Desktop either. That was the first thing I eliminated as a consideration, since that's an obvious point of slowdown. I assume it would be even slower if I had been doing that. Even a `docker build` which does not mount anything from the host was much slower under Docker Desktop than docker under a custom Parallels VM.
The downside, of course, is that it's a little annoying to keep your code in the VM if you'd otherwise have it on the host. But I still find it worth it because of just how much faster it is. Now that I think about it, I wonder how well it would work to do your own host mounting using either the Parallels guest folder sharing feature or SMB. I never tried it.