In this specific case, it's not Linux here that's the problem but Android itself. The kernel has been capable of near-millisecond audio latency for years, even more so with an RT-patched kernel. Google just hasn't cared much about audio latency so far, and their VM probably makes things worse.
The kernel has been capable of sub 10ms audio latency for years, but the audio subsystems that are commonly used in Linux hold Linux back from achieving that level of performance (on general purpose Linux systems at least, embedded systems are a different matter).
ALSA isn't flexible enough, PulseAudio is too slow, JACK is too specialised, etc... CoreAudio on OSX shows what an audio stack can be capable of, all the Linux audio solutions currently available are subpar compared to CoreAudio.
AudioFlinger is Android's audio server. I'd be interested to see how far this can be pushed, but perhaps Google will route around it and provide a different solution for the type of low latency audio that immersive VR requires, IIRC using an alternative audio stack was Samsung's approach with GearVR.
I agree completely about the user-space tools on Linux. My doubt is that Google will come up with a solution that applies anywhere except Android. But given the hopeless situation there at the moment, I'd be okay even with that.
