Anybody have tests of the latency? I haven't found anything better than parsec that is free and open source. Parsec uses their own UDP protocol, and makes use of hardware encoder/decoder.
Moonlight as a protocol is quite cool, made for streaming games in fact. Sunshine is a open source server implementation, and the client was open source to begin with, I think. Sunshine is multi-platform, and so is Moonlight the client. I used it to stream my Linux desktop (and games) to my Android phone.
I also have good experience with Spice. I haven't found a way to set it up as a server, but Qemu uses it for all its VMs, and the performance is great, you can watch videos and everything, if you have the bandwidth.
For WiFi and slower networks, Moonlight was way more performant for me.
The problem with UDP for video is that, on a lossy network, you either end up sending a lot of key frames or reinventing TCP.
There is a trick for achieving low latency video with TCP: set SO_SNDBUF to the lowest possible value and do your own buffering. If your buffer grows too large, lower the bitrate and/or drop frames.
I'd argue that on a lossy network you will never have decent video (with decent latency) so for interactive sessions you don't really have to care about lossy network, it isn't worth the hassle anyway.
That is true, the network needs to be stable in any case, but the keyframes have "bursty" traffic, therefore increasing the latency compared to b-frames (and p-frames)
Ever since h264 there is a feature called intra-refresh, which allows you spread key frames over time, reducing the burstiness of the transmission (and therefore improving latency in most scenarios)
Could be a problem if you need to encode other things, since hardware encoders are artificially limited on nvidia and probably others. Nvidia did increase the amount of concurrent streams at the start of the pandemic though and the limit may also be partly imposed due to patent licensing and not just Nvidia's price discrimination.
When I used Parsec to control a machine on the other side of my room, the monitor and my client were not even 1 frame out of sync. The only downside is the heavy compression to achieve that. There is also no color accuracy.
