I didn't know RFB was a thing. I will be looking into this.
What I really want is to be able to send an initial jpeg keyframe to the client, and then use some application-level heuristic to determine if I should then send a delta-encoded frame (what to change based upon the prior) or another keyframe. For 90%+ of my frames, using a prior frame and indicating deltas would be far more efficient than sending a keyframe each time. Maintaining the previous frames on the server per-client is not a huge deal and would not impact latency if done properly.
I wonder if there is some way I can use jpeg itself to half-ass this sort of approach in the browser. Maybe... encode keyframes at ~80% quality and delta frames at 100% (still not guaranteed, I know). Maybe canvas compositing or simple buffer tricks with bitmaps could get ~95% of the way there. For my application, the pixels don't have to be perfect or accurate to any particular standards.
In case you're interested, here's a permissively licensed server-side implementation of the protocol, including all major codecs, in C: https://github.com/any1/neatvnc
What I really want is to be able to send an initial jpeg keyframe to the client, and then use some application-level heuristic to determine if I should then send a delta-encoded frame (what to change based upon the prior) or another keyframe. For 90%+ of my frames, using a prior frame and indicating deltas would be far more efficient than sending a keyframe each time. Maintaining the previous frames on the server per-client is not a huge deal and would not impact latency if done properly.
I wonder if there is some way I can use jpeg itself to half-ass this sort of approach in the browser. Maybe... encode keyframes at ~80% quality and delta frames at 100% (still not guaranteed, I know). Maybe canvas compositing or simple buffer tricks with bitmaps could get ~95% of the way there. For my application, the pixels don't have to be perfect or accurate to any particular standards.