Hmm, we're getting <200 ms glass-to-glass latency by streaming H.264/MP4 video over a WebSocket/TLS/TCP to MSE in the browser (no WebRTC involved). Of course browser support for this is not universal.
The trick, which maybe you don't want to do in production, is to mux the video on a per-client basis. Every wss-server gets the same H.264 elementary stream with occasional IDRs, the process links with libavformat (or knows how to produce an MP4 frame for an H.264 NAL), and each client receives essentially the same sequence of H.264 NALs but in a MP4 container made just for it, with (very occasional) skipped frames so the server can limit the client-side buffer.
When the client joins, the server starts sending the video starting with the next IDR. The client runs a JavaScript function on a timer that occasionally reports its sourceBuffer duration back to the server via the same WebSocket. If the server is unhappy that the client-side buffer remains too long (e.g. minimum sourceBuffer duration remains over 150 ms for an extended period of time, and we haven't skipped any frames in a while), it just doesn't write the last frame before the IDR into the MP4 and, from an MP4 timestamping perspective, it's like that frame never happened and nothing is missing. At 60 fps and only doing it occasionally this is not easily noticeable, and each frame skip reduces the buffer by about 17 ms. We do the same for the Opus audio (without worrying about IDRs).
In our experience, you can use this to reliably trim the client-side buffer to <70 ms if that's where you want to fall on the latency-vs.-stall tradeoff curve, and the CPU overhead of muxing on a per-client basis is in the noise, but obviously not something today's CDNs will do for you by default. Maybe it's even possible to skip the per-client muxing and just surgically omit the MP4 frame before an IDR (which would lead to a timestamp glitch, but maybe that's ok?), but we haven't tried this. You also want to make sure to go through the (undocumented) hoops to put Chrome's MP4 demuxer in "low delay mode": see https://source.chromium.org/chromium/chromium/src/+/main:med... and https://source.chromium.org/chromium/chromium/src/+/main:med...
We're using the WebSocket technique "in production" at https://puffer.stanford.edu, but without the frame skipping since there we're trying to keep the client's buffer closer to 15 seconds. We've only used the frame-skipping and per-client MP4 muxing in more limited settings (https://taps.stanford.edu/stagecast/, https://stagecast.stanford.edu/) but it worked great when we did. Happy to talk more if anybody is interested.
[If you want lower than 150 ms, I think you're looking at WebRTC/Zoom/FaceTime/other UDP-based techniques (e.g., https://snr.stanford.edu/salsify/), but realistically you start to bump up against capture and display latencies. From a UVC webcam, I don't think we've been able to get an image to the host faster than ~50 ms from start-of-exposure, even capturing at 120 fps with a short exposure time.]
There are already a lot of things that are feasible without any license. In Europe/CEPT, you might have a look at all bands under the provisions of ERC Recommendation 70-03 (https://docdb.cept.org/download/2464).
However, those provisions are made in order to ensure a good/fair access to anyone, and therefore to prevent a single user or single technology from overusing those bands which are meant to be shared. For that purpose, there are associated restrictions (in terms of power/EIRP, duty-cycle) and/or mandatory sharing approaches (Listen-before-talk, detect-and-avoid, etc.). In the case of Wi-Fi, CSMA/CA is a form of listen-before-talk.
Unfortunately, mobile technologies defined at 3GPP (GSM, HSPA, LTE, NR) are not designed to be used in such a way (i.e. they don't have any sharing mechanism such as LBT and they require _by design_ dedicated/licensed bands), which by the way implies some kind of specific coordination at the country borders where two operators are using the same channels... (you might look at ECC recommendation 15-01 for an example of PCI sharing).
LAA is a way to have an LTE carrier within the (shared) 5 GHz band, but it has to rely on an anchor carrier for signaling, which requires licensed spectrum. Multefire is a fully-unlicensed solution, but I doubt many UEs (smartphones) support it, and anyway because it must implement the same power limitations and LBT as wi-fi in order to comply with regulations I doubt it would be much better than wi-fi... (maybe it would in some specific case where deterministic QoS is important)
One more thing : keep in mind that a typical 3G/4G/5G macrocell site (e.g. around 65 dBm EIRP per carrier) is something very expensive : your mileage may vary but it can easily be around 100000 € / site when some construction is required.
It can be relatively inexpensive, depending on which RIR you’re using. There are providers like [Neptune Networks](https://neptunenetworks.org/) or [Vultr](https://www.vultr.com/) that you can peer with from a VPS so you don’t need to get “proper” IP transit in a datacenter.
Here I am, a productive developer eager to deliver maximum value to my customers and apply my innovation to company goals.
Open up JIRA and pick up a unit of work to produce today -- gotta stay faithful to those story points!
Somehow, a magic team of spherical devops in a vacuum created an environment where things are just green, predictable, and are never broken.
Another theoretical team of angels from a parallel universe has materialized into this one, updated all the docs, and beamed back into their universe of productivity and effectiveness.
Unit of work defined, I then proceed to produce it for a few hours completely uninterrupted, since if another human being were to make contact, I'd simply explode and format my hard drive, losing all my productivity.
It's break time -- I ingest 2 story points of coffee and join a game of table tennis. The ball bounces back and forth without ever touching the ground. My movements are productive, and value-adding, just like my partner's.
Back to producing units of work, only half a story point left (sike! no such thing as half a story point, and no such thing as "left", as it's a measure of effort, not time. Almost got you there!).
Git push! Automatic CI gains consciousness and validates that my work unit has no chance of causing a user to break our software. It gives me a thumbs up and discards its human shell to fade back into the ones and zeros. A QA engineer cries out in the distance.
A metric of my productivity is logged to the OKR database.
Nothing breaks, and nothing unexpected happens, as the universe is completely predictable.
This is fairly easy to wire up on Linux using v4l2loopback and pyfakewebcam.
I am currently using a little setup that uses OpenCV to acquire frames from the real camera, TensorFlow/BodyPix to compute an alpha mask for the foreground (me) and then OpenCV again to transform and composite myself behind news desks and into car infotainment screens and the like, eventually writing it to a virtual webcam I can use from Zoom (over its own virt bg feature this adds the layering and perspective transforms), Jitsi, Teams, etc.
The above looks like another fun thing to add. Time to go full Who Framed Roger Rabbit? ...
ChrisFix videos are outstanding and somewhat generalized, he really does fine work.
One of my favourite channels has much lower production values (not bad by any means though!), and is much more of a hacker. He buys something damaged, figures out how to somehow fix it up and does so. Often a bit of a bodge, lol. But the videos come out frequently and are fascinating. This channel is "B is for Build". [1]
Another channel I will recommend, more for the awe of seeing an absolute master craftsman - "Arthur Tussik". This fine Russian gentleman is almost certainly the most skilled panel beater on Youtube. His videos are bodywork restoration of mangled crash wrecks back to perfect in one short video. [2]
And finally a recommendation I'm sure any hacker will enjoy watching - "Bad Obsession Motorsport", and their "Project Binky" series. Rebuild of a classic Mini, with a twist, almost every part needs custom fabrication. Simply amazing. [3]
The trick, which maybe you don't want to do in production, is to mux the video on a per-client basis. Every wss-server gets the same H.264 elementary stream with occasional IDRs, the process links with libavformat (or knows how to produce an MP4 frame for an H.264 NAL), and each client receives essentially the same sequence of H.264 NALs but in a MP4 container made just for it, with (very occasional) skipped frames so the server can limit the client-side buffer.
When the client joins, the server starts sending the video starting with the next IDR. The client runs a JavaScript function on a timer that occasionally reports its sourceBuffer duration back to the server via the same WebSocket. If the server is unhappy that the client-side buffer remains too long (e.g. minimum sourceBuffer duration remains over 150 ms for an extended period of time, and we haven't skipped any frames in a while), it just doesn't write the last frame before the IDR into the MP4 and, from an MP4 timestamping perspective, it's like that frame never happened and nothing is missing. At 60 fps and only doing it occasionally this is not easily noticeable, and each frame skip reduces the buffer by about 17 ms. We do the same for the Opus audio (without worrying about IDRs).
In our experience, you can use this to reliably trim the client-side buffer to <70 ms if that's where you want to fall on the latency-vs.-stall tradeoff curve, and the CPU overhead of muxing on a per-client basis is in the noise, but obviously not something today's CDNs will do for you by default. Maybe it's even possible to skip the per-client muxing and just surgically omit the MP4 frame before an IDR (which would lead to a timestamp glitch, but maybe that's ok?), but we haven't tried this. You also want to make sure to go through the (undocumented) hoops to put Chrome's MP4 demuxer in "low delay mode": see https://source.chromium.org/chromium/chromium/src/+/main:med... and https://source.chromium.org/chromium/chromium/src/+/main:med...
We're using the WebSocket technique "in production" at https://puffer.stanford.edu, but without the frame skipping since there we're trying to keep the client's buffer closer to 15 seconds. We've only used the frame-skipping and per-client MP4 muxing in more limited settings (https://taps.stanford.edu/stagecast/, https://stagecast.stanford.edu/) but it worked great when we did. Happy to talk more if anybody is interested.
[If you want lower than 150 ms, I think you're looking at WebRTC/Zoom/FaceTime/other UDP-based techniques (e.g., https://snr.stanford.edu/salsify/), but realistically you start to bump up against capture and display latencies. From a UVC webcam, I don't think we've been able to get an image to the host faster than ~50 ms from start-of-exposure, even capturing at 120 fps with a short exposure time.]