This is flawed. Turquoise is not blue or green. Also different displays will show different colors. And a lot of displays aren’t great at producing the hues in the green color space. Idk the test seems arbitrary, but I’m not color expert
I'm not sure what's going on, but in Chrome my median point is heavily on the blue side but in Safari it's on the Green side. Also, at least in Safari, reset leads to a series of perceptually unchanging turquois screens -- it seems a bug. Refreshing fixes it for the next run.
I find the title a bit misleading. I think it should be titled It’s Faster to Copy Memory Directly than Send a Protobuf. Which then seems rather obvious that removing a serialization and deserialization step reduces runtime.
Protobuf does something important that copying memory cannot do: a protocol that can be changed separately on either end and things can still work. You have to build for "my client doesn't send some new data" (make a good default), or "I got extra data I don't understand" (ignore it). However the ability to upgrade part of the system is critical when the system is large and complex since you can't fix everything to understand your new feature without making the new feature take ages to roll out.
Protobuf also handles a bunch of languages for you. The other team wants to write in a "stupid language" - you don't have to have a political fight to prove your preferred is best for everything. You just let that team do what they want and they can learn the hard way it was a bad language. Either it isn't really that bad and so the fight was pointless, or it was but management can find other metrics to prove it and it becomes their problem to decide if it is bad enough to be worth fixing.
But something more modern that doesn’t have the encoding/decoding penalty of Protobuf would be better (eg cap’n’proto but there’s a bunch now in this space).
Not that you are wrong, but in the real world this is not significant for most uses. If it is significant you are doing too much IPC. Or maybe using protobuf where you should be making a direct function call. Fix the architecture either way. (similar to how I can make bubble sort faster with careful machine code optimization, but it is hard to make modern tim sort slower in the real world no matter how bad the implementation is)
Whether it is significant is highly specific to the domain. I think protobuf has accumulated enough problems that every large scale application will eventually run into a footgun or two. Almost all of them have to do with the generated types, not the wire format.
Examples that can noticeably slow things down even for “normal” web apps: map types and deeply nested messages.
TIL serializing a protobuf is only 5 times slower than copying memory, which is way faster than I thought it’d be. Impressive given all the other nice things protobuf offers to development teams.
I guess that number is as good or as bad as you want with the right nesting.
Protobuf is likely really close to optimally fast for what it is designed to be, and the flaws and performance losses left are most likely all in the design space, which is why alternatives are a dime a dozen.
That's a 30x faster just by switching to a zero-copy data format that's suitable for both in memory use and network. JSON services spend 20-90% of their compute on serde. A zero copy data format would essentially eliminate it.
I wouldn't hold onto that number as any kind of fixed usable constant since the reality will depend entirely on things like cache locality and concurrency, and the memory bandwidth of the machine you're running on.
Go around doing this kind of pointless thing because "it's only 5x slower" is a bad assumption to make.
Serializing a protobuf can be significantly faster than memcpy, depending. If you have a giant vector of small numbers represented with wide types (4-8 bytes in the machine) then the cost of copying them as variable-length symbols can be less.
5x is pretty slow honestly. Imagine anything happening 5x as slow as you'd expect it to. I mean, for a recent project I had to inline Rust structs rather than parsing JSON too for specific fields, and that definitely sped it up.
Why don't we use standardized zero-copy data formats for this kind of thing? A standardized layout like Arrow means that the data is not tied to the layout/padding of a particular language, potential security problems like bounds checks are automatically handled by the tooling, and it works well over multiple communication channels.
While Arrow is amazing, it is only the C Data Interface that can be FFI'ed, which is pretty low level. If you have something higher-level like a table or a vector of recordbatches, you have to write quite a bit of FFI glue yourself. It is still performant because it's a tiny amount of metadata, but it can still be a bit tedious.
And the reason is ABI compatibility. Reasoning about ABI compatibility across different C++ versions and optimization levels and architectures can be a nightmare, let alone different programming languages.
The reason it works at all for Arrow is that the leaf levels of the data model are large contiguous columnar arrays, so reconstructing the higher layers still gets you a lot of value. The other domains where it works are tensors/DLPack and scientific arrays (Zarr etc). For arbitrary struct layouts across languages/architectures/versions, serdes is way more reliable than a universal ABI.
This is cool, but the UX of the arrows should follow the scroll mode of the device. You drag down on an iPhone to scroll up. Following the arrow and dragging up causes nothing to happen
This is cool, but do you throw away stuff when done with it or do you keep things at another permanent location? Like the tent that isn’t brought on every trip?
But we do. A series of mathematical functions are applied to predict the next tokens. It’s not magic although it seems like it is. People are acting like it’s the dark ages and Merlin made a rabbit disappear in a hat.
Depends on your definition of knowing. Sure, we know it is predicting next tokens, but do we understand why they output the things they do? I am not well versed with LLMs, but I assume even for smaller modles interpretability is a big challenge.
The answer is simple: the set of weights and biases comprise a mathematical function which has been specifically built to approximate the training set. The methods of building such a function are very old and well-known (from calculus).
There's no magic here. Most of people's awestruck reactions are due to our brain's own pattern recognition abilities and our association of language use with intelligence. But there's really no intelligence here at all, just like the "face on Mars" is just a random feature of a desert planet's landscape, not an intelligent life form.
Doesn’t this apply to any process (including human brains) that outputs sequences of words? There is some statistical distribution describing what word will come next.
I think a lot of engineers think, “I thought of a complicated set of abstract ideas that mean nothing to anyone else but will demonstrate my superior intellect” and then make that. It took a lot of self control to not curse.
