Something that’s been on my mind recently is that there’s a need of a high-performance flame graph library for the web. Unfortunately the most popular flame graph as a library / component, basically the react and d3 ones, work fine but the authors don’t actively maintain them anymore and their performance with large profiles is quite poor.
Most people that care about performance either hard-fork the Firefox profiler / speedscope flame graph component or create their own.
Would be nice to have a reusable, high performance flame graph for web platforms.
Haven’t had the chance to play with WezTerm just yet but wanted to share that the author is an incredibly smart, friendly, and humble.
Had the opportunity to work on a project together at work some years back and I can only aspire to be 1/10th as good of an engineer as him. A true hacker.
It is not, programs that are accepted are proved to terminate. Large and more complex programs are accepted by BPF as of now, which might give the impression that it's now Turing complete, when it is definitely not the case.
Personally I’m not a fan of Go’s default zero-initialisation. I’ve seen many bugs caused by adding a new field, forgetting to update constructors to intialise these fields to “non-zero” values which caused bugs. I prefer Rust’s approach where one has to be explicit.
That being said it’s way less complex than C++’s rules and that’s welcomef.
I spent a year and a half writing go code, and I found that it promised simplicity but there an endless number of these kinds of issues where it boils down to "well don't make that mistake".
It turns out that a lot of the complexity of modern programming languages come from the language designers trying to make misaked harder.
If you want to simplyfing by synthesising decades of accumulated knowledge into a coherent language, or to remove depreciated ideas (instead of the evolved spaghetti you get by decades of updating a language) then fine. If your approach to simplicity is to just not include the complexity, you will soon disciplinary that the complexity was there for a reason.
The problem you are describing in Go is rarely a problem in C++. In my experience, a mature code base rarely has things with default constructors, so adding a new field will cause the compiler to complain there's no default constructor for what you added, therefore avoiding this bug. Primitive types like `int` usually have a wrapper around them to clarify what kind of integers, and same with standard library containers like vector.
However I can't help but think that maybe I'm just so fortunate to be able to work in a nice code base optimized for developer productivity like this. C++ is really a nice language for experts.
Compare `int albumId, songId;` versus `AlbumId albumId; SongId songId;`. The former two variables can be assigned to each other causing potential bug and confusion. The latter two will not. Once you have a basic wrapper for integers, further wrappers are just a one-liner so why not. And also in practice making the type more meaningful leads you to shorter variable names because the information is already expressed in types.
Wouldn’t it just be considered bad practice to add a field and not initialize it? That feels strongly like something a code review is intended to catch.
It’s easy to miss this in large codebases. Having to check every single struct initalisation whenever a field is added is not practical. Some folks have mentioned that linters exist to catch implicit initialisation but I would argue this shouldn’t require a 3rd party project which is completely opt-in to install and run.
I just finished this. To each their own of course, but I found the writing too padded and tonally off-putting at times. Some of the stories felt dated both from a technological stance and a cultural stance. I prefer Azure's Cloud Pattern docs myself (though "Release It!" was really good if you prefer a storytelling approach):
This looks incredibly comprehensive, thanks for sharing!
Should have added that I read this book in 2016, and the first edition is even older, so there’s naturally been lots of new (and exciting) developments in this area!
Overall, I am for frame pointers, but after some years working in this space, I thought I would share some thoughts:
* Many frame pointer unwinders don't account for a problem they have that DWARF unwind info doesn't have: the fact that the frame set-up is not atomic, it's done in two instructions, `push $rbp` and `mov $rsp $rbp`, and if when a snapshot is taken we are in the `push`, we'll miss the parent frame. I think this might be able to be fired by inspecting the code, but I think this might only be as good as a heuristic as there could be other `push %rbp` unrelated to the stack frame. I would love to hear if there's a better approach!
* I developed the solution Brendan mentions which allows faster, in-kernel unwinding without frame pointers using BPF [0]. This doesn't use DWARF CFI (the unwind info) as-is but converts it into a random-access format that we can use in BPF. He mentions not supporting JVM languages, and while it's true that right now it only supports JIT sections that have frame pointers, I planned to implement a full JVM interpreter unwinder. I have left Polar Signals since and shifted priorities but it's feasible to get a JVM unwinder to work in lockstep with the native unwinder.
* In an ideal world, enabling frame pointers should be done on a case-by-case. Benchmarking is key, and the tradeoffs that you make might change a lot depending on the industry you are in, and what your software is doing. In the past I have seen large projects enabling/disabling frame pointers not doing an in-depth assessment of losses/gains of performance, observability, and how they connect to business metrics. The Fedora folks have done a superb and rigorous job here.
* Related to the previous point, having a build system that enables you to change this system-wide, including libraries your software depends on can be awesome to not only test these changes but also put them in production.
* Lastly, I am quite excited about SFrame that Indu is working on. It's going to solve a lot of the problems we are facing right now while letting users decide whether they use frame pointers. I can't wait for it, but I am afraid it might take several years until all the infrastructure is in place and everybody upgrades to it.
On the third point, you have to do frame pointers across the whole Linux distro in order to be able to get good flamegraphs. You have to do whole system analysis to really understand what's going on. The way that current binary Linux distros (like Fedora and Debian) works makes any alternative impossible.
It could be one instruction: ENTER N,0 (where N is the amount of stack space to reserve for locals)---this is the same as:
PUSH EBP
MOV ESP,ESP
SUB SP,N
(I don't recall if ENTER is x86-64 or not). But even with this, the frame setup isn't atomic with respect to CALL, and if the snapshot is taken after the CALL but before the ENTER, we still don't get the fame setup.
As for the reason why ENTER isn't used, it was deemed too slow. LEAVE (MOV SP,BP; POP BP) is used as it's just as fast as, if not faster, than the sequence it replaces. If ENTER were just the PUSH/MOV/SUB sequence, it probably would be used, but it's that other operand (which is 0 above in my example) that kills it performance wise (it's for nested functions to gain access to upper stack frames and is every expensive to use).
Great comments, thanks for sharing. The non-atomic frame setup is indeed problematic for CPU profilers, but it's not an issue for allocation profiling, Off-CPU profiling or other types off non-interrupt driven profiling. But as you mentioned, there might be ways to solve that problem.
