Thanks to BPF devs and system folks like Brendan Gregg Linux tracing on the 4.x kernels can rival Solaris/FreeBSD + Dtrace. So my question is where does Sysdig fit in the tracing equation here?
I'm not sure if this fully answers your question, but Loris (the sysdig creator) wrote up a technical analysis of sysdig vs dtrace vs strace a while ago.
practically, sysdig is how something like carbonblack should be doing its monitoring on linux.
so to more directly answer your question; sysdig is the linux equivalent of the kernel module (plus userland grabby bits) needed for the executive pointy-clicky dashboard.
Does Linux perf / bpf tracing allow dealing with userland these days? I wish I could start reading info from live python and ruby like I can with dtrace, but I haven't really seen a way to do that yet.
Yes it does. You can use Userland Statically Defined Trace Points (USDTs) in order to augment arbitrary applications with probe points that can be monitored by tools like BCC 'trace'.
The library that provides the API for these trace points is SystemTap (sys/sdt.h), which is -- AFAICS -- the primary method of USDT instrumenting on Linux. Conveniently, SystemTap's USDT API is the exact same as DTrace's API: you can simply use sprinkle `DTRACE_PROBEn` calls around and it will work fine on either, use the 'dtrace' tool to generate code from a provider, etc etc. Their dtrace tool is just a thin script that's compatible with the FreeBSD variant.
(The nice part about this is that you should just be able to relax e.g. your autoconf checks from enabling DTRACE_PROBEn calls on FreeBSD/Solaris to include Linux as well, so migration should be smooth)
Check out some info here[1].
Alternatively, for "simple" cases where you do not need provider support (e.g. inputs to the trace points), if the executable just has enough symbol information for function names, or the symbol names are exported anyway, you can generally glean some relevant information. For example, the following command uses the 'funclatency' tool to dump a latency histogram of every function call to any function with the word 'poke' in its name, for the application named 'app' in the CWD:
NB: It is also worth noting that 'perf' on Linux has some form of uprobes support; e.g. it can probe functions based on the symbol name, but only for the events/interfaces it has available, I believe.
Was fooled a moment into thinking you had no pricing information on your page. Realized it was there but just not accessible via your menu. My advice, add pricing to your menu. It is one of the first things I search on new products before I try it. Should be accessible from any page in case a reader is interested and wants to check if he can afford your tooling.
Fooled twice :). I do not see pricing in the menu though. I indeed found the page by browsing to another page but my first expectation was to literally find “pricing” in the menu. That page could than start with the open source pricing statement (free) followed by their payed options. Or that is what these kind of products typically do.
The main issue I've had is with the awful documentation. They give a good chunk of examples - but I was unable to easily locate a simple listing of all the different ways of referencing things.
For a tool like this; quality documentation really is a must. And it isn't.
Wouldn't it be the case that adding Python won't deliver the performance needed? I think the choice of Lua is probably because it has to be performant.
