We use eBPF to achieve non-intrusive (we call it `zero-code`) observability without modifying any application code, and have implemented three core features: Universal Map, Distributed Tracing, and Continuous Profiling.
Yes, we have implemented *Distributed* tracing using eBPF. Because of this achievement, we have also published a paper in ACM SIGCOMM 2023.
I have to say I find projects that talk about generic concepts (observability, tracing, eBPF), but then when you dig in the docs it's 100% Kubernetes-specific, to be highly misleading. Not everyone uses Kubernetes, and distributed tracing is a good thing to have regardless of the underlying platform.
Yeah, no. Cloud-native used to mean something even before Kubernetes became mainstream, and technically the CNCF isn't about Kubernetes only, which is why KubeCon and CloudNativeCon are separate events (held together, but separate). Just going to their website shows me two case studies, one is around Kubernetes (Spotify), the other around Vitess (Slack) and has nothing to do with k8s.
If you remove Kubernetes which is way more popular and hiding the numbers of cloud-native, you'll see that cloud-native started being talked about around 2011, with steady small growth untill it explodes alongside Kubernetes later on.
I recall hearing cloud native compared to lift and shift regarding migrating to AWS ~2012-2013.
For distributed tracing, how is deepflow able to correlate an inbound request (eg. client call) with an outbound request (eg. 3rd party API call required to service client call) without being inside the business logic?
tcpdump only uses BPF, not eBPF. BPF is a simpler language that, among other things, is guaranteed to run in finite time because it doesn't have backward jumps, and has limitations on program size (4096 instructions). (The "e" in eBPF stands for "extended", as it extends BPF to remove those limitations, among other changes.)
It compiles your filter expression into a series of instructions, using libpcap. For instance, the output of `tcpdump -d -y EN10MB 'ip and tcp port 80' (which is rather similar to the use case in the OP, but not identical, since it doesn't strip headers), on my machine, is:
# Load the 2-byte ethernet protocol into A (the accumulator register).
(000) ldh [12]
# If IPv4, continue to line 2. Else, jump to line 12.
(001) jeq #0x800 jt 2 jf 12
# Load the one-byte IP protocol into A.
(002) ldb [23]
# If TCP, continue. Else, jump to line 12.
(003) jeq #0x6 jt 4 jf 12
# Load the 2 bytes corresponding to IP flags / fragment offset into A.
(004) ldh [20]
# If the fragment offset is nonzero, jump to line 12. Else, continue.
(005) jset #0x1fff jt 12 jf 6
# Load the internet header length into X. (Note that this is the bottom 4
# bits of the first byte of the IPv4 header, expressed in 4-byte words)
(006) ldxb 4*([14]&0xf)
# Load the source port into A.
(007) ldh [x + 14]
# If 80, jump to 11. Else, continue.
(008) jeq #0x50 jt 11 jf 9
# Load the dest port into A.
(009) ldh [x + 16]
# If 80, jump to 11. Else, jump to 12.
(010) jeq #0x50 jt 11 jf 12
# Accept. Return 262144 bytes, the default snaplen.
(011) ret #262144
# Reject. Literally, return 0 bytes.
(012) ret #0
If you know how to read assembly, it should be fairly straightforward to follow a typical program (you'll need various protocol header wire formats handy if you haven't memorized the offsets).
There are a number of good resources at https://ebf.io, including a couple of links to books. I haven't read those books personally, but I would be surprised if "BPF Performance Tools" by Brendan Gregg isn't worthwhile.
That's a good point. I should clarify that no such verification is necessary in classic BPF due to the absence of backward jumps -- you can trivially show that the maximum steps executed by a BPF program is 4096, since you'll execute each instruction at most once, and there are at most 4096 instructions.
Meanwhile, the verification that an eBPF program terminates is dependent on the correctness of the verifier, and similarly there's no guarantee that a program with appropriately-bounded complexity will be accepted by the verifier.
To be clear: I'm not trying to throw shade at the verifier; to the contrary, I think it's an impressive piece of software. But there's a difference between being able to prove in one sentence that a program always terminates, and needing to rely on the correctness of some verification software.
We use eBPF to achieve non-intrusive (we call it `zero-code`) observability without modifying any application code, and have implemented three core features: Universal Map, Distributed Tracing, and Continuous Profiling.
Yes, we have implemented *Distributed* tracing using eBPF. Because of this achievement, we have also published a paper in ACM SIGCOMM 2023.