Why we didn't write our own eBPF sensor

The runtime agent needs to know how fast a service is answering and how often it's failing, without touching the application. That's an easy sentence to write in a pitch deck. Getting it out of a live process without an SDK, a sidecar, or a restart is a different problem, and for the first few weeks of building the agent we assumed we'd solve it ourselves.

The plan was straightforward on paper: uprobes on SSL_write/SSL_read for TLS traffic, kprobes on tcp_sendmsg for the rest, parse HTTP and gRPC framing in a small eBPF program, ship RED metrics (rate, errors, duration) without the target process ever knowing it was being watched. We got a proof of concept working against a plain Go HTTP server inside a week. Then we pointed it at a real Spring Boot service and it fell over.

Where it actually breaks

Not at the eBPF verifier — that part is genuinely well-designed, and the kernel will reject a program that could loop or crash long before it runs. It breaks at the userspace boundary, in the part nobody puts in the conference talk: uprobe offsets are a function of exact library build. The offset into SSL_write for OpenSSL 3.0.2 on Ubuntu 22.04 is not the offset for OpenSSL 3.0.13 on the same distro six months later, and it's a different number again on Alpine's musl-linked build, and different again once you're inside a JVM doing its own TLS through a bundled provider instead of the system one. Multiply that by every language runtime, every container base image, every minor version a client happens to be running, and you get a matrix of offsets that needs constant re-derivation and a test rig for kernels you don't control.

None of this is a research problem. It's a maintenance problem, forever, and it doesn't stop once you ship — every OpenSSL point release is a potential regression you find out about from a client's Slack message instead of your own CI.

Why we picked Beyla instead

Beyla (Grafana Labs' eBPF instrumentation project) already carries that maintenance burden, in the open, with a community of people whose full-time job is chasing kernel and library drift across the exact environments we'd otherwise be discovering one client at a time. It gives us RED metrics per service — request rate, error rate, p50/p99 duration — with zero code in the target application. We get to spend our own eBPF budget on the part that's actually ours: procnet, our own collector that reads /proc/<pid>/net/tcp to map live TCP connections to containers, which is a narrower, more stable surface than parsing L7 protocols in the kernel.

The honest cost of this decision is that today the agent ships as two containers instead of one — ours, and Beyla's, both with elevated privileges on the host. That's friction we're actively working down (bundling Beyla as a supervised child process of the agent binary is next), but it's a UX problem to solve, not an engineering bet we have to win against a moving target. We'd rather ship that friction honestly than pretend we built something we didn't.

The rule we ended up with

Build the thing that's actually your product. For us that's the graph: the entity resolution that gives "the orders service" the same identity whether it's seen from a Dockerfile, a running container, or a TCP connection, and the correlation logic that decides whether a dependency is real. eBPF at the kernel ABI boundary, across every language and every container base image a client might run, is somebody else's full-time job already — and they're good at it.