Practical Latency Observability with eBPF

Measuring CPU, memory, and I/O latency inside the Linux kernel

01Problem

The delays that decide end-to-end performance are usually invisible from user space. A process waits in the run queue, stalls on a page fault, or blocks on a device, and from the outside all three look the same: the program is slow while the CPU looks fine. Answering which of the three it is needs timestamps taken inside the kernel, on both sides of the event. This toolkit takes those timestamps for scheduling, memory, and block I/O, and reports each as a latency distribution rather than an average.

02What I built

Three eBPF tools — runqlat, memlat, iolat — that time run-queue wait, page-fault handling, and block I/O inside the kernel and export log2 latency histograms.
A shared command-line interface (--duration, --out) and a one-click collection script, so every tool is run the same way and two runs can be compared.
CSV histograms with a matching .summary.json per run, stored next to the environment metadata for that run.

03How it works

Each tool is an eBPF program attached to a pair of kernel tracepoints plus a small user-space control program. The kernel side stores an entry timestamp in a BPF hash map, computes the delta at the exit tracepoint, and increments a log2 bucket in a histogram map. Nothing is sent to user space per event. The Go collector reads the histogram map once the measurement window closes and serialises it. Nothing in the observed application is modified or recompiled.

One measurement path: a tracepoint pair gives the delta, the delta is bucketed in a BPF map inside the kernel, and the Go collector reads the map once per run and writes one CSV and one JSON summary per tool.

1AttachAttach eBPF programs to the entry and exit tracepoints for scheduler wakeup, page fault, and block I/O.

2Time in the kernelStore the entry timestamp in a BPF hash map keyed by the event, compute the delta on exit, and add it to a log2 bucket. Per-event data never crosses into user space.

3CollectAfter a fixed duration the Go collector reads the histogram maps and serialises them, together with the metadata of the machine the run happened on.

4Storescripts/collect_all.sh --duration 30s --out results/... runs all three tools in sequence and writes memlat.csv, runqlat.csv, and iolat.csv with matching .summary.json files.

04Results

Scheduling latency shows a long tail under load even when CPU utilisation looks normal, which is exactly the case an average hides.
Memory access latency stays flat until page faults are induced; then the tail buckets spike while the median barely moves.
Block I/O latency clusters by request size and device behaviour, so the slow requests are identifiable rather than merely present.

Key takeaway The output of every run is a distribution, and the interesting part of all three distributions is the tail. Fixed durations and one output format are what make two tails comparable at all.

05What went wrong

Symptom

Measurements of the same workload disagreed between runs, and there was no way to tell whether a difference was real or an artefact of how the run was started.

Hypotheses

Each tool had its own duration handling, its own output shape, and no record of the machine state, so runs from different sessions were not comparable in the first place.

Tests

Standardised duration and output format across the three tools, then reran identical workloads repeatedly to see how much the histograms varied between runs.

Fix

One CLI, one fixed-duration collection path, and environment metadata written next to every result set.

Takeaway

The measurement method has to be as fixed as the workload. Ad-hoc tracing produces numbers; a fixed method produces numbers you can compare.

06Engineering decisions

Compute deltas in the kernel and export only histograms. Sending one record per event to user space would cost more than the latency being measured.
Bucket by log2 rather than report an average. The tail is the part users feel, and a mean over a long-tailed distribution reports neither the common case nor the bad one.
Give all three tools the same flags and the same output format. Comparing a CPU histogram with an I/O histogram is the whole point, and that only works if they are shaped alike.