Real-Time eBPF Network Observability at Scale

At sustained throughput above 10 Gbps, conventional packet capture tooling collapses under its own overhead. tcpdump relies on AF_PACKET sockets that copy every frame across the kernel-userspace boundary; sidecar proxies in a service mesh add a full TCP/IP round trip per hop; and iptables byte counters give you aggregate numbers with zero contextual metadata. None of these approaches survive contact with a high-cardinality, high-frequency production workload. This is the exact gap that eBPF network observability was engineered to close — instrumenting the kernel’s packet processing path directly, without context switches, without packet duplication, and without a proxy in the data path.

This article breaks down the subsystem architecture required to build a production-grade eBPF network observability pipeline: hook selection, map topology, verifier constraints, and the failure modes that surface once you push this past a proof-of-concept into a fleet running thousands of nodes.

#The Packet Inspection Bottleneck

Traditional observability tooling sits at the wrong layer. A userspace packet sniffer requires the kernel to copy each frame into a ring buffer shared with a socket, then wake a userspace process to drain it — a syscall and a context switch per batch, at minimum. Under packet storms or SYN floods, this pipeline drops samples precisely when visibility matters most. Sidecar-based mesh observability (Envoy, Linkerd-proxy) solves the metadata problem but reintroduces latency: every packet now traverses an additional TCP stack, socket buffer, and TLS termination point.

eBPF network observability inverts this model. Instead of exporting packets to userspace for analysis, you push the analysis logic into the kernel itself, at the exact point the packet is processed, and export only the aggregated result. The kernel becomes the collection agent; userspace becomes a pure consumer of pre-aggregated state.

#Architectural Breakdown: Kernel Hooks and Map Topology

An eBPF program is a restricted, verifier-checked bytecode blob that the kernel JIT-compiles to native machine code before execution. For network observability, the choice of hook point determines both the granularity of data available and the performance ceiling of the collector.

#Hook Point Selection for eBPF Network Observability

• XDP (eXpress Data Path) — attaches at the NIC driver level, before the kernel allocates an sk_buff. Lowest overhead, but limited to the raw frame; no socket context.
• TC (Traffic Control) ingress/egress — attaches after sk_buff allocation, giving access to skb metadata (cgroup, mark, queue mapping) at a modest cost increase over XDP.
• kprobes/tracepoints on functions like tcp_sendmsg or inet_csk_accept — ideal for connection lifecycle and latency observability, but unstable across kernel versions unless you use BTF-backed CO-RE (Compile Once, Run Everywhere).
• Socket filters / cgroup hooks — useful for per-container or per-namespace network observability without touching the physical NIC path.

For line-rate eBPF network observability — counting protocol distribution, detecting SYN floods, or sampling flow five-tuples — XDP in native driver mode is the correct target. It runs before the SKB allocation, avoiding roughly 30–40% of the per-packet cost associated with generic hooks.

The map layer is where most architectural mistakes occur. A naive implementation uses a single BPF_MAP_TYPE_HASH shared across all CPUs, forcing spinlock contention on every packet. Production eBPF network observability pipelines use BPF_MAP_TYPE_PERCPU_HASH or BPF_MAP_TYPE_PERCPU_ARRAY for counters, eliminating cross-CPU locking entirely, and reconcile per-CPU values only when userspace polls the map — typically every 1–5 seconds. For discrete events (new flow detected, anomalous TTL, port scan signature), BPF_MAP_TYPE_RINGBUF is preferred over the older BPF_MAP_TYPE_PERF_EVENT_ARRAY, since it avoids per-CPU buffer fragmentation and guarantees ordering within a single reader.

#Implementation Logic

Building the pipeline follows a consistent sequence regardless of the specific metric being extracted:

• Define the BPF program and map layout in C, targeting the xdp section for native attachment.
• Compile with clang -target bpf and embed BTF (BPF Type Format) so the program can be loaded via CO-RE across kernel versions without recompilation.
• Load and attach the program using libbpf (C) or cilium/ebpf (Go) — both handle relocation, verifier submission, and map creation.
• Pin the maps to /sys/fs/bpf/ so a userspace collector can reattach after a restart without losing accumulated state.
• Poll per-CPU maps on a fixed interval; drain the ring buffer asynchronously via epoll on its associated file descriptor.
• Aggregate and expose the resulting metrics through a Prometheus /metrics endpoint or push to an OpenTelemetry collector.

This design pattern — kernel-side aggregation, userspace-side export — mirrors broader architectural patterns used in distributed systems where you push computation as close to the data source as possible and only transmit reduced state across expensive boundaries.

#Production-Ready Code and Configuration

The following XDP program counts packets by IP protocol into a per-CPU hash map, forming the core of a minimal eBPF network observability agent:

1// xdp_proto_count.c
2#include 
3#include 
4#include 
5#include 
6
7struct {
8    __uint(type, BPF_MAP_TYPE_PERCPU_HASH);
9    __uint(max_entries, 256);
10    __type(key, __u8);
11    __type(value, __u64);
12} proto_counters SEC(".maps");
13
14SEC("xdp")
15int xdp_proto_count(struct xdp_md *ctx)
16{
17    void *data = (void *)(long)ctx->data;
18    void *data_end = (void *)(long)ctx->data_end;
19
20    struct ethhdr *eth = data;
21    if ((void *)(eth + 1) > data_end)
22        return XDP_PASS;
23
24    if (eth->h_proto != __builtin_bswap16(ETH_P_IP))
25        return XDP_PASS;
26
27    struct iphdr *ip = (void *)(eth + 1);
28    if ((void *)(ip + 1) > data_end)
29        return XDP_PASS;
30
31    __u8 proto = ip->protocol;
32    __u64 *count = bpf_map_lookup_elem(&proto_counters, &proto);
33    if (count) {
34        (*count)++;
35    } else {
36        __u64 init = 1;
37        bpf_map_update_elem(&proto_counters, &proto, &init, BPF_ANY);
38    }
39
40    return XDP_PASS;
41}
42
43char _license[] SEC("license") = "GPL";

A Go-based collector using cilium/ebpf loads this object, attaches it in native mode, and polls the map on a fixed interval:

1objs := bpfObjects{}
2if err := loadBpfObjects(&objs, nil); err != nil {
3    log.Fatalf("loading objects: %v", err)
4}
5defer objs.Close()
6
7link, err := link.AttachXDP(link.XDPOptions{
8    Program:   objs.XdpProtoCount,
9    Interface: iface.Index,
10    Flags:     link.XDPDriverMode, // native mode, not generic
11})
12if err != nil {
13    log.Fatalf("attaching XDP: %v", err)
14}
15defer link.Close()
16
17ticker := time.NewTicker(2 * time.Second)
18for range ticker.C {
19    var key uint8
20    var values []uint64
21    iter := objs.ProtoCounters.Iterate()
22    for iter.Next(&key, &values) {
23        var total uint64
24        for _, v := range values { total += v } // sum across CPUs
25        exportMetric(key, total)
26    }
27}

For inspection and debugging in the field, bpftool remains the fastest path to validating that a program is attached correctly and the verifier accepted it without silent truncation:

1ip link set dev eth0 xdp obj xdp_proto_count.o sec xdp
2bpftool prog show
3bpftool map dump name proto_counters
4bpftool net show dev eth0

#Failure Modes and Edge Cases

An eBPF network observability deployment that works in staging can fail in specific, predictable ways once it hits production traffic patterns:

• Verifier rejection on kernel upgrade. The verifier’s complexity budget (historically 1 million processed instructions, with stricter bounds on loop unrolling and stack depth of 512 bytes) can reject a program that previously loaded, if a kernel minor version tightens pointer arithmetic checks. Pin CI to test against your minimum supported kernel, not just your development kernel.
• Driver mode fallback. If the NIC driver lacks native XDP support, the kernel silently falls back to generic (SKB) mode, which reintroduces the SKB allocation cost you were trying to avoid. Always verify attachment mode via bpftool net show rather than assuming success from a zero exit code.
• Map exhaustion. Fixed-size hash maps (five-tuple flow tracking, in particular) will return -E2BIG under a DDoS or port-scan burst, silently dropping new keys once max_entries is reached. Use BPF_MAP_TYPE_LRU_HASH for flow tables that must degrade gracefully instead of failing closed.
• Ring buffer backpressure. If the userspace consumer stalls (GC pause, CPU starvation), the ring buffer fills and the kernel-side bpf_ringbuf_reserve call starts returning NULL, causing silent event loss with no kernel-level error surfaced to the operator unless explicitly counted.
• Per-CPU aggregation skew. Reading a per-CPU map without correctly summing all CPU slots produces undercounted metrics that look plausible but are systematically wrong — a common and hard-to-detect bug in first-pass implementations.

#Scaling and Security Trade-offs for eBPF Network Observability

Once the pipeline moves from a single host to a fleet, the trade-offs shift from correctness to blast radius and privilege boundaries. Reference the kernel’s own subsystem documentation at the official Linux BPF documentation before finalising deployment policy, since verifier behaviour and map semantics do shift between LTS releases.

• Native XDP vs generic vs hardware offload — native mode gives the highest throughput for eBPF network observability but requires driver support (most modern Intel/Mellanox NICs qualify); hardware-offloaded XDP shifts execution to the NIC itself but restricts the instruction set further.
• Privilege boundary — since Linux 5.8, CAP_BPF decouples eBPF loading from full CAP_SYS_ADMIN, meaning your collector daemon no longer needs root-equivalent access, reducing the attack surface if the collector itself is compromised.
• Verifier as an attack surface — the verifier itself has had CVEs (integer overflow bugs enabling out-of-bounds reads); unprivileged eBPF loading should remain disabled (kernel.unprivileged_bpf_disabled=1) on any host processing untrusted workloads.
• Ring buffer vs perf buffer overhead — ring buffer throughput is measurably higher under sustained event rates because it avoids per-CPU buffer duplication, at the cost of single-consumer semantics; perf buffers still suit scenarios needing per-CPU independent consumers.
• Map sizing vs memory pressure — per-CPU maps multiply memory consumption by core count; a 64-core node with a 1M-entry flow table can consume several hundred MB of kernel memory that is not visible to standard cgroup memory accounting in older kernels.
• Cross-node aggregation — eBPF network observability data is inherently per-node; correlating flow state across a cluster still requires a userspace aggregation layer (e.g. Cilium’s Hubble, or a custom OpenTelemetry pipeline), since the kernel provides no native cross-host state sharing.