eBPF Intrusion Detection for K8s Lateral Movement

Traditional host-based intrusion detection relies on either userspace agents polling /proc or auditd rules bolted onto the kernel’s audit subsystem. Both approaches introduce unacceptable blind spots in containerised environments: auditd cannot cheaply attribute a syscall to a specific container namespace without expensive correlation, and userspace polling misses short-lived processes entirely. When an attacker executes a fileless payload that lives for 40 milliseconds inside a compromised pod, sampling-based tooling simply never sees it. This is precisely the gap that eBPF intrusion detection closes by hooking directly into kernel tracepoints and kprobes, giving you deterministic, per-event visibility into syscalls, network flows and process lifecycles without the overhead of a full audit pipeline.

This article walks through a production-grade architecture for deploying eBPF intrusion detection across a multi-tenant Kubernetes cluster, specifically targeting lateral movement detection — the phase of an intrusion where an attacker pivots from an initially compromised pod towards the control plane, a metadata service, or an adjacent namespace.

#The Blind Spot Between Syscalls and Container Runtime

The core architectural problem is one of context loss. The Linux kernel does not natively understand “pod” or “namespace” as a security boundary — it understands PID namespaces, cgroups, and network namespaces as isolated but structurally unlabelled primitives. A syscall like connect() issued from inside a container looks, to a naive tracer, identical to one issued from the host. Without enriching kernel-level events with container metadata at the point of capture, any downstream detection logic is working blind.

Compounding this, lateral movement rarely trips conventional network-layer alerting. An attacker using a stolen service account token to call the Kubernetes API server from within a pod that has never previously made that call is a textbook anomaly, but it is invisible to a firewall rule set that permits all egress to kubernetes.default.svc by default. Detecting this requires syscall-level behavioural baselining, not port-based filtering — which is exactly the terrain eBPF intrusion detection operates in.

#Architecting eBPF Intrusion Detection at the Kernel Level

An eBPF intrusion detection pipeline is composed of four discrete layers, each with distinct failure characteristics and performance costs:

• Instrumentation layer — kprobes/kretprobes and tracepoints attached to syscalls such as execve, connect, openat, and ptrace.
• Enrichment layer — a BPF map lookup that resolves the current cgroup ID to a pod UID, namespace, and container image digest via the kubelet’s cgroup driver metadata.
• Aggregation layer — a ring buffer (BPF_MAP_TYPE_RINGBUF) that streams events to userspace without the lock contention inherent in older perf buffer implementations.
• Detection layer — a rules engine (Falco, Tetragon, or a custom consumer) applying behavioural policy against the enriched event stream.

This layered approach mirrors the broader architectural patterns used in event-driven security tooling generally: decouple capture from correlation, and correlation from alerting, so that each layer can scale or fail independently. Tools such as eBPF’s official documentation describe the verifier and JIT compilation stages that make this instrumentation layer safe to run in a hostile kernel context — critical, because a misbehaving probe on a production node is a self-inflicted denial-of-service.

#Why Kprobes Over Auditd for This Use Case

Auditd rules are evaluated in a linear rule-matching engine and write structured logs to disk via the audit daemon — a synchronous path that adds measurable latency under high syscall volume (observed at 800μs–1.2ms per event on syscall-heavy workloads such as compilers or package managers running inside a pod). An eBPF program attached to the same tracepoint executes in-kernel, writes only the fields required for detection into a ring buffer, and returns control to the syscall path in under 50 nanoseconds in the common case. For eBPF intrusion detection at fleet scale — thousands of nodes, tens of thousands of pods — this difference is the deciding factor between a viable control and one that gets disabled under load-testing pressure from the platform team.

#Implementation Logic: From Kprobes to Alerting

The implementation sequence below assumes a cluster running kernel 5.8+ (for BTF and ring buffer support) and a CNI that exposes cgroup-to-pod mapping, such as Cilium.

1. Attach a kprobe to sys_enter_execve to capture process spawn events, tagging each event with the calling process’s cgroup ID.
2. Attach a kprobe to tcp_connect to capture outbound connection attempts at the socket layer, before DNS or iptables NAT rewriting obscures the original destination.
3. Resolve cgroup ID to pod identity via a BPF hash map populated by a small userspace daemon watching the kubelet’s /sys/fs/cgroup hierarchy.
4. Stream enriched events into a ring buffer consumed by a userspace agent (e.g. libbpfgo or a Tetragon sidecar).
5. Apply behavioural rules — flag any pod issuing connect() to the API server’s ClusterIP that has no prior baseline of doing so, or any execve of a shell binary inside a container whose image manifest declares no interactive entrypoint.
6. Emit structured alerts to a SIEM or a dedicated detection queue for correlation with Kubernetes audit logs.

#Kernel-Side Program

The following is a minimal (illustrative) eBPF C program hooking execve, tagging the event with the cgroup ID for later enrichment. It is compiled with clang against the vmlinux BTF headers and loaded via libbpf.

1SEC("tracepoint/syscalls/sys_enter_execve")
2int trace_execve(struct trace_event_raw_sys_enter *ctx)
3{
4    struct event_t evt = {};
5    evt.pid = bpf_get_current_pid_tgid() >> 32;
6    evt.cgroup_id = bpf_get_current_cgroup_id();
7    bpf_get_current_comm(&evt.comm, sizeof(evt.comm));
8
9    // Push into ring buffer for userspace consumption
10    struct event_t *rb_evt = bpf_ringbuf_reserve(&events, sizeof(evt), 0);
11    if (!rb_evt)
12        return 0;
13
14    __builtin_memcpy(rb_evt, &evt, sizeof(evt));
15    bpf_ringbuf_submit(rb_evt, 0);
16    return 0;
17}

#Behavioural Rule Definition

Once events are enriched with pod identity, detection logic can be expressed declaratively. The following Falco-style rule flags shell execution inside pods labelled as backend microservices, a common lateral movement precursor once an attacker has RCE inside a container.

1- rule: Unexpected Shell in Backend Pod
2  desc: >
3    Detects execve of interactive shell binaries inside pods labelled
4    tier=backend, which should never spawn a shell in normal operation.
5  condition: >
6    spawned_process and
7    k8s.pod.label.tier = "backend" and
8    proc.name in (bash, sh, zsh, ash)
9  output: >
10    Shell spawned in backend pod (pod=%k8s.pod.name ns=%k8s.ns.name
11    proc=%proc.name parent=%proc.pname user=%user.name)
12  priority: CRITICAL
13  tags: [lateral_movement, ebpf_intrusion_detection]

#Alert Schema for Downstream Correlation

To keep the detection layer loosely coupled from the SIEM, emit a normalised JSON payload rather than raw kernel structures. This lets the same eBPF intrusion detection pipeline feed Splunk, Elastic, or a custom Sigma-rule engine without rewriting the consumer.

1{
2  "event_type": "process.exec",
3  "timestamp": "2024-05-14T11:02:31.884Z",
4  "pod": {
5    "name": "checkout-service-7f9b4",
6    "namespace": "prod-payments",
7    "uid": "a13c9f2e-..."
8  },
9  "process": {
10    "comm": "bash",
11    "pid": 48213,
12    "parent_comm": "java"
13  },
14  "cgroup_id": 8842103,
15  "severity": "critical",
16  "detection_source": "ebpf_intrusion_detection"
17}

#Failure Modes and Edge Cases

eBPF intrusion detection is not failure-free, and treating it as an infallible sensor introduces its own risk. The following edge cases must be explicitly handled in production:

• Ring buffer overflow under burst load. A fork bomb or aggressive CI job inside a namespace can generate execve events faster than the userspace consumer can drain the ring buffer, causing silent event drops. Size the buffer generously (8–16MB per node) and instrument drop counters explicitly rather than assuming zero loss.
• Cgroup ID reuse. On high pod-churn nodes, cgroup IDs are recycled quickly once a pod terminates. If your enrichment map is not evicted on pod deletion via a Kubernetes informer, you risk attributing a new pod’s syscalls to a stale identity — a critical correctness bug for any audit trail.
• Kernel version drift across node pools. Mixed-kernel clusters (common in gradual node upgrades) mean BTF availability and supported helper functions differ per node. Programs must be compiled with CO-RE (Compile Once, Run Everywhere) to avoid per-kernel recompilation, per guidance in the kernel BTF documentation.
• Verifier rejection on complex programs. The eBPF verifier enforces a bounded instruction count and loop restrictions. Overly ambitious detection logic pushed into the kernel (rather than userspace) will simply fail to load, and this must be caught in CI, not discovered on a production rollout.
• Privileged sidecar requirement. Most eBPF loaders require CAP_BPF or CAP_SYS_ADMIN on older kernels, which conflicts with strict Pod Security Admission baselines. This forces a trade-off between the security posture of the detection agent itself and its visibility.

#Scaling and Security Trade-offs

Deploying eBPF intrusion detection fleet-wide requires deliberate trade-offs between fidelity, node overhead, and operational blast radius. From an architectural standpoint, the decisions below determine whether the control survives contact with a real production incident or gets rolled back after the first false-positive storm.

• Per-node CPU overhead: a well-tuned eBPF intrusion detection agent adds 1–3% CPU overhead per node under typical syscall volume; naive tracing of every syscall (rather than a targeted set) can push this above 15%, degrading co-located workloads.
• Detection latency vs. batching: ring buffer polling intervals of 100ms give near-real-time detection at the cost of more frequent context switches; batching to 1–2 seconds reduces CPU wake-ups but delays containment response during active lateral movement.
• Centralised vs. node-local rule evaluation: evaluating behavioural rules at the node reduces network egress volume but fragments rule updates across the fleet; centralising evaluation simplifies rule management but requires shipping every raw event off-node, multiplying network and storage costs.
• Privilege scope of the loader daemon: running the loader as a DaemonSet with host PID and CAP_BPF maximises visibility but expands the node’s attack surface if the agent itself is compromised — mitigate by pinning the loader to a minimal, signed image and disabling unnecessary syscalls via seccomp on the agent itself.
• False-positive tuning cost: behavioural baselining for lateral movement detection requires a minimum 7–14 day observation window per workload before enforcement mode is safe to enable; shorter windows produce unacceptable noise, particularly around deployment rollouts and autoscaling events.

None of these trade-offs are resolved generically — they are workload-specific decisions that should be revisited every time a new node pool, CNI version, or kernel upgrade lands in the cluster. Treat the eBPF intrusion detection pipeline as a living control with its own SLOs for event drop rate and detection latency, not a fire-and-forget deployment.