Building a TCP Retransmission Rate Diagnostic Tool

A production API cluster reporting p99 latency spikes of 800ms with zero CPU pressure, zero GC pauses and clean application logs is almost always a transport-layer symptom, not an application bug. The usual suspect is an elevated TCP retransmission rate that never surfaces in APM traces because retransmission happens below the socket abstraction. Application Performance Monitoring tools instrument function calls and database round-trips; they do not instrument the kernel’s retransmission queue. This gap is precisely why teams need a purpose-built diagnostic tool rather than relying on generic dashboards.

This article walks through building a lightweight, low-overhead retransmission rate diagnostic that combines ss socket statistics, targeted tcpdump capture and eBPF

#The Bottleneck: Retransmission Is Invisible to Application Metrics

TCP retransmission occurs when the sender’s retransmission timer (RTO) fires before an ACK arrives, or when three duplicate ACKs trigger fast retransmit. Both mechanisms are entirely transparent to the application; a blocked write() call simply takes longer. From the application’s perspective, a request that took 40ms and one that took 900ms due to a lost segment and RTO backoff look identical — both are just “slow”.

The consequence is that engineering teams chase phantom application bottlenecks — connection pool sizing, thread starvation, database query plans — when the actual fault lies in a congested top-of-rack switch, a misconfigured MTU, or an overloaded conntrack table dropping packets silently. Without a dedicated measurement layer for the TCP retransmission rate, this class of fault is nearly undiagnosable from application telemetry alone.

#

Why Standard Tools Fall Short

netstat -s gives you a cumulative, host-wide retransmission counter since boot. It tells you retransmission is happening somewhere, on some connection, at some point in the past. It gives zero attribution to specific destination IPs, specific ports, or specific time windows. For root-causing an intermittent loss event affecting one downstream dependency, this resolution is useless.

#Architectural Breakdown

The diagnostic tool needs three cooperating layers, each solving a different resolution problem:

• Layer 1 — Socket-level sampling (ss): Polls ss -ti at fixed intervals to extract per-socket retrans and rto fields, giving connection-level granularity without packet capture overhead.
• Layer 2 — Kernel retransmit event tracing (eBPF): Attaches to tcp_retransmit_skb via a kprobe, capturing exact timestamps, socket 4-tuples and sequence numbers with negligible overhead (sub-1% CPU at typical throughput).
• Layer 3 — Targeted packet capture (tcpdump filtered by BPF expression): Triggered conditionally when Layer 2 detects a retransmission burst exceeding a threshold, capturing the surrounding window for offline analysis (duplicate ACK patterns, SACK ranges).

This tiered design avoids the classic diagnostic trade-off: full packet capture gives complete fidelity but is prohibitively expensive to run continuously on high-throughput hosts, while socket sampling is cheap but coarse. Running eBPF continuously as the trigger layer, with tcpdump reserved for anomaly windows, gives both efficiency and forensic depth.

#Implementation Logic

Step one is establishing a baseline sampling loop using ss, which exposes retransmission counters without requiring elevated capture privileges. This runs as a lightweight sidecar or systemd

1#!/usr/bin/env bash
2# poll_retrans.sh - samples per-socket retransmission counters
3INTERVAL=1
4while true; do
5  TS=$(date +%s%N)
6  ss -ti state established '( dport = :443 or sport = :443 )' | 
7    awk -v ts="$TS" '
8      /^tcp/ { split($0, addr, " "); src=addr[4]; dst=addr[5] }
9      /retrans:/ {
10        for (i=1; i<=NF; i++) {
11          if ($i ~ /^retrans:/) { split($i, r, ":"); retrans=r[2] }
12          if ($i ~ /^rto:/) { split($i, rt, ":"); rto=rt[2] }
13        }
14        print ts, src, dst, retrans, rto
15      }'
16  sleep $INTERVAL
17done

The retrans field in ss output is formatted as retransmitted_segments/total_retransmits and resets per-connection lifetime, not per-interval, so the diagnostic tool must compute a first-order derivative across polling intervals to derive an actual TCP retransmission rate (retransmits per second) rather than treating the raw counter as a rate itself. This delta calculation is the single most common implementation error in home-grown retransmission tooling.

#

eBPF Kprobe for Precise Event Capture

For sub-second attribution, attach a kprobe directly to the kernel retransmission path using bpftrace, which avoids the sampling gaps inherent to interval polling:

1bpftrace -e '
2kprobe:tcp_retransmit_skb
3{
4    $sk = (struct sock *)arg0;
5    $inet_family = $sk->__sk_common.skc_family;
6    if ($inet_family == 2) {
7        $daddr = ntop($sk->__sk_common.skc_daddr);
8        $dport = $sk->__sk_common.skc_dport;
9        printf("%s retransmit -> dst=%s dport=%d pid=%dn",
10            strftime("%H:%M:%S", nsecs), $daddr, $dport, pid);
11    }
12}'

This single probe replaces the need for continuous packet capture on most hosts. The overhead is dominated by the kprobe entry/exit cost, typically under 200ns per invocation, making it viable even on hosts pushing tens of thousands of connections. Cross-reference this against the official kernel networking scaling documentation when tuning for high connection-count hosts, since RSS queue affinity affects how evenly retransmit events distribute across cores.

#

Exporting to Prometheus

The eBPF event stream feeds a counter that a small exporter converts into a rate metric, following standard architectural patterns for metrics pipelines — ingest raw events, aggregate into a ring buffer, expose via a scrape endpoint:

1- job_name: 'tcp-retransmit-exporter'
2  scrape_interval: 5s
3  static_configs:
4    - targets: ['localhost:9235']
5  metric_relabel_configs:
6    - source_labels: [__name__]
7      regex: 'tcp_retransmit_rate.*'
8      action: keep

Alerting rules should fire on a sustained TCP retransmission rate exceeding 2% of total segments sent over a rolling five-minute window — below 1% is generally normal background loss on shared infrastructure, while sustained rates above 5% indicate active path degradation warranting immediate investigation.

#Failure Modes and Edge Cases

The diagnostic tool itself introduces failure modes worth accounting for during design:

• Kprobe attach failures on hardened kernels: Grsecurity or lockdown-mode kernels restrict kprobe attachment; the tool must fail gracefully to socket-polling-only mode rather than crash-looping.
• Sampling aliasing: A 1-second polling interval on ss will miss microbursts that occur and clear within the interval, understating the true retransmission rate during transient congestion events. This is why the eBPF layer is not optional for high-fidelity measurement.
• NAT and connection tracking interference: On hosts behind conntrack-heavy NAT gateways, retransmitted segments can be misattributed if the 4-tuple changes mid-connection due to conntrack table eviction, producing phantom retransmission spikes that are actually connection resets.
• Loopback and container network namespace boundaries: eBPF kprobes attached in the host namespace will not see retransmissions occurring purely inside a container’s virtual ethernet pair unless the probe is namespace-aware, a common gap in containerised deployments.

#Diagnostic Correlation Table

#Scaling and Security Trade-offs

Deploying this diagnostic fleet-wide introduces trade-offs that must be weighed against operational value:

• Continuous eBPF vs on-demand tcpdump: Running the kprobe fleet-wide costs roughly 0.5–1% CPU overhead at 10Gbps line rates but provides always-on visibility; on-demand tcpdump is zero-cost when idle but risks missing the triggering event if the burst-detection threshold is misconfigured.
• Privilege escalation surface: Loading eBPF programs requires CAP_BPF or CAP_SYS_ADMIN depending on kernel version; centralising this capability into a single privileged daemon rather than granting it per-container materially reduces the attack surface.
• Packet capture data sensitivity: Triggered tcpdump captures may contain unencrypted payload fragments for non-TLS internal traffic; capture files must be encrypted at rest and access-controlled separately from general metrics storage.
• Cardinality explosion in per-connection metrics: Exporting a distinct time series per source/destination/port tuple at scale will overwhelm most Prometheus deployments; aggregate to per-destination-host granularity before export and retain per-connection detail only in the triggered capture layer.
• Cross-namespace visibility vs container isolation: Attaching kprobes cluster-wide from a privileged DaemonSet grants visibility into every pod’s retransmission behaviour, which is operationally valuable but requires explicit RBAC boundaries to prevent one tenant inspecting another’s traffic patterns in multi-tenant clusters.

The value of building this tool in-house rather than relying solely on cloud provider flow logs is attribution latency — VPC flow logs typically aggregate on 1-to-10-minute windows, whereas the eBPF-driven approach delivers sub-second event timestamps necessary for correlating a retransmission burst with a specific deploy, a specific conntrack eviction, or a specific noisy neighbour on shared infrastructure. Teams running latency-sensitive services at scale should treat TCP retransmission rate as a first-class SLI alongside error rate and request latency, not as a secondary network-team concern surfaced only during incident retrospectives.