Fixing IRQ Affinity Bottlenecks on Home NAS Boxes

A 10GbE NIC pegged at 35% throughput while core0 sits at 100% softirq load is not a driver bug — it is a scheduling problem. This is the single most common cause of mysterious throughput ceilings on self-built NAS units, pfSense/OPNsense routers, and home lab hypervisors: every interrupt from the network card lands on the same CPU core because IRQ affinity tuning was never configured, and the kernel’s default interrupt distribution does not know your workload. The NIC advertises multiple hardware queues, the CPU has eight or sixteen cores available, and yet all the packet processing funnels through one thread of execution. This article walks through why that happens, how to diagnose it with kernel-level tooling, and how to rebalance interrupt load correctly.

#The Bottleneck: Single-Queue Interrupt Pinning

Modern NICs (Intel i225/i226, Mellanox ConnectX, even higher-end Realtek chips) expose multiple RX/TX queues, each backed by its own MSI-X interrupt vector. In theory, the kernel’s Receive Side Scaling (RSS) mechanism hashes incoming packet headers (source/dest IP and port) and distributes them across these queues, which in turn should be serviced by different CPU cores. In practice, three things routinely break this:

• The NIC driver defaults to a conservative queue count (often 1 or 2) unless explicitly told otherwise via ethtool.
• irqbalance is either absent, disabled, or actively fighting a manually configured affinity mask.
• Virtualised NICs (VirtIO, e1000e under KVM/Proxmox) frequently ship with a single queue by default, regardless of the vCPU count assigned to the guest.

The symptom is consistent: /proc/interrupts shows one IRQ line accumulating counts an order of magnitude faster than its siblings, mpstat -P ALL 1 shows one core saturated in the %soft column, and aggregate throughput plateaus well below line rate despite idle cores elsewhere in the system.

#Architectural Breakdown: How Interrupts Map to Cores

Each hardware interrupt vector has an associated SMP affinity mask stored in /proc/irq/<N>/smp_affinity (or the more readable smp_affinity_list). This mask determines which logical CPUs are eligible to service that interrupt. When a NIC driver initialises multiple queues, it typically registers one IRQ per queue and, on well-supported hardware, attempts a sane default distribution. Consumer-grade NICs and virtualised drivers frequently do not, defaulting every vector to CPU 0.

On top of hardware IRQs, the kernel maintains a software layer: RPS (Receive Packet Steering) and RFS (Receive Flow Steering), which redistribute packet processing at the softirq level even when the hardware queue count is limited. RPS hashes flows and dispatches them to a configurable set of CPUs via rps_cpus; RFS goes further by tracking which CPU last handled a given flow’s userspace thread, reducing cache-line bouncing. These are software fallbacks — they consume more CPU aggregate than proper RSS but are essential when the NIC itself cannot expose enough hardware queues (a near-universal constraint on budget home routers).

The correct architectural model treats interrupt distribution as a first-class scheduling concern, on par with process placement. This is the same reasoning applied to broader architectural patterns for resource isolation — you are deciding which execution context owns a given unit of work, and that decision has direct latency and throughput consequences.

#NUMA Locality Matters

On dual-socket boards or higher-end AMD platforms with multiple NUMA nodes, IRQ affinity tuning must also respect NUMA topology. Pinning an interrupt to a core on NUMA node 1 while the NIC’s DMA buffers are allocated on node 0 introduces cross-socket memory latency on every packet — often invisible until you profile with perf stat -e node-loads,node-load-misses. Always confirm the NIC’s local node via cat /sys/class/net/eth0/device/numa_node before assigning affinity masks.

#Implementation Logic: Step-by-Step Rebalancing

The remediation sequence below applies to bare-metal Linux NAS/router builds (Debian, Proxmox host, OPNsense’s FreeBSD variant uses analogous but differently-named tools).

#Step 1: Confirm Queue Count and Current Distribution

1ethtool -l eth0
2# Combined: 1 (max: 8)
3
4ethtool -x eth0
5# RX flow hash indirection table shows all entries mapping to queue 0
6
7cat /proc/interrupts | grep eth0
8# 128:   4839213    0    0    0   PCI-MSI  eth0-rx-0

If Combined reports fewer queues than the driver supports, increase it before touching affinity masks — there is no point balancing interrupts across cores if the hardware only ever generates one queue’s worth of traffic.

#Step 2: Increase Queue Count

1ethtool -L eth0 combined 4
2ethtool -x eth0 hkey $(openssl rand -hex 40) # optional: randomise RSS hash key
3ethtool -X eth0 equal 4

#Step 3: Disable irqbalance for Targeted Vectors

Running irqbalance alongside manual pinning causes it to periodically undo your configuration. Rather than disabling it system-wide (it still helps for unrelated devices), ban the NIC’s IRQs explicitly:

1# /etc/default/irqbalance
2IRQBALANCE_BANNED_CPULIST="0"
3IRQBALANCE_ARGS="--banirq=128 --banirq=129 --banirq=130 --banirq=131"

#Step 4: Set Explicit SMP Affinity

1for irq in 128 129 130 131; do
2  core=$(( (irq - 128) + 1 ))
3  echo $core > /proc/irq/$irq/smp_affinity_list
4done
5
6# Verify
7grep . /proc/irq/12{8,9}/smp_affinity_list /proc/irq/13{0,1}/smp_affinity_list

Note the deliberate avoidance of core 0 — on most distributions, timer interrupts, the scheduler tick, and a disproportionate share of general kernel bookkeeping already live there. Reserving it as a NIC IRQ target compounds contention rather than resolving it.

#Step 5: Configure RPS as a Software Fallback

For NICs that cap out at one or two hardware queues (common on integrated 2.5GbE controllers found in mini-PC NAS builds), RPS extends the balancing further:

1echo f > /sys/class/net/eth0/queues/rx-0/rps_cpus  # binary mask, cores 0-3
2sysctl -w net.core.rps_sock_flow_entries=32768
3echo 32768 > /sys/class/net/eth0/queues/rx-0/rps_flow_cnt

Persist these via a systemd unit or /etc/rc.local equivalent, since they reset on interface reinitialisation.

#Failure Modes and Edge Cases

IRQ affinity tuning is not without hazards, and getting it wrong is often worse than leaving defaults untouched.

IRQ storms under asymmetric load: if you pin all four NIC vectors to cores 1–4 but your workload (e.g. a Samba file share plus a Plex transcode) is also scheduled on those same cores, you create direct contention between userspace and interrupt handling. The fix is to reserve a distinct core set for IRQs versus application threads using taskset or cgroup cpuset constraints — not simply spreading interrupts as widely as possible.

Dropped packets masked as “slow disk”: when a single core cannot drain its softirq queue fast enough, the kernel silently increments rx_missed_errors or rx_fifo_errors in ethtool -S eth0. This frequently gets misdiagnosed as storage latency because the symptom (stalled file transfer) presents identically. Always check NIC statistics before touching the storage stack.

Virtualised NIC queue mismatch: under KVM/Proxmox with VirtIO-net, the guest must explicitly request multiple queues at VM definition time (queues=4 in the VirtIO device XML) — the host bridging multiple vCPUs does nothing if the guest driver only negotiates a single queue. This is a frequent and entirely invisible bottleneck in home lab hypervisor setups.

RSS hash key predictability: on shared or exposed segments, a static, well-known RSS hash key theoretically allows an attacker to craft flows that all hash to the same queue, recreating the exact single-core bottleneck deliberately (a variant of algorithmic complexity attack). Randomising the hash key via ethtool -X mitigates this at negligible cost.

#Scaling and Security Trade-offs

The decision to invest in manual IRQ affinity tuning versus relying on kernel defaults depends heavily on the throughput ceiling you are chasing and the threat model of the box in question.

• Single-queue default: lowest complexity, zero maintenance, but hard-capped throughput around 3–5 Gbps on typical consumer CPUs regardless of link speed — adequate for gigabit links, inadequate for 2.5GbE/10GbE.
• Manual affinity + irqbalance bans: highest throughput ceiling, but introduces a maintenance burden — kernel updates or driver reloads can renumber IRQs, silently reverting your pinning and requiring re-verification after every update cycle.
• RPS/RFS software steering: works on any NIC regardless of hardware queue support, but consumes measurably more aggregate CPU per packet than true hardware RSS — acceptable trade-off on idle home servers, less so on CPU-constrained SBCs.
• irqbalance left fully enabled: safest default for mixed workloads with unpredictable traffic patterns, but its heuristics are tuned for general-purpose servers, not sustained high-throughput NIC saturation, and will actively undermine manual tuning if not explicitly banned per-IRQ.
• Security surface: running irqbalance as root with broad CAP_SYS_ADMIN access is standard on most distributions; auditing its scope matters more on internet-facing router builds (OPNsense/pfSense) than on isolated LAN-only NAS units.

For a canonical reference on the kernel’s packet steering internals, including the exact semantics of RPS, RFS, and RSS interaction, consult the kernel.org networking scaling documentation — it is the authoritative source the driver maintainers themselves reference.

Treat IRQ affinity tuning as an iterative diagnostic exercise rather than a one-off fix: re-verify /proc/interrupts distribution after every kernel update, driver change, or NIC swap, since none of these settings survive a cold reinitialisation of the interface by default.