NUMA-Aware Scheduling
Modern multi-socket servers partition physical memory into banks attached to specific CPU sockets. A core accessing local memory pays a fixed latency; accessing memory attached to a remote socket traverses the inter-socket interconnect (e.g. Intel UPI, AMD Infinity Fabric), incurring 1.5x-3x higher latency and reduced effective bandwidth. The OS scheduler and hypervisor expose this as NUMA nodes, each bundling a set of CPUs, local DRAM, and often a directly attached PCIe root complex (relevant for SR-IOV NICs and NVMe). NUMA-aware scheduling is the discipline of pinning a process’s CPU set, memory allocations, and device interrupts to the same node so the majority of memory accesses stay local.
In Kubernetes, this is implemented through the interaction of the CPU Manager (static policy, exclusive core assignment for Guaranteed QoS pods), Memory Manager, Topology Manager, and Device Manager hints. The Topology Manager acts as an arbiter: each hint provider reports the set of NUMA nodes it could satisfy a resource request from, and the kubelet’s alignment policy (none, best-effort, restricted, single-numa-node) decides whether the pod is admitted, scheduled with a suboptimal placement, or rejected outright. Without this coordination, a container can be granted CPUs on node 0 while its GPU or NIC is on the PCIe root complex of node 1, forcing every DMA transfer and interrupt across the interconnect.
- Silent degradation: utilization dashboards (CPU%, memory%) never surface NUMA locality; only hardware performance counters (e.g.
perf c2c,numastat) or application-level p99 latency regressions reveal the problem. - Bin-packing tension:
single-numa-nodealignment fragments the scheduler’s bin-packing options — a node with free capacity spread across two NUMA domains cannot admit a pod requiring strict single-node alignment, producing scheduling failures despite apparent headroom. - Live migration and hot-add: hypervisor-level vNUMA topology exposed to a guest must match the underlying host topology; mismatched vNUMA after a live migration to different hardware silently breaks the guest OS’s own NUMA-aware allocation logic.
- Interaction with hyperthreading: exclusive core allocation policies must reason about sibling threads on the same physical core, not just logical CPU IDs, or two latency-sensitive containers can be co-scheduled on siblings and contend for shared L2/L1 cache.
The architectural consequence is that NUMA-awareness must be treated as a first-class scheduling constraint for latency-critical, memory-bandwidth-bound workloads — in-memory databases, DPDK-based packet processors, and large ML inference servers — rather than an OS-level implementation detail. Capacity planning for such workloads has to reason in terms of whole-NUMA-node allocatable units, and cluster autoscalers or bin-packers ignorant of topology will systematically underperform, making this one of the few areas where infrastructure-level topology must leak into scheduling policy.