RDMA (Remote Direct Memory Access)

RDMA works by exposing NIC hardware queues (Queue Pairs: send, receive, completion) directly to userspace applications via a verbs API, bypassing the kernel network stack entirely for the data path. Before any transfer, memory regions must be pinned and registered with the NIC, which programs the device’s MMU/IOMMU to translate virtual addresses so the adapter can DMA into or out of that region without page faults. Operations come in two flavors: two-sided (SEND/RECV, which still requires the remote CPU to post a matching receive) and one-sided (RDMA READ/WRITE, which the remote CPU is entirely unaware of). One-sided operations are what give RDMA its defining property — a remote memory access that never schedules, interrupts, or context-switches the remote process.

Transport is carried over one of three physical/link layers: native InfiniBand (lossless fabric with credit-based flow control), RoCE (RDMA over Converged Ethernet, encapsulating InfiniBand transport in UDP/IP, requiring Ethernet to be made lossless via Priority Flow Control and ECN), or iWARP (RDMA over standard TCP/IP, tolerant of loss but with higher latency). RoCEv2 is the dominant deployment choice in hyperscale data centers because it reuses existing Ethernet/IP infrastructure, but it inherits Ethernet’s lossy nature unless PFC/DCQCN congestion control is correctly tuned — a misconfigured fabric produces PFC storms and head-of-line blocking that can cascade across an entire pod, the opposite of the determinism RDMA is meant to provide.

Distributed systems exploit RDMA for exactly the operations where syscall and copy overhead dominate: log replication (Raft/Paxos followers exposing their WAL as an RDMA-writable buffer so the leader pushes entries directly), disaggregated memory and NVMe-oF storage targets, and in-memory databases (Aerospike, FASTER, several HTAP engines) that use RDMA READ to satisfy remote lookups without a remote CPU cycle. This also introduces a fundamentally different failure and consistency model: since the remote CPU never runs code for a one-sided operation, the application cannot use ordinary locking or software-based coordination to protect concurrently accessed memory — protection relies on NIC-level atomics (fetch-add, compare-swap) or careful epoch/versioning schemes layered on top, and a crashed remote process can leave stale registered memory regions that a live NIC will still happily serve reads against.

Operationally, RDMA raises the bar on fabric engineering: memory registration is not free (pinning large regions costs TLB/IOMMU pressure), connection setup (queue pair handshake) is comparatively expensive versus TCP, and the reliable-connected (RC) transport mode scales queue-pair memory linearly with peer count, which becomes a real constraint at data-center fan-out. Debugging is also harder because standard packet-capture and socket-level tooling is blind to the data path; observability requires NIC counters (retransmits, PFC pause frames, ECN marks) rather than application-level logs. For engineers designing latency-critical replication or storage tiers, RDMA is not a drop-in network upgrade — it is a co-design constraint that touches memory management, congestion control, and failure semantics simultaneously, and adopting it without redesigning the surrounding protocol for one-sided-access safety typically yields correctness bugs rather than the expected performance win.