Stopping Swap Thrashing With OOM Killer Tuning

A desktop or build server locking up under memory pressure while free -h still reports available swap is one of the most misdiagnosed failures in Linux

#The Bottleneck: Reclaim Livelock, Not Allocation Failure

The kernel’s memory reclaim path prioritises staying alive over staying responsive. When anonymous memory pressure rises, the kernel does not immediately invoke the OOM killer — it first tries to reclaim page cache, then it starts swapping anonymous pages out under vm.swappiness control, and only escalates to the killer when __alloc_pages_slowpath() exhausts every reclaim attempt and still cannot satisfy an allocation. On a system with several gigabytes of swap, that exhaustion point can be minutes away, and during those minutes every process touching a swapped-out page blocks on I/O. This is reclaim livelock: the system is technically making progress, but at a rate too slow for anything interactive to function. Standard OOM killer tuning approaches that only adjust oom_score_adj values do not solve this, because the killer simply is not being invoked yet. The actual fix requires detecting pressure earlier, via PSI, and killing before the kernel’s own thresholds are breached.

#Architectural Breakdown: PSI, Badness Scores, and the Kill Decision Chain

Effective OOM killer tuning rests on three separate subsystems that must be understood as layers, not a single control knob.

#

1. The Kernel OOM Killer’s Badness Algorithm

When invoked, the kernel computes a badness score per process based on RSS, swap usage, and page table size, then applies oom_score_adj as a linear bias between -1000 and +1000. A value of -1000 makes a process immune; +1000 guarantees it is the first candidate. This is the only tuning surface most administrators touch, and it is necessary but not sufficient.

#

2. Pressure Stall Information (PSI)

Since kernel 4.20, /proc/pressure/memory exposes some and full percentages representing the proportion of time tasks spend stalled waiting on memory reclaim over rolling 10s, 60s, and 300s windows. This is the earliest reliable signal of thrashing, and it is what modern OOM killer tuning strategies key off, because it detects the stall before allocation failure, not after.

#

3. Userspace OOM Daemons

systemd-oomd and Facebook’s oomd poll PSI and cgroup memory pressure and kill processes proactively when thresholds are breached, using the same oom_score_adj biasing but acting far earlier than vmscan would. This is the layer that actually closes the livelock gap.

1watch -n1 cat /proc/pressure/memory
2# some avg10=12.40 avg60=8.10 avg300=2.05 total=184213820
3# full avg10=4.20 avg60=1.90 avg300=0.44 total=52011932

An avg10 value on the full line sustained above roughly 5-10% under normal desktop load is a strong indicator that reclaim livelock has already begun, well before the kernel’s own OOM killer will act.

#OOM Killer Tuning via oom_score_adj and PSI Thresholds

Correct OOM killer tuning treats oom_score_adj as a per-service policy, not a one-off manual override. Systemd

1[Service]
2# Protect the display compositor from early kills
3OOMScoreAdjust=-500
4
5[Unit]
6Description=Wayland compositor session

For batch or build workloads that are expendable — CI runners, transient compile jobs, cache warmers — the inverse policy applies:

1[Service]
2OOMScoreAdjust=800
3ManagedOOMMemoryPressure=kill
4ManagedOOMMemoryPressureLimit=50%

The ManagedOOMMemoryPressure directive is what hands control to systemd-oomd. Without it, OOMScoreAdjust only biases the kernel’s own late-stage killer, which brings you back to the livelock problem. Combining both is the core of a correctly layered OOM killer tuning policy: bias the kernel killer for the worst case, and let the PSI-aware daemon act first.

#

Configuring the Daemon Itself

1sudo systemctl edit --full systemd-oomd.service
2# Confirm PSI monitoring is enabled and check active swap/memory defaults
3systemctl show systemd-oomd -p DefaultMemoryPressureDurationSec
4systemctl status systemd-oomd

On distributions without systemd-oomd enabled by default (many desktop images ship it disabled), earlyoom is a lighter-weight alternative that polls /proc/meminfo directly rather than cgroup PSI files, useful on cgroups v1 hosts where PSI accounting is unavailable per-slice.

#cgroups v2 memory.max as a Hard Boundary

PSI-driven killing is probabilistic and reactive by nature. For workloads where predictability matters more than graceful degradation — multi-tenant build servers, containerised student lab environments — pairing OOM killer tuning with a hard cgroup v2

1mkdir -p /sys/fs/cgroup/build-jobs
2echo "4G" > /sys/fs/cgroup/build-jobs/memory.max
3echo "3.5G" > /sys/fs/cgroup/build-jobs/memory.high
4echo $$ > /sys/fs/cgroup/build-jobs/cgroup.procs

memory.high triggers aggressive reclaim and throttling within the slice before memory.max forces a scoped OOM kill confined to that cgroup only — the rest of the host is untouched. This is the mechanism behind Kubernetes memory limits and is documented in detail in the kernel’s own cgroup v2 admin guide. This approach is directly relevant to the broader architectural patterns used in containerised platform design, where per-tenant memory isolation is non-negotiable.

#Escalation Flow

#Failure Modes and Edge Cases

OOM killer tuning introduces its own failure surface if applied carelessly. The most common production incident is false-positive termination of the wrong process — a database connection pooler with a negative oom_score_adj survives while its parent supervisor, left at the default of 0, gets killed instead, leaving orphaned children and broken health checks. Always audit the entire process tree’s scores, not just the leaf process.

A second edge case involves init systems inside containers. PID 1 inside a container namespace often has oom_score_adj hard-pinned by the container runtime, and attempts to override it via systemd-oomd policies on the host will silently fail because the daemon cannot see into a nested PID namespace without cgroup delegation configured correctly.

A third, subtler failure appears on hosts using zram-backed swap. Because zram compresses pages in memory rather than writing to a block device, PSI’s full stall percentage under-reports actual pressure — the CPU cost of compression does not register as an I/O stall. Systems relying purely on PSI thresholds for OOM killer tuning on zram hosts should lower their trigger thresholds by roughly 30-40% relative to disk-swap baselines, or add a supplementary CPU pressure check, since zram trades I/O stall for CPU stall.

#Mechanism Comparison

#Scaling and Security Trade-offs

• Aggressive PSI thresholds reduce thrash duration but increase the rate of false-positive kills on bursty, legitimate workloads such as compilers or JVM garbage collection pauses.
• Hard memory.max ceilings give predictable multi-tenant isolation but push failure onto the application layer, requiring retry and backoff logic that not every service implements correctly.
• Negative oom_score_adj on privileged daemons improves system stability but, if applied too broadly, can starve the kernel of any killable candidate, forcing a full system freeze rather than a targeted kill.
• zram/zswap combined with OOM killer tuning reduces disk I/O and extends effective memory headroom, but shifts pressure detection onto CPU metrics that most default dashboards do not surface.
• Per-container cgroup delegation for PSI visibility improves kill accuracy inside orchestrated environments but expands the attack surface for cgroup escape if delegation is misconfigured with excessive controller permissions.

None of these mechanisms are mutually exclusive, and treating OOM killer tuning as a single configuration change rather than a layered policy is where most deployments go wrong. A resilient host pairs PSI-aware daemon intervention for early, low-impact kills with hard cgroup ceilings for tenants that must never affect their neighbours, and reserves the kernel’s own badness-score killer purely as the mechanism of absolute last resort. Getting the layering right is less about memorising flag names and more about understanding which subsystem sees the pressure signal first — and ensuring that subsystem is the one authorised to act.