Conflicting OOM Alerts and Healthy Memory Graphs on a Linux Server

Fictional Simulation

This is a fictional simulation created for diagnostic practice. No real systems, credentials, incidents or identifiable organisations are referenced; any resemblance to a real environment is coincidental.

Scenario

A retail company's Linux-based order-processing database (PostgreSQL 15, managed by systemd under cgroup v2) begins restarting overnight. Within the same ten-minute window, the on-call engineer receives two apparently conflicting alerts: the infrastructure dashboard reports "host memory utilisation: 42% — within normal range", while a separate systemd/journal alert reports that a process was terminated by the out-of-memory (OOM) killer. CPU metrics on both alerts remain flat. The engineer's first instinct is to treat the OOM alert as a false positive because the host memory graph looks healthy.

Evidence

• free -h shows roughly 6.2 GB free out of 16 GB total on the host.
• dmesg -T contains kernel oom_reaper messages naming a postgres worker process, with a cgroup path under system.slice/postgresql.service.
• journalctl -u postgresql.service shows repeated "Main process exited, code=killed, status=9/KILL" entries, each followed by an automatic restart.
• Restart timestamps align precisely with the start of the nightly batch import job, not with any host-wide memory pressure event.
• systemctl show postgresql.service -p MemoryMax -p MemoryCurrent reports a slice ceiling of 512 MB, with current usage climbing to that ceiling only during the batch import window.

Hypotheses and Actions

Four immediate, mutually exclusive actions are on the table. Weigh each against the evidence above before selecting one.

• Raise the postgresql.service slice's MemoryMax based on the measured working-set size during batch import, then monitor memory.current against the new ceiling.
• Restart the postgresql.service unit immediately to clear the current OOM condition and monitor for recurrence during the next batch window.
• Provision a larger host with additional total RAM and migrate the database service to it.
• Set a highly negative oom_score_adj for the postgres process so the kernel OOM killer skips it during future memory pressure.

Progressive Hints

• Hint 1: Check whether the dashboard's memory metric is sourced from the host (/proc/meminfo) or from a per-process or per-cgroup source.
• Hint 2: Compare the timing of restarts against the batch import schedule rather than against host-wide load graphs.
• Hint 3: A 512 MB ceiling that was adequate before a new nightly batch feature was added is a strong candidate for a stale configuration assumption.

Reveal

The database process and the host are both fundamentally healthy; the constraint is a systemd-managed cgroup v2 MemoryMax=512M ceiling on the postgresql.service slice, set months earlier for a smaller workload and never revisited when the nightly batch import feature was introduced. The dashboard's "host memory utilisation" metric reads node-level statistics from /proc/meminfo and has no visibility into per-slice cgroup ceilings, so it never reflects the slice-level exhaustion. This is a recognised class of conflicting-evidence incident in cgroup v2 environments, where host-level and slice-level memory accounting diverge and each alert is individually accurate but jointly misleading if read in isolation.

Learning Outcome

The key lesson is epistemic: a host-level memory graph is an observation about the node, not a fact about any single service's memory ceiling. When a process-level OOM alert conflicts with a healthy host-level graph on Linux, the correct inference is to check the relevant cgroup's own accounting (memory.max and memory.current, or the equivalent systemd unit properties) before dismissing the alert. Observable success for this exercise is being able to state, with reference to the evidence above, why the two alerts are not actually contradictory, and to select the single intervention that resolves the confirmed cgroup ceiling rather than a plausible-sounding but unproven alternative such as a hardware upgrade or disabling OOM protection outright.