Skip to main content
daily-triage/linux-log-service-fails-to-write-while-disk-space-appears-available.md
Daily Triage Briefing

A Linux Log Service Fails to Write While Disk Space Appears Available

Impact Summary

A fictional Daily Triage exercise in which a Linux log-forwarding service stops writing due to ENOSPC errors while df -h reports free disk space, revealing an inode-exhaustion fault caused by a logrotate misconfiguration.

SeveritySEV-3
StatusResolved simulation
PlatformOperating Systems
Incident TypeOperating Systems
Published7 Aug 2026
Technologies Involved
Linux

Simulation Label

Fictional simulation. This exercise is a bounded, invented training scenario for practising operational reasoning. No real organisation, incident, telemetry, credentials or production identifiers are involved or implied.

Operational Summary

Root Cause

A logrotate misconfiguration (one-minute rotation interval, 100,000 retained copies, no age-based pruning) exhausted available inodes on the /var/log filesystem, causing application log writes to fail with ENOSPC despite free byte-space.

Detection Method

Evidence-led guided investigation

Current Status

Resolved simulation

Affected Services

Linux

Incident Overview

At 09:12 on a Thursday, the on-call engineer for a fictional platform receives two paging alerts within four minutes. The first reports that the log-forwarding service on host app-log-03 has stopped writing new entries to /var/log/app/current.log.

The second, from a separate monitoring check, reports that the same host's disk usage sits at 61 percent, comfortably below the 90 percent alert threshold. The engineer opens a shell on app-log-03 to begin triage, aware that the two alerts appear to contradict one another: an application that cannot write to disk, on a disk that is not full.

Investigation Options

Review the available operational moves and select the best immediate action.

A

Run df -i to check inode usage on the affected filesystem

B

Delete old log files immediately to free space

C

Restart the logging service and monitor for recurrence

D

Extend or grow the disk partition to add capacity

Tags:Linux