Linux Disk Full but df Shows Free Space: Inode Exhaustion Fix

#Context: why df can look healthy while writes fail

The error No space left on device (ENOSPC) does not always mean that data blocks have run out. A filesystem has at least two finite resources that a write or file creation can exhaust: data blocks and inodes. A plain df reports block usage only. A volume can therefore show plenty of free capacity while every attempt to create a file fails.

Two causes account for most cases of this symptom, and they need different fixes:

• Inode exhaustion. The filesystem has no free inodes, typically because of very large numbers of small files such as session files, mail queue entries, cache fragments or unrotated artefacts.
• Deleted-but-open files. A file was unlinked, but a process still holds an open descriptor. The blocks stay allocated until the last descriptor closes. In this case df reports more usage than du can account for.

Other contributors exist, including reserved blocks, project or user quotas, and a different mount than the one you are inspecting. This article treats them as exclusions to rule out, not as the main path.

Assumptions: a LinuxThe KBY LexiconLinuxLinux is the open-source kernel underlying most server, container and cloud infrastructure; distinct from the distributions built around it. Read Full Context host with root or equivalent privilege, a local block-backed filesystem, and a validation host or maintenance window for any change. Filesystem-specific inode behaviour varies by type and version, so confirm details against your distribution documentation for your kernel and filesystem tools.

#Architecture: what is being exhausted

An inode is the metadata record for a file: ownership, mode, timestamps and pointers to data. A directory entry maps a name to an inode. Two properties matter for diagnosis:

1. On several traditional filesystems, such as ext4 in its default configuration, the number of inodes is fixed when the filesystem is created. Free blocks cannot be converted into inodes later without recreating or resizing the filesystem. Other filesystems allocate inodes dynamically, but may still enforce an upper bound. Check the behaviour for your filesystem type.
2. A file’s blocks are released only when its link count is zero and no process holds it open. Removing the name is not enough.

This gives a clear separation of observations. If df -i shows 100% inode use while block use is low, the inode table is full. If df shows high block use that du -x cannot explain, look for unlinked files that are still open. If neither is true, consider quotas, reserved space and mount boundaries.

#Implementation: a bounded diagnostic sequence

Work from read-only evidence towards any change. Identify the filesystem that holds the failing path first, because a path under /var may sit on its own mount.

#

Step 1: identify the mount and compare blocks with inodes

Run df -hT and df -i against the failing path. Record the mount point, the filesystem type, block use and inode use. Inode use near 100% with free blocks points to the first cause. Mismatched block use points to the second.

#

Step 2: locate directories with the most entries

Counting entries across a large filesystem is itself I/O intensive. Restrict it to one filesystem with -xdev, start with the likely subtree, and run it at a quiet time. Counting parent directories of files shows where entries concentrate. Repeat on the heaviest subtree to drill down until you reach a single directory, which is often an application cache, spool or temporary directory.

Before removing anything, establish what writes there, whether it is safe to purge, and whether an application-level cleanup tool exists. Prefer the application’s own retention mechanism over manual deletion. Deleting by hand can break references the application expects.

#

Step 3: look for deleted-but-open files

The command lsof +L1 lists open files whose link count is below one, which means they have been unlinked. Restricting it to the mount point keeps output relevant. Note the process name, PID, descriptor and size. A log file removed by hand while its daemon still writes to it is the classic example. Rotating by deletion rather than by a mechanism that signals the process to reopen its log is a common root cause.

#

Step 4: choose the least disruptive release

Releasing a deleted-but-open file normally requires the owning process to close the descriptor, usually by a controlled restart or a reload that reopens files. Where a service is managed by systemdThe KBY Lexiconsystemdsystemd is the Linux init system and service manager that starts, supervises and coordinates units of work such as services, sockets and timers. Read Full Context , use the service manager so that dependent units and restart policies behave predictably. The systemd manual pages document unit behaviour and service management, but check the exact semantics for your installed version. Because this is a state change on a production service, agree the window, the owner and the fallback first.

Related background on diagnosing storage health before blaming capacity is in Reading SMART Attribute Thresholds Before Failure. If a restart leads to memory pressure on the same host, see Stopping Swap Thrashing With OOM Killer Tuning.

#Validation: how to confirm the diagnosis and the fix

Success is observable. Before any change, capture a baseline: inode use, block use, the count of entries in the heaviest directory, and the list of unlinked open files. After the change, repeat the same measurements and compare.

• For inode exhaustion, df -i shows a lower IUse% on the same mount, and creating a small test file as the affected application user succeeds.
• For deleted-but-open files, lsof +L1 no longer lists the holder, df block use falls, and the difference between df and du -x narrows.
• The application error rate for ENOSPC returns to its normal baseline in its own logs.

Keep the test file trivial and remove only that file afterwards. Do not generalise from a single successful write, because inode headroom may still be thin. Record the headroom and set an alert threshold, as described in the final section.

#Failure modes

• Misreading the mount. Checking / when the application writes to a separate volume gives a false negative. Resolve the actual mount first.
• Cleanup that regrows. If a producer is creating files faster than any purge, space returns briefly and exhaustion recurs. Find and fix the producer.
• Restart with no effect. If a child process or another service inherited the descriptor, the holder survives. Re-run lsof +L1 after the restart.
• Expensive scans. Counting millions of entries can add I/O load to an already stressed host. Scope by subtree and schedule it carefully.
• Quota or reserved-block confusion. Normal inode and block figures with failing writes indicate a different limit. Check quotas and reserved-block settings for your filesystem.
• Permanent shortage. If the workload legitimately needs more inodes than the filesystem provides, cleanup is only a stopgap. Fixing it needs a planned migration to a filesystem created with more inodes, or a filesystem type with dynamic allocation.

#Security

Apply least privilege. Read-only diagnostics such as df need no special rights, while lsof and a full directory scan generally need elevated privilege to see other users’ processes and files. Use a time-limited privileged session and avoid pasting output containing file names, user names or paths into public tickets without review, because they can disclose business or personal data.

Treat unexpected mass file creation as a possible security signal, not only a hygiene issue. A burst of tiny files can come from a compromised service, a fork-bomb-like script or an abused upload endpoint. Preserve a sample of directory listings and timestamps before purging, and involve the security team if the pattern is unexplained. Do not run bulk-removal commands copied from forums against paths you have not verified. Deleting from the wrong directory is not reversible without a backupThe KBY LexiconBackupA causally disconnected, point-in-time copy of system state, tagged with a consistency marker, that lets you recover from logical corruption or data loss independent of the source system's health. Read Full Context . This article therefore provides no deletion command.

#Recovery

Recovery depends on the cause, and every change needs a boundary and a way back.

• Inode exhaustion. Remove or archive only data that the application owner has confirmed is disposable. Take a backup or snapshot of the directory first where the data has any value. If a purge removes something needed, restore from that copy. Stop the purge if the application reports missing-file errors.
• Deleted-but-open files. Restart or reload the holding service through its service manager, in an agreed window. Stop and escalate if the service fails to return to a healthy state. Roll back by starting the previous unit configuration and confirm health through the application’s own checks. Data in the unlinked file is lost on release, so confirm that it is only log or scratch content.
• Persistent shortage. Escalate to a human owner for a capacity decision. Recreating or migrating a filesystem is a planned project with backups and a tested restore, not an incident-time action.

If the evidence is ambiguous, or the host holds irreplaceable data without a verified backup, stop making changes and escalate to the storage or platform owner.

#Operational checks after an ENOSPC incident on Linux

Close the incident with measurable checks rather than an assumption that the problem has gone.

1. Record post-change df and df -i values for the affected mount, and compare them with the baseline.
2. Confirm that lsof +L1 on the mount shows no large unexpected holders.
3. Add monitoring on both block and inode use, with a warning level that leaves time to act.
4. Fix log rotation so that it signals the process or uses a copy-and-truncate approach appropriate to the application, rather than deleting a live file.
5. Assign an owner for the producer of small files and set retention for its directory.

The next safe decision is simple. If the producer is understood and capped, close the incident. If not, keep monitoring and schedule a capacity review. Version-specific filesystem and tool behaviour should be confirmed in your distribution documentation and the manual pages for the tools used. The systemd manual pages listed in the sources cover service management only.