Linux / Editorial solution guide

Distinguish full storage from exhausted inodes

Diagnose a no-space write failure by identifying the affected filesystem, allocation type, quotas, and hidden consumers.

About 3 min readContent reviewed

Read-only knowledge base · No public comments

Recognize the symptoms

A fictional report job cannot create its next output and reports No space left on device. The machine may still show free gigabytes elsewhere, or df and du may disagree. A failure to create a tiny file can be caused by exhausted inode capacity rather than large content volume.

Understand the likely causes

  • The failing path may reside on a different mount than expected. Other possibilities include exhausted blocks, exhausted inodes, a user or project quota, reserved allocation, or deleted-but-open files still holding blocks. Snapshots, sparse files, compression, and filesystem-specific accounting affect comparisons. An error message narrows the investigation but does not identify a safe file to delete.

1. Identify the exact failing destination

Record the application, account, timestamp, and complete destination path. Inspect the nearest existing parent if the new file does not exist. Map that path to a mount before looking at any capacity percentage. A host can have free root storage while a separate /var, home, tmpfs, or container writable layer is full. Perform the checks from the workload’s mount context when isolation is involved.

bash Read both capacity dimensions for a safe lab path 
findmnt --target /tmp
df -h /tmp
df -i /tmp

Replace /tmp only with an authorized lab destination. The commands are read-only and do not allocate space or simulate exhaustion on the host.

2. Classify the exhausted resource

If available blocks are near zero, investigate byte consumers. If free inodes are near zero while bytes remain, investigate large populations of small files. Some filesystems allocate metadata dynamically or report inode values differently, so use their documentation. Check applicable quotas with the distribution’s supported read-only tools; a user may hit a quota while global space remains. Do not change reserved-space or quota policy merely to make the immediate error disappear.

3. Reconcile allocation with visible files

A bounded du -x view can help locate reachable usage while staying within one filesystem, but permission failures mean its total can be incomplete. Deleted files still open by a process can explain high df with lower du. Use approved process-file inspection to identify the owning application, then arrange its documented close, rotation, or controlled restart through the authorized operator. Do not kill arbitrary processes or erase active log files. Also review snapshots and mount-hidden data using filesystem-specific documentation.

bash Study a bounded directory inventory 
du -x -h --max-depth=1 ./synthetic-output

This GNU du illustration assumes a small prepared synthetic-output directory. It summarizes one level and avoids crossing filesystem boundaries; it does not delete any file.

4. Fix retention and the producing behavior

Agree which synthetic outputs are disposable, preserve required evidence, and apply the application’s documented retention or archival mechanism in the lab. For inode exhaustion, focus on file count and creation patterns rather than only large files. For open deleted data, release the application’s reference through its supported lifecycle. Ensure enough room remains for recovery work. The durable fix should address why data accumulated, including failed rotation, runaway retries, or unbounded temporary-file generation.

Verify the fix

Repeat df for bytes and inodes on the same affected mount and check the relevant quota. Run one bounded synthetic output operation as the original identity, verify its content and exit status, and confirm usage stabilizes across another normal cycle. Freed bytes on a different filesystem do not prove recovery.

Keep it from recurring

  • Monitor both byte and inode trends, bound temporary-file and log retention, and alert before the operational reserve disappears. Include quotas and container storage in the inventory. Test restore procedures before deleting valuable data, and document which application owns cleanup instead of relying on ad hoc filesystem deletion.

Official references