Linux Disk Full Recovery Without Losing Service Data

On Linux, disk full recovery starts with protecting services, not deleting the largest file. A filesystem at 100% can block database writes, interrupt email delivery, or prevent applications from saving sessions. The wrong cleanup can destroy recovery records while leaving the real storage problem untouched.

This guide focuses on an active capacity incident. The goal is to restore working space, preserve business data, and confirm service recovery before attempting deeper cleanup.

Stabilize Services Before Reclaiming Disk Space

First, identify the business impact. Check whether users can complete transactions, upload documents, receive messages, or save changes. An application may still load pages while failing every new write.

• Record the incident time, affected services, and recent changes.
• Keep your existing administrative session open.
• Confirm console access before any action that might interrupt remote access.
• Pause approved, nonessential jobs that generate data, such as exports or local backups.
• Avoid upgrades, software installation, and blanket service restarts.

Use application maintenance mode or approved traffic controls if continued writes threaten data integrity. Coordinate this with the service owner. Do not stop a database casually; shutdown and startup may require additional working space.

Capture essential evidence on your workstation or another healthy destination. Writing a large diagnostic archive onto the full filesystem makes the incident worse.

Locate the Full Filesystem, Not Just the Full Server

Linux can place application data, logs, and temporary files on separate filesystems. Freeing space under /home will not help a separate, full /var filesystem.

These inspection commands help identify the affected location. Commands here assume common Linux tools; availability and permissions vary.

df -hT
 df -i
 findmnt -T /var/lib

Run each command separately, without the leading indentation. Replace /var/lib with the application’s actual data path. Check filesystem type, mount location, available bytes, and inode use. Inodes hold file metadata; exhausting them can prevent new files despite free bytes.

If inode use reaches its limit, follow the separate inode exhaustion diagnostic guide. This incident requires a different cleanup strategy from removing a few large files.

Also check error messages for read-only mounts or quota limits. Those problems can resemble full-disk failures. If capacity looks normal, use the Linux write-failure checklist rather than deleting unrelated data.

Find a Small, Defensible Reclamation Target

Effective disk full recovery creates breathing room before pursuing a complete storage audit. Choose known, disposable data on the affected filesystem.

Start with a shallow scan of the relevant directory. For example, if /var is full:

sudo du -x -h --max-depth=1 /var

The -x option keeps this scan within one filesystem. Repeat the scan inside the largest relevant directory. Even read-only scans can cause substantial storage activity, so avoid repeatedly scanning the entire server.

For a bounded search for large regular files:

sudo find /var/log -xdev -type f -size +100M -ls

This example finds files above a screening size, not files that are safe to delete. Confirm each file’s owner, purpose, retention requirements, and active use.

A large file is not automatically the cause. Compare its size and modification time with the incident timeline. Reclaim enough space for controlled recovery, then investigate growth without rushing into broader deletion.

Handle Logs Without Erasing the Evidence

Preserve a small, relevant error sample before removing logs. Repeating failures often explain both the outage and rapid storage growth.

On systems using the systemd journal, inspect its storage use:

journalctl --disk-usage

Journal vacuum operations remove archived journal files, not active ones. Rotation can make active files eligible, but rotation itself may fail when space is critically low. Review retention needs and the journalctl manual before choosing vacuum options.

For plain-text logs, prefer the application’s documented rotation and reopen procedure. Forced rotation may need temporary space. Compression also needs working room, so it may fail during an emergency.

Do not truncate a file merely because its name ends in .log. Transaction logs and application journals may contain essential recovery data. Even truncating a confirmed text log destroys evidence and can interact badly with its writer.

After reclaiming space, watch whether the same log immediately grows again. Fixing the repeated error matters more than repeatedly removing its output.

Check Deleted Files That Processes Still Hold Open

Sometimes df reports far more usage than directory scans explain. One possible cause is a deleted file that a running process still holds open.

If lsof is already available, inspect unlinked open files:

sudo lsof +L1

Match the reported device and file to the affected filesystem. Inspect the owning process and apparent size. Multiple open descriptors may reference the same file, so do not blindly add every reported size.

Unlinking an active log does not necessarily release its storage. The filesystem generally reclaims that space when the final reference closes.

Use the service’s documented log-reopen action when appropriate. Otherwise, plan a controlled restart after confirming dependencies and recovery requirements. Do not kill an unfamiliar process simply to release space.

If deleted-open files do not explain the difference, stop guessing. Filesystem snapshots, reserved space, hidden files beneath mounts, and storage-layer accounting can require separate investigation.

Separate Package Caches From Business Data

Downloaded package archives can offer a lower-risk cleanup target. They differ from installed software and package-management records.

Check the operating system and cache location first. On Debian-family systems, apt-get clean removes downloaded package archives. Use it only after confirming those archives are disposable and no package operation is running.

Do not remove the package database, installed libraries, or old kernels manually. An emergency capacity incident is a poor time to change the software baseline.

Other possible targets include verified obsolete exports and expired application caches. Confirm that regeneration will not overload services or consume the recovered space immediately.

Moving a file within the same filesystem does not free capacity. Cross-filesystem moves copy data before removing the source. Verify the destination’s capacity and the copied data before deleting the original. Never relocate active application data this way without a supported procedure.

Protect Databases and Their Recovery Files

Database storage needs special handling. Never manually delete table files, write-ahead logs, redo logs, or replication records to make room.

These files may support crash recovery, replicas, or point-in-time restores. Their names and roles differ across database engines.

• Identify the database engine and its actual data filesystem.
• Confirm backup status and recovery requirements with the database owner.
• Check replica dependencies before any supported log purge.
• Avoid table rebuilds, optimization jobs, and local database dumps during critical capacity pressure.

Maintenance operations can require substantial extra space. A database restart may also trigger recovery work rather than immediately restoring service.

If no disposable files exist, adding capacity may be safer than deleting data. Verify the complete storage path first: cloud volume, partition, logical volume, and filesystem. Expansion steps depend on the layout and filesystem. A larger virtual disk alone does not guarantee more usable filesystem space.

Verify Recovery Before Ending the Incident

Successful disk full recovery means more than seeing usage fall below 100%. Confirm that the affected services can write correctly and remain stable.

1. Recheck free bytes and inode availability on the original filesystem.
2. Perform an approved application transaction, such as saving a test record.
3. Inspect service errors, database health, and queued work.
4. Resume paused jobs gradually while watching storage growth.
5. Record what you removed, changed, or restarted.

Choose a working-space margin based on normal growth and peak workload. A few free megabytes may disappear before a recovery task finishes. Set alerts that consider available capacity and growth rate, not percentage alone.

Finally, correct the source: failed retention, repeated logging, abandoned exports, backup placement, or genuine capacity demand. Keep the incident notes for future maintenance planning.

Professional remote assistance is appropriate when critical databases, uncertain file ownership, or storage expansion complicate recovery. Tech Rescue Ops LLC provides remote server support to help assess service impact and plan controlled reclamation without treating business data as disposable.