Linux OOM recovery starts with evidence, not a reboot. When a server exhausts usable memory, Linux may terminate processes to protect the system. A website might fail while SSH still works. A database might disappear, then return through an automatic restart. Those symptoms alone do not establish the cause.
This guide focuses on recovering from a specific out-of-memory incident. It separates host-wide pressure from service limits, then connects the event to workload. The goal is to restore service without destroying useful evidence or repeating the same failure.
Preserve Linux OOM Evidence Before Restarting Anything
Record the failure time, affected services, and recent changes. Note the server’s time zone so application logs and monitoring align. If access remains stable, collect a small snapshot before making changes.
date -Is
uptime
free -h
swapon --show
systemctl --failed
journalctl -k --since "30 minutes ago" --no-pagerThese examples assume a systemd-based Linux host. Log access may require elevated privileges. Review output before sharing it because logs can contain customer details or credentials.
Avoid installing diagnostic packages or launching heavy scans during severe pressure. Use existing monitoring and the tools already available. If SSH becomes unreliable, use your hosting console or another approved recovery path.
Do not reboot simply because memory looks low. A restart removes live process evidence and may erase logs without persistent storage. If business impact demands immediate action, capture the smallest useful snapshot first.
Read OOM-Killer Evidence in Context
OOM means out of memory. The kernel’s OOM killer selects processes for termination when certain memory allocation failures require intervention. Search the kernel journal, but keep the surrounding entries:
journalctl -k --since "2 hours ago" --no-pager | grep -Ei 'out of memory|oom|killed process'Use the matched timestamp to retrieve a wider window from the unfiltered journal. Look for the killed process name, process ID, memory figures, and references to a memory control group.
- Global OOM: The host faced memory pressure beyond what it could satisfy.
- Memory-cgroup OOM: A constrained group reached its limit, even if the host had spare memory.
- Userspace termination: A manager such as systemd-oomd may kill workloads before a kernel OOM event.
The victim does not necessarily identify the original cause. Another workload may have consumed memory first. Likewise, exit code 137 indicates a SIGKILL in common shell conventions, not proof of OOM.
Check service logs and the journalctl reference for time-window and boot selection options. Missing kernel messages do not rule out a memory-related termination.
Separate Host Memory Pressure From Service Limits
Linux OOM recovery takes different paths depending on the boundary that failed. Adding host memory will not automatically fix an unchanged container limit.
For a systemd service, replace example.service with the actual unit name:
systemctl status example.service --no-pager
systemctl show example.service -p Result -p ControlGroup -p MemoryCurrent -p MemoryHigh -p MemoryMax
systemctl cat example.serviceMemoryHigh applies reclaim pressure and throttling. MemoryMax sets a hard memory boundary. Parent groups can impose tighter effective limits than the service’s own settings.
A control group, or cgroup, accounts for and limits a set of processes. On cgroup v2 hosts, inspect memory.events, memory.current, and memory.max beneath the actual group path.
The oom_kill counter records OOM kills affecting the group. Counters alone do not establish when an event occurred; compare them with logs or earlier samples.
For containers, also inspect runtime or orchestration events and memory settings. Do not assume a container can see the host journal. Review the systemctl manual when interpreting unit state.
Identify Memory Consumers Without Blaming the Victim
Inspect surviving processes after preserving the event record:
ps -eo pid,ppid,user,comm,rss,vsz --sort=-rss | head -n 21
free -hRSS means resident set size: memory currently resident in RAM for a process. The ps figures use KiB here. VSZ represents virtual address space, not actual RAM consumption.
Shared memory complicates totals, so do not simply add every process’s RSS. Group related workers by service or container. Ten moderate workers can exceed the footprint of one large process.
In free, focus on available rather than only free. Linux uses otherwise idle RAM for caches and can reclaim some of it. Low free memory alone does not prove exhaustion.
After an OOM kill, current usage may look healthy because the largest consumer no longer exists. Historical graphs and the kernel’s event snapshot therefore matter more than a single post-incident reading.
If repeated samples show unexplained growth under comparable load, continue with a separate memory leak investigation. One OOM incident does not establish a leak.
Check Swap and Reclaim Pressure Before Changing Capacity
Swap provides backing storage for some memory pages. It can absorb certain demand spikes, but it cannot replace enough RAM for an active workload.
swapon --show
vmstat 1 5
cat /proc/pressure/memoryIn vmstat, examine swap-in and swap-out activity alongside CPU wait and workload response. The first report commonly summarizes activity since boot; later reports cover each sampling interval.
The pressure file, where available, reports time lost to memory stalls. Sustained pressure helps distinguish a server struggling to reclaim memory from one merely holding a large cache.
Check container swap restrictions separately from host swap availability. A workload may lack swap access even when the host has configured it.
Do not run swapoff during exhaustion without a reviewed plan. Moving swapped pages into RAM can worsen pressure. Likewise, avoid adding swap blindly: storage capacity, performance, encryption, and hosting rules all matter.
Correlate the Kill With Workload and Recent Changes
Build a short timeline around the first sign of pressure, not just the final kill. Compare the event with:
- Backup, compression, reporting, or indexing jobs.
- Traffic bursts and growing application queues.
- Deployments that changed worker counts or cache sizes.
- Database maintenance or unusually large queries.
- Multiple scheduled tasks starting together.
- Automatic restarts that repeatedly reload large datasets.
Compare concurrency as well as request volume. A service may handle normal traffic until too many memory-heavy requests run together. Queued work can then amplify the problem.
If a service keeps returning and dying, build a service restart timeline. Separate the original memory failure from later restart attempts.
Record a testable explanation. For example, investigate whether overlapping jobs exceeded a group’s hard limit. Do not label the event a leak unless the trend supports that conclusion.
Restore Service With One Controlled Change
Effective Linux OOM recovery reduces demand or corrects a verified constraint before restarting the affected service. Choose the smallest change that addresses the evidence.
- Pause a confirmed noncritical batch job through its normal controls.
- Reduce application concurrency using supported settings.
- Separate overlapping scheduled tasks.
- Raise a mistaken service limit only after checking host headroom.
- Add capacity when legitimate peak demand exceeds the current budget.
Avoid disabling the OOM killer, dropping caches as a routine fix, or removing all memory limits. Those actions can shift the outage to other services.
Before restarting, consider active transactions, recovery behavior, and dependent services. Obtain approval where downtime requires it. Watch memory, pressure, error rates, and queue depth throughout recovery.
Confirm the affected business function, not just an active service status. Then monitor through the next comparable workload peak. Document the event, change, rollback path, and remaining uncertainty.
Professional remote assistance may help when access remains unstable, critical databases need recovery, or multiple limits obscure the cause. Tech Rescue Ops LLC can assist with Linux server diagnosis and recovery, using collected evidence to plan the next step.