Linux reboot evidence can disappear while a server appears healthy again. Logs rotate, management consoles expire events, and another restart changes which boot counts as “previous.” Start by preserving records, not by guessing which component failed.
This guide focuses on building an evidence packet for an administrator, hosting provider, or hardware vendor. It covers the collection sequence rather than a broad repair checklist. The goal is a trustworthy timeline that separates confirmed events from plausible explanations.
Stabilize Service Without Erasing Reboot Evidence
First, confirm whether essential services work and whether the server continues to restart. Record the business impact, discovery time, hostname, and affected applications. Note every recovery action, including who performed it and when.
Avoid another reboot, package upgrade, log cleanup, or diagnostic stress test unless recovery requires it. Those actions can remove useful context or add new faults. If safety or service restoration takes priority, document that decision.
- Use an authorized account and an approved administrative connection.
- Check available storage before creating large exports.
- Choose a restricted evidence directory or approved external destination.
- Record the server identifier and whether it is physical or virtual.
- Request provider and hardware logs early, before their retention windows expire.
Commands below assume a systemd-based Linux server. Permissions and available tools vary. Read operations still create some operational activity, including authentication records and disk reads. Suspected compromise or legal evidence requirements call for a specialist collection process.
Record the Boot Identity and Clock Context
Boot identifiers distinguish one running instance of the kernel from another. Capture them before a second restart makes relative references confusing.
date --iso-8601=seconds
timedatectl
uptime -s
cat /proc/sys/kernel/random/boot_id
uname -a
sudo journalctl --list-boots --no-pagerSave the output with the collection time and operator name. The boot list shows available journal boots, their identifiers, and recorded time ranges. It does not guarantee complete coverage.
Record the server timezone, clock synchronization state, and any known clock changes. A synchronized clock now does not prove it stayed accurate before the reboot.
Ask providers for timestamps with explicit timezone offsets. Keep original timestamps alongside any UTC conversion. If clocks disagree, record the uncertainty rather than forcing events into an exact sequence.
For each source, note what its timestamp means. A monitoring alert may mark the detection time, not the moment the server stopped responding.
Export the Previous-Boot Journal Before Filtering
The systemd journal stores service and kernel messages. Start with the complete relevant boot, then create smaller working extracts. Filtering too early can hide the event that explains a later warning.
sudo journalctl -b -1 --no-pager -o short-iso-precise
sudo journalctl -k -b -1 --no-pager -o short-iso-preciseThese commands display the previous recorded boot and its kernel messages. Redirect output only to a verified destination with adequate space. Preserve the command, collection time, and any errors with each export.
For repeatable collection, replace -1 with the relevant boot identifier from the boot list. Otherwise, another reboot changes the meaning of “previous.”
Where appropriate, retain a structured export using -o export as well as readable text. It preserves journal fields that plain text may omit. Follow your organization’s process for copying original journal files.
Consult the journalctl manual for supported options. If no prior boot appears, document the gap. Volatile storage or retention limits may explain it. Enabling persistent logging now cannot recover missing history.
Read the Final Events Without Overstating Them
Inspect the end of the preserved journal. Look for an orderly shutdown sequence, service stops, filesystem unmounts, and a reboot request. Then inspect the minutes before that sequence.
An orderly shutdown suggests that software initiated a controlled transition. It does not identify the person, scheduler, or management system that requested it.
A sudden journal cutoff supports an abrupt interruption, but it does not prove power loss. A kernel crash, host reset, storage failure, or missing final writes can leave similar gaps.
Collect last -x output if available. Treat it as supporting evidence from login accounting records, not a definitive cause report. Rotation and incomplete records can limit its value.
Keep observations separate from conclusions:
- Observation: The journal ends during normal application activity.
- Corroboration needed: Check host, power, and console records for the same interval.
- Unresolved: The guest journal alone cannot establish the reset source.
Preserve Kernel, OOM, and Watchdog Clues
Search working copies for kernel panic, lockup, hung-task, machine-check, storage-error, and out-of-memory messages. Keep surrounding lines and timestamps. A single matching line rarely explains the entire incident.
An out-of-memory event, or OOM event, means the system encountered a memory allocation crisis. The kernel may kill a process without rebooting the server. Correlate any kill with later panic, watchdog, or automation records.
Record relevant panic-on-OOM and watchdog settings, but label them as current settings. They may differ from those active before the reboot. For response guidance, see recovering from OOM incidents without blind restarts.
A watchdog monitors responsiveness and can trigger recovery when checks fail. Distinguish a service watchdog from a system or hardware watchdog. A service restart does not establish a whole-server reset.
Ask whether the system configured crash dumps before the incident. Preserve existing dump files and persistent crash records, such as pstore data, through an approved process. Do not trigger a test crash on production to validate collection during the investigation.
Collect Hardware, Hypervisor, and Power Records
A virtual machine cannot fully explain events outside its boundary. Request hypervisor records for resets, host restarts, migration, storage interruption, and operator actions. A hypervisor is the system that runs virtual machines.
Record the virtual machine’s unique identifier, not just its display name. Ask whether neighboring machines experienced the same interruption. Shared timing can narrow the investigation without proving a common cause.
For physical servers, export the management controller’s event log before clearing anything. Look for power transitions, temperature alarms, memory faults, and watchdog resets. Preserve raw event details and controller clock settings.
Collect uninterruptible power supply records and relevant facility maintenance history. A power alarm at a nearby time needs correlation with the server’s actual power path.
Use the broader guide to separating kernel, hardware, and power causes after securing these records. Evidence preservation should come before component changes or repeated reboot tests.
Reconcile Maintenance and Administrative Activity
Review package-manager history, automatic update logs, scheduled tasks, systemd timers, and configuration-management runs. Include backup platforms and hosting control panels that can request restarts.
Capture relevant authentication and privilege-use records where available. Shell history alone cannot establish who rebooted a server. It may omit commands, lack timestamps, or reflect a shared account.
Compare maintenance tickets with actual execution records. An approved window does not prove that a particular job ran. Likewise, a new kernel after startup does not prove an automatic update initiated the reboot.
Keep the distinction between a service restart and a machine reboot clear. If only an application restarted, use a service restart timeline instead of treating the incident as a host reset.
Package the Evidence for a Clear Handoff
A useful Linux reboot evidence packet contains more than a folder of logs. Add a short index that explains each file and its source.
- Incident window, timezone, affected services, and business impact.
- Host identity, boot identifiers, and platform details.
- Original exports, collection commands, timestamps, and access errors.
- A timeline linking each observation to its source.
- Recovery actions, known gaps, and questions for the provider.
Keep originals restricted and create separate redacted copies for sharing. Logs can contain usernames, addresses, tokens, and customer data. File hashes help detect later changes, but they do not prove that the source records were complete.
If the cause remains unclear, request professional remote assistance before making disruptive changes. Tech Rescue Ops LLC can help correlate guest, host, and maintenance records while keeping evidence and recovery decisions distinct.