The first five minutes on a sick server

Here is a triage routine that gives a broad picture of a Linux system in about five minutes. Run the commands in this order and note anything unusual before going deeper.

uptime
dmesg -T | tail -30
journalctl -p warning --since "30 min ago" --no-pager | tail -40
vmstat 1 5
mpstat -P ALL 1 3
pidstat 1 3
iostat -xz 1 3
free -m
ss -s
sar -n DEV,TCP,ETCP 1 3

mpstat, pidstat, iostat and sar come from the sysstat package. Install it on every server in advance; you do not want to install packages during an incident.

What each command tells you

• uptime: load averages over 1, 5 and 15 minutes. Rising or falling? On Linux, load counts runnable tasks and tasks in uninterruptible sleep, often waiting for disk, so high load does not always mean CPU.
• dmesg: kernel messages, such as OOM kills, disk errors, network link changes and hung task warnings. Errors here change everything.
• journalctl -p warning: recent warnings and errors from services.
• vmstat 1: r is the run queue (tasks waiting for CPU), b is blocked tasks, si/so show swapping, and CPU columns us, sy, id, wa, st split time between user, system, idle, I/O wait and stolen by the hypervisor.
• mpstat -P ALL: per CPU usage. One CPU at 100 percent and others idle suggests a single threaded bottleneck.
• pidstat: which processes use the CPU.
• iostat -xz: per device utilisation, queue size and average wait times (r_await, w_await).
• free -m: memory. Focus on available, not free; the page cache is supposed to use spare memory.
• ss -s: socket summary. Huge numbers of connections in particular states are suspicious.
• sar -n: network throughput, TCP connection rates and retransmits.

Pressure stall information

Modern kernels expose how much time tasks spend stalled on CPU, memory and I/O:

cat /proc/pressure/cpu /proc/pressure/memory /proc/pressure/io

Non zero some or full averages show real contention, a direct measure of saturation that is often clearer than utilisation figures.

After the routine

You should now be able to say which resource, if any, is saturated, and whether the kernel or services are reporting errors. Pick the most suspicious finding and go deeper with the dedicated sections of this course. If nothing stands out, the problem may be outside this host: a dependency, the network or the client.