Why a method beats a list of tools

When a system misbehaves, it is tempting to run every tool you know and hope something looks wrong. That approach is slow, and it often finds something that looks odd but is not the cause. Experienced engineers follow a method instead.

Start from the problem statement

Before touching the server, clarify:

• What exactly is wrong? Slow responses, errors, timeouts, a crash?
• Since when? A sudden change points to a deployment, configuration change or traffic shift. A gradual one points to growth, leaks or data accumulation.
• For whom? All users, one endpoint, one region, one server?
• What changed? Releases, configuration, kernel updates, traffic, data volume.

Many incidents are solved at this stage, because the timeline points straight at a change.

The USE method

For every resource, such as CPUs, memory, disks, network interfaces and also software resources like connection pools and file descriptors, check three things:

• Utilisation: how busy the resource is, as a percentage of time or capacity.
• Saturation: how much work is waiting because the resource is busy, usually a queue.
• Errors: error counts reported by the resource.

The method, described by Brendan Gregg, is valuable because it is complete: it forces you to look at every resource rather than only the ones you suspect. Saturation is often the key signal. A disk at 70 percent utilisation with a long queue is a bigger problem than one at 95 percent with no queue.

Work from broad to narrow

1. System wide: is any resource saturated or reporting errors?
2. Per process: which process is using that resource?
3. Inside the process: which code path, system call or request type?

Each step narrows the search. Jumping straight to step three without the first two wastes time on details that may be irrelevant.

Write notes as you go

Keep a running log: time, command, observation, interpretation. It keeps you honest, helps colleagues join an incident without interrupting you, and becomes the basis of the post incident review.

The lab

The course lab is a virtual machine with scripts that create specific problems on demand. Each section starts by running one, and you practise the method before reading the explanation.