SynologyRS
Performance

Profiling a slow box: perf, iostat and bpftrace in the real world

📅 2025-02-01✎ 9 min read

How I found a 0.1% of traffic path that was eating all the CPU.

“The server is slow” is the least useful bug report, and the most common. Before touching configuration, I find where the time actually goes. Usually it is one path, not the whole system.

Start with what is obvious

top -bn1 | head -15
iostat -dx 1 5

top still finds the loud process. The interesting cases are quiet ones — a request path that is rarely used but expensive, hiding behind mostly-idle averages.

perf for the CPU

perf record -g --call-graph dwarf -a -C 0 -- sleep 30
perf report --stdio | head -40

Run a short sampled profile under representative load and look at the flame graph. You are hunting for a single function eating a surprising slice, usually in code you would never have guessed.

iostat and the disk story

iostat -mxY 1
# watch for %util 100 with high await
iostat -cxz 1

If CPU looks idle but things feel slow, the disk is the suspect. iostat -mx 1 shows saturation CPU that averages hide. A device sitting at 100% util with high await is your bottleneck, full stop.

Do not optimize the thing that is loud. Optimize the thing the averages are hiding.