debugging · advanced

Debugging: perf + Flame Graph — CPU Profiling, Hotspots, and Bound Type Diagnosis

perf is the Linux kernel performance profiler using hardware performance counters (PMU). perf stat: runs app and reports IPC, cycles, instructions, cache-misses, branch-misses — diagnose bound type (IPC<1.5=memory, branch-miss>2%=branch, IPC>3=compute). perf record -g -F 99: sample at 99 Hz; -g captures call graphs; writes perf.data. perf report --stdio: functions sorted by self%; top entry = hotspot. perf script | stackcollapse-perf.pl | flamegraph.pl > flame.svg: visual call tree; width proportional to time; click to zoom in browser. perf annotate -s fn: per-assembly-instruction hit counts; pointer chase = cache miss chain. perf record -p PID: attach to running process; pair with DEBUGINFOD or objcopy for symbols. eBPF/bcc tools: same capability without kernel patches, works on stripped binaries.

The code

# perf — Linux kernel sampling profiler

# 1. Stat: high-level hardware counters
perf stat ./app
# Shows: instructions, cycles, IPC, cache-misses, branch-misses

# 2. Record: sample call stacks at 99 Hz
perf record -g -F 99 ./app
# -g: capture call graphs (stack traces)  -F: samples/sec

# 3. Report: text summary sorted by self%
perf report --stdio | head -40

# 4. Flame graph (Brendan Gregg's toolkit)
perf script | stackcollapse-perf.pl | flamegraph.pl > flame.svg
# Open flame.svg in browser — click to zoom into hot paths

# 5. Annotate: assembly-level breakdown
perf annotate -s MyClass::process
# Shows which instruction costs the most cycles

# Common perf stat output fields to watch:
# IPC < 1.5: memory bound (increase cache efficiency)
# branch-miss > 1%: high branch misprediction
# cache-misses > 1%: cache inefficiency
# LLC-load-misses / loads: last-level cache miss rate

# Differential profiling: A/B compare two binaries
perf diff perf.data.before perf.data.after

# eBPF / bcc tools (modern alternative for production)
profile-bpfcc -f 9999 ./app  # same as perf record but no kernel patching

What this lesson walks through

1. 01perf stat — hardware counters in 2 seconds
2. 02perf record + report — find the hot function
3. 03Flame graph — visualize the full call tree
4. 04perf annotate — assembly-level hotspot
5. 05Reading perf stat — diagnose bound type
6. 06Production profiling with perf + no symbols

perf stat runs your program and collects hardware performance counters from the CPU: instructions executed, cycles used, IPC (instructions per cycle), cache misses, branch mispredictions. This is always the first profiling step — it tells you if you're CPU-bound, memory-bound, or branch-prediction-bound, guiding which optimization path to take.