debugging · advanced

Debugging: Live Deployment — Crash, CPU, Memory, Hung Process, Deploy Checklist

Live production debugging workflow for C++ backends. Crash: check journalctl → /var/crash/ for core dump → gdb ./app core → bt → frame → fix. High CPU: top -H -p PID (per-thread) → perf top -p PID (live hotspot) → perf record → flamegraph. Memory growth: watch /proc/PID/status VmRSS; /proc/PID/smaps Private_Dirty; fd count for fd leaks. Hung process: strace -p PID → look for futex_wait repeated = deadlock; gdb -p PID → thread apply all bt → identify which threads hold which locks. Network: ss -tnp | grep app; check connection count, TIME_WAIT accumulation. Runtime signals: SIGUSR1/SIGHUP for log-level/config reload without restart. Deploy: cmake Release → objcopy debug symbols out → strip binary → rolling deploy → health check → atomic symlink rollback if health check fails.

The code

# Live production debug checklist (C++ backend)

# 1. First: is it a crash? Check for core dumps / crash logs
ls /var/crash/ && journalctl -u myapp -n 100 --no-pager
gdb ./myapp /var/crash/core.1234  # post-mortem

# 2. High CPU? Find the process
top -H -p $(pgrep myapp)   # -H: show threads
perf top -p PID             # live hotspot view

# 3. Memory growth? Check RSS trend
cat /proc/PID/status | grep VmRSS
cat /proc/PID/smaps | grep Private_Dirty

# 4. Hung process? Stuck in what syscall?
strace -p PID -e trace=all -f 2>&1 | tail -20
# -f: follow threads  -e: filter syscalls
ls -la /proc/PID/fd | head -30   # check file descriptor leak
cat /proc/PID/limits              # resource limits

# 5. Network issues?
ss -tnp | grep myapp  # all TCP connections
netstat -snp | grep myapp

# 6. Deadlock / high lock contention
sudo perf record -g -p PID -- sleep 10
# look for futex_wait in perf report → deadlock or lock contention
gdb -p PID → thread apply all bt  # attach to live process

# 7. Logging without restart (runtime log level)
kill -SIGUSR1 PID  # if SIGUSR1 raises log level in your app
kill -SIGHUP  PID  # if SIGHUP triggers config reload

# 8. Build and deploy checklist
# - Build: cmake --build . --config Release -- -j$(nproc)
# - Strip: objcopy --only-keep-debug app app.debug && strip app
# - Deploy: rsync + restart with rolling update
# - Verify: curl -s http://localhost/health | jq .
# - Rollback: symlink swap (atomic) if health check fails

What this lesson walks through

1. 01Live debug checklist — first 60 seconds
2. 02Crash diagnosis — core dump + GDB post-mortem
3. 03High CPU — find the hot thread
4. 04Memory growth — /proc/PID/status + smaps
5. 05Hung process — strace + GDB live attach
6. 06Deploy checklist — build, strip, verify, rollback

A live production issue follows a standard triage order. First: is the process running or dead? Check core dumps and logs. Second: CPU or memory? Use top -H and /proc/PID/status. Third: stuck in a syscall? Use strace -p. Fourth: network/fd leak? Check /proc/PID/fd. This structured approach avoids random guessing under pressure.