debugging · advanced

Debugging: Service Isn't Receiving Data — Trace the Receive Path

The 'my service stopped receiving data' production scenario, solved by walking the receive path instead of guessing. Layer 0: confirm the process is up and bound to a reachable address (0.0.0.0, not 127.0.0.1) on the right port (ss -lunp). Layer 1 (wire/NIC): tcpdump proves whether packets arrive; ethtool -S / /proc/net/dev expose rx_dropped / rx_missed_errors (NIC ring overrun). Layer 2-3 (kernel/socket): netstat -su 'receive buffer errors' and ss -m skmem reveal a full socket buffer dropping datagrams — raise SO_RCVBUF / net.core.rmem_max or drain faster. Layer 4 (app): strace -p on recvfrom/recvmsg — a growing Recv-Q with no recvfrom returning means the app's reader thread is stuck (mutex, full downstream queue), not the network. Cross-cutting: a firewall DROP rule (iptables -L -v -n counters) or a routing/rp_filter surprise (ip route get). Fix the first layer that shows missing or dropped data.

The code

# 'My application stopped receiving data.' Walk DOWN the receive path and
# find the FIRST layer where the data is missing — don't guess.

#  wire → NIC → kernel rx queue → socket recv buffer → app read()

# 0. Is the process even up and listening on the right address?
pgrep -a myapp
ss -lunp 'sport = :9000'   # UDP feed   (ss -ltnp for TCP)
#  bound to 0.0.0.0:9000 (ok) or 127.0.0.1:9000 (unreachable!) ?

# 1. Are packets arriving on the wire at all?
sudo tcpdump -i eth0 -nn host 10.0.0.9 and port 9000 -c 20
ethtool -S eth0 | grep -i 'drop\|miss\|err\|fifo'
cat /proc/net/dev   # rx packets / rx drop per interface

# 2. Is the KERNEL/socket dropping after the NIC?
ss -uap | grep :9000        # Recv-Q on the socket
netstat -su | grep -i 'receive errors\|buffer errors\|dropped'
netstat -s | grep -i 'pruned\|collapsed\|overflow\|listen'

# 3. Is the APP actually reading?
sudo strace -p $(pgrep myapp) -f -e trace=recvfrom,recvmsg,read,epoll_wait
#  Recv-Q grows but no recvfrom() returns  ->  app is stuck, not the network

# 4. Is a firewall / route silently dropping it?
sudo iptables -L -v -n            # watch the pkts/bytes counters move
sudo nft list ruleset            # nftables equivalent
ip route get 10.0.0.9            # which interface/route is chosen

# 5. Socket buffer too small for a bursty feed?
ss -m -uap | grep -A1 :9000      # skmem: rcv buffer usage
sysctl net.core.rmem_max net.core.rmem_default

What this lesson walks through

1. 01Stop guessing — walk the receive path
2. 02Layer 0–1: is it on the wire at all?
3. 03Layer 2–3: kernel & socket buffer drops
4. 04Layer 4: is the app actually reading?

When a service 'stops receiving data', don't guess. Data flows through fixed layers: wire → NIC → kernel rx queue → socket recv buffer → your read(). Walk DOWN and find the FIRST layer where the data is already missing.