First look at a misbehaving process with strace and tcpdump

Goal

Answer, within minutes and without changing code, two questions about a process that hangs or fails: what is it waiting for, and is the network peer responding?

Prerequisites

Root, or CAP_SYS_PTRACE for strace and CAP_NET_RAW for tcpdump. The kernel's Yama module (kernel.yama.ptrace_scope, documented in the cited page) restricts attaching to unrelated processes when set to 1 or higher, so an unprivileged attach may fail even for your own process. A rough idea of the process ID and the port involved.

Steps

1. Get the PID and its current state: ps -o pid,stat,wchan:32,cmd -p PID; state D or a wchan naming a network function already narrows the search.
2. Attach briefly: timeout 20 strace -f -p PID -T -e trace=%network,%file -o /tmp/trace.txt (-f / --follow-forks follows threads and children, -T prints the time spent in each call, %network and %file select call groups). A process blocked in read, recvfrom or futex for the whole window is waiting, not working.
3. Read the file for the last completed calls before the block: which descriptor, which address (connect shows the peer), which path. Map descriptors with ls -l /proc/PID/fd.
4. Capture the corresponding traffic with a narrow filter and a bound: tcpdump -ni any -c 200 -w /tmp/cap.pcap 'host 203.0.113.5 and port 5432'. -n avoids DNS lookups, -c stops after 200 packets, -w writes raw packets (the default snaplen of 262144 bytes keeps whole packets).
5. Inspect with tcpdump -nr /tmp/cap.pcap or a graphical analyser. Look for repeated SYNs without SYN-ACK (nothing listening or filtered), retransmissions (loss), RST (the peer refused or reset) or a request with no reply within the application's timeout.
6. Stop both tools, note the answer, and only then consider deeper tools (perf, application profilers, longer captures with -C and -W for rotating files).

Expected result

A one-line diagnosis of the form "blocked in recvfrom on the connection to X; X never sends a reply" or "the request never leaves the host", which points at the next component to inspect.

Limits and test basis

strace(1) states that a traced process runs more slowly than a non-traced one; do not leave it attached to a production hot path. The manual's mitigation, --seccomp-bpf, only works when strace starts the command itself with -f; it is not applicable to a process attached with -p, so the attach-and-detach window above is the only bound. Captures contain payloads and secrets; treat pcap files as sensitive and delete them. TLS traffic shows only handshake and sizes. Tool options are taken from the cited manuals; no timing or overhead figures are claimed.