Python subprocess pipes: drain both streams before waiting indefinitely

What it is

Python's subprocess documentation warns that Popen.wait can deadlock when stdout or stderr is a pipe and the child fills an OS pipe buffer. communicate coordinates input and output, but buffers captured data in memory. The documentation therefore distinguishes a convenient bounded-output path from workloads requiring deliberate streaming and storage limits. Python subprocess

Why it matters

An agent may increase a timeout while the parent and child are waiting for each other. First inspect which party must make progress: the parent draining output, the child reading input, or the child exiting. A longer wait does not repair a missing consumer.

How to apply

• Write down how stdin, stdout and stderr are connected. Check whether the parent reads one stream completely before touching the other or waits for exit before reading either.
• For known bounded output, use the documented high-level invocation or communicate path. Set a task-appropriate timeout and handle the resulting failure explicitly.
• For potentially large output, design concurrent draining to bounded storage or a bounded processing pipeline. Specify what happens when output exceeds its budget rather than accumulating it indefinitely.
• Distinguish subprocess.run timeout behavior from direct Popen.communicate timeout handling. The latter requires an explicit cleanup decision; follow the documented terminate or kill and drain sequence for the intended policy.
• Propose a child fixture that writes substantial data to both streams, plus fixtures for early exit and timeout. Check process cleanup as well as captured content.

Pitfalls

Pipe capacity is an implementation detail, so avoid a production dependency on one measured buffer size. A timeout is not a universal process-tree termination policy. Consider grandchildren separately when the tool can spawn them. The proposed fixtures are bounded diagnostic tests, not a claim that any specific command has been tested or that capture_output is safe for unlimited output.