Why Python subprocess Hangs When Reading stdout and How to Fix It

If subprocess hangs while reading stdout, check three common causes: the child process has filled stderr and blocked while writing; the parent is waiting for a complete line with readline(), but the child process does not send \n or does not flush its own buffer; or the parent called wait() without draining channels created with PIPE. For a final result, usually use subprocess.run() or Popen.communicate(). For streaming output, service all active channels concurrently.

Applicable versions and limitations

The main examples target Python 3.7 and later: this version added the capture\_output argument to subprocess.run(), while text became a supported alias for universal\_newlines. In Python 3.6, you can use explicit stdout=subprocess.PIPE, stderr=subprocess.PIPE, and universal\_newlines=True.

import subprocess
result = subprocess.run(
["some-command"],
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
universal_newlines=True,
check=False,
)

The deadlock-prevention principle is the same for these versions: a channel that is actively filling must not be left unread while the parent waits for the child process to terminate or for EOF on another channel.

Why deadlock occurs with stdout=PIPE and stderr=PIPE

With stdout=subprocess.PIPE and stderr=subprocess.PIPE, the operating system creates channels with limited capacity. The child process writes data to the channels, and the parent must read it. If one channel fills up, the child process may block on its next write until space becomes available.

A dangerous pattern looks like this:

import subprocess
proc = subprocess.Popen(
["some-command"],
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
text=True,
)
stdout = proc.stdout.read()
stderr = proc.stderr.read()
proc.wait()

While the parent reads stdout until EOF, the child program may write heavily to stderr. Once stderr fills up, the child stops while trying to write and cannot terminate. As a result, it does not close stdout, while the parent keeps waiting for EOF on stdout.

If stdout=PIPE and stderr=PIPE are used at the same time, do not read these channels sequentially until EOF and do not call wait() while leaving them unread. To collect a final result, use communicate() or run(); for streaming logs, service both channels concurrently.

Use subprocess.run() for final output

If you only need the data after the command has finished, it is simpler not to manage Popen manually:

import subprocess
result = subprocess.run(
["some-command", "--option"],
capture_output=True,
text=True,
check=False,
)
print("return code:", result.returncode)
print("stdout:", result.stdout)
print("stderr:", result.stderr)

capture\_output=True configures capture of standard output and the standard error stream. run() waits for the command to finish and returns the collected data in a CompletedProcess object.

This approach is suitable when the output is finite and can reasonably be stored in memory. For very large or infinite streams, collecting all data is not appropriate: output should be processed incrementally.

When using Popen, use communicate()

If you need a Popen object but do not need to process lines while the program is running, use communicate():

import subprocess
proc = subprocess.Popen(
["some-command"],
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
text=True,
)
stdout, stderr = proc.communicate()
print("return code:", proc.returncode)
print("stdout:", stdout)
print("stderr:", stderr)

The official documentation specifically warns that Popen.wait() can cause a deadlock when stdout=PIPE or stderr=PIPE is used and the child process writes enough data to fill the channel. In this case, the documentation recommends using Popen.communicate().

Do not use the following order with a potentially active PIPE:

# Dangerous if the child process can fill PIPE
proc.wait()
stdout = proc.stdout.read()

If the process has already blocked while writing to a full channel, it cannot terminate, so wait() will not return either.

If you do not need to separate stderr and stdout, merge them

When a single combined log is sufficient, you can redirect stderr to stdout:

import subprocess
proc = subprocess.Popen(
["some-command"],
stdout=subprocess.PIPE,
stderr=subprocess.STDOUT,
text=True,
)
output, _ = proc.communicate()
print(output)

As a result, the parent only needs to service one channel. The limitation is clear: after merging the streams, you can no longer determine whether a particular fragment was originally sent to stdout or stderr.

readline() may wait for a newline

Stopping at the following operation does not necessarily mean there is a deadlock:

line = proc.stdout.readline()

readline() reads a line. If the child process has sent part of the data without a newline character and continues running, the call may wait for the rest of the line or for EOF.

For example:

import time
print("starting...", end="", flush=True)
time.sleep(60)

Here the data is actually flushed to standard output because of flush=True, but there is no \n character. Therefore, a parent using line-oriented readline() must not treat the appearance of individual bytes as a guarantee that a complete line is available.

If the protocol is intended to be line-oriented, the child program must produce complete lines:

import sys
sys.stdout.write("starting...\n")
sys.stdout.flush()

Buffering in a child Python process and the -u option

Even if the child code has produced data, it may remain in the child process's own buffers for some time. If you control the Python program, use explicit flush=True, call flush(), or use an appropriate execution mode.

The Python interpreter supports the -u option, which makes the standard stdout and stderr streams unbuffered at the corresponding level described in the Python command-line documentation:

import subprocess
import sys
proc = subprocess.Popen(
[sys.executable, "-u", "worker.py"],
stdout=subprocess.PIPE,
stderr=subprocess.STDOUT,
text=True,
)

For programs written in other languages, the rules depend on their runtime environment and output implementation. Popen settings are not a universal way to force an arbitrary child process to call flush() after every write.

Do not confuse bufsize with buffering in the child program

The bufsize argument in Popen is passed when creating pipe file objects on the parent Python process side. It does not automatically disable internal buffering in the launched program.

proc = subprocess.Popen(
    ["some-command"],
    stdout=subprocess.PIPE,
    text=True,
    bufsize=1,
)

This setting alone does not guarantee that the child program will start sending data immediately. If messages only appear just before the child process terminates, check the buffering mechanism in the child process itself.

For streaming logs, read stdout and stderr concurrently

If you need to receive lines while the process is running and it is important to keep the two streams separate, both channels must be drained regularly. A portable approach is to use a separate reader thread for each channel:

import subprocess
import threading
def consume(name, stream):
try:
for line in stream:
print(f"{name}: {line}", end="")
finally:
stream.close()
proc = subprocess.Popen(
["some-command"],
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
text=True,
)
stdout_thread = threading.Thread(
target=consume,
args=("stdout", proc.stdout),
)
stderr_thread = threading.Thread(
target=consume,
args=("stderr", proc.stderr),
)
stdout_thread.start()
stderr_thread.start()
returncode = proc.wait()
stdout_thread.join()
stderr_thread.join()
print("return code:", returncode)

While the child process is running, one thread reads stdout and the other reads stderr, so one channel is not left completely unserviced while the other is being read.

This example assumes a line-oriented text protocol. For binary data or messages that are not separated by \n, choose the reading method according to the actual format.

Two threads also do not preserve an exact global event order between stdout and stderr: the scheduler may run the parent reader threads in a different order. If a single observable stream of messages is important, using stderr=subprocess.STDOUT is simpler, with the understanding that the original source of each line is then lost.

If capture is not required, do not create a PIPE

When the only goal is to run a program and show its output in the same terminal, PIPE is unnecessary:

import subprocess
result = subprocess.run(
["some-command"],
check=False,
)
print("return code:", result.returncode)

Without explicit redirection, the child process inherits the corresponding standard streams from the parent. This is also a useful diagnostic check: if the command finishes without PIPE but stops after manual pipe reading is added, investigate the channel-reading pattern and buffering.

Add a controlled timeout

A timeout does not fix incorrect pipe handling, but it lets you limit how long an external command can run.

import subprocess
try:
result = subprocess.run(
["some-command"],
capture_output=True,
text=True,
timeout=30,
check=False,
)
except subprocess.TimeoutExpired:
print("process timeout")

When using Popen.communicate(), after TimeoutExpired you can explicitly terminate the process according to the application's policy and then call communicate() again to wait for it to finish and drain the channels:

import subprocess
proc = subprocess.Popen(
["some-command"],
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
text=True,
)
try:
stdout, stderr = proc.communicate(timeout=30)
except subprocess.TimeoutExpired:
proc.kill()
stdout, stderr = proc.communicate()
print("return code:", proc.returncode)

Reproducible test: filling stderr

You do not need an external command to test a deadlock. Create a child\_stderr.py file that repeatedly writes to stderr and then prints a marker to stdout:

import sys
chunk = "x" * 4096
for _ in range(10000):
sys.stderr.write(chunk)
sys.stderr.flush()
print("finished")

The system pipe size depends on the platform, so the test should not rely on a specific capacity. This example intentionally generates far more output than a typical small stream of diagnostic messages.

Do not use sequential reading of stdout and then stderr for such a child process. A safe test with communicate():

import subprocess
import sys
proc = subprocess.Popen(
[sys.executable, "child_stderr.py"],
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
text=True,
)
stdout, stderr = proc.communicate(timeout=30)
print("return code:", proc.returncode)
print("stdout:", stdout.strip())
print("stderr chars:", len(stderr))

The verification criterion here is not a specific number of characters, but successful process termination and the presence of the finished line without hanging when the second channel fills up.

Reproducible test: data without a newline

To test readline() behavior separately, create child\_no\_newline.py:

import sys
import time
sys.stdout.write("partial")
sys.stdout.flush()
time.sleep(5)
sys.stdout.write("\n")
sys.stdout.flush()

Parent code:

import subprocess
import sys
import time
proc = subprocess.Popen(
[sys.executable, "child_no_newline.py"],
stdout=subprocess.PIPE,
text=True,
)
started = time.monotonic()
line = proc.stdout.readline()
elapsed = time.monotonic() - started
print("received:", repr(line))
print("seconds:", round(elapsed, 1))
proc.wait()

The child process immediately flushes the text partial, but sends the newline only after the pause. Therefore, readline() returns a complete line only after \n appears. This test helps distinguish waiting for a line boundary from a deadlock caused by a full stderr channel.

How to identify the specific cause

Check the Popen parameters. If both stdout=PIPE and stderr=PIPE are specified, find the code responsible for servicing both channels.
If wait() is called before the channels are read, replace that pattern with communicate() or concurrent reading.
If the code first performs stdout.read() until EOF and then stderr.read(), eliminate sequential reading of two channels that may both be actively filling.
If execution stops at readline(), check for the presence of \n and whether the child program actually flushes its data.
If messages only appear when the child program terminates, check its own buffering behavior.
If keeping the streams separate is unnecessary, consider stderr=STDOUT.
If output capture is not needed at all, do not create a PIPE.

Final checklist

The parent does not call wait() while leaving actively filling PIPE channels unread.
stdout and stderr are not read sequentially until EOF when both streams may be used heavily.
run() or communicate() is used for a final result.
For streaming processing, all active channels are serviced concurrently.
Code with two reader threads does not assume that the exact global order of stdout and stderr will be preserved.
When readline() is used, the child protocol actually produces newline-terminated lines.
The parent Popen bufsize setting is not used as a substitute for controlling buffering in the child program.
Potentially long-running external commands have an appropriate timeout and process-termination policy.
The fix is tested separately with a heavy stderr write test and a test that emits output without an immediate \n.