It is understood that in exceptional circumstances, a read may block after select(2) declares that data is available for reading for something like a network socket, where checksum failures or other buffer manipulations may cause data to be discarded between select and read. However, I would not expect that to happen for a Python program when dealing only with standard pipes. Consider the following: with subprocess.Popen(["openssl", "speed"], text=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE) as process: # Something weird is going on. `fd.read()` below blocks even if `select` declares # that there is data available. The `set_blocking` calls below partially help this, # but now there is a strange delay at program start. os.set_blocking(process.stdout.fileno(), False) os.set_blocking(process.stderr.fileno(), False) buffer = StringIO() while process.poll() is None: rfds, _, _ = select.select([process.stdout, process.stderr], [], []) for fd in rfds: if fd == process.stdout: chunk = fd.read() if DEBUG: sys.stdout.write(chunk) buffer.write(chunk) elif fd == process.stderr: sys.stdout.write(fd.read()) I would expect this to work without the set_blocking calls, but it doesn't. I have also tried with standard binary streams (text=False) and the results are the same. Why is read blocking when select says it shouldn't? Continue reading...
