[Python] Why does opening a subprocess with universal_newlines cause a unicode decode...

I am using the subprocess module to run a child job, and collecting its output and error streams with subprocess.PIPE's. To avoid deadlock I continually read from those streams on a separate thread. This works, except sometimes the program crashes due to a decoding issue:

`UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 483: ordinal not in range(128
​

At a high level, I understand that Python is probably trying to convert to a string using the ASCII codec, and that I need to call decode somewhere, I'm just not sure where. When I create my subprocess job, I specify universal_newlines to be True. I thought this meant, return stdout/stderr as unicode, not binary:

self.p = subprocess.Popen(self.command, shell=self.shell, stdout=subprocess.PIPE, stderr=subprocess.PIPE, universal_newlines=True)


The crash happens in my reading thread function:

def standardOutHandler(standardOut):
# Crash happens on the following line:
for line in iter(standardOut.readline, ''):
writerLock.acquire()
stdout_file.write(line)
if self.echoOutput:
sys.stdout.write(line)
sys.stdout.flush()
writerLock.release()


Its not clear why readline is throwing a decoding exception here; as I stated, I thought universal_newlines being true was already returning me decoded data.

What is going on here and what can I do to correct this?

Here is the full traceback

Exception in thread Thread-5:
Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/threading.py", line 920, in _bootstrap_inner
self.run()
File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/threading.py", line 868, in run
self._target(*self._args, **self._kwargs)
File "/Users/lzrd/my_process.py", line 61, in standardOutHandler
for line in iter(standardOut.readline, ''):
File "/Users/lzrd/Envs/my_env/bin/../lib/python3.4/encodings/ascii.py", line 26, in decode
return codecs.ascii_decode(input, self.errors)[0]
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 483: ordinal not in range(128)

Continue reading...