[Python] How to open a textfile in python on windows with the correct encoding?

I am aware that correctly determining the encoding of text is impossible. I notice however, that programs such as notepad can correctly read a file encoded in utf-8, and I want to do the same in python.

open can not do it, it is documented in python3.11 to use the systems local encoding as default, which is 'cp1252' on my machine, so the behaviour does not depend on any BOM or other magic that might be in the file or its metadata.

The following code acts the same on python-versions 3.7, 3.10 and 3.11 (which is all that I tried):


fname = "tmpWindowsOut.txt"
with open(fname, "w", encoding="utf-8") as f:
# if encoding wasn't specified in `open` this fails with the infamous
# UnicodeEncodeError: 'charmap' codec can't encode character '\u03b2' in position 1: character maps to <undefined>
f.write("βKHα")

with open(fname, "r") as f:
print(f.read()) # prints βKHα

# opening the file with notepad shows the correct output


I know I can correctly read the file by passing the encoding-parameter to the other open-call, the question is not how to read a file that is known to be encoded in utf-8. The question is how to either:

1. do the read-portion correctly without specifying the encoding
2. do the write-portion in a way that makes No.1 easier (or even just possible)

Continue reading...