[Python] PyPy3 on Windows 11 doesn't display non-ASCII characters correctly

PyPy3 doesn't display non-ASCII UNICODE characters correctly.

A simple example, the following:

b'\xce\x9e\xce\xad\xce\xbd\xce\xb7 \xce\x93\xce\xae\xce\xb9\xce\xbd\xce\xbf\xcf\x82'.decode('utf8')


Should evaluate to my user name: 'Ξένη Γήινος'.

But the output of PyPy3 is this:

In [1]: b'\xce\x9e\xce\xad\xce\xbd\xce\xb7 \xce\x93\xce\xae\xce\xb9\xce\xbd\xce\xbf\xcf\x82'.decode('utf8')
Out[1]: '╬×╬¡╬¢╬À ╬ô╬«╬╣╬¢╬┐¤é'


There is very little information on this I can find through Google searching, but I have found this:

https://github.com/pypy/pypy/issues/4948

So this is a known issue, and it hasn't been fixed.

I tried to fix this issue using information I found from the linked page, the encoding and locale used by CPython are:

In [1]: import locale, os, sys

In [2]: locale.getdefaultlocale()
<ipython-input-2-64720e52add3>:1: DeprecationWarning: 'locale.getdefaultlocale' is deprecated and slated for removal in Python 3.15. Use setlocale(), getencoding() and getlocale() instead.
locale.getdefaultlocale()
Out[2]: ('en_US', 'cp1252')

In [3]: locale.getlocale()
Out[3]: ('English_United States', '1252')

In [4]: locale.getencoding()
Out[4]: 'cp1252'

In [5]: locale.LC_ALL
Out[5]: 0

In [6]: sys.getdefaultencoding()
Out[6]: 'utf-8'


And these are what's used by PyPy3:

In [2]: import sys, locale

In [3]: locale.getencoding()
Out[3]: 'utf-8'

In [4]: locale.getlocale()
Out[4]: ('English_United States', '1252')

In [5]: sys.getdefaultencoding()
Out[5]: 'utf-8'


So it seems evident that the issue is caused by the mismatch between the code page Windows uses and the code page PyPy3 uses, Windows uses 'cp1252' and CPython uses the same code page, but PyPy3 doesn't. Thus the fix is to either make PyPy3 use 'cp1252' code page or make Windows console use 'utf-8'.

I tried many ways to fix it, setting the environment variable doesn't work:

Set-ItemProperty -Path "HKLM:\SYSTEM\CurrentControlSet\Control\Session Manager\Environment" -Name "PYTHONIOENCODING" -Type STRING -Value "UTF-8"


The following also doesn't work:

chcp 65001


There is no reload in Python 3 and therefore I cannot use sys.setdefaultencoding, and locale.setlocale(0) doesn't work either.

The above are just a few of the methods I have tried.

But the following works:

PS C:\Users\xenig> [Console]::InputEncoding = New-Object System.Text.UTF8Encoding
PS C:\Users\xenig> [Console]::OutputEncoding = New-Object System.Text.UTF8Encoding
PS C:\Users\xenig> D:\Programs\pypy3\Scripts\ipython.exe
Python 3.11.11 (0253c85bf5f8, Feb 26 2025, 10:43:25)
Type 'copyright', 'credits' or 'license' for more information
IPython 9.0.2 -- An enhanced Interactive Python. Type '?' for help.
Tip: IPython 9.0+ have hooks to integrate AI/LLM completions.

In [1]: b'\xce\x9e\xce\xad\xce\xbd\xce\xb7 \xce\x93\xce\xae\xce\xb9\xce\xbd\xce\xbf\xcf\x82'.decode('utf8')
Out[1]: 'Ξένη Γήινος'


Okay, now, how can I make it so I can directly launch PyPy3 in Windows Terminal without launching PowerShell first and make PyPy3 display UNICODE characters correctly?


Oh, and the output for the first code block from the linked GitHub page is:

pypy win32 3.11.11 (0253c85bf5f8, Feb 26 2025, 10:43:25)
[PyPy 7.3.19 with MSC v.1941 64 bit (AMD64)]
os.device_encoding(0)='cp850'
os.device_encoding(1)='cp850'
sys.getdefaultencoding()='utf-8'
sys.getfilesystemencoding()='utf-8'
locale.getpreferredencoding()='utf-8'
locale.getencoding()='utf-8'
locale.getlocale()=('English_United States', '1252')
locale.getlocale()=('English_United States', '1252')

Continue reading...