[Python] UnicodeEncodeError when processing JSON with emojis in Python on Windows

I’m trying to load and process a JSON file in Python (3.12) on Windows using VS Code.

My JSON file contains text with emojis and special characters, like this:

[
{
"text": "Just saw a LinkedIn Influencer with 'Organic Growth' written in the profile with 65K+ followers.",
"engagement": 90
},
{
"text": "Jobseekers, this one’s for you. ",
"engagement": 347
}
]


Here’s my Python code:

import json

def process_posts(raw_file_path, processed_file_path=None):
with open(raw_file_path, encoding='utf-8') as file:
posts = json.load(file)
print(posts)

if __name__ == "__main__":
process_posts("data/raw_posts.json", "data/processed_posts.json")


When I run this, I get this error due to print(posts)

UnicodeEncodeError: 'charmap' codec can't encode character '\U0001f33b' in position 748: character maps to <undefined>

I tried using:

import sys
import io

sys.stdout = io.TextIOWrapper(sys.stdout.buffer, encoding='utf-8')



and printing with:

print(json.dumps(posts, ensure_ascii=False))


This works — but when I later pass the same JSON into an LLM (large language model) to extract certain values, I get another error:

'utf-8' codec can't encode character '\ud83e' in position 983: surrogates not allowed

I have read that this may be related to Windows terminal encoding or to how Python handles surrogates.

Question: What is the best way to process and print JSON that contains emojis and special characters in Python on Windows — especially when passing it to other systems (like LLMs)? How can I avoid these UnicodeEncodeError or surrogate errors?

Continue reading...