[Python] Handling UTF-8 Special Characters in GitLab CI/CD Pipeline on Windows

I'm creating a merge request in gitlab with a gitlab pipeline. I set special characters in .gitlab-ci.yml and provide it to a python script, which is creating the merge request.
But in the merge request, the characters "äöüß" are displayed as "├ñ├╢├╝├ƒ"

.gitlab-ci.yml


test-job:
stage: test
script:
- python script.py -Test "äöüß"

tags:
- shellowershell
- toolsython

script.py


import argparse
import os
import gitlab

# arguments
parser = argparse.ArgumentParser()
parser.add_argument("-Test", type=str, required=True)
args = parser.parse_args()

# gitlab project
gl = gitlab.Gitlab("https://xxx/", private_token=os.getenv("xxx"), ssl_verify=os.getenv("GITLAB_SSL_CERT"))
project = gl.projects.get(os.getenv("CI_PROJECT_ID"))

# create test merge request
mr = project.mergerequests.create({
"source_branch": os.getenv("CI_COMMIT_BRANCH"),
"target_branch": "master",
"title": "Test-Title",
"description": args.Test
})


If I add print(f"input: {args.Test}") to script.py it returns this error:

Traceback (most recent call last):
File "D:\GitLab-Runner\builds\yxFfC2Egu\0\SPS_Entwicklung\test\ci-dev-test\script.py", line 14, in <module>
print(f"input: {args.Test}")
~~~~~^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Program Files\Python313\Lib\encodings\cp1252.py", line 19, in encode
return codecs.charmap_encode(input,self.errors,encoding_table)[0]
~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
UnicodeEncodeError: 'charmap' codec can't encode character '\u251c' in position 7: character maps to <undefined>


I have a Windows 11 runner with powershell as shell.

• .gitlab.ci.yml is encoded as utf-8
• I have tried different ways of encode()/decode()
• checked the encoding with chardet, what returned utf-8.

I have no clue what I can try next or what I am doing wrong or what I'm not understanding here. Strings in the .gitlab.ci.yml and in python should be handled as utf-8, right? But powershell handels strings as windows-1252 per default? Is this the Problem? I tried also [Console]::OutputEncoding = [Text.UTF8Encoding]::UTF8 but nothing changed. I am happy for every hint.

Continue reading...