[Python] Why am I getting a leading non printable character using split? [duplicate]

I have a input file which has a sequence of grouped data like this

1234567890={
name=Tom
employee=Dick
employee=Harry
}


I read the lines sequentially and to get the group key from the first line of the group record, which should be 1234567890 in the example, I use this snippet where line is simply the next line read from the input file

fields = line.split('=')

if len(fields) == 2 and fields[1].strip() == '{':
# start of group is 2 fields, where the 1st is the key and the 2nd is a opening {
processing_group = True
group_key = fields[0].strip()


But instead of getting 1234567890 the group_key has a non printable ascii character inserted before the actual group key.

The input data definitely does not have any non printable characters in front of the group keys.

So I have to do the following to remove the non printable character

group_key = fields[0].strip()[1:]


This is driving me mad, I have never noticed this behaviour before but perhaps that's because I have always used such extracted data in ways I would never see this behaviour, such as using the data internally or printing to terminal which eats the non printable characters.

However now I am using the extracted group key externally to the python program that creates it, by writing it to a new file which is then further processed by other programs which do not expect a non printable character in the data.

I also have another python program which extracts the group keys for a bash shell script to construct filenames and that also adds a non printable character to the key which didn't show up before because linux simply creates the filename with the non printable character and never displays it in a terminal file listing or in my file explorer, and every editor I used to review the data didn't care about the filename.

I only stumbled on this behaviour because I used vscode one day which wouldn't open the files because although it's internal file explorer could see them, it's editors couldn't open them.

So after vscode revealed the problem I tried to manipulate the files in bash and that too obviously can't see the filenames I am typing because of the non printable characters, so I had to use find -type f -iname = "*1234567890*" -exec some program "{}" \; to process them.

The cooked data decks this program creates are fed into another program which can also be fed data files from other programs that don't have this problem, so I can end up dropping the first valid character of a group key if I use [1:] in the secondary program.

One thought I had is perhaps this non printable character has something to do with the file encoding because I think this behaviour only seems to be showing up in programs that process files with single groups and the group starts on the first line. In my environment, some files are utf-8 bom encoded and some just simple utf-8?

I open the files using

try:
with open(groups_input_pathname, 'r') as groups_file:
lines = groups_file.readlines()

with open(groups_output_pathname, 'w') as groups_out_file:
for line in lines:


I found an answer somewhere that said to use

f = io.open("test", mode="r", encoding="utf-8")


to open the file but I don't know in advance if the file is utf-8 or utf-8 bom

So why is the split or strip adding the non printable character and how do I avoid this so I don't have to use [1:]?

If the problem is indeed caused by file encoding, how can I open the files in a encoding neutral way so as to avoid the issue?

Continue reading...