[Python] Strange Regex Behaviour When Using Character Sets ([]) and Em-Dashes using the Pandas...

For some reason conducting a regex search using the em dash (–) in a character set using the pandas str.contains() function leads to strange results.

I am attempting to sort some military ranks into lower, mid and upper enlisted ranks based on user-typed responses. For this, the pandas python library for parsing the data and am using its built-in regex implementation for sorting.

I have the filter filter_lower_enlisted = r'e[ -–]*[123]' (the second dash is the em dash) set up to filter the bottom ranks, hopefully capturing entries such as e3, e 3, e-2, e--2, e––1 amongst others. For some reason though, entries such as E-6/ Petty Officer 1st class (the - here being a regular en dash instead) or even dummy inputs such as 'eajucbniuawbnciu1' keep popping up in my sorting. And stranger, removing the em dash or the space removes that entry from the sort (even though the problem strings do not contain one), while of course removing the functionality of detecting lower ranks that use either of those characters. Upon further examination, I noticed that the issue would occur when: a) Using a character set with length greater than 3 and b) The character set ends in an em dash

With this knowledge filters such as r'e[- –]*[123]' or r'e[ –]*[123]|e[- ]*[123]' (hard to tell with this font but the ems are present) would provide the sorting behavior I would expect.

My full dummy code and output is listed below.

Code:

data = {
'Rank': ['E-3', 'e2', 'E 1', 'E7', 'E-5 ','e-6 petty officer 1st class'],
}

import pandas as pd

filter_lower_enlisted = r'e[ -–]*[123]'

participants = pd.DataFrame(data)

parts_enlisted_low = participants[participants['Rank'].str.contains(filter_lower_enlisted, case=False, na=False)].reset_index(drop=True)
print(parts_enlisted_low['Rank'])


Output:

0 E-3
1 e2
2 E 1
3 e-6 petty officer 1st class
Name: Rank, dtype: object


Why is this a problem and is there any known solution to this behaviour?

Continue reading...