I have a data frame in pyspark. This data frame has say some columns with special characters. cols = df.schema.names cols ['abc test', 'test*abc', 'eng)test', 'abc_&test'] reps = ((' ', '_&'), ('(', '*_'), (')', '_*'), ('{', '#_'), ('}', '_#'), (';', '_##'), ('.', '_$'), (',', '_$$'), ('=', '_**')) def col_rename(x): new_cols = reduce(lambda a, kv: a.replace(*kv), reps, x) for i in cols: df = df.withColumnRenamed(i, col_rename(cols, i)) return df Now I want to see if after replacing the special characters in the column names if there are any duplicate columns. As we can see there is a duplicate of columns in the new_cols abc_&test I want to return extra _ underscore when this happens. My new_cols shoul be like below ['abc__&test', 'test*_abc', 'eng_*test', 'abc_&test'] How can I achieve what I want? Continue reading...
