[Python] Problem of assigning different indexes to different unseen labels in Pyspark and...

I wonder if there is any algorithm or method to assign different indexes to different unseen labels, and therefore, addressing the 'cold problem' properly.

For example if I use the following Pyspark code

indexers = [
StringIndexer(inputCol=c,
outputCol="id_{}".format(c)).setHandleInvalid("keep")
for c in cols
]
pipeline = Pipeline(stages=indexers)

indexer_model = pipeline.fit(data\
.where(F.col('period')=='OLD'))

data = indexer_model.transform(data)


I do not solve the problem. For instance if I have 1,000 unseen values, all of them will share exactly the same new index, instead of getting a unique different index each.

Something similar happens with Scikit-learn:

from sklearn.preprocessing import OrdinalEncoder


ordinal_encoder = OrdinalEncoder(handle_unknown='use_encoded_value',
unknown_value=-1)

Continue reading...