I am learning the concept of target encoding, specifically M-Estimate Encoding. Firstly, the tutorial demonstrates mean encoding. autos["make_encoded"] = autos.groupby("make")["price"].transform("mean") Then it mentions there are some issues. An encoding like this presents a couple of problems, however. First are unknown categories. Target encodings create a special risk of overfitting, which means they need to be trained on an independent "encoding" split. When you join the encoding to future splits, Pandas will fill in missing values for any categories not present in the encoding split. These missing values you would have to impute somehow. They say these issues are fixed by splitting the dataset. X = df.copy() y = X.pop('Rating') X_encode = X.sample(frac=0.25) y_encode = y[X_encode.index] X_pretrain = X.drop(X_encode.index) y_train = y[X_pretrain.index] from category_encoders import MEstimateEncoder # Create the encoder instance. Choose m to control noise. encoder = MEstimateEncoder(cols=["Zipcode"], m=5.0) # Fit the encoder on the encoding split. encoder.fit(X_encode, y_encode) # Encode the Zipcode column to create the final training data X_train = encoder.transform(X_pretrain) I don't understand what the issue is and how does splitting solve the problem. Why is mean encoding not sufficient? What is the risk of overfitting? Continue reading...
