[Python] Code after using Stratified K-Fold: Do we need to specify which fold?

I have been using simple techniques to split the dataset. However I would love to progress with more advanced techniques. Such as Stratified K-Fold for imbalanced data. However, once the data is splitted such as the code I provide. How do we continue with the preprocessing phase?

Example: we can not use the whole dataset to impute the NaN values. How do we code it?

# Assume sem_df is your DataFrame and 'target' is the target column
X = sem_df.drop('target', axis=1)
y = sem_df['target']

n_total = len(sem_df)

# 1. Split the data into 80% training and 20% temporary (which will be split into validation and test)
X_train, X_test, y_train, y_test = train_test_split(
X, y,
test_size=0.20,
stratify=y,
random_state=42
)

# Print shapes with percentage information
print(f"Train set shape: {X_train.shape} ({X_train.shape[0] / n_total * 100:.1f}%)")
print(f"Test set shape: {X_test.shape} ({X_test.shape[0] / n_total * 100:.1f}%)")


Here is the imputation problem:

# Note: different values from train and test
train_num_medians = X_train[num_cols].median()
train_cat_modes = X_train[cat_cols].mode().iloc[0]

# Note: different values from train and test
test_num_medians = X_test[num_cols].median()
test_cat_modes = X_test[cat_cols].mode().iloc[0]

# Impute missing values in the training set using .loc and without inplace
X_train.loc[:, num_cols] = X_train.loc[:, num_cols].fillna(train_num_medians)
X_train.loc[:, cat_cols] = X_train.loc[:, cat_cols].fillna(train_cat_modes)

# Impute missing values in the test set using training imputation values
X_test.loc[:, num_cols] = X_test.loc[:, num_cols].fillna(test_num_medians)
X_test.loc[:, cat_cols] = X_test.loc[:, cat_cols].fillna(test_cat_modes)


Or even for the normalization part:

# Now safely perform scaling:
scaler = StandardScaler()
X_train[norm_col] = scaler.fit_transform(X_train[norm_col])
#X_val.loc[:, norm_col] = scaler.transform(X_val[to_norm])
X_test[norm_col] = scaler.transform(X_test[norm_col])


When performing the data split with cross validation do we have to change the classical step?

train -> fit_transform
test -> transform

Continue reading...