[Python] ValueError: X has 7 features, but ColumnTransformer expects 13 features

I have the following code where I try to predict price of tools for which I use poisson regression.

# --- Load and Prepare Data ---
y = train['PriceToday']
X = train.drop(columns=['PriceToday'])

# Define non-standard types
non_standard_types = ["Nar", "Orch", "Fru","Comp"]

# Create a flag feature for non-standard
X["Non_Standard_Flag"] = X["Type_LS"].isin(non_standard_types).astype(int)

# Identify numerical and categorical columns
num_features = ["AGE", "POWER", "Hours", "Non_Standard_Flag"]
cat_features = ["BRAND", "Country", "Final_Trans"]

# Define preprocessing pipeline
preprocessor = ColumnTransformer(
transformers=[
('num', StandardScaler(), num_features),
('cat', OneHotEncoder(handle_unknown='ignore'), cat_features)
], remainder="drop"
)

# --- Train/Test Split ---
# Create a weight column
train["sample_weight"] = train["Type_LS"].apply(lambda x: 1 if x == "Standard" else 5)

train["stratify_group"] = train["BRAND"].astype(str)
X_train, X_val, y_train, y_val, train_weights, val_weights = train_test_split(
X, y, train["sample_weight"], test_size=0.2, random_state=42, stratify=train["stratify_group"]
)
# Fit the preprocessor once on training data
X_train_preprocessed = preprocessor.fit_transform(X_train)
X_val_preprocessed = preprocessor.transform(X_val)

# Define models
models = {
"Poisson": PoissonRegressor(alpha=0.01)
}

# Train and evaluate models
model_results = {}

for model_name, model in models.items():
model.fit(X_train_preprocessed, y_train, sample_weight=train_weights)

# Predictions
predictions = model.predict(X_val_preprocessed)
# Calculate Metrics
r2 = r2_score(y_val, predictions)

model_results[model_name] = {
"model": model,
"R2": r2
}



I have a test data for which I want to compare the prices of them with the predicted price from the model. My test data is something like this:

# Ensure new data has the correct format
new_data = pd.DataFrame({
"AGE": [12, 24, 36, 48, 60, 72, 84, 12, 24, 36, 48, 60, 72, 84],
"Hours": [500, 1000, 1500, 2000, 2500, 3000, 3500, 500, 1000, 1500, 2000, 2500, 3000, 3500],
"BRAND": ["NH"] * 14,
"POWER": [150] * 7 + [80] * 7,
"Final_Trans": ["Cv"] * 14,
"Country": ["DEU"] * 14,
"Type_LS": [Nar, Nar, Nar, ST, ST, ST, ST, ST,ST, ST, ST, ST, ST, ST]
"Current_Pred": [105614, 96681, 88504, 81018, 74165, 67892, 62150, 42608, 39728, 37043, 34540, 32206, 30029, 28000]
})
``



My code is:

new_df = pd.DataFrame(new_data)
# Create the 'Non_Standard_Flag'
new_df["Non_Standard_Flag"] = new_df["Type_LS"].isin(non_standard_types).astype(int)

# Select the columns required by the preprocessor
X_new = new_df[['AGE', 'POWER', 'Hours', 'Non_Standard_Flag', 'BRAND', 'Country', 'Final_Trans']]

X_new_preprocessed = preprocessor.transform(X_new)

# Get the column names after one-hot encoding from the training data
ohe = preprocessor.named_transformers_['cat']
encoded_cat_columns = ohe.get_feature_names_out(cat_features)

# Create column names for numeric features
num_columns = num_features

# Combine the column names
all_columns = num_columns + list(encoded_cat_columns)

# Create a DataFrame from the preprocessed data
X_new_preprocessed_df = pd.DataFrame(X_new_preprocessed, columns=all_columns)

# --- Predict with the Poisson Model ---
poisson_model = model_results["Poisson"]["model"] # Access the trained Poisson model
predicted_prices = poisson_model.predict(X_new_preprocessed_df)

# Compare and Store Results ---
new_df['Predicted_Price'] = predicted_prices

# Calculate the difference between predicted and current prices
new_df['Price_Difference'] = new_df['Predicted_Price'] - new_df['Current_Pred']



But after doing this I get an error:X has 7 features, but ColumnTransformer expects 13 features I have the same number of columns so I do no understand why I have this error. Any help would be appreciated.

Continue reading...