[Python] Issues with Balancing an Imbalanced Dataset Using ADASYN and Random Undersampling

Dataset Overview and Goals: I am working with the CICEVSE2024 dataset, which contains network traffic data from electric vehicle charging stations, labeled as either benign or attack traffic. The dataset originally consists of multiple CSV files, but I have merged them into a single file for ease of processing.

My goal is to balance the dataset before training a machine learning model to detect cyberattacks. The extreme class imbalance poses a challenge in model training, as it can lead to biased predictions favoring the dominant class.

Class Imbalance Issue: The dataset contains:

81 benign samples

Over 2.7 million attack samples

Since the attack class heavily outweighs benign traffic, training a model on this raw dataset would result in severe class bias, where the model simply predicts "attack" for everything.

To address this, I applied random undersampling to reduce the number of attacks, followed by ADASYN oversampling to generate synthetic benign samples, aiming to create a balanced dataset.

Current Code and Approach

import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler, LabelEncoder
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import accuracy_score, classification_report, roc_auc_score
from imblearn.under_sampling import RandomUnderSampler
from imblearn.over_sampling import ADASYN

# Load dataset (merged from multiple files)
data = pd.read_csv("CICEVSE2024.csv", low_memory=False)

# Handle missing values for categorical columns
categorical_cols = data.select_dtypes(include=['object']).columns
data[categorical_cols] = data[categorical_cols].fillna("Missing")

# Remove duplicate entries
data.drop_duplicates(inplace=True)

# Process attack labels (simplify multi-class labels)
data['label_mul'] = data['label_mul'].str.replace(r'\d+$', '', regex=True)

# Convert to binary classification: 0 (benign), 1 (attack)
data['label_bi'] = np.where(data['label_mul'].isin(['idle benign', 'Charging Benign']), 0, 1)

# Drop unnecessary columns (identifiers & network-specific attributes)
drop_cols = ['id', 'expiration_id', 'src_mac', 'src_oui', 'dst_mac', 'dst_oui', 'src_ip', 'dst_ip', 'src_port', 'dst_port']
data.drop(columns=drop_cols, errors='ignore', inplace=True)

# Convert numerical columns to float32 (memory optimization)
data.fillna(data.mean(), inplace=True)
numerical_cols = data.select_dtypes(include=['number']).columns
data[numerical_cols] = data[numerical_cols].astype('float32')

# Encode categorical columns
for col in categorical_cols:
if col in data.columns:
le = LabelEncoder()
data[col] = le.fit_transform(data[col])

# Ensure label column exists
if 'label_bi' not in data.columns:
raise KeyError("Column 'label_bi' not found in dataset.")

# Define features and target
X = data.drop(columns=['label_bi'], errors='ignore')
y = data['label_bi']

# Step 1: Random Undersampling (reduce attack samples)
attack_classes = data[data['label_bi'] == 1]['label_mul'].value_counts().index
undersample_ratio = {attack: 0.5 for attack in attack_classes} # Reduce attack samples by 50%
undersampler = RandomUnderSampler(sampling_strategy=undersample_ratio, random_state=42)
X_under, y_under = undersampler.fit_resample(X, y)

# Step 2: ADASYN Oversampling to Generate Synthetic Benign Samples
batch_size = 5000 # Process in batches for efficiency
adasyn = ADASYN(random_state=42, n_neighbors=5)
X_adasyn, y_adasyn = [], []

for i in range(0, len(X_under), batch_size):
X_batch = X_under.iloc[i:i + batch_size]
y_batch = y_under.iloc[i:i + batch_size]
if len(y_batch[y_batch == 1]) > 1: # Ensure attack samples exist
X_resampled, y_resampled = adasyn.fit_resample(X_batch, y_batch)
X_adasyn.append(X_resampled)
y_adasyn.append(y_resampled)

X_adasyn = pd.concat(X_adasyn, ignore_index=True)
y_adasyn = pd.concat(y_adasyn, ignore_index=True)

# Step 3: Additional Oversampling for Benign Samples
benign_samples_needed = y_adasyn.value_counts()[1] - y_adasyn.value_counts()[0]
benign_samples = X_adasyn[y_adasyn == 0].sample(n=min(benign_samples_needed, len(X_adasyn[y_adasyn == 0])), replace=False, random_state=42)
X_balanced = pd.concat([X_adasyn, benign_samples], ignore_index=True)
y_balanced = pd.concat([y_adasyn, pd.Series([0] * len(benign_samples))], ignore_index=True)

# Step 4: Check Final Class Balance
assert y_balanced.value_counts()[0] == y_balanced.value_counts()[1], "Benign and attack samples are not balanced."

# Step 5: Shuffle Data and Save
balanced_df = X_balanced.copy()
balanced_df['label_bi'] = y_balanced
balanced_df = balanced_df.sample(frac=1, random_state=42).reset_index(drop=True)
balanced_df.to_csv("CICEVSE2024_Balanced.csv", index=False)

# Show final class distribution
print("Final Class Distribution:")
print(balanced_df['label_bi'].value_counts())

# Step 6: Train/Test Split
X_train, X_test, y_train, y_test = train_test_split(X_balanced, y_balanced, test_size=0.2, random_state=42)

# Train a Simple Machine Learning Model
model = RandomForestClassifier(n_estimators=50, random_state=42)
model.fit(X_train, y_train)

# Make Predictions
y_pred = model.predict(X_test)

# Evaluate Model Performance
print("Model Accuracy:", accuracy_score(y_test, y_pred))
print("Classification Report:\n", classification_report(y_test, y_pred))
print("AUC-ROC Score:", roc_auc_score(y_test, y_pred))


Questions & Areas for Improvement: Why is my model performing too well?

Could the oversampling strategy (ADASYN in batches) be causing data leakage?

How can I ensure the train/test set remains perfectly balanced?

Should I be performing stratified sampling after resampling?

Are there better ways to balance the dataset?

Would SMOTE be a better alternative to ADASYN?

Is there a better undersampling method than a fixed 50% reduction per attack type?

Could feature leakage be a problem?

Should I check if some features contain duplicate/misleading information?

I would appreciate any suggestions, alternative approaches, or debugging ideas to improve the balancing process while avoiding unrealistic model performance.

Continue reading...