[Python] How to train XGBoost on a large dataset and improve fraud detection?

I'm just starting with ML, so I'd appreciate any advice.

I'm trying to train a model for fraud detection in transactions. The data is highly imbalanced (~96% normal vs. ~4% fraudulent). First issue – memory consumption The training file is 32 GB, but even when reading just 1 million rows, I get a memory allocation error:

xgboost.core.XGBoostError: bad_malloc: Failed to allocate 25547999900 bytes.


Second issue – poor prediction quality I'm training on 100k rows, but no matter how I tune XGBoost, the model barely detects fraud cases.

What are the best class balancing techniques for XGBoost in such a scenario? How should I handle a dataset this large? What would you recommend changing? enter image description here

df = pd.read_csv('train.csv', nrows=100000)

df.drop(['transaction_id', 'card_holder_first_name', 'card_holder_last_name', 'is_verified', 'browser', 'browser_version',
'operating_system', 'operating_system_version', 'card_id', 'ip_address', 'merchant_customer_id', 'merchant_id', 'user_agent',
'merchant_customer_last_name', 'merchant_customer_first_name', 'merchant_customer_phone', 'merchant_customer_email', 'bin','device',
'traffic_source', 'transaction_source', 'merchant_city', 'merchant_shop_id', 'merchant_shop_name', 'order_number'],
axis=1, inplace=True)

df['bank'].replace(' ', '_', regex=True, inplace=True)

df['created_at'] = pd.to_datetime(df['created_at'])
df['seconds_since_midnight'] = df['created_at'].dt.hour * 3600 + df['created_at'].dt.minute * 60 + df['created_at'].dt.second
df['day_of_week'] = df['created_at'].dt.weekday

df.drop('created_at', axis=1, inplace=True)

df.loc[pd.isna(df['merchant_language']), 'merchant_language'] = 'unknown'

df.loc[pd.isna(df['payment_type']), 'payment_type'] = 0

X = df.drop('is_fraud', axis=1).copy()

y = df['is_fraud'].copy()

X_encoded = pd.get_dummies(X, columns=['merchant_country',
'transaction_type',
'merchant_language',
'platform',
'ip_country',
'bank',
'cardbrand',
'cardcountry',
'cardtype',
'payment_type'])


X_train, X_test, y_train, y_test = train_test_split(X_encoded, y, random_state=42, stratify=y)


clf_xgb = xgb.XGBClassifier(
objective="binary:logistic",
seed=42,
eval_metric="aucpr",
early_stopping_rounds=10,
max_depth=6,
subsample=0.8,
colsample_bytree=0.8
)

clf_xgb.fit(
X_train,
y_train,
eval_set=[(X_test, y_test)],
verbose=True
)

disp = ConfusionMatrixDisplay.from_estimator(
clf_xgb,
X_test,
y_test,
display_labels=["Not fraudsters", "Fraud"],
cmap="Blues"
)
disp.plot(values_format='d')

plt.show()


I'm new to ML, so I haven't tried many techniques yet. I experimented with different XGBoost parameters and reduced the dataset size to fit in memory, but the model still struggles to detect fraud cases. I'm not sure what the best approach is for handling such a large and imbalanced dataset, so I'd appreciate any advice.

Continue reading...