[Python] Why are two different Logistic Regression models producing vastly different results...

I've searched for similar questions, and none of the answers address my specific situation.

I'm working with NFL data, predicting game outcomes. My source data (used for both processes below) consists of ~3,800 rows of 20 individual features (ten for each team, plus a date, game id, home and away team, and home win/away win columns).

Model "A" cycles through specific date ranges for a Train/Test split. That is, I use hard cutoffs, so each set is contiguous date-wise. I use dates that reflect 50%, 55% etc up to 85% split. With this model, each of the 8 accuracy results are ~ 61%, which is typical for NFL predictions (maybe a bit on the high side).

Model "B" uses the train_test_split module from sklearn to split data in the same percentages as above. I realize this method shuffles the data, so I know each split is working with different data than model A. But using this method, the percent accuracy for each cycle is ~ 82%, which is WAY higher than model A, and seemingly unreasonable. So I'm looking to understand if I'm implementing one (or both) of these methods correctly.

(NOTE: I'm fairly new to Python and Machine Learning and both of these models are edited versions of other existing work). Any help would be greatly appreciated. And if someone can show me how to attach my source csv, I'd be happy to do so.

Relevant portion of Model A

...

train_test_date_range = ['2017-11-02', '2018-10-04', '2019-09-05', '2019-12-01', '2020-11-05', '2021-10-03', '2022-01-02', '2022-11-24']

features = [ 'team_1_average_ptsScored_last_five', 'team_1_average_feature_1_last_five',

FEATURE LIST ABBREVIATED FOR CLARITY

'team_2_average_feature_9_allowed_last_three_away',
'team_2_average_feature_10_allowed_last_three_away'


]

for tt_line_date in train_test_date_range :

train_data = match_stats[match_stats['date'] < tt_line_date]
test_data = match_stats[match_stats['date'] >= tt_line_date]

X_train = train_data[features]
X_test = test_data[features]
Y_train = train_data['team_1_result']
Y_test = test_data['team_1_result']

names = ["Log Reg"]

classifiers = [
LogisticRegression()
]

for name, clf in zip(names, classifiers):
# Fit the classifier on the training data and make predictions
clf.fit(X_train, Y_train)
test_data[name + '_team_1_result'] = clf.predict(X_test)
accuracy = clf.score(X_test, Y_test)

print(name + ' accuracy at T/T_Line ' + tt_line_date + ' =', format(accuracy, ".2%"))


...

Output from model "A" Log Reg accuracy at T/T_Line 2017-11-02 = 62.11% Log Reg accuracy at T/T_Line 2018-10-04 = 61.83% etc.

Relevant portion of model "B"

...

train_test_split_range = [0.5, 0.55, 0.6, 0.65, 0.7, 0.75, 0.8, 0.85]

for tt_line in train_test_split_range: X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=tt_line, random_state=42)

model = LogisticRegression()

scaler = StandardScaler()

scaled_train = scaler.fit_transform(X_train)
scaled_test = scaler.fit_transform(X_test)

model.fit(X_train, y_train)
y_pred_proba = model.predict_proba(X_train)[:, 1]

y_pred = model.predict(X_test)

accuracy = accuracy_score(y_test, y_pred)
precision = precision_score(y_test, y_pred)
recall = recall_score(y_test, y_pred)
f1 = f1_score(y_test, y_pred)
roc_auc = roc_auc_score(y_test, model.predict_proba(X_test)[:, 1])

# Calculate pseudo r squared
# Modified from: https://github.com/scikit-learn/scikit-learn/issues/25982
ll_null = log_loss(y_train, [y_train.mean()] * len(y_train))
ll_model = log_loss(y_train, y_pred_proba)
pseudo_r2 = 1 - ll_model / ll_null

# Print the metrics
print(f"\033[1m\033[4mResults at " + str(tt_line) + " train/test split\033[0m")
print(f"Accuracy: {accuracy:.3f}")
print(f"Precision: {precision:.3f}")
print(f"Recall: {recall:.3f}")
print(f"F1-score: {f1:.3f}")
print(f"ROC AUC: {roc_auc:.3f}")
print(f"Pseudo R-squared: {pseudo_r2:.3f}")
print()


...

Output from model "B" (1 of the 8, but typical) Results at 0.5 train/test split Accuracy: 0.820 Precision: 0.828 Recall: 0.854 F1-score: 0.841 ROC AUC: 0.896 Pseudo R-squared: 0.490

Continue reading...