[Python] Why does my PyTorch model reach 100% accuracy quickly while the Keras version trains...

I'm implementing the same CNN architecture for binary classification on a filtered version of the FashionMNIST dataset (Coat vs Pullover) using:

1. keras package
2. pytorch package

Both models have the same structure:

• 2x convolutional layers
• List item
• Batch normalization
• Dropout
• Gaussian noise at the input
• List item
• A dense layer with 32 units
• Final sigmoid output

using Keras, I get this training result:

Epoch 1/5
270/270 - 5s - 18ms/step - accuracy: 0.6794 - loss: 1.1801 - val_accuracy: 0.5042 - val_loss: 1.1561

Epoch 2/5
270/270 - 2s - 6ms/step - accuracy: 0.7513 - loss: 0.8556 - val_accuracy: 0.7010 - val_loss: 0.7886

Epoch 3/5
270/270 - 2s - 6ms/step - accuracy: 0.7752 - loss: 0.6989 - val_accuracy: 0.7927 - val_loss: 0.6180

Epoch 4/5
270/270 - 2s - 6ms/step - accuracy: 0.7971 - loss: 0.5981 - val_accuracy: 0.8344 - val_loss: 0.5222

Epoch 5/5
270/270 - 2s - 6ms/step - accuracy: 0.8065 - loss: 0.5404 - val_accuracy: 0.8021 - val_loss: 0.5670


But using PyTorch, the model reaches ~100% accuracy within just 2–3 epochs:

Epoch 1/5 - Train Loss: 0.4841 - Train Acc: 0.7904 - Val Loss: 0.1818 - Val Acc: 1.0000

Epoch 2/5 - Train Loss: 0.0785 - Train Acc: 0.9972 - Val Loss: 0.0257 - Val Acc: 1.0000

Epoch 3/5 - Train Loss: 0.0226 - Train Acc: 1.0000 - Val Loss: 0.0131 - Val Acc: 1.0000

Epoch 4/5 - Train Loss: 0.0141 - Train Acc: 1.0000 - Val Loss: 0.0099 - Val Acc: 1.0000

Epoch 5/5 - Train Loss: 0.0122 - Train Acc: 1.0000 - Val Loss: 0.0092 - Val Acc: 1.0000


I'm confused because the models are architecturally identical. However, PyTorch seems to overfit extremely fast.

Here’s what I’ve already ensured:

• 
  Normalized pixel values to [0, 1]
  
  
• 
  Reshaped images to (N, 1, 28, 28) in PyTorch
  
  
• 
  Used BCELoss() with PyTorch and binary_crossentropy with Keras
  
  
• 
  Matched dropout rates and regularization (used weight_decay=0.01 in Adam)
  

Keras model implementation:

import tensorflow as tf
from tensorflow.keras import layers, regularizers

def simple_cnn_model():
model = tf.keras.Sequential([
layers.GaussianNoise(0.1), # Adds slight noise to improve robustness

layers.Conv2D(8, (3,3), activation='relu', input_shape=(28,28,1),
kernel_regularizer=regularizers.l2(0.01)),
layers.BatchNormalization(),
layers.MaxPooling2D((2,2)),
layers.Dropout(0.5), # Reduced dropout (0.6 → 0.5)

layers.Conv2D(16, (3,3), activation='relu',
kernel_regularizer=regularizers.l2(0.01)),
layers.BatchNormalization(),
layers.MaxPooling2D((2,2)),
layers.Dropout(0.5), # Reduced dropout (0.7 → 0.6)

layers.Flatten(),
layers.Dense(32, activation='relu', kernel_regularizer=regularizers.l2(0.01)),
layers.BatchNormalization(),
layers.Dropout(0.4), # Less dropout in Dense layer (0.7 → 0.5)

layers.Dense(1, activation='sigmoid') # 1 neuron for binary classification
])

model.compile(optimizer='adam',
loss='binary_crossentropy',
metrics=['accuracy'])
return model

# Create and compile model
model = simple_cnn_model()


model=simple_cnn_model()

history = model.fit(
x_train, y_train,
validation_split=0.1,
epochs=5,
batch_size=32,
shuffle=True,
verbose=2
)


Pytorch model implementation:

import torch.nn as nn
import torch.nn.functional as F

from sklearn.model_selection import train_test_split

x_train_final, x_val, y_train_final, y_val = train_test_split(
x_train, y_train, test_size=0.1, stratify=y_train, random_state=42
)

train_dataset = torch.utils.data.TensorDataset(torch.tensor(x_train_final), torch.tensor(y_train_final).float())
val_dataset = torch.utils.data.TensorDataset(torch.tensor(x_val), torch.tensor(y_val).float())

train_loader = DataLoader(train_dataset, batch_size=32, shuffle=True)
val_loader = DataLoader(val_dataset, batch_size=32, shuffle=False)



class SimpleCNN(nn.Module):
def __init__(self):
super(SimpleCNN, self).__init__()
self.noise_std = 0.1

self.conv1 = nn.Conv2d(1, 8, kernel_size=3, padding=0)
self.bn1 = nn.BatchNorm2d(8)
self.dropout1 = nn.Dropout(0.5)

self.conv2 = nn.Conv2d(8, 16, kernel_size=3, padding=0)
self.bn2 = nn.BatchNorm2d(16)
self.dropout2 = nn.Dropout(0.5)

self.fc1 = nn.Linear(16 * 5 * 5, 32) # 28x28 → 26x26 → 13x13 → 11x11 → 5x5
self.bn3 = nn.BatchNorm1d(32)
self.dropout3 = nn.Dropout(0.4)

self.fc2 = nn.Linear(32, 1)

def forward(self, x):
# Add Gaussian Noise manually
if self.training:
noise = torch.randn_like(x) * self.noise_std
x = x + noise

x = self.conv1(x)
x = F.relu(x)
x = self.bn1(x)
x = F.max_pool2d(x, 2)
x = self.dropout1(x)

x = self.conv2(x)
x = F.relu(x)
x = self.bn2(x)
x = F.max_pool2d(x, 2)
x = self.dropout2(x)

x = torch.flatten(x, 1)
x = self.fc1(x)
x = F.relu(x)
x = self.bn3(x)
x = self.dropout3(x)

x = self.fc2(x)
return torch.sigmoid(x)


# Convert numpy arrays to PyTorch tensors
train_tensor = torch.utils.data.TensorDataset(torch.tensor(x_train), torch.tensor(y_train).float())
train_loader = DataLoader(train_tensor, batch_size=32, shuffle=True)
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

model = SimpleCNN().to(device)
optimizer = torch.optim.Adam(model.parameters(), lr=0.001, weight_decay=0.01) # matches Keras regularization
criterion = nn.BCELoss()

for epoch in range(5):
# ---- Training ----
model.train()
running_loss = 0.0
correct = 0
total = 0

for inputs, labels in train_loader:
inputs, labels = inputs.to(device), labels.to(device).unsqueeze(1)
optimizer.zero_grad()
outputs = model(inputs)
loss = criterion(outputs, labels)
loss.backward()
optimizer.step()

running_loss += loss.item() * inputs.size(0) # multiply by batch size
predicted = (outputs > 0.5).float()
correct += (predicted == labels).sum().item()
total += labels.size(0)

train_loss = running_loss / total
train_acc = correct / total

# ---- Validation ----
model.eval()
val_loss = 0.0
val_correct = 0
val_total = 0

with torch.no_grad():
for inputs, labels in val_loader:
inputs, labels = inputs.to(device), labels.to(device).unsqueeze(1)
outputs = model(inputs)
loss = criterion(outputs, labels)

val_loss += loss.item() * inputs.size(0)
predicted = (outputs > 0.5).float()
val_correct += (predicted == labels).sum().item()
val_total += labels.size(0)

val_loss /= val_total
val_acc = val_correct / val_total

print(f"Epoch {epoch+1}/5 - Train Loss: {train_loss:.4f} - Train Acc: {train_acc:.4f} - Val Loss: {val_loss:.4f} - Val Acc: {val_acc:.4f}")


Is there something I'm missing in terms of how the frameworks handle:

• Loss computation
• BatchNorm and Dropout behavior
• Validation split vs. full training loop
• Regularization
• etc

Any tips for making the PyTorch training behave more realistically like the Keras version?

Continue reading...