[Python] Why does my CNN model's loss keep increasing when training on traffic sign dataset?

I’m training a CNN model for traffic sign classification and encountering an issue where the loss keeps increasing instead of decreasing with each epoch.

GitHub repository with dataset and code:
https://github.com/bilenevgin/traffic-sign-classifier

Highlights:

• Training loss starts around ~0.00307 and increases
• Validation loss also increases.
• Accuracy stays very low throughout.

Environment & Setup:

• 
  Framework: Pytorch
  
  
• 
  Dataset: Custom traffic sign images (~40 classes), included in the repo
  
  
• 
  Model: CNN (Conv2D, MaxPooling, Flatten, Dense)
  
  
• 
  Loss Function: Categorical Crossentropy
  
  
• 
  Optimizer: Adam (lr=0.001)
  
  
• 
  Batch Size: 32
  
  
• 
  Epochs: 20
  
  
• 
  Reduced the learning rate to 0.0001.
  
  
• 
  Switched optimizer from Adam to SGD.
  

Expected outcome: Loss should decrease gradually as the model learns, and accuracy should improve over epochs. However, the loss continues to increase.

Continue reading...