[Python] Why is my models classification performance so much worse than its regression...

I'm comparing different models on their performance for breathing detection. For every model, i try to predict a continouous breathing signal as regression task as well as a binary classification for every time point (inhale or exhale), using video data as input, so seq-to-seq, 480 timesteps per sample. I train all models completely new for each test, with MSE for regression and (weighted) categorical cross entropy for classification. Now, usually this works quite good, meaning models that perform well/bad on regression task also perform well/bad on classification. The only difference in my models is the last fully connected linear layer, where for regression I put an output dim of 1 and for classification a 4.

However, for two models, the regression task performs very well, but the classification task performs quite bad. There still is some meaningful output but just much worse than expected.

After excluding all kinds of typical error sources, it seems they overfit on the classification task, but not on the regression task. Is this something to be expected? I tried a LOT around with weight clipping, decay, decreasing layer sizes, learning rates etc. etc., but everytime it either overfits or, after too much regularization, it does not learn anything (obviously). The only thing I can think of now is to pretrain on the regression task and then use the weights and only change the last layer; however, I was wondering if this behaviour is somewhat "typical" or some freak property of these nets?

The nets in questions are ordinary 3D-CNNs, one for example with a LSTM layer after the conv part and then a linear output layer; LSTMs (with some preprocessed feature sequences as input instead of images) and a video transformer work great, so it does not seem like a problem of data itself /size of dataset /model size.

I'm happy about every idea and opinion, also for literature on that matter!

Continue reading...