[Python] Choice of Activation Function for Output Layer in Supervised Neural Network...

I'm working on a supervised neural network regression task where my target variable typically resembles the following pattern:

Click to see the target variable image here

I'm currently evaluating different architectures and approaches, and I'm particularly unsure about the best choice for the activation function in the output layer. Here are my considerations:

1. 
   Linear Activation: My initial thought was to use a linear activation function since regression tasks often require outputting a continuous range of values. However, I'm not entirely convinced this is the best approach.
   
   
2. 
   Other Activations: I've also read that using non-linear activation functions (like ReLU or sigmoid) might offer certain benefits in specific scenarios, but I'm unclear when that would be appropriate for regression.
   
   
3. 
   No Activation: I’ve seen suggestions to leave the output layer unactivated, effectively treating it as a linear function.
   

Given the nature of my target signal and the goal of my regression task, could anyone provide insights on the following:

Which activation function (linear, non-linear, or none) is generally preferred for regression problems? Are there specific scenarios or characteristics of the target distribution that would favor one option over the others? What considerations should I keep in mind when making this decision? Any insights or experiences would be greatly appreciated!

Continue reading...