What is the appropriate way to implement a differentiable variant of L0 regularizer (count the non-zero values in a Conv layer / matrix) to keras layer? I was thinking of using r(x) = tanh(abs(f*x)) for x the matrix, where f is some factor that scales the range in which the result will be considered small (and tan(0.5) ~ 0.55, so for f=1 : r(0.55)=0.5). The loss + regularizer result will be simply added for the training loop? Is the result of the regularizer function divided by the size of the matrix (to get a mean, like the loss function)? I have not seen that in the example L1 or L2 class, there it is the sum. I want to have a function independent of layer size. https://keras.io/api/layers/regularizers/#creating-custom-regularizers Continue reading...
