I have just seen some code which has sparked my interest. with tf.GradientTape() as g: y = f(x) dy_dx = g.gradient(y, x) (Code loosely taken from this reference.) There are two things which I think are interesting about this: The use of a context manager The fact that when the context manager has closed the gradient value(s) are available What interests me is understanding the software behind what might look like "magic code". You might ask questions such as why is there a with keyword involved here? if I evaluated f(x), then how is it that a value for a gradient (which is a different function to f) become available? I think the context manager is fairly easy to figure out. It causes the magic dunder functions __enter__ and __exit__ to be triggered, and so I am fairly confident this is a convenient way of marking where the beginning and end of gradient calculation. I would guess this is to make the overall code more efficient - there is no point calculating gradients if they are not needed, and therefore we need a way to mark where we should start and finish gradient calculations. The API could just as well have exposed a .start() and .stop() function. The second point is less easy to understand. My initial assumption was that perhaps the gradients were being approximated numerically. However, after doing some research I understand this is not the case, and that something called Automatic Differentiation is used instead. Numerical methods suffer from numerical errors due to finite precision, so it is not particularly surprising that a potentially numerically unstable algorithm is not used here. What I don't understand is how to connect the dots. A context manager is used to mark the beginning and end of where some gradient calculations must be performed, and those calculations are calculated using Automatic Differentiation. I don't think I have a particularly intuitive understanding of how Automatic Differentiation works, either. I understand that the chain rule is used, but that it is not symbolic differentiation. So what does Tensorflow do here? Does it effectively have a big list of if-elseif statements to calculate the right function? For example, if Tensorflow sees an expression like matmul(x, y) does it just contain logical rules which say that d_matmul_d_arg1 = arg2, and d_matmul_d_arg2 = arg1? And then to actually calculate the value of the gradient, does it just run the calculation for d_matmul_d_arg1(arg2) in the same way that it would calculate the value of matmul(x, y) with x=..., y=... using the usual numerical algorithm for matrix multiplication? Basically what is it doing, and how does it work at a low level? Continue reading...
