I am working on an online softmax calculation and have implemented two different recursion methods. The first one is the original implementation of online softmax, while the second method is a variant that I discovered. Through testing, I found that both methods produce equivalent results, but I am struggling to understand the underlying reason for their equivalence. Here are the two recursive methods for reference: import numpy as np # First recursion method def recursion_method_1(z): n = len(z) d = np.zeros(n) m = np.zeros(n) # Initialize m_1 and d_1 m[0] = z[0] d[0] = np.exp(z[0] - m[0]) for k in range(1, n): m[k] = max(z[k], m[k-1]) d[k] = d[k-1] * np.exp(m[k-1] - m[k]) + np.exp(z[k] - m[k]) return d # Second recursion method def recursion_method_2(z): n = len(z) d = np.zeros(n) m = np.zeros(n) # Initialize m_1 and d_1 m[0] = z[0] d[0] = np.exp(z[0] - m[0]) for k in range(1, n): m[k] = max(z[k], m[k-1]) d[k] = d[k-1] * np.exp(z[k-1] - m[k]) + np.exp(z[k] - m[k]) return d The key difference between the two methods is that in the second method, I replaced the term $m_{k-1}$ with $z_{k-1}$ in the recurrence formula, as suggested by the original online softmax approach (see paper). I've compared the outputs using various inputs and confirmed that both methods yield the same results. Despite this, I can't seem to pinpoint why these two recursion approaches are mathematically equivalent. Does anyone have insight into this? Is there a deeper mathematical principle that explains why these two methods produce the same outcome? Continue reading...
