I am implementing a simple Reinforcement Learning algorithm, namely Q Learning in a simple game setting. While the general ideas are clear, I have stumbled upon a technical detail, which is bothering me. I don't fully grasp the difference between the implementations with / without a history of states & actions. without history it would look like this: def play(...) action = player.select_optimal_action(...) next_state, reward, done = step(action) player.update_Q(state, action, next_state, reward, done) and update_Q would have something like reward = reward if done else reward + gamma * max(Q[next_state]) So basically an update of Q after each player turn. Now I have seen implementations which use a history of states and only backpropagate them when the terminal state is reached like def play(...) action = player.select_optimal_action(...) if has_terminated_after_action: player.collect_terminal_reward() and the player would have something like state_history = [] ... state_history.append(best_action) # at the end of select_optimal_action def collect_terminal_reward(...) ... for state, action in reversed(state_history) ... My question is about the equivalence of both approaches. Are the approaches the same if the terminal reward is not discounted? please see the pseudocode snippets Continue reading...
