I am trying to understand how the DQN algorithm with RNNs works in PyTorch's RL API through this tutorial. However, the way some of the classes handle episodes and batches during training are unclear to me. I noticed that SyncDataCollector stitches episodes together into the same TensorDict until it reaches the desired number of frames per batch. I am wondering if this means that during training, the DQNLoss treats all these stitched episodes as a single trajectory or if it knows to separate them back into individual episodes. If they are treated as one, wouldn't this mix different trajectories and affect learning? Additionally, does the replay buffer randomly sample states and actions without tracking which episode they belong to, potentially mixing unrelated data? Continue reading...
