[Python] RuntimeError: Trying to backward through the graph a second time on loss tensor

I have the following training code. I am quite sure I call loss.backward() just once, and yet I am getting the error from the title. What am I doing wrong? Note that the X_train_tensor is output from another graph calculation, so it has required_grad=True as you can see in the print statement. Is this the source of the problem, and if so, how can I change it? It won't allow me to toggle it directly on the tensor.

for iter in range(max_iters):
start_ix = 0
loss = None

while start_ix < len(X_train_tensor):
loss = None
end_ix = min(start_ix + batch_size, len(X_train_tensor))
out, loss, accuracy = model(X_train_tensor[start_ix:end_ix], y_train_tensor[start_ix:end_ix])

# every once in a while evaluate the loss on train and val sets
if (start_ix==0) and (iter % 10 == 0 or iter == max_iters - 1):
out_val, loss_val, accuracy_val = model(X_val_tensor, y_val_tensor)
print(f"step {iter}: train loss={loss:.2f} train_acc={accuracy:.3f} | val loss={loss_val:.2f} val_acc={accuracy_val:.3f} {datetime.datetime.now()}")


optimizer.zero_grad(set_to_none=True)
print (iter, start_ix, X_train_tensor.requires_grad, y_train_tensor.requires_grad, loss.requires_grad)
loss.backward()
optimizer.step()
start_ix = end_ix + 1


This is the error:

RuntimeError: Trying to backward through the graph a second time (or directly access saved tensors after they have already been freed). Saved intermediate values of the graph are freed when you call .backward() or autograd.grad(). Specify retain_graph=True if you need to backward through the graph a second time or if you need to access saved tensors after calling backward.


Update: this is where the model input tensors are coming from, as output of other (autoencoder) model:

autoencoder.eval()
with torch.no_grad(): # it seems like adding this line solves the problem?
X_train_encoded, loss = autoencoder(X_train_tensor)
X_val_encoded, loss = autoencoder(X_val_tensor)
X_test_encoded, loss = autoencoder(X_test_tensor)


Adding the with torch.no_grad() line above solves the issue, but I don't understand why. Does it actually change how the outputs are generated, how does that work?

Continue reading...