[Python] Pytorch compute graph don't release as expected

import torch

def analyze_your_actual_code(): """分析你的实际代码"""

a = torch.randn(4, requires_grad=True)

print(f"a: {a}")
print(f"a.grad: {a.grad}")

loss = a.sum()
loss.backward()
print(f" a.grad: {a.grad}")

loss.backward()
print(f" a.grad: {a.grad}")

analyze_your_actual_code()


Why I try to call loss.backward twice, but the gradients still accumulate. The compute graph don't release as expected?

Code output:

a: tensor([-0.3121, -0.2331, 0.9317, -0.5075], requires_grad=True)

a.grad: None a.grad: tensor([1., 1., 1., 1.])

a.grad: tensor([2., 2., 2., 2.])

Continue reading...