[Python] How to update a leaf variable in PyTorch

I am trying to implement simple gradient descent to find the root of a quadratic equation using PyTorch. I'm doing this to get a better sense of how the autograd function works but it's not going very well. Let's say that I want to find the roots of

y = 3x^2 + 4x + 9

as a random example. Below was my first attempt to run one step of gradient descent and re-calculate the gradient:

import torch

# Step size
alpha = 0.1

# Random starting point
x = torch.tensor([42.0], requires_grad=True)

# Function y = 3 * x^2 + 4 * x + 9
# Find the minimum of this with gradient descent
y = 3 * x ** 2 + 4 * x + 9
y.backward()
print(x.grad)

with torch.no_grad():
x -= alpha * x.grad

y.backward()
print(x.grad)


This didn't like calling .backward() multiple times, so I updated it to this:

import torch

# Step size
alpha = 0.1

# Random starting point
x = torch.tensor([42.0], requires_grad=True)

# Function y = 3 * x^2 + 4 * x + 9
# Find the minimum of this with gradient descent
y = 3 * x ** 2 + 4 * x + 9
y.backward(retain_graph=True) # <--- Change here
print(x.grad)

with torch.no_grad():
x -= alpha * x.grad

y.backward(retain_graph=True) # <--- And here
print(x.grad)


and I get the error:

RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.FloatTensor [1]] is at version 1; expected version 0 instead. Hint: enable anomaly detection to find the operation that failed to compute its gradient, with torch.autograd.set_detect_anomaly(True).


I get the sense that I am fundamentally misunderstanding something about PyTorch here. How would I do this correctly?

Continue reading...