[Python] How to calculate FPS in image classification?

Please, I would appreciate some help and will be grateful for any help.

I have some doubts about how to calculate the frame rate (FPS) in image classification for training an testing routines. I've already searched internet, github, chatgpt etc, but I'm still not sure and confident in the correct way to calculate FPS in image classification.

Here is the recursive routine use for calling the train and test fuctions:

# Treinamento e teste
train_losses = []
test_losses = []
train_acc = []
test_acc = []
train_times = []
test_times = []
training_times = []
testing_times = []

for epoch in tqdm(range(epochs)):
# Train the model
train_loss, train_accuracy, train_time, training_time = train(model, trainloader, criterion, optimizer, device)
# Evaluate the model
test_loss, test_accuracy, test_time, testing_time, initial_accuracy = test(model, testloader, criterion, device)

# Appened the loss, accuracy and execution time values
train_losses.append(train_loss)
test_losses.append(test_loss)
train_acc.append(train_accuracy)
test_acc.append(test_accuracy)
train_times.append(train_time)
test_times.append(test_time)
training_times.append(training_time)
testing_times.append(testing_time)
test_times.append(test_time)


Here is the TRAINING function:

def train(model, trainloader, criterion, optimizer, device):
model.train()
running_loss = 0.0 # keep track of running loss
correct = 0
total = 0

start_time = time.time()
for i, data in enumerate(trainloader, 0): # iterate over data
inputs, labels = data[0].to(device), data[1].to(device) # get inputs and labels

# zero the parameter gradients (otherwise they are accumulated)
optimizer.zero_grad()

# forward + backward + optimize
outputs = model(inputs) # forward pass
loss = criterion(outputs, labels) # compute loss
loss.backward() # backward pass
optimizer.step() # optimize
_, predicted = torch.max(outputs, 1)

# print statistics
total += labels.size(0)
correct += (predicted == labels).sum().item()
running_loss += loss.item()
#if i % 2000 == 1999: # print every 2000 mini-batches
# print('[%d, %5d] loss: %.3f' %
# (epoch + 1, i + 1, running_loss / 2000))
# running_loss = 0.0

training_time = time.time() - start_time # Total elapsed time
scheduler.step() # Adjust the learning rate based on the scheduler
train_loss = running_loss / len(trainloader)
train_accuracy = 100 * (correct / total)
execution_time = get_execution_time(start_time)
return train_loss, train_accuracy, execution_time, training_time


Here is the TESTING function:

def test(model, testloader, criterion, device):
running_loss = 0.0
correct = 0
total = 0
class_correct = list(0. for i in range(len(classes)))
class_total = list(0. for i in range(len(classes)))
accuracy_dict = {}

model.eval()
start_time = time.time()
with torch.no_grad(): # don't compute gradients
for data in testloader: # iterate over data
images, labels = data[0].to(device), data[1].to(device) # get inputs and labels
outputs = model(images) # get the network's predictions
loss = criterion(outputs, labels)
_, predicted = torch.max(outputs.data, 1) # get the class with the highest score

total += labels.size(0) # increment total by the number of labels
correct += (predicted == labels).sum().item() # increment correct by the number of correct predictions
running_loss += loss.item()

c = (predicted == labels).squeeze()
#for i in range(len(labels)):
# label = labels
# class_correct[label] += c.item() #erro
# class_total[label] += 1
for i in range(len(labels)):
label = labels
if (c.ndim == 0):
class_correct[label] += c.item()
else:
class_correct[label] += c.item()
class_total[label] += 1

testing_time = time.time() - start_time # Total elapsed time


test_loss = running_loss / len(testloader)
test_accuracy = 100 * (correct / total)
execution_time = get_execution_time(start_time)
return test_loss, test_accuracy, execution_time, testing_time, accuracy_dict


Thanks.

I am calculating in this way:

Training:

total_train_images = len(trainloader.dataset) * epochs # Total images processed
train_fps = total_train_images / sum(training_times) #if training_time > 0 else 0

print(f"total_train_images: {total_train_images: .2f}")
print(f"Total training_times: {sum(training_times): .2f} seconds")
print(f"Training FPS: {train_fps: .2f}")


Testing:

total_test_images = len(testloader.dataset) * epochs # Total number of images in test set
test_fps = total_test_images / sum(testing_times) #if testing_time > 0 else 0

print(f"total_test_images: {total_test_images: .2f}")
print(f"Total testing_times: {sum(testing_times): .2f} seconds")
print(f"Testing FPS: {test_fps: .2f}")


I have found other calcuations including the batch_size:

print("Train Throughput: {:.5f} (images/s)".format((len(trainloader) * epochs * batch_size) / sum(train_times)))
print("Test Throughput: {:.5f} (images/s)".format((len(testloader) * epochs * batch_size) / sum(test_times)))


Which one is right? Should I consider the number of epochs in multiplication for training and testing? What about the batch size? Thanks.

Continue reading...