[Python] Why is the Panda's apply function so slow when iterating over an entire row, rather...

My intuition when using Pandas is that, if you have to use df.apply, it would be more optimal to group all the apply operations into one call. This was further reinforced by me learning that NumPy stores data in row-major form, and thus, calling apply across an entire row would benefit from spatial temporality. However, in my experience, it is significantly slower to combine apply into one function, and it is more optimal to invoke apply separately on each column.

The below code on my machine takes 0.0383 seconds when apply-ing across an entire row at once, but 0.0013 seconds when calling apply for each column of interest, a ~30x increase.

import pandas as pd
import numpy as np
import time

df = pd.DataFrame(
data = np.random.randint(0, 100, size=(1000, 5)).astype(float),
columns = [f'col{i}' for i in range(1, 6)]
)

def apply_row(row):
row['col1'] = row['col1'] * 2 + 3
row['col2'] = row['col2'] + 5
row['col3'] = row['col3'] - 1
row['col4'] = row['col4'] / 2
row['col5'] = row['col5'] ** 2
return row

start = time.time()
df.apply(apply_row, axis=1)
end = time.time()
print(f"Row-based time: {end - start} seconds")

start = time.time()
df['col1'] = df['col1'].apply(lambda x: x* 2 + 3)
df['col2'] = df['col2'].apply(lambda x: x + 5)
df['col3'] = df['col3'].apply(lambda x: x - 1)
df['col4'] = df['col4'].apply(lambda x: x / 2)
df['col5'] = df['col5'].apply(lambda x: x ** 2)
end = time.time()
print(f"Column-based time: {end - start} seconds")


I realize, of course, the operations in the above snippet don't require apply. But the point remains, and, I imagine that string operations see even more significant performance differences.

For reference, this is with Python 3.13.5, Pandas 2.3.1, NumPy 2.3.2.

Continue reading...