[Python] Is there a perfomance benifit in Polars in regards to parallelism using select over...

Is there a difference in regards to parallelism in polars when comparing these two methods which gives the same result? Method 1 assings in select is an intermediate step which shows that x_1 and x_2 can be calculated in parallel. Method 2 just uses a full expression. It is both in lazy-execution allowing for optimizations.

import polars as pl

df = pl.DataFrame({
"a": pl.int_range(0,100_000_000,eager=True),
"b": pl.int_range(0,100_000_000,eager=True),
"c": pl.int_range(0,100_000_000,eager=True)}).lazy()

# Method 1
(
df.select(
x_1 = pl.col("a") + pl.col("b"),
x_2 = pl.col("b")+ pl.col("c"))
.select(target = pl.col("x_1") + pl.col("x_2"))
.collect()
)

# Method 2
(
df.select(
target = (pl.col("a") + pl.col("b")) + (pl.col("b")+ pl.col("c"))
)
.collect()
)




I have tried timing and Method 2 comes out on-top but I'm unsure why. So do you know if parallelism happens in the calculation in Method 2 or is that inheirtly single-threaded?

Continue reading...