[Python] What is the fastest way to build/instantiate a Python Polars Dataframe from a...

I have a function that calls a loop that generates new rows of data, 4 at a time. I'm trying to turn all of the rows generated into a Polars dataframe as fast as possible.

My current code creates a list of rows (tuples) for the DataFrame, and extends it with each batch of new rows. Then, it instantiates a polars DataFrame from the list of the tuples.

Current code

rows = []

def make_df(x):
for data in x: # This loops ~500k times
new_rows = process(data) # 4 new rows created
# new_rows is a tuple of length 4
# Each element (row) of the tuple is another tuple of length 10
rows.extend(new_rows)

return pl.DataFrame(rows, orient='row') # This line is very slow (4-5 sec)


For my DataFrame (about 2 million rows, 10 columns), this is taking ~4-5 seconds just to instantiate the DF. This seems slow and I'm wondering if there's a faster way to do this. I've tried using a generator (example below), but it seems to take the same amount of time, maybe even a bit slower.

Generator (not working any faster)

def g(x):
for data in x: # This loops ~500k times
new_rows = process(data) # 4 new rows created
# new_rows is a tuple of length 4
# Each element of the tuple is another tuple of length 10
yield new_rows[0]
yield new_rows[1]
yield new_rows[2]
yield new_rows[3]

def make_df(x):
return pl.DataFrame(g(x), orient='row') # Slows down the loop by 5-6 seconds


Any faster way to do this?

Update

A reproducible example is below:

import polars as pl
import datetime
date = datetime.datetime(2024, 7, 14, 12, 30)
rows = []

def make_df():
for data in range(500000): # This loops ~500k times
new_row = (1, date, 3, 4.0, 5.0, 6.0, 7.0, 8.0, 9.0, date)
new_rows = ((new_row),) * 4 # 4 new rows created
# new_rows is a tuple of length 4
# Each element (row) of the tuple is another tuple of length 10
rows.extend(new_rows)

return pl.DataFrame(
rows,
orient='row',
schema={
'a': pl.Int64,
'b': pl.Datetime,
'c': pl.Int64,
'd': pl.Float64,
'e': pl.Float64,
'f': pl.Float64,
'g': pl.Float64,
'h': pl.Float64,
'i': pl.Float64,
'j': pl.Datetime,
},
) # This line is very slow (4-5 sec)

make_df()

Continue reading...