[Python] Efficient way to get the minimal tuple from a set of columns in polars

What is the most efficient way in polars to do this:

import polars as pl
import numpy as np
rng = np.random.default_rng()

df = pl.DataFrame([
pl.Series('a', rng.normal(size=10_000_000)),
pl.Series('b', rng.normal(size=10_000_000)),
])

df.sort('a', 'b').head(1)


I.e. I want to find the smallest tuple of numbers (a, b) based on lexicographical ordering. Finding the minimum does not require a full sort, so the above code is very inefficient. I tried

df.lazy().sort('a', 'b').head(1).collect()


but this does not lead to a significant speedup. Creating a single column of type pl.Struct does not help either since polars does not seem to define a notion of ordering on structs.

Update


Thanks to the answer from ΩΠΟΚΕΚΡΥΜΜΕΝΟΣ I have now settled for the following solution:

def lexicographic_min(df):
columns = list(df.columns)
for col in columns:
if df.select(pl.col(col).is_null().any()).row(0)[0]:
df = df.filter(pl.col(col).is_null())
else:
df = df.filter(pl.col(col) == pl.col(col).min())
return df.row(0)

def lexicographic_max(df):
columns = list(df.columns)
for col in columns:
df = df.filter(pl.col(col) == pl.col(col).max())
return df.row(0)


This version handles null values by considering them 'smaller' than any non-null values and does not require any invocations of .sort().

Continue reading...