[Python] Why `.first()`, and why before `.over()`, in `with_columns` expression function...

new to Polars, seeking help understanding why part of the function composition for the expression in the .with_columns() snippet below has to be done in that particular order.

Specifically, I don't understand why the .first() call is necessary, nor (given that it appears to be necessary) why it must come before the .over("instance") call.

(The goal of the code is to compute the factor by which each instance's value differs from the value of the same instance in category A; yes, the result is irrelevant for rows in category A.)

>>> df = pl.DataFrame({"category": ["A", "A", "A", "A", "B", "B", "B", "B", "C", "C", "C", "C", "D", "D", "D", "D"], "instance": [1,2,3,4,1,2,3,4,1,2,3,4,1,2,3,4], "value": [7,444,0,99,50,60,70,80,90,80,70,60,50,60,70,80]})
>>> df
shape: (16, 3)
┌──────────┬──────────┬───────┐
│ category ┆ instance ┆ value │
│ --- ┆ --- ┆ --- │
│ str ┆ i64 ┆ i64 │
╞══════════╪══════════╪═══════╡
│ A ┆ 1 ┆ 7 │
│ A ┆ 2 ┆ 444 │
│ A ┆ 3 ┆ 0 │
│ A ┆ 4 ┆ 99 │
│ B ┆ 1 ┆ 50 │
│ B ┆ 2 ┆ 60 │
│ B ┆ 3 ┆ 70 │
│ B ┆ 4 ┆ 80 │
│ C ┆ 1 ┆ 90 │
│ C ┆ 2 ┆ 80 │
│ C ┆ 3 ┆ 70 │
│ C ┆ 4 ┆ 60 │
│ D ┆ 1 ┆ 50 │
│ D ┆ 2 ┆ 60 │
│ D ┆ 3 ┆ 70 │
│ D ┆ 4 ┆ 80 │
└──────────┴──────────┴───────┘
>>> df.with_columns(
... (
... pl.col("value")
... .filter(pl.col("category") == "A")
... .first()
... .over("instance")
... / pl.col("value")
... ).alias("factor")
... )
shape: (16, 4)
┌──────────┬──────────┬───────┬──────────┐
│ category ┆ instance ┆ value ┆ factor │
│ --- ┆ --- ┆ --- ┆ --- │
│ str ┆ i64 ┆ i64 ┆ f64 │
╞══════════╪══════════╪═══════╪══════════╡
│ A ┆ 1 ┆ 7 ┆ 1.0 │
│ A ┆ 2 ┆ 444 ┆ 1.0 │
│ A ┆ 3 ┆ 0 ┆ NaN │
│ A ┆ 4 ┆ 99 ┆ 1.0 │
│ B ┆ 1 ┆ 50 ┆ 0.14 │
│ B ┆ 2 ┆ 60 ┆ 7.4 │
│ B ┆ 3 ┆ 70 ┆ 0.0 │
│ B ┆ 4 ┆ 80 ┆ 1.2375 │
│ C ┆ 1 ┆ 90 ┆ 0.077778 │
│ C ┆ 2 ┆ 80 ┆ 5.55 │
│ C ┆ 3 ┆ 70 ┆ 0.0 │
│ C ┆ 4 ┆ 60 ┆ 1.65 │
│ D ┆ 1 ┆ 50 ┆ 0.14 │
│ D ┆ 2 ┆ 60 ┆ 7.4 │
│ D ┆ 3 ┆ 70 ┆ 0.0 │
│ D ┆ 4 ┆ 80 ┆ 1.2375 │
└──────────┴──────────┴───────┴──────────┘
>>>


To explore, I've tried switching the ordering to be [...].over("instance").first()[...], which seems more intuitive to me, but then receive a ShapeError. I've also tried .item() instead of .first(), which seems better since it guarantees only one value exists---a guarantee I'd love to have---but unfortunately .item() can only be called on a DataFrame and not within an expression. (Is there a way to include such a guarantee "inline" within the expression?) I've read the documentation of each of the functions, but I having trouble putting it all together conceptually.

Continue reading...