[Python] How can matrix be stored efficiently in Polars dataframe and performance matrix...

I have a lot of dates. For each date, there is a vector v (with length n) and a square matrix M (with dimension n by n). v, M and n will vary by date in terms of both values and lengths/dimensions.

The task is for each date, I want to perform a matrix operation to generate a constant scalar - transpose(v) * M * v.

The naïve way is to do this through a for loop, but the computation time will be huge given it is sequential.

I am wondering if I can store all information into a single Polars Dataframe such that I can do something like df.group_by("date").agg(...) which is parallel and efficient.

To give a concrete example:

• For date 2020-01-01, v = [1, 2], M = [[1, 2], [3, 4]], the indices for v and M are ["a", "b"].
• For date 2020-02-01, v = [1, 2, 3], M = [[1, 2, 3], [3, 4, 5], [4, 5, 6]], the indices for v and M are ["a", "b", "c"].
• For date 2020-03-01, v = [1, 3, 5], M = [[1, 5, 9], [2, 4, 6], [3, 6, 9]], the indices for v and M are ["b", "d", "e"].
• Result:

Continue reading...