[Python] Polars creates copies of series in each row of a column

I am disappointed at Polars or maybe just my own misunderstanding on the copy-on-write principle:

import numpy as np
import polars as pl
np.random.seed(123)

x = pl.DataFrame({
'a': np.array([0, 1, 2], dtype = np.uint32),
'b': [
pl.Series(np.random.uniform(size = int(1e6))),
pl.Series(np.random.uniform(size = int(1e6))),
pl.Series(np.random.uniform(size = int(1e6)))
]
})

y = pl.DataFrame({
'a': np.random.choice(3, size = 30).astype(np.uint32)
})

xy = x.join(y, on = "a", how = "inner")
xy.estimated_size() / 1e6 # About 240 MB, not 1e6 * 3 * 8 / 1e6 = 24MB
with open('../temp/tmp.pkl', 'wb') as o:
pickle.dump(xy, o) # The size of the pickle file is also 240 MB

z = x['b'][y['a']] # A Polars Series of 30 list[f64]
z.estimated_size() / 1e6 # About 240 MB, not 1e6 * 3 * 8 / 1e6 = 24 MB
with open('../temp/z.pkl', 'wb') as o:
pickle.dump(z, o) # The size of the pickle file is also 240 MB

w = x['b'].to_numpy()[y['a']] # A numpy array of 30 objects.
with open('../temp/w.pkl', 'wb') as o:
pickle.dump(w, o) # The size of the pickle file is only 24 MB

p = pl.DataFrame({
'w': w
})
p.estimated_size() / 1e6 # About 0.00024 MB.
with open('../temp/p.pkl', 'wb') as o:
pickle.dump(p, o) # Exception: Cannot be pickled.


I want to have a Polars column just like w (the numpy array), where it basically contains only 3 concrete blocks of data and the rest 27 elements behave like references.

Is this possible?

Continue reading...