I'm trying to flatten a DynamoDB JSON file with Polars. My JSON file is 1GB, but in memory when I read it, it uses 16GB. Because the data comes from DynamoDB, my columns can have multiple types. 1 year ago it was a String, today it's a boolean for example. I try to build 2 files, with 2000 rows. The first one, all columns have 1 type. The second one, I can have multiple type, as my real use case. For the same volume, the second file took less space in memory. Can you help me to understand why I have this difference ? How Polars manage his memory ? Continue reading...
