I am currently working with larger-than-ram data. Therefore, I am relying on the Streaming engine to run all operations. The problem is that, given the complexity of the process, my kernel crashes (silently). To address this, I collect some intermediate steps, to convert it right after to LazyFrame. I have found that for my processes, a better approach would be to sink_parquet and then scan_parquet, as in this way I avoid writing the intermediate DataFrame to disk after collecting -and before turning it a lazyframe-. The problem I face is that this process "pollutes" my storage system. Therefore I am wondering if there is any way to sink-scan a temporary parquet file, avoiding writing any intermediate heavy dataframe. Continue reading...
