I'm testing the following script using pytest-memray: import pytest import pandas as pd import polars as pl @pytest.fixture def path() -> str: return "files/rows.csv" @pytest.fixture def path_xlsx() -> str: return "files/bankdataset.xlsx" def test_load_pandas(path: str): pd.read_csv(path) def test_load_polars(path: str): pl.scan_csv(path, low_memory=True).collect() def test_load_pandas_xlsx(path_xlsx: str): pd.read_excel(path_xlsx, sheet_name=None) def test_load_polars_xlsx(path_xlsx: str): pl.read_excel(path_xlsx, sheet_name=None) def test_load_pandas_partial(path: str): df = pd.read_csv(path, nrows=20) assert len(df) == 20 def test_load_polars_partial(path: str): df = pl.scan_csv(path).head(n=20).collect() assert len(df) == 20 def test_load_pandas_partial_xlsx(path_xlsx: str): df = pd.read_excel(path_xlsx, nrows=20) assert len(df) == 20 def test_load_polars_partial_xlsx(path_xlsx: str): df = pl.read_excel(path_xlsx).head(n=20) assert len(df) == 20 files/rows.csv is around 30k rows, and files/bankdataset.xlsx is 1 million rows. This outputs: ================================================================================================================ MEMRAY REPORT ================================================================================================================ Allocation results for tests/test_load.py::test_load_polars_xlsx at the high watermark Total memory allocated: 473.5MiB Total allocations: 21 Histogram of allocation sizes: |▁█▆ | Biggest allocating functions: - load_sheet_eager:/home/monopoly/workspace/toys/sheetz/.venv/lib/python3.13/site-packages/fastexcel/__init__.py:424 -> 473.4MiB - _call_with_frames_removed:<frozen importlib._bootstrap>:488 -> 20.4KiB - read_excel:/home/monopoly/workspace/toys/sheetz/.venv/lib/python3.13/site-packages/fastexcel/__init__.py:514 -> 13.9KiB - _compile_bytecode:<frozen importlib._bootstrap_external>:784 -> 9.6KiB - inner:/home/linuxbrew/.linuxbrew/opt/python@3.13/lib/python3.13/typing.py:429 -> 9.0KiB Allocation results for tests/test_load.py::test_load_polars_partial_xlsx at the high watermark Total memory allocated: 473.4MiB Total allocations: 7 Histogram of allocation sizes: |█ | Biggest allocating functions: - load_sheet_eager:/home/monopoly/workspace/toys/sheetz/.venv/lib/python3.13/site-packages/fastexcel/__init__.py:424 -> 473.4MiB - read_excel:/home/monopoly/workspace/toys/sheetz/.venv/lib/python3.13/site-packages/fastexcel/__init__.py:514 -> 13.9KiB - _read_spreadsheet:/home/monopoly/workspace/toys/sheetz/.venv/lib/python3.13/site-packages/polars/io/spreadsheet/functions.py:684 -> 536.0B Allocation results for tests/test_load.py::test_load_pandas_xlsx at the high watermark Total memory allocated: 426.2MiB Total allocations: 871 Histogram of allocation sizes: | █▂ | Biggest allocating functions: - feed:/home/linuxbrew/.linuxbrew/opt/python@3.13/lib/python3.13/xml/etree/ElementTree.py:1291 -> 245.1MiB - parse_cell:/home/monopoly/workspace/toys/sheetz/.venv/lib/python3.13/site-packages/openpyxl/worksheet/_reader.py:244 -> 63.0MiB - maybe_infer_to_datetimelike:/home/monopoly/workspace/toys/sheetz/.venv/lib/python3.13/site-packages/pandas/core/dtypes/cast.py:1198 -> 39.3MiB - _rows_to_cols:/home/monopoly/workspace/toys/sheetz/.venv/lib/python3.13/site-packages/pandas/io/parsers/python_parser.py:1066 -> 38.3MiB - _infer_types:/home/monopoly/workspace/toys/sheetz/.venv/lib/python3.13/site-packages/pandas/io/parsers/base_parser.py:720 -> 15.3MiB Allocation results for tests/test_load.py::test_load_polars at the high watermark Total memory allocated: 172.5MiB Total allocations: 92 Histogram of allocation sizes: |█ | Biggest allocating functions: - collect:/home/monopoly/workspace/toys/sheetz/.venv/lib/python3.13/site-packages/polars/lazyframe/frame.py:2332 -> 96.0B Allocation results for tests/test_load.py::test_load_pandas at the high watermark Total memory allocated: 24.6MiB Total allocations: 25 Histogram of allocation sizes: |█ ▁| Biggest allocating functions: - read:/home/monopoly/workspace/toys/sheetz/.venv/lib/python3.13/site-packages/pandas/io/parsers/c_parser_wrapper.py:234 -> 18.6MiB - __init__:/home/monopoly/workspace/toys/sheetz/.venv/lib/python3.13/site-packages/pandas/io/parsers/c_parser_wrapper.py:93 -> 6.0MiB - get_handle:/home/monopoly/workspace/toys/sheetz/.venv/lib/python3.13/site-packages/pandas/io/common.py:873 -> 4.0KiB - _clean_options:/home/monopoly/workspace/toys/sheetz/.venv/lib/python3.13/site-packages/pandas/io/parsers/readers.py:1688 -> 1.5KiB - read_csv:/home/monopoly/workspace/toys/sheetz/.venv/lib/python3.13/site-packages/pandas/io/parsers/readers.py:1009 -> 1.5KiB It appears that pandas beats polars in every aspect as far as memory efficiency. Is there something that I'm doing wrong? I'm trying to take advantage of LazyFrames with pl.scan_csv, but even that doesn't seem to help. Continue reading...
