[Python] How can I use Python Polars to stream the contents of a Parquet file as CSV text to...

Discussão em 'Python' iniciado por Stack, Dezembro 21, 2024.

  1. Stack

    Stack Membro Participativo

    Using Python Polars, how can I modify the following script to stream the contents of a Parquet file as CSV text to standard output?

    import polars as pl
    import sys
    pl.scan_parquet("BTCUSDT-trades-2022-01.parquet").sink_csv(sys.stdout)


    Python complains that Lazyframe.sink_csv expects a string argument and not a TextIOWrapper:

    Traceback (most recent call last):
    File "/mnt/storage/Data/Binance/Market Data/Polars/select_btcusdt_aggtrades.py", line 4, in <module>
    pl.scan_parquet("../BTCUSDT/2022/BTCUSDT-trades-2022-01.parquet").sink_csv(sys.stdout)
    File "/home/derek/.cache/uv/archive-v0/bOaA51uU_dQEF2peOOxqI/lib/python3.12/site-packages/polars/_utils/unstable.py", line 58, in wrapper
    return function(*args, **kwargs)
    ^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/home/derek/.cache/uv/archive-v0/bOaA51uU_dQEF2peOOxqI/lib/python3.12/site-packages/polars/lazyframe/frame.py", line 2717, in sink_csv
    path=normalize_filepath(path),
    ^^^^^^^^^^^^^^^^^^^^^^^^
    File "/home/derek/.cache/uv/archive-v0/bOaA51uU_dQEF2peOOxqI/lib/python3.12/site-packages/polars/_utils/various.py", line 225, in normalize_filepath
    path = os.path.expanduser(path) # noqa: PTH111
    ^^^^^^^^^^^^^^^^^^^^^^^^
    File "<frozen posixpath>", line 259, in expanduser
    TypeError: expected str, bytes or os.PathLike object, not TextIOWrapper


    (Note that I wish to stream the Parquet file and not read the entire file into a DataFrame in memory because each input file is quite large and there are twelve files for each year, so reading all of the the input data for a year would quickly consume all available memory in my laptop.)

    BTCUSDT-trades-2022-01.zip contains the trades in CSV format between Bitcoin and USDT on the Binance cryptocurrency exchange in January 2022. BTCUSDT-trades-2022-01.parquet contains this trade data in Parquet format.

    Continue reading...

Compartilhe esta Página