I have a device on 32-bit arm with custom page_size(32k) I could compile polars against armv7-unknown-linux-gnueabihf with mimalloc it works on simple examples, but on the code whole_data_df = pl.scan_csv( index_buffer_file_path, schema=schema_as_dict, has_header=False ) whole_data_df = whole_data_df.update( existing_data_df, on=index_column_name, how="left", include_nulls=False ) merged_data = whole_data_df.collect( engine="streaming" ) it looks like it cannot allocate heap: 2025-04-08T14:42:51.626749560Z mimalloc: warning: thread 0x55BF3250: unable to allocate aligned OS memory directly, fall back to over-allocation (size: 0x8000000 bytes, address: 0x4C768000, alignment: 0x400000 , commit: 1) 2025-04-08T14:42:51.626832340Z mimalloc: reserved 131072 KiB memory 2025-04-08T14:42:51.665466280Z mimalloc: reserved 131072 KiB memory 2025-04-08T14:42:51.704161900Z mimalloc: reserved 131072 KiB memory 2025-04-08T14:42:51.748628280Z mimalloc: reserved 131072 KiB memory 2025-04-08T14:42:51.788860500Z mimalloc: reserved 131072 KiB memory 2025-04-08T14:42:51.834474620Z mimalloc: reserved 131072 KiB memory 2025-04-08T14:42:51.874195640Z mimalloc: reserved 131072 KiB memory 2025-04-08T14:42:51.925063160Z mimalloc: reserved 262144 KiB memory 2025-04-08T14:42:55.804869000Z mimalloc: error: thread 0x55BF3250: unable to allocate memory (1345920 bytes) 2025-04-08T14:42:55.805469360Z memory allocation of 1345920 bytes failed the code works perfectly on x86_64 My ideas have ended, so maybe it just not possible I've tried using https://github.com/rust-cross/manylinux-cross/tree/main?tab=readme-ov-file image to cross-compile from source tried to compile py-polars with zig Thread 5 (Thread 0x530c7450 (LWP 23) "python"): #0 __libc_do_syscall () at ../sysdeps/unix/sysv/linux/arm/libc-do-syscall.S:47 #1 0x75ac6a20 in __libc_signal_restore_set (set=0x530c198c) at ../sysdeps/unix/sysv/linux/internal-signals.h:86 #2 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:48 #3 0x75ab7322 in __GI_abort () at abort.c:79 #4 0x6e650cdc in std::sys:al::unix::abort_internal () from /app/.venv/lib/python3.12/site-packages/polars/polars.abi3.so #5 0x6e64fe58 in std:rocess::abort () from /app/.venv/lib/python3.12/site-packages/polars/polars.abi3.so #6 0x6e651678 in std::alloc::rust_oom () from /app/.venv/lib/python3.12/site-packages/polars/polars.abi3.so #7 0x6e651698 in __rg_oom () from /app/.venv/lib/python3.12/site-packages/polars/polars.abi3.so --Type <RET> for more, q to quit, c to continue without paging-- #8 0x6e436b08 in __rust_alloc_error_handler () from /app/.venv/lib/python3.12/site-packages/polars/polars.abi3.so #9 0x6e43714c in alloc::alloc::handle_alloc_error () from /app/.venv/lib/python3.12/site-packages/polars/polars.abi3.so #10 0x6e437134 in alloc::raw_vec::handle_error () from /app/.venv/lib/python3.12/site-packages/polars/polars.abi3.so #11 0x71ccd77c in polars_plan:lans::conversion::expr_expansion::rewrite_projections::h0ef9bafe764f9a64 () from /app/.venv/lib/python3.12/site-packages/polars/polars.abi3.so #12 0x71cd41dc in <polars_plan:lans::iterator::ExprMapper<F> as polars_plan:lans::visitor::visitors::RewritingVisitor>::mutate () from /app/.venv/lib/python3.12/site-packages/polars/polars.abi3.so #13 0x71e48df0 in polars_plan:lans::visitor::visitors::TreeWalker::rewrite () from /app/.venv/lib/python3.12/site-packages/polars/polars.abi3.so #14 0x71ddbbf8 in polars_utils::functions::try_arc_map () from /app/.venv/lib/python3.12/site-packages/polars/polars.abi3.so #15 0x71e26154 in polars_plan:lans::visitor::expr::<impl polars_plan:lans::visitor::visitors::TreeWalker for polars_plan::dsl::expr::Expr>::map_children () from /app/.venv/lib/python3.12/site-packages/polars/polars.abi3.so #16 0x71e48d40 in polars_plan:lans::visitor::visitors::TreeWalker::rewrite () from /app/.venv/lib/python3.12/site-packages/polars/polars.abi3.so #17 0x71ccd884 in polars_plan:lans::conversion::expr_expansion::rewrite_projections::h0ef9bafe764f9a64 () from /app/.venv/lib/python3.12/site-packages/polars/polars.abi3.so #18 0x71cca3a8 in polars_plan:lans::conversion::expr_expansion:repare_projection () from /app/.venv/lib/python3.12/site-packages/polars/polars.abi3.so #19 0x71cc7b98 in polars_plan:lans::conversion::dsl_to_ir::resolve_with_columns () from /app/.venv/lib/python3.12/site-packages/polars/polars.abi3.so #20 0x71ce54b0 in polars_plan:lans::conversion::dsl_to_ir::to_alp_impl::_$u7b$$u7b$closure$u7d$$u7d$ --Type <RET> for more, q to quit, c to continue without paging-- ::h6b1c3ff6d8dd9278 () from /app/.venv/lib/python3.12/site-packages/polars/polars.abi3.so #21 0x71ce1b64 in polars_plan:lans::conversion::dsl_to_ir::to_alp_impl () from /app/.venv/lib/python3.12/site-packages/polars/polars.abi3.so #22 0x71ce3a84 in polars_plan:lans::conversion::dsl_to_ir::to_alp_impl::_$u7b$$u7b$closure$u7d$$u7d$::h6b1c3ff6d8dd9278 () from /app/.venv/lib/python3.12/site-packages/polars/polars.abi3.so #23 0x71ce1b64 in polars_plan:lans::conversion::dsl_to_ir::to_alp_impl () from /app/.venv/lib/python3.12/site-packages/polars/polars.abi3.so #24 0x71ce37b0 in polars_plan:lans::conversion::dsl_to_ir::to_alp_impl::_$u7b$$u7b$closure$u7d$$u7d$::h6b1c3ff6d8dd9278 () from /app/.venv/lib/python3.12/site-packages/polars/polars.abi3.so #25 0x71ce1b64 in polars_plan:lans::conversion::dsl_to_ir::to_alp_impl () from /app/.venv/lib/python3.12/site-packages/polars/polars.abi3.so #26 0x71cc6084 in polars_plan:lans::conversion::dsl_to_ir::to_alp () from /app/.venv/lib/python3.12/site-packages/polars/polars.abi3.so #27 0x71c5735c in polars_plan:lans:ptimizer:ptimize () from /app/.venv/lib/python3.12/site-packages/polars/polars.abi3.so #28 0x705e86a8 in polars_lazy::frame::LazyFrame:ptimize_with_scratch () from /app/.venv/lib/python3.12/site-packages/polars/polars.abi3.so #29 0x705e81c0 in polars_lazy::frame::LazyFrame::to_alp_optimized () from /app/.venv/lib/python3.12/site-packages/polars/polars.abi3.so #30 0x705e8e18 in polars_lazy::frame::LazyFrame::collect_with_engine () from /app/.venv/lib/python3.12/site-packages/polars/polars.abi3.so #31 0x73041e04 in polars_error::signals::catch_keyboard_interrupt () from /app/.venv/lib/python3.12/site-packages/polars/polars.abi3.so #32 0x72c74ae8 in <pyo3::marker:ython as polars_python::utils::EnterPolarsExt>::enter_polars () from /app/.venv/lib/python3.12/site-packages/polars/polars.abi3.so looks like nothing special actually - just trying to allocate vec and OOM. But it definitely OOMs way earlier than container limit is reached Continue reading...
