I have a script to load individual files into xarray for a custom data format which works fine when loading in individual files, however the moment I try to either load using open_mfdataset() or combine_by_coords() it falls over. Here's the minimal example I can produce: paths = pathlib.Path("/simulations").glob("*.sdf") datasets = [] for path in paths: ds = xr.open_dataset(path, chunks={}) datasets.append(ds) # Fails here when trying to combine by coords due to large amount of data being loaded into RAM combined = xr.combine_by_coords(datasets) There are quite a lot of coordinates in each file: "X_Grid": 2501, "Y_Grid": 10001, "X_Grid_mid": 2500, "Y_Grid_mid": 10000, "Px_px_py_Electron": 1000, "Py_px_py_Electron": 1000, "X_x_px_Electron": 2500, "Px_x_px_Electron": 1000, "Px_px_py_Electron_mid": 999, "Py_px_py_Electron_mid": 999, "X_x_px_Electron_mid": 2499, "Px_x_px_Electron_mid": 999, Each of these dimensions is no more than 20kB on average. Each file is approximately 314MB and I'm only loading in 2 files in this example. How can I load in these files and combine across these dimensions without them ballooning in size and leading to out of memory issues? Running this on my supercomputing cluster leads to the error: numpy._core._exceptions._ArrayMemoryError: Unable to allocate 7.27 TiB for an array with shape (999, 1000, 1000, 1000, 1) and data type float64 Continue reading...
