I am working on project where I get time series data on change from a different database and dumping via delta-rs to a azure storage account data lake gen 2. I am currently running Z-Order and vaccum every 20 iterations and then resuming. Main question was does z-order load all data for optimising? Currently the data isn't that much but soon over time it will grow very large. My Schema is given below and I z-order based on all of those columns. {"id":int,"name":string,"timestamp":datetime.datetime } Also are optimise operations acid complaint? What if I optimise via another job while I'm still appending every minute to the table Continue reading...
