I need to calculate the size of all top-level folders (i.e., non-nested folders) inside an Azure Data Lake Gen2 container. My storage account contains millions of blobs and approximately 300 TB of data. Current Setup: I'm splitting the data by container and top-level folders. I'm using Databricks with dbutils.fs.ls() and multi-threading. However, due to ABFSS listing limitations (returns only 1000 blobs per call), performance is a major bottleneck. I've also tried the Data Lake Gen2 SDK and PowerShell, but both approaches are too slow. Azure Monitor / Log Analytics is not enabled, so please do not suggest that. Question: What is the most efficient way to get the size of top-level folders in Azure Data Lake Gen2, given the scale of the data, without hitting the listing limits or suffering from long execution times? Continue reading...
