[Python] Identify changed directories in Object Storage since a specific datetime with Python

Having an s3 object storage, I want to know which directories in a base directory have changed since a give datetime.

It would work similar to get_changed_directories:

bucket_directory = "your_bucket_name/base_directory"
since_datetime = datetime(2023, 1, 1, tzinfo=timezone.utc)

changed_dirs = get_changed_directories(s3_client, bucket_directory, since_datetime)
>>> ["your_bucket_name/base_directory/subdir_1", "your_bucket_name/base_directory/subdir_2", "your_bucket_name/base_directory/subdir_4"]


The s3_client can be any client; for example boto3, but async aiboto3 or s3fs is usually faster.

notes

After some experiments, this seems to be the fastest method to list details in a directory:

s3_file.ls(directory, detail=True, refresh=True)

Continue reading...