What is a 100% secure way to move files between S3 buckets with boto3 in Python, without risking unwanted copies, and without relying on stuff outside of the code base (e.g., credential files on disk)? Anybody ever implemented a database transaction concept, that either successfully completes ALL necessary steps, or does a full rollback and NO action is being taken? Or other ways to guarantee 'move or nothing'? The challenge There is no straight-out move command in boto3. Lots has been written/discussed about how to implement moving objects between buckets. Common ideas often fall into 2 categories (also discussed here): A sequence of copy and delete [s3.meta.client.copy(...) and client.delete_object(...)] Using an AWS CLI command create_clidriver().main('s3 mv ...') Both approaches have a major downside: Copying & deleting is not a SAFE way to move. If the copy is successful, but the deletion fails (for whatever reason: connection drops, not permitted, typo in kwargs, ...), then you end up with a copy. Using the CLI takes a detour via the OS and requires an additional authentication - even if your boto3 session is authenticated, you need to have credentials elsewhere for CLI to work. I can't find any "best practice" - but I'm sure many have done it. How? Continue reading...
