We need to copy the contents of a bucket to a new account in a different AWS region. The bucket contains ±200K objects, most of which are archived (Glacier), and a large portion of them are quite big (>10GB). At first we tried the following: Batch script to restore all the objects. Configure a Lambda trigger to handle restore completion per object. Execute the copy method with boto3 on the Lambda function. This works well for smaller objects (up to ~2GB), but larger objects take longer than the maximum allowed Lambda duration (15min). To complete the rest of them (>10K objects) we must use some VM, however VMs are forced to be in a VPC and for some reason boto3 can't handle cross-region copy commands from within a VPC: An error occurred (AccessDenied) when calling the UploadPartCopy operation: VPC endpoints do not support cross-region requests So we resorted to download and upload each object. This takes too long for the amount of objects we need to copy. Can anyone offer a better solution? NOTE #1: AWS has something called Batch Jobs which may be relevant, but it was a bit difficult to configure and for some reason requires both source and destination buckets to enable versioning (which is disable in this case). NOTE #2: We have a role on the destination account which has access to the source account and S3 requests are called with the requester paying for them. Continue reading...
