Mass Batch for S3 Objects

The new Amazon Batch Operations feature was also presented at re:Invent 2018 and is now available as a preview. This feature lets admins manage billions of objects stored in S3 with a single API call or a few mouse clicks in the S3 console and allows object properties or metadata to be change for any number of S3 objects. This approach also applies to copying objects between buckets or replacing tag sets, changing access controls, or retrieving/restoring objects from S3 Glacier in minutes rather than months.

Until now, companies have often had to spend months of development time writing optimized application software that could apply the required API actions to S3 objects on a massive scale. The Batch Operations feature can also be used to perform custom Lambda functions on billions or trillions of S3 objects, enabling highly complex tasks, such as image or video transcoding. Specifically, the feature takes care of retries, tracks progress, sends notifications, generates final reports, and delivers the events for all changes made and tasks performed to CloudTrail.

For a start, users can specify a list of target objects in an S3 inventory report that lists all objects of an S3 bucket or prefix. Optionally, you can specify your own list of target objects. You then select the desired API action from a prefilled options menu in the S3 Management Console. New S3 Batch Operations [3] are available in all AWS regions now. Operations are charged at $0.25/job or $1.00/million object operations performed, on top of charges associated with any operation S3 Batch Operations performs for you (e.g., data transfer, requests, and other charges).

Conclusions

Amazon S3 is far more than a file storage facility on the Internet, and even experienced users often don't know all of its capabilities, especially because AWS is constantly adding new features. Those who know the access patterns to their data can save a lot of money. Additionally, AWS now provides a degree of automation with the new S3 Intelligent-Tiering memory class (at an extra charge).