An open source object storage solution

Object Lesson

Object Versioning and Lifecycle Policies

Object versioning and lifecycle policies provide additional layers of data protection and help manage storage costs. Enabling object versioning to maintain multiple versions of objects in your MinIO buckets allows you to recover previous versions in case of accidental deletion or modification.

To enable object versioning, use the MinIO Client as follows:

mc versioning mycluster1/my-minio-bucket enable

To create and apply lifecycle policies, use mc ilm; for example:

mc ilm add mycluster1/my-minio-bucket --id "ExpireOldVersions" --status "Enabled" --expiry-days 30

With this command, you can automate the management of object versions, such as transitioning older versions to a different storage class or deleting them after a specified period.

Migration from S3

Organizations might consider migrating from S3 to MinIO for three main reasons: cost savings, improved performance, and data sovereignty requirements. Although both services are S3 compatible, migrating data between them requires careful planning and execution. The following steps are by no means exhaustive; rather, they present a basic outline to help you migrate your data from Amazon S3 to MinIO.

The first step is to install MinIO and define aliases in mc – both for MinIO and S3:

mc alias set myminio http://<your-minio-server>: 9000<minio-access-key> <minio-secret-key>
mc alias set aws https://s3.amazonaws.com <aws-access-key> <aws-secret-key>

Be sure to replace the placeholders (<>) with the appropriate access keys, secret keys, and server addresses.

Next, use the mc mirror command to transfer data from your Amazon S3 buckets to your MinIO server. This command synchronizes the source and destination, copying objects that don't exist in the destination or have a different size or modification time:

mc mirror --overwrite --remove aws/your-s3-bucket myminio/your-minio-bucket

The --overwrite flag tells mc to overwrite the destination objects if they differ from the source objects, whereas the --remove flag allows mc to delete destination objects that no longer exist in the source.

Note that transferring large amounts of data can take a significant amount of time depending on your network and storage conditions. Consider running the mc mirror command during periods of low system usage or scheduling it to run during off-peak hours. Also, mind your AWS bill – egress fees are notoriously high, and if you have particularly large volumes of data that are not particularly important, it might make more sense just to delete them.

After migrating your data, update your applications to use the MinIO server instead of Amazon S3. This step might involve changing the S3 endpoint, access keys, and secret keys in your application's configuration. Because both Amazon S3 and MinIO use the S3 API, no major code changes should be necessary.

Once your data has been migrated and your applications have been updated, it's essential to verify data integrity, which you can do by:

comparing the object counts and total size of your Amazon S3 buckets and MinIO buckets,
performing test queries and operations on the data in your MinIO server to ensure that it behaves as expected, and
validating the migrated data with application-specific tests, such as searching for particular records, generating reports, or processing data.

After verifying that your data has been successfully migrated to MinIO and that your applications are functioning correctly, you can decommission your Amazon S3 buckets. Be sure to delete any unnecessary objects, versioned objects, and bucket configurations before deleting the bucket itself.

In real-life scenarios, migration is rarely that simple, though. You might have a huge number of buckets with objects in them, each with a different access configuration by AWS Identity and Access Management (IAM) user, group, and role policies, as well as individual bucket policies, not to mention organizational-level policies (e.g., service control policies (SCPs)).

Replicating all this complexity might be difficult, even if you use IAM Roles Anywhere. If you don't have proper tag policies in place, you could even have a hard time figuring out who is the owner of a given bucket, and contacting them might not be easy. When you take all this and other aspects into consideration, the process of copying data itself might be the easiest part.

Logging and Monitoring

Monitoring and auditing MinIO server activity can help identify security threats and ensure compliance with data protection policies. MinIO supports logging of audit events and integration with monitoring tools like Prometheus for real-time insights.

To enable audit logging, add the --audit flag when starting the MinIO server:

minio server --audit /data

Also, consider monitoring MinIO performance with built-in Prometheus metrics, which provide insights into system health, resource usage, and performance metrics. To enable these metrics, add the --metrics prometheus flag when starting the MinIO server: