Copy Data from S3 Object Storage to Different S3 Object Storage Provider

MinIO is an Amazon S3 compatible server-side software storage stack, it can handle unstructured data such as photos, videos, log files, backups, and container images with currently the maximum supported object size of 5TB.

In this example, this is a perfect simple script to either backup or migrate your s3 object storage data between buckets (on the same provider — or different providers).

The first step is to download the MinIO client.

$ wget https://dl.minio.io/client/mc/release/linux-amd64/mc
$ chmod 755 mc
$ ./mc --help

If you’re using different s3 storage providers just change the URL:
https://s3.amazonaws.com

$ ./mc config host add <bucket_name> https://s3.amazonaws.com <access_key> <secret_key>

$ ./mc config host add <another_bucket_name> https://s3.wasabisys.com <access_key> <secret_key>

If you’re connected to your server using Putty — I recommend executing this command from the Console rather than Putty, so it’s safe if the Putty terminal window loses connection.

Create a terminal script and run this from the console. The final command copy.sh executes the copy script.

$ echo './mc mirror bucket_name/ another_bucket_name/' > copy.sh$ chmod u+x ./copy.sh$ ./copy.sh

Tip:

Remember, if you’re migrating storage buckets to another provider, the storage buckets in each provider must have identical names (similar to a domain name) if you’re making a direct copy — otherwise there will be permission related errors due to the fact the bucket is owned by someone else. In this case, you would need to create a new storage bucket with a different name on the newly created S3 Storage Provider.

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Ian Arman

Hey I'm @ianarman, I only clap about the cool stuff on @Medium!