Skip to main content

Migrate from another storage provider to Contabo Object Storage

Contabo Object Storage is compatible with the S3 API, which is why migrating from another object storage provider like Amazon S3 or DigitalOcean Spaces is simple and fast using rclone. With rclone, you can manage your data on different cloud storages directly from your command line.

To get started, make sure you have rclone installed on your system. If you haven't, head over to the rclone installation guide and follow the instructions.

This example shows you how to migrate from Amazon S3 to Contabo Object Storage, but any other cloud storages compatible with rclone can be used as well.

Prepare rclone

Before you can manage external storages with rclone, they must be added to rclone's configuration. That's where all the required information about the storages is stored.

Configure the source storage

Start the rclone configuration by running rclone config. Then select n for New remote.

  1. Choose a name for the remote storage, e.g. aws-s3
  2. Select the type of storage you want to add, for Amazon S3 that's Amazon S3 Compliant Storage Providers including AWS, ...
  3. Select your S3 provider: Amazon Web Services (AWS) S3
  4. Enter your credentials by selecting Enter AWS credentials in the next step and type in your access_key_id and secret_access_key in the upcoming prompts
  5. Select the region
  6. Specify the endpoint for S3 API. Skip (return) this step to use the default AWS endpoint for the region specified in the previous step
  7. You can skip (return) the following steps acl, server_side_encryption, and sse_kms_key_id because you don't want to create files on your Amazon S3 storage
  8. Select your object storage class. If you have a default S3 storage, select Default (select Glacier Storage class if you have a S3 Glacier storage)
  9. If you get prompted to edit the advanced config, select n for No
  10. Verify the information and confirm with y

Finished rclone configuration for AWS S3

Now the first remote storage is configured, repeat the process for the storage you want to migrate your data to. In this case, Contabo Object Storage.

Configure Contabo Object Storage

Start the rclone configuration again (rclone config) and create a new remote (select n for new).

  1. Choose a name for the remote storage, e.g. contabo-storage
  2. Because the object storage is combatible with S3, select Amazon S3 Compliant Storage Providers including AWS, ...
  3. Select Ceph Object Storage as your provider
  4. Enter your credentials by selecting Enter AWS credentials in the next step and type in your access_key_id and secret_access_key in the upcoming prompts. If you don't have these keys, generate them first in your Contabo Customer Panel.
  5. Skip the region selection by pressing return
  6. Specify the S3 API Endpoint which you can find in the Object Storage Overview (e.g. https://eu2.contabostorage.com)
  7. You can skip (return) the following steps acl, server_side_encryption, and sse_kms_key_id
  8. If you get prompted to edit the advanced config, select n for No
  9. Verify the information and confirm with y

For a more detailed guide, please refer to rclone with Contabo Object Storage.

Both remote storages are now added to the rclone config. To verify that everything works, use the rclone ls command to list the objects inside a bucket:

# rclone ls remote-name:bucket-name
rclone ls aws-s3:bucket-name

List of all files in a specific buckets

You can use lsf instead to list both objects and directories.

Copy the data

With both remote storages added to rclone, copying the data from one to another isn't a big deal anymore. rclone's built-in copy command copies all the data inside one bucket to another. Use the following command to copy the data from your Amazon S3 bucket to your new bucket on Contabo Object Storage:

# rclone copy [source] [destination]
rclone copy aws-s3:bucket-name contabo-storage:bucket-name --s3-no-head
info

If the destination already has content inside, keep the following in mind:

  • Existing files on the destination that don't exist on the source are ignored and won't be deleted
  • Existing files on the destination that are identical to the files on the source will be ignored
  • Existing files on the destination that are not identical to the files on the source will be overwritten

This may take a while depending on the size of the files needing to be copied. Note that rclone downloads the files to your computer before uploading them to the destination.