How Can I Control Space Being Used On My Object Storage?

Sometimes you may encounter with S3 compatible Object Storage in general that more storage space is reported as being used than you obviously have. In practice it has the following reasons:

• Aborted multipart uploads
• Enabled Versioning

The next sections will cover the different topics in more detail. In the following we assume aws cli to be installed and configured properly.

What are aborted multipart uploads?

Usually uploads of larger files are chunked into smaller pieces e.g. 10 MB and uploaded one by one. Those pieces of files are not shown by default but of course occupy storage space. Multipart uploads are managed by buckets. 

How can I find aborted multipart uploads?

You can run this request:

aws --profile eu2 --region default --endpoint-url https://eu2.contabostorage.com s3api list-multipart-uploads --bucket mybucket

This will return a JSON output containing files currently being uploaded or having been aborted. If you get nothing back then there are no multipart uploads going on. 

How do I delete aborted multipart uploads?

In order to abort a single upload please copy the key (file name) and the upload id and invoke and run the following:

aws --profile eu2 --region default --endpoint-url https://eu2.contabostorage.com s3api abort-multipart-upload --bucket mybucket --key filename.ext --upload-id 2~Pw5t8pu1uBnxu_qMyJRnjnZKYvav41c

You can run the below script to automate the process of deleting every multipart upload in progress or which has been aborted:

BUCKETNAME=mybucket
aws --profile eu2 --region default --endpoint-url https://eu2.contabostorage.com s3api list-multipart-uploads --bucket $BUCKETNAME \
| jq -r '.Uploads[] | "--key \"\(.Key)\" --upload-id \(.UploadId)"' \
| while read -r param; do
eval "aws --profile eu2 --region default --endpoint-url https://eu2.contabostorage.com s3api abort-multipart-upload --bucket $BUCKETNAME $param";
done

What is enabled versioning?

Enabled or once enabled Versioning holds also storage space as there are many versions of a file. Those files are usually not visible.

How do I delete an old version of a file?

You can run the following request to delete old versions of a file that may be taking up space:

aws --profile eu2 --region default --endpoint-url https://eu2.contabostorage.com s3api list-object-versions --bucket mybucket
aws --profile eu2 --region default --endpoint-url https://eu2.contabostorage.com s3api delete-object --bucket mybucket --key file-to-delete.ext --version-id 3424sadfds2344