Skip to main content

Control Storage Space

Sometimes you may encounter with S3 compatible Object Storage in general that more storage space is reported as being used than you obviously have. In practice it has the following reasons:

The next sections will cover the different topics in more detail. In the following we assume aws cli to be installed and configured properly.

Aborted multipart uploads

Usually uploads of larger files are chunked into smaller pieces e.g. 10 MB and uploaded one by one. Those pieces of files are not shown by default but of course occupy storage space. Multipart uploads are managed by buckets.

Display multipart uploads

aws --profile eu2 --region default --endpoint-url s3api list-multipart-uploads --bucket mybucket

This will return a JSON output containing files currently being uploaded or having been aborted. If you get nothing back then there are no multipart uploads going on.

Aborting / Deleting multipart uploads

In order to abort a single upload please copy the key (file name) and the upload id and invoke, e.g.

aws --profile eu2 --region default --endpoint-url s3api abort-multipart-upload --bucket mybucket --key filename.ext --upload-id 2~Pw5t8pu1uBnxu_qMyJRnjnZKYvav41c

Aborting all multipart uploads of a bucket

The following script automates the process of deleting every multipart upload in progress or which has been aborted.

aws --profile eu2 --region default --endpoint-url s3api list-multipart-uploads --bucket $BUCKETNAME \
| jq -r '.Uploads[] | "--key \"\(.Key)\" --upload-id \(.UploadId)"' \
| while read -r param; do
eval "aws --profile eu2 --region default --endpoint-url s3api abort-multipart-upload --bucket $BUCKETNAME $param";

Enabled Versioning

Enabled or once enabled Versioning holds also storage space as there are many versions of a file. Those files are usually not visible.

Please refer to Versioning for how to enable and disable it. In the following we will just handle the deletion of older versions to reclaim storage space.

Delete an old version of a file

aws --profile eu2 --region default --endpoint-url s3api list-object-versions --bucket mybucket
aws --profile eu2 --region default --endpoint-url s3api delete-object --bucket mybucket --key file-to-delete.ext --version-id 3424sadfds2344

Tools like Cyberduck are making these tasks very easy. Just do a Info on a bucket and check Bucket Versioning. Please also enable via View -> Show Hidden Files the display of all versions. You might also delete older versions very easily.