Client support for Amazon S3 multipart uploads (files >5GB)

I recently had a need to upload a few large (11GB) files to S3 for offsite archive purposes. Amazon announced support for objects >5GB back in November 2010, so I had assumed that most S3 clients and libraries had support for it already.  I was wrong.  It turns out that in order to support large files, you need to use the new “multipart upload” feature.  What I found was that many S3 clients do not yet support multipart uploads.

In case this helps save anyone else a few hours of time, here is a summary of what I found regarding multipart upload support in the various S3 clients:

As of Feb 18, 2011:

To solve my immediate need I used S3 Browser on Windows 7.  It was the only free client I found that supported multipart uploads.  I mostly use a Mac, however, so if someone knows of a free S3 client for Mac please let me know.

I did not look at which S3 libraries support multipart uploads yet.  I have heard that my favorite python library – boto – supports it already.  And I have seen some reports that RightScale’s right_aws ruby library may support it on some platforms.  I’m not sure about the others.  Post in the comments if you want to confirm support for multipart uploads in your favorite S3 client or library.

Update (3/10/2010): Cyberduck 4.x was released this week and includes support for multipart uploads and 5TB uploads.  A Windows version is also available now.  Pretty cool, good job Cyberduck.

Update (4/18/2012): s3cmd 1.1.0-beta2 is available and supports multipart uploads along with many other new S3 and Cloudfront features. Grab the beta or wait for 1.1.0 final to be released

Posted Friday, February 18th, 2011 under AWS, internet services, macosx, software.
  • http://s3browser.com S3 Browser Team

    Hi Joe,

    thank you for mentioning S3 Browser in your blog.

    We have mono-compatible version of S3 Browser that works with some limitations (IAM features) via mono. We have tested it under Ubuntu and should also work under Mac.

    So, you may contact us and we will send you the link to mono-compatible version of S3 Browser.

    Regards,
    S3 Browser Team.

  • Nathan Hall

    I’d be very interested in how long your 11gb upload took. I have the need to archive 30gb worth of SQL Server backups on a daily basis – I’m wondering if this is a viable solution.

    • jmiller

      I was uploading from a 5Mbps connection, so I was limited by that. I was able to achieve the full 5Mbps for the entire upload, however. I’m sure many other factors come into play, such as your location and proximity to the S3 location you’re uploading to.

  • Bucket Explorer

    Bucket Explorer tool now support multipart upload, You can upload you %GB or Greater file on Amazon..
    http://www.bucketexplorer.com/documentation/amazon-s3–how-to-upload-big-file-in-parts.html