S3 Backup Destination - 5GB file size limit
Open Discussion
S3 currently only allows transfer of files smaller than 5GB.
The WHM S3 Backup Destination is failing to send some of our backups to S3 because of this (no errors that we've seen either).
Please either provide proper notification when files aren't sent (and do not have them deleted if they haven't been sent), or provide an option to split files into 5GB chunks before sending.
I like this idea 
cPanel, WebHost Manager and WHM are registered trademarks of cPanel, L.L.C. for providing its computer software that facilitates the management and configuration of Internet web servers.
Sebastian, can you open a tech support ticket for this? We would like to take a look at your server, and see if there is more information that will help us fix this.
We are currently working on making large Amazon S3 backups more reliable (by sending in smaller chunks, as you describe).
Sebastian, can you open a tech support ticket for this? We would like to take a look at your server, and see if there is more information that will help us fix this.
We are currently working on making large Amazon S3 backups more reliable (by sending in smaller chunks, as you describe).
Thanks Phil, but I no longer see any warning about size transfer limits in the S3 documentation. So perhaps Amazon now allows transfers up to 5TB in size.
We are having some trouble getting some rather large backups sent to our S3 bucket however - in the ~80GB range. All the config looks right, and the backup logs show the backups completed... unfortunately they disappear from the WHM server (we don't have retention enabled to save space), but never show up on the S3 Bucket.
I'll open a support ticket for this, but could we also get an answer here as to whether the backup system should work with sending backups to S3 larger than 5GB now? Or maybe where to find logs? Thanks!
Thanks Phil, but I no longer see any warning about size transfer limits in the S3 documentation. So perhaps Amazon now allows transfers up to 5TB in size.
We are having some trouble getting some rather large backups sent to our S3 bucket however - in the ~80GB range. All the config looks right, and the backup logs show the backups completed... unfortunately they disappear from the WHM server (we don't have retention enabled to save space), but never show up on the S3 Bucket.
I'll open a support ticket for this, but could we also get an answer here as to whether the backup system should work with sending backups to S3 larger than 5GB now? Or maybe where to find logs? Thanks!
Ah.... I finally found where they document the limit again:
http://docs.aws.amazon.com/AmazonS3/latest/dev/UploadingObjects.html
And I've opened a support ticket so we can help with the multipart upload.
Ah.... I finally found where they document the limit again:
http://docs.aws.amazon.com/AmazonS3/latest/dev/UploadingObjects.html
And I've opened a support ticket so we can help with the multipart upload.
So to confirm what's in the documentation (at time of writing, April 2015)
- a single PUT operation to an Amazon S3 bucket has a 5Gb limit.
- a multi-part upload has a 5Tb limit.
So the trick clearly is to permit a multi-part upload using whatever backup software you're using. We are currently evaluating Server Backup Manager from http://www.r1soft.com/ as a likely superior solution to WHM's own limited-feature-set remote backup solution.
So to confirm what's in the documentation (at time of writing, April 2015)
- a single PUT operation to an Amazon S3 bucket has a 5Gb limit.
- a multi-part upload has a 5Tb limit.
So the trick clearly is to permit a multi-part upload using whatever backup software you're using. We are currently evaluating Server Backup Manager from http://www.r1soft.com/ as a likely superior solution to WHM's own limited-feature-set remote backup solution.
Replies have been locked on this page!