Our features site is undergoing a refresh! Be sure to explore the revamped site and discover our latest product roadmap launching here on Monday, March 18th.

Amazon S3 Adjustable Chunk Size

Jeremy shared this idea 8 years ago
Open Discussion

The Amazon S3 backup script hard codes chunck size to 20MB. This causes excessive PUT transactions that can make small daily backups exceed free tier and raise the costs of S3 backups in general.


This combined with the other feature request for retry chunck instead of entire file would make S3 backups more cost efficient.

https://features.cpanel.net/topic/amazon-s3-cpanel-backup-transporter-code-to-retry-failed-uploads-in-chunks-rather-than-entire-file


Please include a WHM Backup advanced option to adjust upload_chunk_size.


  1. /usr/local/cpanel/Cpanel/Transport/Files/AmazonS3.pm
  2. # The size we break the file into for multipart upload
  3. # We hardcode this for now; but we may make this a config
  4. # option if there is demand for it.
  5. $OPTS->{'upload_chunk_size'} = 20 * 1024**2;

Leave a Comment
 
Attach a file