If I'm not mistaken, the backup schedule/frequency for Amazon S3 remote backups in WHM is currently 'locked' to be the same as the daily schedule/frequency for local backups.
What we need is to be able to schedule remote backups completely independently of the local backup schedule.
Right now, if I want to send backups to Amazon S3 weekly, I can't do this unless I set the daily local backups to only occur once per week - which is of course crazy :-)
I like this idea 
I would like to keep only the most recent backup on my local server, and a weeks worth on the remote server.
I would like to keep only the most recent backup on my local server, and a weeks worth on the remote server.
Much needed. We have plenty of free space in the remote storage there would like to keep many backups, but only limited space locally.
Ideally, we want to keep 2 daily backups locally (for fast access) and 7 daily/4 weekly remote.
Much needed. We have plenty of free space in the remote storage there would like to keep many backups, but only limited space locally.
Ideally, we want to keep 2 daily backups locally (for fast access) and 7 daily/4 weekly remote.
The cost of storage is vastly different as is the need to store for different time periods. I would love to be able to store long term offsite files and a few onsite files. I think this feature is well worth it.
The cost of storage is vastly different as is the need to store for different time periods. I would love to be able to store long term offsite files and a few onsite files. I think this feature is well worth it.
You can set up auto-delete rules within Amazon S3. They call it Lifecycle Management. Set the rules up on your whole bucket, or individual folders for when files should be automatically deleted (e.g. 35 days old).
Instructions are here:http://docs.aws.amazon.com/AmazonS3/latest/UG/lifecycle-configuration-bucket-no-versioning.html
You can set up auto-delete rules within Amazon S3. They call it Lifecycle Management. Set the rules up on your whole bucket, or individual folders for when files should be automatically deleted (e.g. 35 days old).
Instructions are here:http://docs.aws.amazon.com/AmazonS3/latest/UG/lifecycle-configuration-bucket-no-versioning.html
I noticed that my local retention rules are being applied to S3.
Example:
I set WHM to retain 7 daily backups (they are stored on a separate drive) and also transfer them over to S3.
My S3 bucket has a lifecycle setting that permanently deletes backups after 31 days.
I would like to be able to set remote retention rules to a different value than local rules, not the same!
I noticed that my local retention rules are being applied to S3.
Example:
I set WHM to retain 7 daily backups (they are stored on a separate drive) and also transfer them over to S3.
My S3 bucket has a lifecycle setting that permanently deletes backups after 31 days.
I would like to be able to set remote retention rules to a different value than local rules, not the same!
This is important. Local and remote resources are not always the same and bandwidth costs are an issue for some.
This is important. Local and remote resources are not always the same and bandwidth costs are an issue for some.
I need option for only monthly backup to FTP.
I need option for only monthly backup to FTP.
Custom retention for S3 would be nice, but cPanel does already apply local retention rules to S3. Only the number of day's I've specified to retain backups are being retained in S3.
Custom retention for S3 would be nice, but cPanel does already apply local retention rules to S3. Only the number of day's I've specified to retain backups are being retained in S3.
Retention can be quite complex (as versioning affects it too) but is easily managed per bucket using rules.
I think it's a bad habit to delete backups from S3 manually (as WHM does with its retention). Also it wouldn't be useful to be able to change the rules for S3 retention (or glacier archiving and versioning for that matter) unless the policy set in AWS has the leading role, eg. when you change the policy in AWS then those changes will reflect in WHM also. Even then, Amazon is rapidly developing new features all the time like Glacier, encryption and even more retention options. To be useful (and prevent conflicts) those all need to be implemented in WHM and kept up to date.
It's better to set it once, configure it well and forget about it. If you watch their news you'll see when a new feature has been implemented and whether it's worth using it in your policy.
Let's not forget there still isn't support for the not even so new authentication system that uses AWS4-HMAC-SHA25 (protocol v4) and is the only available authentication type in newer datacenters. More important things have to be done right now.
Retention can be quite complex (as versioning affects it too) but is easily managed per bucket using rules.
I think it's a bad habit to delete backups from S3 manually (as WHM does with its retention). Also it wouldn't be useful to be able to change the rules for S3 retention (or glacier archiving and versioning for that matter) unless the policy set in AWS has the leading role, eg. when you change the policy in AWS then those changes will reflect in WHM also. Even then, Amazon is rapidly developing new features all the time like Glacier, encryption and even more retention options. To be useful (and prevent conflicts) those all need to be implemented in WHM and kept up to date.
It's better to set it once, configure it well and forget about it. If you watch their news you'll see when a new feature has been implemented and whether it's worth using it in your policy.
Let's not forget there still isn't support for the not even so new authentication system that uses AWS4-HMAC-SHA25 (protocol v4) and is the only available authentication type in newer datacenters. More important things have to be done right now.
You can control the retention in S3 using versioning and object life cycle.
More information here: http://docs.aws.amazon.com/AmazonS3/latest/UG/enable-bucket-versioning.html, http://docs.aws.amazon.com/AmazonS3/latest/dev/object-lifecycle-mgmt.html and http://docs.aws.amazon.com/AmazonS3/latest/dev/intro-lifecycle-rules.html
You can control the retention in S3 using versioning and object life cycle.
More information here: http://docs.aws.amazon.com/AmazonS3/latest/UG/enable-bucket-versioning.html, http://docs.aws.amazon.com/AmazonS3/latest/dev/object-lifecycle-mgmt.html and http://docs.aws.amazon.com/AmazonS3/latest/dev/intro-lifecycle-rules.html
People pushing AWS retention rules don't really get one of the problems.
On local server i don't have unlimited storage, so i have to set cPanel to delete backups after 1-2-3-x days.
On S3 i have unlimited space, but cPanel will still delete them based on local delete rules ( 1-2-3-x day), so i can say goodbye to AWS retention rules.
Also i don't really care what AWS has/can do.
You could setup Apache/PHP on your own, yet if you are using cPanel you expect to do them for you.
cPanel's backup should be a standalone application which doesn't require me to login to other places, and and add some manual config.
People pushing AWS retention rules don't really get one of the problems.
On local server i don't have unlimited storage, so i have to set cPanel to delete backups after 1-2-3-x days.
On S3 i have unlimited space, but cPanel will still delete them based on local delete rules ( 1-2-3-x day), so i can say goodbye to AWS retention rules.
Also i don't really care what AWS has/can do.
You could setup Apache/PHP on your own, yet if you are using cPanel you expect to do them for you.
cPanel's backup should be a standalone application which doesn't require me to login to other places, and and add some manual config.
We have set up our S3 bucket with a few rules to make it easier to manage costs. This can be done on a bucket-level inside the S3 management console:
* Move files to glacier after two weeks
* Remove files after 100 days
The move to glacier helped us greatly reduce cost while still retaining backups for a long time.
We have set up our S3 bucket with a few rules to make it easier to manage costs. This can be done on a bucket-level inside the S3 management console:
* Move files to glacier after two weeks
* Remove files after 100 days
The move to glacier helped us greatly reduce cost while still retaining backups for a long time.
Is there any update on this? And any changes to this should affect not just S3 backups, but other off-site backups as well.
Is there any update on this? And any changes to this should affect not just S3 backups, but other off-site backups as well.
No update yet, but we are definitely aiming to spend some significant time on the backup system in 2017. As soon as anything specific to this is done, I'll let everyone know!
No update yet, but we are definitely aiming to spend some significant time on the backup system in 2017. As soon as anything specific to this is done, I'll let everyone know!
This would be ideal. We run a VM running on limited SSD storage. We only have enough space to keep 3x backups locally and ideally we want to keep them on S3 for longer. Obviously we need more than 3 day's worth of backups so it involves manual work to retain them. The ability to set custom retention periods for S3 would be ideal.
This would be ideal. We run a VM running on limited SSD storage. We only have enough space to keep 3x backups locally and ideally we want to keep them on S3 for longer. Obviously we need more than 3 day's worth of backups so it involves manual work to retain them. The ability to set custom retention periods for S3 would be ideal.
This would be a nice convenience feature. Right now I just have a bucket policy erasing all backups older than x number of days. It works but is not as "friendly" as a built-in cPanel feature.
This would be a nice convenience feature. Right now I just have a bucket policy erasing all backups older than x number of days. It works but is not as "friendly" as a built-in cPanel feature.
I'll expand the original request to simply be that remote destinations should be able to override the default retention with a custom one. For instance, we are testing adding B2 but S3 has been our main backup destination for the last few years. Our retention is 5 dailys, 3 weeklys, and 3 monthlys. Local backups are deleted due to lack of space. I want to add B2 (since it's so cheap) instead of replacing S3 since I want more redundancy. But it would be nice to leave the normal retention in place for the new B2 but switch S3 over to only accepting and retaining dailys (to shave cost). Being able to override retention policy at the destination level makes sense to me and allow for tiering across unrelated destinations. The comments above about being able to 'program' the behavior within S3 & Galcier is noted, but I want my server to manage it, not the storage provider.
I'll expand the original request to simply be that remote destinations should be able to override the default retention with a custom one. For instance, we are testing adding B2 but S3 has been our main backup destination for the last few years. Our retention is 5 dailys, 3 weeklys, and 3 monthlys. Local backups are deleted due to lack of space. I want to add B2 (since it's so cheap) instead of replacing S3 since I want more redundancy. But it would be nice to leave the normal retention in place for the new B2 but switch S3 over to only accepting and retaining dailys (to shave cost). Being able to override retention policy at the destination level makes sense to me and allow for tiering across unrelated destinations. The comments above about being able to 'program' the behavior within S3 & Galcier is noted, but I want my server to manage it, not the storage provider.
Is there any movement on this?
I also would like to keep only a copy or two locally, but 15 days remotely on S3. I still don't see any settings about separate retention periods.
Is there any movement on this?
I also would like to keep only a copy or two locally, but 15 days remotely on S3. I still don't see any settings about separate retention periods.
Replies have been locked on this page!