Delete old remote backups before uploading new ones
When someone saves the backups to remote destination and keeps daily, weekly and monthly backups, he should always have 3 times the size of the backup of free space in remote destination.
That's because if it is the 1st day of the month, and it is the day that weekly backup is made too, the daily backup is uploaded to the remote destination, then copied to weekly and monthly folders, and then old daily, weekly and monthly backups are deleted.
So that day there should be at least 3 times the size of the backup disk space available at the remote destination.
As you may understand, this is a very big waste of space and money, especially if the daily backup is a lot of GB (we have 50GB daily backup)!
You could have the option in backup configuration like 'Delete backups from remote destination before uploading new ones'.
I like this idea 
cPanel, WebHost Manager and WHM are registered trademarks of cPanel, L.L.C. for providing its computer software that facilitates the management and configuration of Internet web servers.
I am interested to hear others' feedback on this feature request and whether they would like this feature.
I am personally apprehensive about it and would be very worried that unsuspecting server owners would be setting themselves up for potential data loss/angry customers at a later date. It would be a very real possibility for the below situation to happen, should the feature you've requested be implemented and then utilized by a server owner:
Now you've defeated the benefit of remote backups by deleting your old ones before you had confirmed the new ones were sent.
Same goes for if the problem isn't as catastrophic.
How does cPanel & WHM handle the backup now? At that point you have no good backup on the remote system. You only have the staged one that was attempting to transfer remotely. Do we store the backup locally and send an alert to the admin? What if the admin takes no action before next backup run? Do we delete the only good local backups we have to generate new ones and hope they remote transfer?
These are all problems that could result in catastrophic loss of customer data. This is not to say that you feature request can't or won't be implemented, but they are very real scenarios that I want to make sure you're aware of when requesting this feature for the convenience of using less disk space for a few hours. It seems like a drastic trade off that risks customer backup integrity. Again, I'd love to hear feedback from other admins and whether they have similar concerns or if they're okay with the risks.
I am interested to hear others' feedback on this feature request and whether they would like this feature.
I am personally apprehensive about it and would be very worried that unsuspecting server owners would be setting themselves up for potential data loss/angry customers at a later date. It would be a very real possibility for the below situation to happen, should the feature you've requested be implemented and then utilized by a server owner:
Now you've defeated the benefit of remote backups by deleting your old ones before you had confirmed the new ones were sent.
Same goes for if the problem isn't as catastrophic.
How does cPanel & WHM handle the backup now? At that point you have no good backup on the remote system. You only have the staged one that was attempting to transfer remotely. Do we store the backup locally and send an alert to the admin? What if the admin takes no action before next backup run? Do we delete the only good local backups we have to generate new ones and hope they remote transfer?
These are all problems that could result in catastrophic loss of customer data. This is not to say that you feature request can't or won't be implemented, but they are very real scenarios that I want to make sure you're aware of when requesting this feature for the convenience of using less disk space for a few hours. It seems like a drastic trade off that risks customer backup integrity. Again, I'd love to hear feedback from other admins and whether they have similar concerns or if they're okay with the risks.
I am interested to hear others' feedback on this feature request and whether they would like this feature.
I am personally apprehensive about it and would be very worried that unsuspecting server owners would be setting themselves up for potential data loss/angry customers at a later date. It would be a very real possibility for the below situation to happen, should the feature you've requested be implemented and then utilized by a server owner:
Now you've defeated the benefit of remote backups by deleting your old ones before you had confirmed the new ones were sent.
Same goes for if the problem isn't as catastrophic.
How does cPanel & WHM handle the backup now? At that point you have no good backup on the remote system. You only have the staged one that was attempting to transfer remotely. Do we store the backup locally and send an alert to the admin? What if the admin takes no action before next backup run? Do we delete the only good local backups we have to generate new ones and hope they remote transfer?
These are all problems that could result in catastrophic loss of customer data. This is not to say that you feature request can't or won't be implemented, but they are very real scenarios that I want to make sure you're aware of when requesting this feature for the convenience of using less disk space for a few hours. It seems like a drastic trade off that risks customer backup integrity. Again, I'd love to hear feedback from other admins and whether they have similar concerns or if they're okay with the risks.
I am interested to hear others' feedback on this feature request and whether they would like this feature.
I am personally apprehensive about it and would be very worried that unsuspecting server owners would be setting themselves up for potential data loss/angry customers at a later date. It would be a very real possibility for the below situation to happen, should the feature you've requested be implemented and then utilized by a server owner:
Now you've defeated the benefit of remote backups by deleting your old ones before you had confirmed the new ones were sent.
Same goes for if the problem isn't as catastrophic.
How does cPanel & WHM handle the backup now? At that point you have no good backup on the remote system. You only have the staged one that was attempting to transfer remotely. Do we store the backup locally and send an alert to the admin? What if the admin takes no action before next backup run? Do we delete the only good local backups we have to generate new ones and hope they remote transfer?
These are all problems that could result in catastrophic loss of customer data. This is not to say that you feature request can't or won't be implemented, but they are very real scenarios that I want to make sure you're aware of when requesting this feature for the convenience of using less disk space for a few hours. It seems like a drastic trade off that risks customer backup integrity. Again, I'd love to hear feedback from other admins and whether they have similar concerns or if they're okay with the risks.
I have to agree with Brian. I have been testing the Remote Backup extensively to resolve a timeout issue affecting large 13-20GB backups per account. Having had remote Backup transfer interrupted I know I would not my backup deleted before the new one has been successfully transferred. Currently the transport does not resume interrupted transfer(s). That being said I agree disk space is a real concern and think that the old backup should be removed before uploading next backup but only after successful remote transfer of the new backup. I also would consider not backing up Daily, Weekly & Monthly. In the interest of conserving disk space I think 2 out of 3 should suffice.
I have to agree with Brian. I have been testing the Remote Backup extensively to resolve a timeout issue affecting large 13-20GB backups per account. Having had remote Backup transfer interrupted I know I would not my backup deleted before the new one has been successfully transferred. Currently the transport does not resume interrupted transfer(s). That being said I agree disk space is a real concern and think that the old backup should be removed before uploading next backup but only after successful remote transfer of the new backup. I also would consider not backing up Daily, Weekly & Monthly. In the interest of conserving disk space I think 2 out of 3 should suffice.
Good grief - just delete the old cpanel account from the old backup set before creating the new one - At least then the worst thing is that one account is affected - that's far better than having an entire backup fail because it ran our of space.
Good grief - just delete the old cpanel account from the old backup set before creating the new one - At least then the worst thing is that one account is affected - that's far better than having an entire backup fail because it ran our of space.
This too has caught me now and I am spending mass amounts of cash to have extra buffer space just to make sure the backups run successfully.
@PbG -"...the old backup should be removed before uploading next backup but only after successful remote transfer of the new backup."
Your comment doesn't make any sense. You are contradicting yourself here. First you say it should be removed BEFORE uploading (aka transferring the file) but then you say it should only be removed AFTER it has been successfully transferred. 0.o
I agree with Mr Sant. But I think we should do it slightly differently. The transfer of each backup file should be a two-step process.
This way you are keeping the disk space optimised during the backup process without running out of space AND you are ensuring that the old backup data isn't lost in the event of a failure on the WHM/cPanel server in the event of a failure.
This is vital in my opinion
This too has caught me now and I am spending mass amounts of cash to have extra buffer space just to make sure the backups run successfully.
@PbG -"...the old backup should be removed before uploading next backup but only after successful remote transfer of the new backup."
Your comment doesn't make any sense. You are contradicting yourself here. First you say it should be removed BEFORE uploading (aka transferring the file) but then you say it should only be removed AFTER it has been successfully transferred. 0.o
I agree with Mr Sant. But I think we should do it slightly differently. The transfer of each backup file should be a two-step process.
This way you are keeping the disk space optimised during the backup process without running out of space AND you are ensuring that the old backup data isn't lost in the event of a failure on the WHM/cPanel server in the event of a failure.
This is vital in my opinion
Hello, I was about to post the same like BlueSteam did. I second this as this method will help save disk space while ensuring the whole backup process completeness.
This can't add any overhead nor extra time to add up to the whole time the process take.
I hope you can take this into consideration as it will help many servers with not enough free disk space.
Hello, I was about to post the same like BlueSteam did. I second this as this method will help save disk space while ensuring the whole backup process completeness.
This can't add any overhead nor extra time to add up to the whole time the process take.
I hope you can take this into consideration as it will help many servers with not enough free disk space.
EVERY SINGLE week I have to manually copy over backups to my backup server because of this flaw in the backup process. I CANNOT believe the developers can't see this as a problem. It baflles me beyond belief!!
EVERY SINGLE week I have to manually copy over backups to my backup server because of this flaw in the backup process. I CANNOT believe the developers can't see this as a problem. It baflles me beyond belief!!
I think the problem here is that, I believe, the backup -job- is treated as an object here rather than each account.
Having enough space for 3 copies of the currently-transferring account is easy. Having enough space for 3 copies of the entire backup set is the problem.
I think the problem here is that, I believe, the backup -job- is treated as an object here rather than each account.
Having enough space for 3 copies of the currently-transferring account is easy. Having enough space for 3 copies of the entire backup set is the problem.
I agree with BlueSteam and Kent. It should be managed per account.
Also, if it is implemented as were requested, shouldn't be catastrophic as Brian stated. I have 6 days backup plus 4 weeks backups. What dangerous is to delete the oldest day (day 1) and upload the day 6? if something goes wrong (as usual) we are losing the oldest one of a serie of 6, the same as weekly backups.
Then I disagree with Brian who imagine a scenario with ONE backup, when most of Hosting companies at least implement 5 days backups, when implement.
I agree with BlueSteam and Kent. It should be managed per account.
Also, if it is implemented as were requested, shouldn't be catastrophic as Brian stated. I have 6 days backup plus 4 weeks backups. What dangerous is to delete the oldest day (day 1) and upload the day 6? if something goes wrong (as usual) we are losing the oldest one of a serie of 6, the same as weekly backups.
Then I disagree with Brian who imagine a scenario with ONE backup, when most of Hosting companies at least implement 5 days backups, when implement.
On a related note, I've been messing with failed backup transfer logs telling me the backup was "unable to send" to my backup FTP server. I had to sift through the /usr/local/cpanel/logs/cpbackup_transporter.log log file to find "warn [cpbackup_transporter] Upload attempt failed: 183142215: No space left on device".
As backups are scheduled overnight, when I check the FTP site during the day there _seems_ to be enough space to hold another backup. But judging from this feature request, there _isn't_, during the ftp backup.
My cents are with Jose, BlueSteam and Kent as well. :)
On a related note, I've been messing with failed backup transfer logs telling me the backup was "unable to send" to my backup FTP server. I had to sift through the /usr/local/cpanel/logs/cpbackup_transporter.log log file to find "warn [cpbackup_transporter] Upload attempt failed: 183142215: No space left on device".
As backups are scheduled overnight, when I check the FTP site during the day there _seems_ to be enough space to hold another backup. But judging from this feature request, there _isn't_, during the ftp backup.
My cents are with Jose, BlueSteam and Kent as well. :)
What's more, there is no official opinions from cPanel staff in this thread. Can anybody give any signals of at least trying to reproduce the scenario?
What's more, there is no official opinions from cPanel staff in this thread. Can anybody give any signals of at least trying to reproduce the scenario?
No.
What happens when the backup gets "...first..." deleted, then the upload of the new one fails or is corrupt? ;)
No.
What happens when the backup gets "...first..." deleted, then the upload of the new one fails or is corrupt? ;)
I've commented on a few of these backup threads in the past. I disagree with the concept of deleting a backup before successful transfer of the new backup completes. I have some 16 copies of my server off-site. Storage is pretty cheap these days.
That being said, the per-account post-transfer cleanup instead of the post-batch cleanup definitely makes sense and should be an option here.
I've commented on a few of these backup threads in the past. I disagree with the concept of deleting a backup before successful transfer of the new backup completes. I have some 16 copies of my server off-site. Storage is pretty cheap these days.
That being said, the per-account post-transfer cleanup instead of the post-batch cleanup definitely makes sense and should be an option here.
I'm in favour of this idea; but not limited to remote backups only.
I take backups once a week and have retention set at 2; but this essentially means I have to be able to hold 3 backups. I understand the risks of 'deleting before creating' - but as has been mentioned above in this thread; its not always an issue.
Sure if you only have 1 backup, then this "could" (not necessarily, "will") be a problem. But in my case, simply having the option for cPanel to delete the oldest backup first before running the newest backup ensures that I only need to have "2x" the disk space as opposed to "3x" the disk space required.
Im at the stage where I wont have the option to hold "3x" backups anymore anyway which means I'll have to reduce my retention to "1". So as you can see, having the above option to 'delete before creating' can actually be rather beneficial.
I'm in favour of this idea; but not limited to remote backups only.
I take backups once a week and have retention set at 2; but this essentially means I have to be able to hold 3 backups. I understand the risks of 'deleting before creating' - but as has been mentioned above in this thread; its not always an issue.
Sure if you only have 1 backup, then this "could" (not necessarily, "will") be a problem. But in my case, simply having the option for cPanel to delete the oldest backup first before running the newest backup ensures that I only need to have "2x" the disk space as opposed to "3x" the disk space required.
Im at the stage where I wont have the option to hold "3x" backups anymore anyway which means I'll have to reduce my retention to "1". So as you can see, having the above option to 'delete before creating' can actually be rather beneficial.
Hello, I would like this feature too!
I get 500GB free FTP backup storage with my Server and my sites are over 250GB. Which means the second backup to FTP fails.
I would like option to delete backup first. I understand the risk. But a) i keep a local copy of the backup b) i have option to send backup to Google drive of elsewhere.
So for the FTP Destination i want to delete before upload. Most Datacenters offer limited FTP storage so its good idea.
thanks
/J
Hello, I would like this feature too!
I get 500GB free FTP backup storage with my Server and my sites are over 250GB. Which means the second backup to FTP fails.
I would like option to delete backup first. I understand the risk. But a) i keep a local copy of the backup b) i have option to send backup to Google drive of elsewhere.
So for the FTP Destination i want to delete before upload. Most Datacenters offer limited FTP storage so its good idea.
thanks
/J
Replies have been locked on this page!