Transfer all backups over a single TCP/IP connection
Currently, our server spend a lot of time backing up to 3 different sources. Knowing that our server previously had a cap of 250 Mbit/s outgoing bandwidth and one of our remote backup servers it was and is 2 Gbit/s.
So I wondered why it took so long.
After a recent upgrade, where we increased the cap from 250 Mbit/s to 1 Gbit/s, and where we got SSD's in RAID 10 instead of SAS 15K RPM in RAID 10, we got backup generation time down from an hour or so to a few minutes.
However backup transportation to remote hosts still take as long time.
Reviewing the cpbackup_transporter.log file, I can see that it take 1 file and first transfer to each backup destination, then the next file, and so on.
This means that there is created a TCP/IP connection to the backup server, 1 file is transferred, and the connection is closed again. Sometimes only a few MB.
If instead 1 TCP/IP connection was made, where all backups to that destination is transferred at once, that would remove the overhead of creating and destroying TCP/IP connections, and at the same time make it possible to transfer the backups faster.
This is because (as far as I know) TCP/IP connections start out slow, then going faster and faster as it realizes it can use more bandwidth.
However it'll never realize that when it only transfer tons of small files.
I like this idea 
cPanel, WebHost Manager and WHM are registered trademarks of cPanel, L.L.C. for providing its computer software that facilitates the management and configuration of Internet web servers.
How do you propose this be accomplished? In the current form of backup generation and remote storage, we leverage existing technologies (FTP, SFTP, WebDAV, etc). Therefore we're limited by the implementation of those technologies in how to transmit data.
The concept of trying to implement some constant single socket stream of all backups is pretty complex and prone to problems. It would also likely need to be an entirely in-house written and developed daemon since I'm not aware of any existing technologies that would fit in and provide this. It's something not frequently done due to the complexities and the fact that alternative solutions seem to be more effective.
The alternative solutions I speak of are the obvious counter to your "lots of files" statement. Using compressed backups mean one file per account regardless the size/amount of files within that cPanel account. This substantially increases throughput reliability for the reason you mentioned where having to frequently open/close connections works against you speed wise over time.
If you can propose of a workable, simple, reliable method of what you're requesting, we'd be open to hearing more on it. I can't think of one at this time, as the complexities of what you're asking for is why this is not a commonplace feature in pretty much any file transfer daemon/utility used in production.
How do you propose this be accomplished? In the current form of backup generation and remote storage, we leverage existing technologies (FTP, SFTP, WebDAV, etc). Therefore we're limited by the implementation of those technologies in how to transmit data.
The concept of trying to implement some constant single socket stream of all backups is pretty complex and prone to problems. It would also likely need to be an entirely in-house written and developed daemon since I'm not aware of any existing technologies that would fit in and provide this. It's something not frequently done due to the complexities and the fact that alternative solutions seem to be more effective.
The alternative solutions I speak of are the obvious counter to your "lots of files" statement. Using compressed backups mean one file per account regardless the size/amount of files within that cPanel account. This substantially increases throughput reliability for the reason you mentioned where having to frequently open/close connections works against you speed wise over time.
If you can propose of a workable, simple, reliable method of what you're requesting, we'd be open to hearing more on it. I can't think of one at this time, as the complexities of what you're asking for is why this is not a commonplace feature in pretty much any file transfer daemon/utility used in production.
How do you propose this be accomplished? In the current form of backup generation and remote storage, we leverage existing technologies (FTP, SFTP, WebDAV, etc). Therefore we're limited by the implementation of those technologies in how to transmit data.
The concept of trying to implement some constant single socket stream of all backups is pretty complex and prone to problems. It would also likely need to be an entirely in-house written and developed daemon since I'm not aware of any existing technologies that would fit in and provide this. It's something not frequently done due to the complexities and the fact that alternative solutions seem to be more effective.
The alternative solutions I speak of are the obvious counter to your "lots of files" statement. Using compressed backups mean one file per account regardless the size/amount of files within that cPanel account. This substantially increases throughput reliability for the reason you mentioned where having to frequently open/close connections works against you speed wise over time.
If you can propose of a workable, simple, reliable method of what you're requesting, we'd be open to hearing more on it. I can't think of one at this time, as the complexities of what you're asking for is why this is not a commonplace feature in pretty much any file transfer daemon/utility used in production.
How do you propose this be accomplished? In the current form of backup generation and remote storage, we leverage existing technologies (FTP, SFTP, WebDAV, etc). Therefore we're limited by the implementation of those technologies in how to transmit data.
The concept of trying to implement some constant single socket stream of all backups is pretty complex and prone to problems. It would also likely need to be an entirely in-house written and developed daemon since I'm not aware of any existing technologies that would fit in and provide this. It's something not frequently done due to the complexities and the fact that alternative solutions seem to be more effective.
The alternative solutions I speak of are the obvious counter to your "lots of files" statement. Using compressed backups mean one file per account regardless the size/amount of files within that cPanel account. This substantially increases throughput reliability for the reason you mentioned where having to frequently open/close connections works against you speed wise over time.
If you can propose of a workable, simple, reliable method of what you're requesting, we'd be open to hearing more on it. I can't think of one at this time, as the complexities of what you're asking for is why this is not a commonplace feature in pretty much any file transfer daemon/utility used in production.
Hello,
A simple solution could be to compress the account backups so x accounts go in one file or x GB of data and then it's split or something. Of course do an md5 check afterwards to make sure it is correctly transferred, and perhaps unpack it correctly.
Another way, however, less efficient, (so would prefer the solution above) would be to simply swap it around in when a backup is backed up, so it first takes all backups to one destination - then to another destination and then to the third.
You'd save the login overhead of FTP and such, but not really more than that. So it would be preferable if the backups could be combined in larger files thus getting transferred faster and then perhaps unpacked at the end.
Hello,
A simple solution could be to compress the account backups so x accounts go in one file or x GB of data and then it's split or something. Of course do an md5 check afterwards to make sure it is correctly transferred, and perhaps unpack it correctly.
Another way, however, less efficient, (so would prefer the solution above) would be to simply swap it around in when a backup is backed up, so it first takes all backups to one destination - then to another destination and then to the third.
You'd save the login overhead of FTP and such, but not really more than that. So it would be preferable if the backups could be combined in larger files thus getting transferred faster and then perhaps unpacked at the end.
Replies have been locked on this page!