Our features site is undergoing a refresh! Be sure to explore the revamped site and discover our latest product roadmap launching here on Monday, March 18th.
New Topic
cPanel & WHM Feature RequestsTopicBackups
This object is in archive! 
Properties
Category Backups
Tags pigz, backup, 55535

Faster cPanel Backup by Replacing gzip with pigz

Gareth-AWD shared this idea 12 years ago
Completed

Gzip doesn't make good use of multi-processor/multi-core machines, which results in long backup time.


Fortunately, there's an in-place replacement, pigz


"pigz, which stands for parallel implementation of gzip, is a fully functional replacement for gzip that exploits multiple processors and multiple cores to the hilt when compressing data."


Also, files compressed with pigz can be decompressed with gunzip, and unpigz can decompress gzip compressed files.


I've done a couple of tests on an 8 cores machine, and here are the results:


  1. [root@ns3 tests]# time gzip test1.tar


    real 1m 7.176s

    user 0m55.119s

    sys 0m1.069s


    [root@ns3 tests]# time pigz test2.tar


    real 0m 12.096s

    user 1m15.924s

    sys 0m1.109s


    [root@ns3 tests]# time pigz -1 test3.tar


    real 0m 7.141s

    user 0m44.020s

    sys 0m1.349s


    -rw-r--r-- 1 root root 315M Nov 10 04:54 test1.tar.gz

    -rw-r--r-- 1 root root 315M Nov 10 04:54 test2.tar.gz

    -rw-r--r-- 1 root root 346M Nov 10 04:54 test3.tar.gz

    -rw-r--r-- 1 root root 788M Nov 10 04:54 test.tar


In layman's terms "to get more votes :)" .. pigz compressed a 788 MB file, using fastest compression and all 8 cores, in 7 seconds, while it took gzip 1 minute and 7 seconds .. you do the math.


BTW, pigz was written by Mark Adler.


Adler is the author of the Adler-32 hash function, a co-author of the zlib compression library and gzip, has contributed to Info-ZIP, and has participated in developing the Portable Network Graphics (PNG) image format. Adler was also the Spirit Mission Manager for the Mars Exploration Rover mission.


In short .. he knows what he's doing :)


Original Thread: Faster cPanel Backup by replacing gzip with pigz

45 I like this idea 0  
Best Answer
photo
Aila ● 11 years ago

pigz integration with pkgacct and the backup system is complete. Barring incident it will be available with cPanel & WHM 11.38.

Replies (21)

photo
2
WhiteDog ● 12 years ago

Any ETA yet when this will make it into production?

photo
2
sardelich ● 12 years ago

i think request together with remote incremental backup is number 1 priority in reducing overall load and io problems...but balance on number of cores used should be really really well calculated...

photo
2
WanderingZombie ● 12 years ago

We've given up with making backups as they take too long... the best part of 6 hours. This then means the copying of the files to a different server happens during our busier hours and the load sky rockets despite all attempts to fine tune rsync to keep load low

photo
2
Gareth-AWD ● 12 years ago

Hurry up with this please.

photo
1
WhiteDog ● 12 years ago

WanderingZombie wrote:

We've given up with making backups as they take too long... the best part of 6 hours. This then means the copying of the files to a different server happens during our busier hours and the load sky rockets despite all attempts to fine tune rsync to keep load low
LOL. I can understand the frustration but what is your backup plan in case of a crash? Also you don't have to "wait" for pigz. Instructions are available in the thread (it's not that hard to implement) :)

photo
3
WanderingZombie ● 11 years ago

WhiteDog wrote:

LOL. I can understand the frustration but what is your backup plan in case of a crash? Also you don't have to "wait" for pigz. Instructions are available in the thread (it's not that hard to implement) :)
On a highly dynamic website, a backup from (perhaps) many hours before the problem occurred would require new hardware (cause of failure), a restore (un-proven to work) of the account, then get MySQL replicating again. Downtime from hardware fix alone would be an issue, let alone flaky restores. Unless you have the simplest of accounts in cPanel with no additional software installed, I wouldn't be much on the restore digging me out of a hole. cPanel's own services share the same instance of MySQL that we use, so who knows what'll happen after a restore. We have enough problems with EasyApache and other updates that change service passwords which then replicate to other servers and screw up their accounts. We could stop mysql.* tables from replicating, but we actually want them to for our purposes. cPanel gets in the way too much... it's often a toss of a coin to ditch or keep it tbh. As far as backup goes... we have all our code offsite and can FTP it to a replacement server. Our images are mirrored to a CDN. Email loss would be a problem, but hey, some people have far worse things in their daily life to deal with. I guess the major problem is that the cPanel backup just cannot be tweaked enough to backup what we need backing up whilst ignoring other things. There's going to be some config file somewhere that I could create I'm sure which might help, but it's bound to be far worse than the hoops you have to jump through just to setup some account specific Apache configuration. Every setting should be changeable via WHM or cPanel screens, not command line only, and only then if you have the documentation to hand.

photo
1
Aila ● 11 years ago ●

pigz integration with pkgacct and the backup system is complete. Barring incident it will be available with cPanel & WHM 11.38.

photo
1
WanderingZombie ● 11 years ago

Well after one of our servers got hacked after cPanel support spilled our root password into the wild we had to nuke it and start from scratch. A 'simple' process of restoring the accounts from backup our hosts said once the server had been setup with Centos and cPanel. If only! The restore process failed to setup SSL properly and it took many attempts to get it working. The new DHTML/AJAX certificate page is a nightmare, pulling in outdated (bad browser cache management?) certificate files which didn't match the key being installed. Careful paste, spot the reload and then paste the certificate back in again got around that. This server is unbelievably simple - a couple of accounts, each with just 2 PHP pages, a css file and a couple of images, and self-signed SSL certificates (fine for what we need). Yet the restore process couldn't deal with it. Would I trust it to our big live sites... no chance.

photo
1
lukapaunovic ● 11 years ago

Any news?!

photo
3
lukapaunovic ● 11 years ago

When cPanel & WHM 11.38 will be released?!

photo
1
WireNine ● 11 years ago

lukapaunovic wrote:

When cPanel & WHM 11.38 will be released?!
this.

photo
1
lukapaunovic ● 11 years ago

WireNine wrote:

this.
?

photo
2
WanderingZombie ● 11 years ago

Looks like we might get this sooner than 11.38? Have just received this email from cPanel...We are excited to announce cPanel & WHM software version 11.37.0.5 has gone to the EDGE Tier.


If you are not yet ready to receive the 11.37 series, this is your last chance to change your update preferences.


In our newest version of cPanel & WHM software, you will find improvements in:


- SSL Management System

- Backup System

- Other Improvements:

- Able to configure hosts used for Email Autodiscovery and Autoconfiguration

- Support for pigz in Account Transfer and Backups

- New cPanel & WHM installs will feature MySQL5.5

- Improved jailshell functionality

- Added jailshell mount configuration to Tweak Settings

- Jail Apache Virtual Hosts Via mod_ruid2 and cPanel Jailshell

- TailWatch Updates

- Integration Focused Improvements

- Improvements to Apache Configuration

photo
1
lukapaunovic ● 11 years ago

I updated, everything works just fine, I am waiting tomorrow to see backup results.

But I am really angry!

You didn't implemented pigz for "Backup Wizard" in cPanel, it still uses gzip, why?

photo
1
Aila ● 11 years ago

WanderingZombie wrote:

Looks like we might get this sooner than 11.38?
11.37 is the development release of 11.38. Please read our documentation on software versions for more information: http://docs.cpanel.net/twiki/bin/view/AllDocumentation/InstallationGuide/CpanelProductVersions

photo
2
Aila ● 11 years ago

lukapaunovic wrote:

I updated, everything works just fine, I am waiting tomorrow to see backup results.

But I am really angry!

You didn't implemented pigz for "Backup Wizard" in cPanel, it still uses gzip, why?

Once enabled in tweak settings, then all backup functions that use pkgacct will use pigz, this includes the Full Backup functionality in the Backup Wizard.


Any compressing activities that occur outside pkgacct need addressed as separate requests.

photo
1
lukapaunovic ● 11 years ago

Kenneth Power wrote:

Once enabled in tweak settings, then all backup functions that use pkgacct will use pigz, this includes the Full Backup functionality in the Backup Wizard.


Any compressing activities that occur outside pkgacct need addressed as separate requests.

Thanks

photo
1
Hostiso ● 11 years ago

Activated on all our servers, nice improvments :)

photo
1
Eric Caldwell ● 11 years ago

WanderingZombie wrote:

Well after one of our servers got hacked after cPanel support spilled our root password into the wild we had to nuke it and start from scratch. A 'simple' process of restoring the accounts from backup our hosts said once the server had been setup with Centos and cPanel. If only! The restore process failed to setup SSL properly and it took many attempts to get it working. The new DHTML/AJAX certificate page is a nightmare, pulling in outdated (bad browser cache management?) certificate files which didn't match the key being installed. Careful paste, spot the reload and then paste the certificate back in again got around that. This server is unbelievably simple - a couple of accounts, each with just 2 PHP pages, a css file and a couple of images, and self-signed SSL certificates (fine for what we need). Yet the restore process couldn't deal with it. Would I trust it to our big live sites... no chance.
We had the same thing happen a couple of years ago because we setup WHMCS incorrectly. Server got rooted. The restores worked perfect for us. Some lessons we learned.


1. Change/rotate your root password every time you give it out for support.

2. Never have your server set for SSH password auth, ONLY use ssh keys.

3. Set your SSHD to listen on a random port! Don't use 22.

4. If you don't offer Jailed shells (we don't), further lock SSHD down via hosts.allow for ONLY IP's you know.


Those 4 steps alone help tighten up your servers. Nothing is hack proof but, it stops all the script kiddie crap that happens.

photo
1
Sysop-Legionnaire ● 10 years ago

Any updates on this request? cPanel please update now!

photo
1
Aila ● 10 years ago

Sysop-BRAZIL wrote:

Any updates on this request? cPanel please update now!
Pigz support was delivered in cPanel & WHM version 11.38, one year ago.

Replies have been locked on this page!

cPanel, WebHost Manager and WHM are registered trademarks of cPanel, L.L.C. for providing its computer software that facilitates the management and configuration of Internet web servers.