Optimization: Unused md5 generated for every pkgacct file during backup

Austin Morton shared this idea ago

Needs Feedback

In the file /usr/local/cpanel/bin/backup when a file is setup to be transported to remote destinations an md5 is generated. If no remote destinations are setup, this md5 is entirely unused from what I can tell. Even with remote destinations enabled I was unable to find a place that made use of the md5.

sub transport_file {
my ($args_ref) = @_;
$logger->info("Queuing transport of file: $args_ref->{'local_path'}");
my $queue_entry = {
'cmd' => 'copy',
'user' => $args_ref->{'user'},
'md5' => Cpanel::MD5::getmd5sum( $args_ref->{'local_path'} ),
'time' => time,
'local_path' => $args_ref->{'local_path'},
'remote_path' => $args_ref->{'remote_path'},
'keep_local' => $args_ref->{'keep_local'},
'session_id' => $args_ref->{'session_id'}
};
return queue_backup_transport_item( $queue_entry, $args_ref->{'queue'} );
}

The biggest issue for me is that we are generating backups onto NFS storage, so generating this md5 causes the entire archive to be read back over the network again.

I like this idea 0

Replies (3)

mike_ ●

We have the same issue - backups are stored on an NFS volume, and the md5 sum wastes considerable time and bandwidth by reading back all of the archives.

Reply URL

Stewart Adam ●

This is a big deal for us as well, as we are using SSHFS for backups.

Reply URL

Stewart Adam ●

I spoke with cPanel support about this and they've provided, in my opinion, an adequate workaround: we can patch out the md5 checksum manually and have it automatically apply to the script using a 'postupcp' hook.

I've implemented this technique as follows:

cat << EOF >> /scripts/postupcp
run-parts /etc/postupcp.d
EOF
chmod +x /scripts/postupcp
mkdir /etc/postupcp.d
chmod 700 /etc/postupcp.d
cat << EOF > /etc/postupcp.d/backup-no-md5
#!/bin/sh
sed -i'.bak' -e "s/Cpanel::MD5::getmd5sum( \\\$args_ref->{'local_path'} )/''/g" /usr/local/cpanel/bin/backup
EOF
chmod +x /etc/postupcp.d/backup-no-md5

This will setup a folder /etc/postupcp.d and any scripts in it will be run after running "upcp". A script is established in this folder to patch out the md5 calls from the backup script.

Alternatively, one can also patch the backup script manually and then just mark it as excluded from cpsync: https://documentation.cpanel.net/display/ALD/The+cpanelsync.exclude+File

Reply URL