New Topic

cPanel & WHM Feature Requests Topic Email

Category Email

Tags compress existing mail, email compression, mbox, mdbox

Ability to compress existing email

cPanelMichael shared this idea ago

Open Discussion

As a web hosting provider and system administrator, I would like the Compress Messages option in WHM >> Mailserver Configuration expanded to include additional functionality that compresses all existing messages stored on the server in order to maximize disk space savings.

--------------------------------------------------------------------

The Compress Messages option in WHM >> Mailserver Configuration enables compression for recently created and delivered messages. I would like to see this option expanded to cover all existing messages stored on the server.

Forums thread: https://forums.cpanel.net/threads/mailbox-compression-and-maildir-mdbox-conversion.623391

I like this idea 0

Replies (8)

cpanelnick ●

As a workaround, you could convert from maildir to mdbox and back once compression is on.

Reply URL

benito ●

This is safe? I read some comments about loosing flags or something.

Thanks

URL

Jay Bernier ●

You can do the conversion, but (see below) from my testing it messed up all the client-based flags in Mac Mail.

we experienced success using impacsync with no side-effects whatsoever (again see my comments below)

URL

Morten ●

Yes, I read that some places also. But that is alot of work if you have hundreds of servers.

Would be much easier to run a script that does compress all old emails on the whole server:

/scripts/compressmail :-)

Reply URL

Greg ●

Yes please add this feature.

Being able to compress all existing messages without having to convert the mailbox to and from would be so much better and dramatically reduce the possibility of errors and problems during the conversion.

Reply URL

Jay Bernier ●

We have a few really huge mailboxes. like many GB is size. It would be fantastic to have this "compress old email" option, even if it were a script to run in the shell.

Reply URL

Jonno ●

This is a needed feature. Often we migrate cPanel accounts from other legacy servers or hosts who don't use compression and have huge 20-40gb mailboxes. Is this in the roadmap yet? Thanks!

Reply URL

benny@cpanel.net ●

Nope, not yet. When it is, we’ll make sure to update folks here!

URL

overseer ●

Just to chime in -- please add this feature to cPanel, since the rest of the framework is already in place, it shouldn't be too great of an engineering challenge to implement. I manage several cPanel / WHM servers and this is a BADLY needed feature. One server (a VPS where I have root WHM access) has been running for 8 years and seen many cPanel upgrades, so now it has 250+ IMAP e-mail accounts that are ever growing larger. To compress them would represent HUGE disk savings. The other server is a reseller server with dozens of hosting accounts, and again over 5 years the mailboxes have ballooned in size. On that server, I only have limited WHM access, but will be migrating to a VPS this quarter. I really need to compress old mail messages in order to realize disk usage savings -- which directly translates to financial savings due to disk quota allotments -- $10 per 10GB/mo. I could be saving $30-50/mo if this feature was implement!

URL

Steven Dunn ●

I’m in the same situation with client’s who have extremely large email accounts. This would result in a massive storage space savings.

URL

Peter Armstrong ●

@Benny, did this discussion ever progress further within cPanel? It could potentially make a huge disk space saving to many of our clients.

URL

Jay Bernier ●

FWIW, we've had success approaching this a different way, using: http://imapsync.lamiral.info/#DOC_BASIC_UNIX

It's not without it's drawbacks though, namely, requiring a 2nd server and also requiring a notification to go out to the users of the mailbox you need to move informing them that due to system maintenance their mail account is being migrated to a more efficient format (which is true) and they will need to log into webmail to reset their password back to whatever it was after the migration.

You then perform a pkgacct and restorepkg of their account to a 2nd cpanel server (either temporarily setup for this or if you have a 2nd production cpanel server), reset their password via the webmail interface in source and destination to your temporary password, and run this script (with the appropriate flags) to migrate just the changes since the account backup and restore. You could then either leave them on your 2nd production server and change DNS records, or migrate them back from your temporary server after removing the account and recreating it to get the compression.

I don't believe this would work using the same domain (even if you used a different account) on the same server. A 2nd server is required and also outage to the client, which is not ideal. Depends upon your client I suppose, you may get some of them to accept this and get the savings.

I can also confirm the client-side flags are unharmed using this method because the scripts acts as an IMAP client.

URL

maquinadigital ●

You don't have to have a second server if you have ssh root on that server.

I use IMAP sync all the time to get IMAP accounts into our servers and it's a bit slow, with message per message sync.

A proper conversion is still needed and I don't really understand why this features has so few votes.

Regarding performance, are the compressed mailboxes bringing to much load to the servers? Does anyone have feedback on this?

Another thing, is it possible to have just a few compressed mailboxes or we have to activate that at the server level and there is no other way to try compression?

I'd like to try with a couple of mailboxes before going balistic with a complete server.

URL

Nayemur ●

+1 for this one. Due to the lack of this feature, disk usage is showing data for individual mail boxes, when old emails are not compressed. This is very misleading for end cPanel user.

Reply URL

Jay Bernier ●

I thought I would leave another comment on this thread that I originally commented on ~5 months ago. I did some of my own testing with our top 6 accounts size-wise. I took the packaged up accounts (from pkgacct) restored them to a test server with email compression enabled, then ran the conversion from maildir -> mdbox -> maildir.

Prior to the conversion I verified each set of folders inside each mail box and the total message count for each.

I then validated the same post-conversion to mdbox and back.

Here's what I found:

0) We definitely saved tons of space doing this.

1) There seemed to be no data loss (only counted the folders and message counts for each so I cannot comment on the actual emails themselves as I'd still be checking them!)

2) I DID find however that when connected to a converted mailbox (I did my own to test it out too), my client-side flags were all borked up. I happen to use Mac Mail that gives me multi-colored flags (at least on the Desktop client) and those were all completely different. It appears the conversion process changes some "ID" in the email file that means the client has the wrong emails 'flagged'. From my research I think this was discussed already online - but I don't have the link for reference. The only other solution I saw was a client-side bit of code that would preserve (rewrite) the old ID so as to avoid that issue. That looked a little too intense at the time though so I did not pursue it.

I guess the net of my comment would be: Even if they "enabled" this feature, I would suspect they'd need to crack the nut of keeping the IDs the same or at least restoring them automatically after compression. So the "tools" may already be there, but there's some other magic to really make it work seamlessly to the end user.

At least these are my thoughts based on my experiences..

Cheers.

Reply URL

Peter Zieseniss ●

Thank you Jay for this comment!

Your 2nd note is *extremely important* to me, and a total deal-breaker on any conversion...

My clients have 20-to-70 GB email accounts so the temptation to do something to limit disk space is immense. But 70GB worth of emails getting marked as unread, or dis-organized in any way, would definitely lose me my clients, permanently.

Also, for those hoping that compression is a panacea, please remember that compression usually does not apply to attachements (gzip won't save you any space on a jpeg, it might actually make it bigger). So, really, the aforementioned compression really only applies to a tiny portion of the usual space usage of emails..

URL

Joe Wasicek ●

Another vote for this. It's not fun telling the owner of a business they're paying $30 in disk storage fees just because they have 50GB worth of company email that could be compressed if they're willing to lose all the flags.

URL

Chris Gebhardt ●

This would be a huge feature add and I'm surprised it hasn't gotten more attention here. Who doesn't want to save space?

URL

Koree A. Smith ●

This is something we are looking at for 2022. Not sure exactly how it will look, but I will update here as we move forward.

Koree A. Smith

Product Owner

cPanel, LLC

Reply URL

Jay Bernier ●

This is great to hear Koree. This could save folks tons of space and would be greatly appreciated.

URL