[MlMt] MailMate so slow it locks up

[Bill Cole]

On 15 Jun 2016, at 11:10, Alex Bligh wrote:

> I'm having great difficulty moving from Mail.app & Thunderbird to 
> MailMate, because MailMate is so so slow it locks up.
>
> Here's the background. I am currently using Mail.app and Thunderbind 
> to access a couple of IMAP accounts. The IMAP accounts are LARGE. The 
> smaller one has I think 200,000 - 300,000 messages in it, and the 
> large one probably five times that (1,000,000 - 1,500,000). I am on a 
> high powered Mac Pro with SSD disks running El Capitan, lots of CPU, 
> lots of RAM and lots of bandwidth. Mail.app and Thunderbird work fine. 
> Because Mail.app does not support IMAP subscriptions (and I don't want 
> all 1m+ messages on the SSD syncing all the time) I have a homebrewed 
> IMAP filter between Mail.app and my server (so this isn't a strictly 
> fair comparison for Mail.app), but Thunderbird accesses both accounts 
> directly in the normal way.

I'm unclear on why it is problematic for Mail.app to have all of the 
million messages on disk. Obviously you understand IMAP well enough to 
have built the filter so I won't presume to get didactic about the 
protocol details, except to point out that the work needed to keep a 
complete client-side cache in sync with the server state isn't a 
function of message count, but rather of how more complicated issues 
like how many mailboxes exist and how many of those are subject to 
frequent content and/or flag changes. Is there some quirk of Mail.app 
that makes it sensitive to message count?

> On installing MailMate, I set up the smaller account first (300,000 
> messages). It correctly imported the settings, but after that there 
> was a world of pain. It sync'd all the mail, but only by running 2 CPU 
> cores at 100% continuously for over 24 hours.

Based on my experience with MailMate indexing database rebuilds from 
existing local stores of similar raw message counts, this implies that 
the bottleneck is data flow from the server. I last did a rebuild of the 
index DB for a machine with a set of accounts with ~500k messages using 
~7GB of disk, and on 2009's finest iMac (albeit with 32GB RAM, much of 
it unused) it took less than 2 hours, during which I was able to do 
other things without noticing any slowness. This implies that MM is 
waiting VERY HARD for data from the server, which isn't giving MM 
message data as fast as MM can process it. OR: there's something about 
your 300k messages that is starkly different from my 500k and is causing 
the MM indexing procedure to work spectacularly slowly. An example of 
how mail content might be able to slow indexing would be non-standard 
headers with large and highly varied content, because unlike most (maybe 
all?) other MUAs, MailMate indexes *ALL* headers as well as message 
bodies.