[MlMt] MailMate so slow it locks up

[Alex Bligh]

> On 16 Jun 2016, at 13:43, Bill Cole <mmlist-20120120 at billmail.scconsult.com> wrote:
> 
> On 15 Jun 2016, at 11:10, Alex Bligh wrote:
> 
>> I'm having great difficulty moving from Mail.app & Thunderbird to MailMate, because MailMate is so so slow it locks up.
>> 
>> Here's the background. I am currently using Mail.app and Thunderbind to access a couple of IMAP accounts. The IMAP accounts are LARGE. The smaller one has I think 200,000 - 300,000 messages in it, and the large one probably five times that (1,000,000 - 1,500,000). I am on a high powered Mac Pro with SSD disks running El Capitan, lots of CPU, lots of RAM and lots of bandwidth. Mail.app and Thunderbird work fine. Because Mail.app does not support IMAP subscriptions (and I don't want all 1m+ messages on the SSD syncing all the time) I have a homebrewed IMAP filter between Mail.app and my server (so this isn't a strictly fair comparison for Mail.app), but Thunderbird accesses both accounts directly in the normal way.
> 
> I'm unclear on why it is problematic for Mail.app to have all of the million messages on disk. Obviously you understand IMAP well enough to have built the filter so I won't presume to get didactic about the protocol details, except to point out that the work needed to keep a complete client-side cache in sync with the server state isn't a function of message count, but rather of how more complicated issues like how many mailboxes exist and how many of those are subject to frequent content and/or flag changes. Is there some quirk of Mail.app that makes it sensitive to message count?

It's more that Mail.app syncs either everything or nothing. I don't want it syncing the linux kernel mailing list (for instance) over my 3G connection, or it sits there sucking bandwidth. Nor do I want the gigabytes of my SSD taken up by archives. This is why an MUA that handled subscriptions would be good.

All the filter thing does is cause it to look at a subset of the mailboxes there, which is what subscriptions are for which Apple seem to have ignored in their wisdom. Indeed I was originally going to make the filter inject an LSUB and only list subscribed mailboxes.

>> On installing MailMate, I set up the smaller account first (300,000 messages). It correctly imported the settings, but after that there was a world of pain. It sync'd all the mail, but only by running 2 CPU cores at 100% continuously for over 24 hours.
> 
> Based on my experience with MailMate indexing database rebuilds from existing local stores of similar raw message counts, this implies that the bottleneck is data flow from the server. I last did a rebuild of the index DB for a machine with a set of accounts with ~500k messages using ~7GB of disk, and on 2009's finest iMac (albeit with 32GB RAM, much of it unused) it took less than 2 hours, during which I was able to do other things without noticing any slowness. This implies that MM is waiting VERY HARD for data from the server, which isn't giving MM message data as fast as MM can process it.

That would imply MM was waiting in a busy loop. I can get a 70Mb/s TCP connection to the server, so I doubt this is the issue. Also I've (admittedly not for a few months now) done resyncs of Mail.app over the same connection, and it's been pretty fast (a few hours).

An alternative is that the problem is in part that my connection is FASTER. If the network was the bottleneck, perhaps MM would spend the idle time in the UI redraw loop etc.

> OR: there's something about your 300k messages that is starkly different from my 500k and is causing the MM indexing procedure to work spectacularly slowly. An example of how mail content might be able to slow indexing would be non-standard headers with large and highly varied content, because unlike most (maybe all?) other MUAs, MailMate indexes *ALL* headers as well as message bodies.

Mmm... well I don't insert any headers myself on the MTA (apart from a Received: line or two). The vast majority of the messages are a variety of technical email lists, if that helps.

-- 
Alex Bligh