[MlMt] MailMate so slow it locks up

[Alex Bligh]

Benny,

Thanks for your reply.

> On 15 Jun 2016, at 22:36, Benny Kjær Nielsen <mailinglist at freron.com> wrote:
> 
> On 15 Jun 2016, at 17:10, Alex Bligh wrote:
> 
> I'm afraid I cannot provide you with a solution, but at least I can provide a few comments/explanations. It may not be clear below, but in general I'm always embarrassed when performance clearly could be better.
> 
>> ... Thunderbird accesses both accounts directly in the normal way.
> 
> But maybe not in a fully offline mode?

Thunderbird and Mail.app access the smaller account in the normal way, i.e. all folders are synced and work online and offline.

On the larger account Thunderbird is only subscribed to some of the mailboxes (in this respect just like Mail.app) and works online and offline. Mail.app can't cope with that which is why I use the filtering app I wrote.

> MailMate is a so-called offline IMAP email client and there are no options to change that. This means that MailMate fetches and stores a local copy of every email in the account(s).

Like Mail.app and like, (I believe) Thunderbird in respect of subscribed mailboxes. My main client is Mail.app, and I'd be happy if MailMate worked just like Mail.app in respect of subscribed mailboxes and ignored everything else.

> In addition to that, every header of every email is indexed.

I believe the same is true in Mail.app and Thunderbird; Mail.app (at least) also indexes the bodies.

> Although optimizations mean that MailMate can handle fairly large email accounts (some users have more than a million),

Elsewhere on a list populated by Mac users with large mail spools, there seems to be a binary split between people who've got Mailmate to work with an enormous number of messages, and those who haven't. This split is not based on spool size. Most of them have (hence I got a recommendation to use it). It may be that I'm doing something wrong.

> MailMate was not really designed to handle “huge” accounts. There is likely more that I could do to handle large accounts more efficiently, but few of them are likely to be quick fixes. In short, MailMate is extremely flexible, but some parts do not scale well.

I think one thing that is problematic and is possibly easy to fix is simply that when downloading it does not 'yield' to the UI often enough. I would care less about the high CPU if the GUI would redraw. Oh, and also if I could see what it was actually doing and how far it was through. I tried the activity window but it appears to show IMAP instructions without showing which mailboxes it is syncing.

>> On installing MailMate, I set up the smaller account first (300,000 messages). It correctly imported the settings, but after that there was a world of pain. It sync'd all the mail, but only by running 2 CPU cores at 100% continuously for over 24 hours. Neither Mail.app nor Thunderbird do this; they import in the background, and (for that mail account) take only a few hours. Saying that, after 24 hours (perhaps a bit more), it had pulled all the mail in and appeared to work.
> 
> That's more time than I would have expected if you are on a fast connection and it isn't Gmail (Google throttles traffic), but since it did finish then some kind of looping behavior is probably unlikely.

It can pull about 50-70Mb/s from the mail server. There is no throttling in place. It's writing to a fast SSD. I'm guessing some sort of O(n^2) problem is in play.

>> I then enabled the larger account. [...] MailMate was fantastically unresponsive during this time - clearly importing is a real CPU and memory hog.
> 
> Yes, it can be pretty bad, and this is probably beyond the number of messages MailMate can handle.

Right, but as soon as I added it, I changed the option in the account subscriptions to only sync things which were subscribed on the server; in terms of subscribed mailboxes (server side subscriptions) the number of items should be manageable. I don't know whether it continued despite the UI change to try to sync everything, which might explain the issue.

>> The mail server is running cyrus imapd 2.4, has many users on it, and not one of them has seen any problems with any other MUA to my knowledge.
> 
> Most IMAP email clients are “online” in some sense or they only index a few headers of each message. And some email clients are probably just better than MailMate.

I *believe* Mail.app indexes everything, in that you can search mail effectively with no internet connection. I can't speak for Thunderbird, but inspection suggests it uses some composite strategy.

>> I regularly use Mail.app, Thunderbird (on OS-X) an iPhone, and until a couple of years ago Mulberry, with no issues. I know others are running all sorts of stuff (so imap logs tell me). The problem is not the mail server, and it's certainly not the client - Thunderbird is pretty fast, and Mail.app is positively zippy.
> 
> If I understood correctly then you don't have millions of messages in Mail.app, but I get the point.

In Mail.app I have the smaller account there in its entirety - this has 200k - 300k messages, and was already making MailMate choke. Mail.app has no difficulty here.

In Mail.app I have a filtered version of the larger account - in order to get around the criminal omission of any form of subscription handling, I wrote a filter (https://github.com/abligh/goimapfilter and various perl based antecedents) which intercepts the IMAP LIST and LSUB requests, and filters out particular mailboxes to hide them. The filtered version of the larger account is probably 500k messages or so, perhaps a bit more. The filtered version pretty much corresponds with what MailMate should see if it only looked at subscribed messages.

>> In the brief time I used MailMate on the smaller account, I really liked it. But even if I could get over the import issue, the fact that if I ever lose my mail spool I'd need to last 48 hours+ without a mail client that works is a red flag for me.
> 
> Very understandable.
> 
>> Any ideas what this might be and/or how to fix it?
> 
> MailMate does support subscriptions which means that if you don't really need all those messages at your fingertips then you should take advantage of that. Note that you can edit subscriptions before adding the account.

This is exactly what I was trying to do. I don't need my mail archive at my fingertips, would be happy to have it not available all the time, and would even be happy to use a different MUA to access it (that's what I current use Thunderbird for, and previously used Mulberry after my switch to Mail.app).

However, I can't get the subscription thing to work 'before adding the account'. I have set up the subscription correctly (checked with both Thunderbird and WebMail), and when I import the account into MailMate, it prefers its idea of local subscriptions to server side subscriptions. There is an option at the bottom of the subscriptions dialog I have to switch off in order for it to show the server side subscription list ungreyed. By that time it has already started downloading things. I don't know if this is part of the issue.

> I did not design MailMate to handle millions of messages (or even hundreds of thousands). Internally, all messages are put into one big pool and everything else is based on queries within this pool —— including all IMAP mailboxes. This only works with large message stores, because many parts of MailMate are actually quite fast. But if there is one thing I've learned then it's that every time I optimize MailMate to handle X number of messages then someone comes around asking for 2*X :-)

If the scalability reached that of Mail.app and no more, I'd still be interested. I don't much like using my IMAP protocol hack, and Mail.app seems more broken on every revision.

-- 
Alex Bligh