[MlMt] MailMate so slow it locks up

[Alex Bligh]

On 18 Jun 2016, at 00:51, Bill Cole <mmlist-20120120 at billmail.scconsult.com> wrote:

>>> That's more time than I would have expected if you are on a fast connection and it isn't Gmail (Google throttles traffic), but since it did finish then some kind of looping behavior is probably unlikely.
>> 
>> It can pull about 50-70Mb/s from the mail server.
> 
> Is that through a single IMAP session retrieving messages or just pumping a flood of bits?

Test of a TCP connection (HTTP), though downloading large attachments over IMAP is similar.

> IMAP servers use all sorts of different storage formats and I don't recall the details on Cyrus, but it is certain that you cannot pull 10,000 10KB messages in a tree of 100 mailboxes down over one IMAP connection as fast as you can transfer a single 100MB file via HTTP or FTP or SCP.

Undoubtedly. The point I was making is that it isn't a 'slow connection' problem.

> My 1st guess would be that you only have one IMAP session open and could benefit from opening more concurrent IMAP sessions to the server, but that you cannot for some reason. By default, MM will use up to 3 concurrent connections per account, and you can increase that for a specific source account by manually adding a "maxNumberOfConnections" integer key to the account's dictionary in ~/Library/Application Support/MailMate/Sources.plist.

OK I can look at that. Mail.app seems to run with a maximum of 4-6 but about 2 when idle.

> MailMate will silently reduce it's operational limit when it gets an indication from the server that it has hit a session limit (Benny magic... there's no consistency in how servers handle session limits.) If you've got multiple MUAs online with the same account simultaneously, it's not hard to hit a server-side limit, which (when they exist) tend to be ~10 but may be as high as 30. The fact that you're only eating 2 cores during an initial sync and not 3 or 4 may indicate that Cyrus is telling MM to get lost on the 2nd and 3rd connection attempts and MM is making do with one. On the other hand, I could be wrong: Benny may have MM behaving in a special way for a 1st synch.

I'm pretty sure this is not Cyrus saying 'go away' as (per above) Mail.app happily launches multiple connections. My proxy even reports regularly how many connections are open when using that.

>> There is no throttling in place. It's writing to a fast SSD. I'm guessing some sort of O(n^2) problem is in play.
> 
> That does not seem consistent with the fact that I can rebuild a MM index database from ~500k messages on local spinning rust on a vintage (late 2009 i7) iMac in under 2 hours while your MacPro took 24 hours to import 300k via IMAP. The only advantage I can see on my side to explain that is how fast I can pull message data from disk compared to how fast you can pull message data over IMAP.

It's strange isn't it? And I know plenty of people with large spools for whom MM works well.

> The rebuild task provides a progress window showing which mailbox it is working on, which strongly suggests that it is a fully serialized process: reading one message at a time and indexing it, very possibly using a single thread for that work. This makes it easy for me to test the lower bound for how fast a rebuild could be if it was entirely i/o bound by serially reading every message in my whole local MM spool It turns out I can read 525 messages/second, ~4MB/s. That would be an order of magnitude higher if HFS+ didn't suck, but it's still probably much faster than you can retrieve and store messages in a single IMAP session. Maybe even faster than you could in 3 sessions.
> 
> Can you try again and check how many sessions MM is actually opening and using?

Presumably you want me to wipe the local copy of the mailspool first? I can do that if you tell me how (starting Mailmate just beachballs at the moment).

-- 
Alex Bligh