[MlMt] What takes time at startup?

[Helen Holzgrafe]

Benny,

Thanks for letting me know I was right. I'd feel bad if I had steered 
people wrong.

Just for your information, with my set up of about 350000 messages (far 
less than a third of what Patrick and Steven have). My startup was 
around 10 minutes, too, like theirs. So, I think it's more about the 
chunks of data MailMate needs to keep in memory at a time rather than 
how much data in total.

Based on what I saw while trying to get my times down, the main problem 
was very large folders used as search folders.  I also did not get any 
benefit from using a search query as my mailbox to search. For example, 
if I used the search term "all messages but none in "deep archive" in a 
smart folder it did not cut down the time at all. But, if I made a smart 
folder with that same query and then used that smart folder by name in 
all my other smart folders I got a significant time savings.

I don't know what database you are using, but many of the better ones 
have a whole set of "queries" that you use instead of the basic one that 
are designed to bring in a part of the data at a time and as you use the 
data releases the memory and rolls in more automatically, thus reducing 
memory thrash.  I cannot recall the name for those kinds of queries and 
they are not that easy to use, but may be necessary to resolve these 
issues. I believe the Apple Developer standard database does have those, 
but I am not totally certain.

-Helen

On 6 Feb 2016, at 14:40, Benny Kjær Nielsen wrote:

> On 6 Feb 2016, at 22:03, Steven M. Bellovin wrote:
>
>> I have 1.4M messages, and while startup takes a while it doesn't take 
>> nearly as long as Patrik reports.
>
> I've had reports about this number of messages before, but I'm still 
> surprised it works at all with such a number of messages.
>
>> On 6 Feb 2016, at 15:49, Patrik Fältström wrote:
>>
>>> 350k email messages is nothing.
>>>
>>> I have just below 2 million. 12k added each month, approximately.
>
> I'm even more surprised this works although given the startup time one 
> could claim that it doesn't really work.
>
>>> There is something happening at startup that takes very long time. 
>>> And I am interesting in knowing what happens.
>
> I'll try to explain further below.
>
>>> On 6 Feb 2016, at 19:43, Helen Holzgrafe wrote:
>>>
>>>>> Why does this work? My educated guess:
>>>>>
>>>>> My guess is that Mailmate somehow must be recreating the contents 
>>>>> lists for each smart mailbox on the fly each time it starts up, 
>>>>> rather than keeping that as part of its database all the time. 
>>>>> Very time consuming at start up, but it does guarantee accuracy.
>>>>>
>>>>> [...]
>>>>>
>>>>> Benny, how close did I come to explaining this problem correctly?
>
> That's very exact. MailMate stores a copy of all headers (and some 
> virtual headers) of all emails and these are used on startup to build 
> *all* mailboxes. It's not just smart mailboxes. An IMAP mailbox is a 
> query for matches on a virtual header with a specific IMAP location.
>
> Everything in MailMate is based on these queries and with more than a 
> million messages then this can be slow on startup. It can also take up 
> a considerable amount of memory.
>
> Patrick: Now, 8 minutes is a lot and it could be interesting for me to 
> see where the bottlenecks are for this kind of problem size. You can 
> generate a log for me by pasting these lines in a Terminal window:
>
> 	defaults write com.freron.MailMate TimersEnabled -bool YES
> 	/Applications/MailMate.app/Contents/MacOS/MailMate
>
> When the launch has finished then quit MailMate (to make MailMate 
> summarize the results) and then send me the Terminal output. Send it 
> off list.
>
> (I cannot really promise to make more improvements soon, but there 
> might be some simple things you can change to speed up the launch.)
>
> -- 
> Benny
> _______________________________________________
> mailmate mailing list
> mailmate at lists.freron.com
> http://lists.freron.com/listinfo/mailmate
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.freron.com/pipermail/mailmate/attachments/20160206/419efc26/attachment.html>