[MlMt] What takes time at startup?

[Helen Holzgrafe]

Patrick,

Do you have any smart folders? If you do, cut down the number of 
messages searched to create them the in a similar manner to what I did. 
I am sure as I explained below that MailMate is doing a lengthy analysis 
of every single mail message for every single smart folder if you  are 
using "all messages" as your search mailbox for them. Also, if "all 
messages" is 2 million messages it's likely searching 2 million messages 
every time for every single smart mailbox.

That's overwhelming for your computer if you are short of RAM and pretty 
inefficient on MailMate's part. Also, it may not be able to keep info 
for that many messages at a time in RAM and so must swap that in and out 
multiple times to get the job down. That's called thrashing in Computer 
Science nomenclature.  In bad cases it can causes slowdowns that seem 
beyond reason (1000 times as slow as you would expect, or so).

If you don't have any smart folders at all, then I think my analysis may 
be short of what is actually happening.  Perhaps, Mailmate creates its 
entire database, not just its smart folders each time it starts up.

To me it is very clear that MailMate just thrashes during startup, 
particularly if you have large folders it must analyze.

My suggestion would be to make all your folders - IMAP and Smart - 
smaller. For example, if you keep messages by year, then switch to 
quarterly or monthly or weekly to get the number of messages MailMate 
feels it needs to work with at any one time below 100000 (or better yet 
below 50000).

Based on my analysis of MailMate on my machine, I do believe it is 
spending a lot of extra time when it is forced to deal with very large 
folders. I am sure this is a kernel thrashing issue and the only way 
around it would be for Benny to spend some time on new algorithms that 
directly cut down on the thrashing.

If you only have 4gigs of RAM in your machine, run out today and up it 
to 8 or 16. I really think you will see a marked improvement.

Benny may tell us all that my analysis is incorrect, but either way, I 
think you should look at your folder structure and make all your folders 
much smaller and see if that helps you. I know it did for me.

-Helen

On 6 Feb 2016, at 13:03, Steven M. Bellovin wrote:

> I have 1.4M messages, and while startup takes a while it doesn't take 
> nearly as long as Patrik reports.
>
> On 6 Feb 2016, at 15:49, Patrik Fältström wrote:
>
>> 350k email messages is nothing.
>>
>> I have just below 2 million. 12k added each month, approximately.
>>
>> There is something happening at startup that takes very long time. 
>> And I am interesting in knowing what happens.
>>
>> Patrik
>>
>> On 6 Feb 2016, at 19:43, Helen Holzgrafe wrote:
>>
>>> Hi,
>>>
>>> Also, if you have Apple's "Activity Monitor" (inside the 
>>> Applications/Utility folder) running when you start up Mailmate, you 
>>> can directly see that Mailmate is using 100% of your computer's 
>>> available  CPU cycles during its startup.
>>>
>>> That's why a Mailmate Startup window that shows some form of 
>>> progress bar, the name of each folder as it is being processed, or 
>>> "25 of 300 Smart Folders verified" or some other indicator gives the 
>>> user faith that MailMate is actually doing something and not 
>>> hanging. This window also gives some clue as to how long until 
>>> MailMate is done with start up.
>>>
>>> -Helen
>>>
>>> On 6 Feb 2016, at 10:04, Helen Holzgrafe wrote:
>>>
>>>> Hi,
>>>>
>>>> I have a partial explanation from my experience.  I have around 
>>>> 350000 mail messages, mostly archived, and quite a few smart 
>>>> mailboxes.  For a while it was taking over 10 minutes to start up, 
>>>> which was getting incredibly annoying.  I used the following method 
>>>> to get the time down to a manageable 30 seconds.
>>>>
>>>> I created essentially two archives. I have 50000 or so messages in 
>>>> the regular archive. These messages I actually want to search a lot 
>>>> and create smart folders from. I also created another folder inside 
>>>> the real archive folder that I call "Deep Archive". This is not a 
>>>> smart folder, but an actual iMAP folder. It contains around 300000 
>>>> messages I must keep, but I don't expect to search or create smart 
>>>> folders from.
>>>>
>>>> Then, I created a smart folder called "All Accounts - No Deep 
>>>> Archive" which contains all messages in all of my accounts and 
>>>> excludes the "Deep Archive" folder.  Now all my smart folders use 
>>>> "All Accounts - No Deep Archive" for the Mailboxes to search in 
>>>> (the first tab on "edit Mailbox" window).  I no longer search "All 
>>>> Messages" for each smart mailbox.  Just in case it matters, which I 
>>>> do not know for sure, I also placed this smart folder at the top of 
>>>> my folder list so the odds are greater that it will be created 
>>>> first before all the other smart folders.  Only Benny can tell us 
>>>> if this placement at the top actually helped.
>>>>
>>>> Why does this work? My educated guess:
>>>>
>>>> My guess is that Mailmate somehow must be recreating the contents 
>>>> lists for each smart mailbox on the fly each time it starts up, 
>>>> rather than keeping that as part of its database all the time. Very 
>>>> time consuming at start up, but it does guarantee accuracy.
>>>>
>>>> Remember that messages can come in or you can mess directly with 
>>>> your IMAP folders while MailMate is not running.  So Mailmate must 
>>>> do something at start up to make sure all those smart folders have 
>>>> correct information.  The problem for me was that it was searching 
>>>> 350000 messages for every smart folder that used "All messages" as 
>>>> the mailbox it searched through.  That's a lot of messages.
>>>>
>>>> I also think Mailmate was also thrashing in some fashion trying to 
>>>> bring that many messages in and out for search. So, I also suspect 
>>>> that how much RAM memory you have in your machine will also affect 
>>>> how much MailMate goes into overdrive to get this start up done.
>>>>
>>>> It might be good for Benny to put up a window during this startup 
>>>> that shows how much time it is spending creating your smart folders 
>>>> or some other method of letting you you that it's actually working 
>>>> and not actually "not responding" as the operating system reports.
>>>>
>>>> Benny, how close did I come to explaining this problem correctly?
>>>>
>>>> -Helen
>>>>
>>>> On 6 Feb 2016, at 9:12, John Cooper wrote:
>>>>
>>>>> Have you tried uninstalling MailMate, including its support 
>>>>> folders, and then reinstalling it?
>>>>>
>>>>> Patrik Fältström wrote (at 8:54 on 6 Feb 2016):
>>>>>
>>>>>> On 6 Feb 2016, at 17:45, John Cooper wrote:
>>>>>>
>>>>>>> Patrik Fältström wrote (at 4:01 on 6 Feb 2016):
>>>>>>>
>>>>>>>
>>>>>>>> When starting MailMate it does something. For a very long time.
>>>>>>>>
>>>>>>>> What's up?
>>>>>>>
>>>>>>>
>>>>>>> What's it say in the Activity Viewer while this is happening? 
>>>>>>> (Window > Show Activity Viewer)
>>>>>>
>>>>>> There is nothing, no response even from the application. No menus 
>>>>>> nothing. "Application not responding" in the mach kernel although 
>>>>>> it do send events.
>>>>>>
>>>>>> Patrik
>>>>>> _______________________________________________
>>>>>> mailmate mailing list
>>>>>> mailmate at lists.freron.com
>>>>>> http://lists.freron.com/listinfo/mailmate
>>>>> _______________________________________________
>>>>> mailmate mailing list
>>>>> mailmate at lists.freron.com
>>>>> http://lists.freron.com/listinfo/mailmate
>>>> _______________________________________________
>>>> mailmate mailing list
>>>> mailmate at lists.freron.com
>>>> http://lists.freron.com/listinfo/mailmate
>>> _______________________________________________
>>> mailmate mailing list
>>> mailmate at lists.freron.com
>>> http://lists.freron.com/listinfo/mailmate
>> _______________________________________________
>> mailmate mailing list
>> mailmate at lists.freron.com
>> http://lists.freron.com/listinfo/mailmate
>
> _______________________________________________
> mailmate mailing list
> mailmate at lists.freron.com
> http://lists.freron.com/listinfo/mailmate
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.freron.com/pipermail/mailmate/attachments/20160206/1ad69c13/attachment-0001.html>