[MlMt] Memory usage

[Bill Cole]

On 2021-05-11 at 07:17:25 UTC-0400 (Tue, 11 May 2021 13:17:25 +0200)
Charlie Clark <mailmate at lists.freron.com>
is rumored to have said:

> On 11 May 2021, at 9:10, Patrik Fältström via mailmate wrote:
>
>> Question, have anyone of you like myself had issues with memory and 
>> looked into what might causing it?
>
> It's almost bound to be indexes. There will be indexes for the headers 
> and presumably a full-text one as well. I would not be surprised with 
> a 1:10 relation from index to total mail size, for the text parts 
> (you'll see similar on a DB).

MM's indices are much less efficient, due to their comprehensive 
coverage and the nature of email. 1:3.2 is the ratio on my primary 
machine. One reason for that is that MM indexes every email header it 
ever sees and many headers are very rare, but it is more significant 
that it maintains both case-preserving and case-squashed indices for 
body text and the most common headers, doubling the size of the largest 
index files.

> Not sure if you can do anything about that but Benny's probably the 
> only person in a position to know.

Benny has actually made substantial improvements in how MM manages 
memory over the past few years. It is much better about how much of the 
full index it loads at startup and while it does tend to load more over 
time, it doesn't behave as greedily as it used to, preventing OS memory 
compression and seemingly never using "purgeable" memory. If he has some 
user-accessible trick for reducing memory usage, he has kept it a secret 
despite multiple threads here on the issue.

I suspect that the bottom-line summary is that high memory use is the 
unavoidable price of MailMate's ridiculously fast and flexible searching 
and smart mailbox functionality. There is no magical way around the fact 
that big data is big.

-- 
Bill Cole
bill at scconsult.com or billcole at apache.org
(AKA @grumpybozo and many *@billmail.scconsult.com addresses)
Not Currently Available For Hire