[MlMt] Improving search performance?

[Bill Cole]

On 2023-06-16 at 06:26:11 UTC-0400 (Fri, 16 Jun 2023 12:26:11 +0200)
Robert M. Münch <mailmate at lists.freron.com>
is rumored to have said:

> I have the feeling the search is taking considerable longer than in the past.
>
> IMO MM flexible, easy to use search is a key USP.
>
> @Benny: Would be nice to speed things up (again). Not sure how MM does searching, but using SQLite for full-text indexing, or special indexing should help a lot.

MM uses an indexing mechanism that appears to be custom-designed for the specific purpose of searching email. You can see the artifacts of that in ~/Library/Application Support/MailMate/Database.noindex/. Only Benny could conceivably explain the details, but it seems to me to be unlikely that he would get much from ripping all that out and replacing it with SQLite or some other off-the-shelf tool.

One serious issue with indexing email is that email is highly divergent in data structure, and while you can do a simple index for basic standard mail metadata, "full text" and "all headers" search for mail is a nightmare because real-world mail breaks almost every rule theoretically governing it and it is not a simple matter to determine what is or is not body text. Email typically arrives with multiple alternative parts theoretically representing the same message, possibly QP or B64 encoded and usually including one version with HTML markup. And that markup can be bad, wrong, or even intentionally malicious. Very large mail stores are inherently tough to search.


-- 
Bill Cole
bill at scconsult.com or billcole at apache.org
(AKA @grumpybozo and many *@billmail.scconsult.com addresses)
Not Currently Available For Hire