[MlMt] Local email archiving status/options?

[Bill Cole]

On 25 Aug 2015, at 10:58, David O'Donnell wrote:

> On 25 Aug 2015, at 7:03, Brian Scholl wrote:
>
>> Also, a more theoretical postscript: It seems to me that Benny's 
>> reluctance to pursue any sort of .mbox export […]
>
> I can’t speak to the IMAP-*v*-POP3 debate, but I would really love 
> the ability to export a series of messages as a .mbox file in the same 
> format that Apple Mail does.

Isn't mbox great: one always has to specify a particular tool's 
flavor...

If you want a reasonably portable mbox on your Desktop containing the 
messages in a MM IMAP (not "Smart") folder, it's easily done with a 
simple bit of shell calling the crufty old "formail" tool:

cd "~/Library/Application 
Support/MailMate/Messages/IMAP/[account at server]/[path-to-folder]/Messages/"
for fname in *.eml ;  do formail < $fname ; done > 
~/Desktop/Exported.mbox

If you want the delimiter lines' timestamps derived from Date headers 
instead of now, the conversion gets more arcane and will not like 
improper Date headers:

for fname in *.eml; do formail -a Date: < $fname ; done |
sed '/^From /s/, \(..\) \(...\) \(....\) \(..:..:..\) .*/ \2 \1 \4 \3/' 
 > ~/Desktop/Exported.mbox

Unfortunately, there's no sound way to get the proper delivery date into 
the delimiter lines unless you are an IMAP client, so Mail.app does one 
thing unequivocally better than formail. Also, unlike Mail.app, formail 
uses (CORRECTLY) the Return-Path header (if present) to derive the 
delimiter line address instead of Mail.app's simple use of the From 
message header and only adds blank lines ahead of delimiter lines on an 
as-needed basis.

So, while MailMate won't duplicate what Apple Mail does, one can get 
pretty close using the MailMate message store without MailMate doing the 
actual work.

> In addition to using SpamSieve on my Mac (which is quite good), I 
> maintain my own mail server (Mac OS X 10.6.8, until I am forced to 
> “upgrade”)

It is mildly amusing to learn that, as it means I'm not alone and in 
respectable company. In my case the excuse is that the original Core Duo 
can't do 64-bit mode and so can't run 10.7 or later, and I can't bear to 
junk that old machine...

(Tangent: Senior mail admins are insanely over-represented on this 
mailing list.)

> and use SpamAssassin there to try to intercept as much crud as 
> possible *before* it gets to SpamSieve. Apple Mail produces the 
> perfect .mbox files for feeding to SpamAssassin’s spam-learning 
> routines, so periodically I have to haul Mail out, select all the junk 
> that SpamAssassin needs to learn how to intercept, and File > Save 
> As… (raw source) to a file that I can then drag to the server and 
> run learnspam against.

I'm surprised that no one in this thread has yet mentioned the simple 
ability to select messages in a MM message list and drag them to a 
Finder window: a very fast way to create a file-per-message offline 
local archive.

As someone who has retained email for 20+ years including a substantial 
spam corpus (it's a professional focus) I share the desire for a local, 
integrated, purely private, and reliable mail archive. Even though I 
have my own Dovecot server running on a machine in the same room as my 
main desktop, I don't keep my biggest archive there. Instead, after 
years of working with various tools on a semi-converted pile of Eudora 
almost-mbox files, I finally bought a license for Emailchemy 
(http://www.weirdkid.com/products/emailchemy/index.html) a tool that can 
actually recover and convert Eudora's quirky Classic Mac format 
(CR-delimited mboxes with resource fork indexing and split attachments 
pointed to by HFS CNIDs) as well as just about any other mail format one 
might have into a variety of file-per-message and standard mbox formats 
INCLUDING a maildir-like tree with a trivial read-only IMAP interface 
meant to be used as a "Import Server." Since my "old archive" is just 
that, I had Emailchemy do the conversion and run the Import Server for 
long enough to have MailMate import the whole tree as a new IMAP 
account. That account has been in offline mode in MM for a couple of 
years now, happily challenging the robustness of MailMate's indexing & 
searching capabilities as a huge collection of very weird mail, much of 
it intentionally and maliciously in violation of any known standard. 
Works marvelously, except that MM eats a lot of RAM (unavoidable with 
400k+ messages) and is a bit sluggish for some searches.

Another approach I've played with but not really exercised hard is a 
zombie "IMAP Account" permanently offline as a local archive. The 
connection settings don't work and never have, but I can create folders 
and subfolders and MM creates the directories just as it would for a 
normal account. Benny has vaguely warned that this might not be a 
reliable approach, but I think that's only because it has no backend and 
ultimately would be lost if MM decided its local cache of messages was 
not to be trusted. Tags may not work also...

I think to some degree the objection to MM not having an explicit 
mechanism for local archive is a phantom issue perpetuated by Benny's 
politely overt hostility to the concept. Every IMAP folder you can see 
in MM has a complete local cache in the form of a folder full of plain 
text files with .eml extensions in pristine RFC822 format under 
~/Library/Application Support/MailMate/Messages/. Unless you exclude 
that MM Messages tree from backups or you don't do backups, you have 
backups of that cache. If you want a snapshot of what's in a particular 
folder right now to save for later outside of MM because you're about to 
whack it off your IMAP account, you can copy a whole folder elsewhere in 
the Finder. MM even has a command to show you any selected message in 
the Finder so you don't have to unhide ~/Library or type a long odd path 
into Terminal. MM even has a Spotlight importer now so that if you were 
to copy a folder out of the MM "cache" and then wipe it from the server, 
you could still use Spotlight to find those messages, which open in MM 
when double-clicked. If you wiped a folder last week from your IMAP 
account and now really want it back, fire up Time Machine, find it in 
the cache, restore it to someplace OUTSIDE of the cache, and you've got 
your local archive independent of MM. Import it into a zombie account, 
and it's a local archive directly accessible by MM.