[MlMt] Moving large IMAP folders

[Bill Cole]

[ Sorry about the double-post. The backstory involves the indirect 
interaction of Little Snitch and a snowblower.]

On 11 Jan 2014, at 8:49, Kee Hinckley wrote:

>> On Jan 11, 2014, at 2:31 AM, "Bill Cole" 
>> <mmlist-20120120 at billmail.scconsult.com> wrote:
>>
>>> On 3 Jan 2014, at 14:04, Benny Kjær Nielsen wrote:
>>>
>>>> On 3 Jan 2014, at 17:08, Kee Hinckley wrote:
>>>>
>>>> Is there in fact an underlying IMAP command to move a folder 
>>>> hierarchy (as opposed to rename)? Or if I tell MailMate to move a 
>>>> folder, is it going to copy and delete?
>>>
>>> It is going to be copy and delete. Many servers though do that just 
>>> as efficiently as if it was a move (but not all servers).
>>
>> Does the IMAP RENAME command not actually work reliably to move 
>> folders and their children? 
>> http://tools.ietf.org/html/rfc3501#section-6.3.5 seems to say it 
>> should.
>
> I ran into an issue with this the other day and meant to comment on 
> it.
>
> Apparently IMAP folders aren't really hierarchical in a true sense, so 
> if you rename a folder and expect it to do what a directory does when 
> you rename it (have all the same folders under it, not just the 
> immediate files (messages)) you have to change the path of each 
> folder.

That is addressed in the RFC at the section I gave the link to:

    If the name has inferior hierarchical names, then the inferior
    hierarchical names MUST also be renamed.  For example, a rename of
    "foo" to "zap" will rename "foo/bar" (assuming "/" is the
    hierarchy delimiter character) to "zap/bar".

In other words: the protocol spec requires the *server* to do any 
downstream renaming. A server that only renames an enclosing folder 
while leaving its subfolders at their old names is broken.

Administering mail servers has been a substantial part of my work for 
many years, which is why I know which RFC to go to for IMAP4rev1 
details, but unlike Benny I don't maintain code that actually has to use 
IMAP4rev1 as a client with a diverse jungle of servers. I could manually 
test what a handful of different IMAP servers actually do, but that 
would not answer the question of whether MM uses a copy+delete process 
out of necessity because RENAME is broken often enough to make it 
useless or out of a misunderstanding of what RENAME actually does.

It is useful to understand that for many years of the early life of 
IMAP4, there was a widely deployed diversity of formal breakage in 
servers and clients which created a culture of ad hoc pragmatic 
workarounds that lives on in botches like GMail's faux-IMAP and the maze 
of client adaptations to it. The hierarchical model is one area where 
this has been focused, a consequence of the many edge and corner cases 
of multiple uncoordinated clients (and often mutually oblivious 
server-side processes) having simultaneous access with modification 
powers to a directory tree.

> Mailmate's rename is only renaming the top-level folder and not 
> telling IMAP to rename those under it. So you end up with the top 
> level folder having a new name, but all the folder children still 
> having the old path.

The way to do that with a correctly-behaving IMAP server is to create a 
new folder, copy all of the existing messages into it, and delete them 
from the original folder. This seems to be what MM does and it is a less 
risky and more universally workable strategy, but in principle it should 
be possible to do a real tree rearrangement with one RENAME command 
instead. I am interested in *why* Benny chose not to use RENAME.

> I can't think of any case where that's the preferred behavior. I know 
> the recursive rename is the default in my webmail. I ended up going 
> there to rename my folder.
>
> (The efficiency issue is a separate one. I can imagine that if 
> MailMate issues the rename, it could be smart enough to know that all 
> it has to do locally is move the messages. I don't know I'd it does, I 
> imagine that logic could get tricky.)

Historically, IMAP servers have mostly mimicked a hierarchical 
filesystem for mailboxes rather than actually using their real 
filesystems as a directly mapped hierarchy. This solves some server-side 
problems (performance, access control, concurrent acess, metadata 
storage, etc.) at the cost of making logically atomic operations on the 
directory tree non-atomic at the real filesystem level. Depending on 
their specific storage models, servers using file-per-message schemes 
("Maildir" and related schemes) may have as many distinct filesystem 
operations as there are leaf nodes or branch points in a sub-tree being 
moved with RENAME, which must seem atomic to clients.