[MlMt] High Sierra, APFS, Time Machine, and MailMate...

Steven M. Bellovin smb at cs.columbia.edu
Sun Dec 17 12:56:03 EST 2017

On 12 Dec 2017, at 20:10, Steven M. Bellovin wrote:

> On 12 Dec 2017, at 17:48, Bill Cole wrote:
>> On 12 Dec 2017, at 1:10 (-0500), Steven M. Bellovin wrote:
>>> On 11 Dec 2017, at 23:26, Bill Cole wrote:
>>>> On 10 Dec 2017, at 21:14 (-0500), Steven M. Bellovin wrote:
>>>>> My suspicion is that the problem has to do with very large 
>>>>> directories on APFS file systems
>>>> This would be shocking. One of the rationales for APFS existing is 
>>>> that the HFS foundation was played out for dealing with large 
>>>> directories efficiently. I haven't looked into the details (life is 
>>>> short...) but if APFS is *worse* than HFS{+,X} with large 
>>>> directories then Apple is in a worse state than I had thought...
>>> Yah. I have no other explanation, though. To give a current example, 
>>> on a machine -- an old one, to be sure -- a Time Machine backup 
>>> started almost 10 hours ago. It's dumped 77.5 MB -- out of a total 
>>> of 152.7 MB -- in that time, and it's been at about 77 MB for the 
>>> last ~7-8 hours. At some point, though, it will pass the expensive 
>>> point and run at a reasonable rate. This dump is to a directly 
>>> connected USB 2.0 drive. And the CPU is about 96% idle, according to 
>>> 'top'.
>>> Btw: by "big", I mean that I have one mailbox with 114K messages; 
>>> the directory itself is 3.6 MB. No other mailbox is more than half 
>>> that size, though I have four that are over 1 MB.
>> Oh my.
> Yah. I knew some were large, but I didn't think *that* large. Worse 
> yet, one of the top few is my inbox, which I haven't been cleaning out 
> of late. I've been following the MailMate mantra: just create smart 
> folders...
>> Since the backup disk can't be APFS (Time Machine relies on 
>> hard-linked directories, which APFS won't do) you're still dealing 
>> with that huge directory in HFS+ on the write side. If that directory 
>> has changes it is going to be spectacularly slow for TM to do 114k 
>> file hard links and copy a handful of changed files into a new 
>> directory.
> Right, which explains older slowness, but not the sudden problem.

OK. I split up the really largest mailboxes, which has taken care of the 
immediate problem: I can now do backups in finite time. I still suspect 
an APFS issue which suddenly made the problem critical, but at least I'm 
back on the air. (And I'm building a Python script to ease mailbox 

         --Steve Bellovin, https://www.cs.columbia.edu/~smb

More information about the mailmate mailing list