[MlMt] High Sierra, APFS, Time Machine, and MailMate...

[Steven M. Bellovin]

On 12 Dec 2017, at 20:10, Steven M. Bellovin wrote:

> On 12 Dec 2017, at 17:48, Bill Cole wrote:
>
>> On 12 Dec 2017, at 1:10 (-0500), Steven M. Bellovin wrote:
>>
>>> On 11 Dec 2017, at 23:26, Bill Cole wrote:
>>>
>>>> On 10 Dec 2017, at 21:14 (-0500), Steven M. Bellovin wrote:
>>>>
>>>>> My suspicion is that the problem has to do with very large 
>>>>> directories on APFS file systems
>>>>
>>>> This would be shocking. One of the rationales for APFS existing is 
>>>> that the HFS foundation was played out for dealing with large 
>>>> directories efficiently. I haven't looked into the details (life is 
>>>> short...) but if APFS is *worse* than HFS{+,X} with large 
>>>> directories then Apple is in a worse state than I had thought...
>>>
>>> Yah. I have no other explanation, though. To give a current example, 
>>> on a machine -- an old one, to be sure -- a Time Machine backup 
>>> started almost 10 hours ago. It's dumped 77.5 MB -- out of a total 
>>> of 152.7 MB -- in that time, and it's been at about 77 MB for the 
>>> last ~7-8 hours. At some point, though, it will pass the expensive 
>>> point and run at a reasonable rate. This dump is to a directly 
>>> connected USB 2.0 drive. And the CPU is about 96% idle, according to 
>>> 'top'.
>>>
>>> Btw: by "big", I mean that I have one mailbox with 114K messages; 
>>> the directory itself is 3.6 MB. No other mailbox is more than half 
>>> that size, though I have four that are over 1 MB.
>>
>> Oh my.
>
> Yah. I knew some were large, but I didn't think *that* large. Worse 
> yet, one of the top few is my inbox, which I haven't been cleaning out 
> of late. I've been following the MailMate mantra: just create smart 
> folders...
>>
>> Since the backup disk can't be APFS (Time Machine relies on 
>> hard-linked directories, which APFS won't do) you're still dealing 
>> with that huge directory in HFS+ on the write side. If that directory 
>> has changes it is going to be spectacularly slow for TM to do 114k 
>> file hard links and copy a handful of changed files into a new 
>> directory.
>
> Right, which explains older slowness, but not the sudden problem.
>>

OK. I split up the really largest mailboxes, which has taken care of the 
immediate problem: I can now do backups in finite time. I still suspect 
an APFS issue which suddenly made the problem critical, but at least I'm 
back on the air. (And I'm building a Python script to ease mailbox 
splitting...)


         --Steve Bellovin, https://www.cs.columbia.edu/~smb