[MlMt] lock ups

[Bill Cole]

On 28 Nov 2012, at 12:11, Benny Kjær Nielsen wrote:

> On 28 Nov 2012, at 17:29, Timothy Wright wrote:
>
>> It is definitely high on the memory side.
>> When I started this email, it was using 1.3GB of RAM. 5 minutes later 
>> it is
>> sitting at 1.71GB. It shows that I have 30mb out of 8GB of RAM free. 
>> Also
>> regularly sits at 50% CPU.

There's something else involved here. 30MB free in a 8GB system 
indicates severe systemwide RAM pressure. It is likely that the system 
is spending a lot of time trying to meet the demands of all the running 
processes for RAM. That 30MB is mostly transient little regions of RAM 
freshly freed by page-outs and about to be allocated to a hungry 
process. MM is surely one of those, but you must have something else 
contributing to overall RAM starvation. A system in that state is likely 
to respond slowly and show beachballs for any app switch. Common 
collaborators with MailMate in getting a Mac to that state include 
iTunes, Safari, Firefox, Chrome, and VMWare Fusion. It's also possible 
to make Terminal a memory troublemaker, but you have to push it very 
hard to get there.

> The CPU is fine. The memory usage is high, but the main problem is if 
> it continues to increase. That would indicate that MailMate caches too 
> much information in memory while importing. I'm not sure how much 
> memory is used in regular use with 200K messages, but I wouldn't be 
> surprised if it's more than 1GB. I have >30K messages and right now 
> it's at 170MB.

Data point from 'top' output on a machine with >390K messages:

PID    COMMAND   %CPU TIME     #TH  #WQ #PORTS #MREGS RPRVT  RSHRD  
RSIZE  VPRVT  VSIZE
13146- MailMate  0.0  03:03:09 30    1   353+   3924+ 1820M+ 78M+   
1654M+ 2662M+ 3544M+

That was after 55 hours of runtime on a Lion machine with 8GB RAM and no 
overall memory pressure (>2GB free). MM has 8 source accounts (7 in use, 
1 testing account mostly offline). Database.noindex eats 1.6GB and 
Messages 3.0GB

There are a few things here which seem a bit odd:

PORTS: Not insanely high, but 354 puts MM in the top 10 on this machine 
and that number seems to grow (leak?) slowly over time.
MREGS: A long-running MM process always ends up with a huge number of 
memory regions, even beating out sloppy apps like Safari and iTunes. 
Also grows slowly over time and never shrinks much, so there may be a 
leak. 3 hours after that sample, the count is now 3959.
RPRVT/RSHRD/RSIZE: I have no idea how RPRVT>RSIZE can be, but it 
routinely is for MM. Obviously this is partly a top bug, but whatever 
mis-counting causes top to violate the concept of RPRVT+RSHRD=RSIZE is 
uniquely exercised by MM.

After a few hours of running, MM's "private" memory use grows to make MM 
persistently the 2nd largest process on the machine (behind Spotlight's 
'mds', which has a pathologically large data set on this machine and 
hence a pathologically large virtual size.) There seems to be some slow 
leak because both RPRVT and VPRVT keep growing. 3 hours after the sample 
shown, RPRVT has grown by 40M and VPRVT by 24MB. Also, after a long 
runtime MM can take many minutes to quit with top showing it very slowly 
shedding MREGS and VSIZE. I can count on MM causing a restart or logout 
to timeout if I don't manually quit it first.