[bitbake-devel] [OE-core] Bitbake Memory Usage

[Phil Blundell]

On Sun, 2012-02-19 at 15:41 +0000, Richard Purdie wrote:
> I just did (to 4k) and it drops the overall RSS from 150M to 143M but
> otherwise no significant difference. For the record in the archives, the
> rather nasty hacky code I used was:
> 
> from ctypes import *
> cdll.LoadLibrary("libc.so.6") 
> libc = CDLL("libc.so.6") 
> print libc.mallopt(-3, 4*1024)

FWIW, you can just say "export MALLOC_THRESHOLD=4096" before running
bitbake, which should achieve roughly the same thing with slightly less
typing. 

> I just looked at the caches in question again and I now suspect the
> strings are under the 256b limit, I was thinking the data was slightly
> different.
> 
> If that is the case, it means the memory is being returned to python's
> pool allocator. Unfortunately that will suffer the same fragmentation
> problem described above.

Yeah.  I guess it would be possible to make the pool allocator smarter
to cut down on or ameliorate the fragmentation.  But I don't know
whether that is feasible/straightforward without patching Python itself.
Some further investigation required I suppose.

p.