[bitbake-devel] [OE-core] Bitbake Memory Usage

[Richard Purdie]

On Sun, 2012-02-19 at 08:34 +0000, Phil Blundell wrote:
> On Sun, 2012-02-19 at 02:55 +0000, Richard Purdie wrote:
> > The problem is then going to be memory fragmentation since whilst we can
> > free the memory used by the objects, its extremely unlikely everything
> > to the end of the data segment is likely to be free so sbrk() cannot be
> > rolled backwards to free memory.
> 
> Sounds like it.  Have you tried running python with a lower
> MMAP_THRESHOLD?

I just did (to 4k) and it drops the overall RSS from 150M to 143M but
otherwise no significant difference. For the record in the archives, the
rather nasty hacky code I used was:

from ctypes import *
cdll.LoadLibrary("libc.so.6") 
libc = CDLL("libc.so.6") 
print libc.mallopt(-3, 4*1024)

I just looked at the caches in question again and I now suspect the
strings are under the 256b limit, I was thinking the data was slightly
different.

If that is the case, it means the memory is being returned to python's
pool allocator. Unfortunately that will suffer the same fragmentation
problem described above.

Either way, I'm pretty sure memory fragmentation is the problem here.
One solution we might consider putting the codeparser into a separate
process accessed over a socket. At the (mostly) face to face TSC meeting
earlier in the week I did briefly touch on the future with bitbake. I'm
planning to summarise this on the mailing list when I get a chance but
one of the ideas discussed was putting the data store into a separate
process accessed over sockets as a solution to some of the memory use
issues. There are problems with this such as the anonymous python
functions and where in the system these would run but its certainly
worth thinking along those lines. It would solve some of the nasty cache
issues we currently have with the interaction between multiple parsing
threads and the codeparser cache needing to get "merged" at the end of
parsing.

Cheers,

Richard