[bitbake-devel] [OE-core] Bitbake Memory Usage

[Chris Larson]

On Mon, Feb 20, 2012 at 7:05 AM, Colin Walters <walters at verbum.org> wrote:
> On Sat, 2012-02-18 at 23:36 +0000, Richard Purdie wrote:
>> As soon as the child starts trying to remove
>> things from memory, we lose the benefits of CoW and USS and PSS rise.
>
> Note that even leaving out the garbage collector, the cPython VM
> incrementing/decrementing the refcount of objects will force the copy.
>
> The python multiprocessing module has some classes which allow
> explicitly sharing memory.  Certainly for various caches e.g. if
> they're just read-only after a certain point it might make sense
> to wrap them in e.g. a multiprocessing.Value.

Hmm, that's quite interesting, thanks for that tidbit. We should
consider looking into that. On a related note, I think we should sit
down and basically review every use of a cache anywhere in bitbake and
rethink their operation. We still have multiple caches stored as
globals with no expiration policy of any sort, they accrue
indefinitely, until process exit. As we review them, we could consider
this as well on a case-by-case basis.
-- 
Christopher Larson