[bitbake-devel] [OE-core] Bitbake Memory Usage

[Richard Purdie]

On Mon, 2012-02-20 at 12:57 -0700, Chris Larson wrote:
> On Mon, Feb 20, 2012 at 7:05 AM, Colin Walters <walters at verbum.org> wrote:
> > On Sat, 2012-02-18 at 23:36 +0000, Richard Purdie wrote:
> >> As soon as the child starts trying to remove
> >> things from memory, we lose the benefits of CoW and USS and PSS rise.
> >
> > Note that even leaving out the garbage collector, the cPython VM
> > incrementing/decrementing the refcount of objects will force the copy.
> >
> > The python multiprocessing module has some classes which allow
> > explicitly sharing memory.  Certainly for various caches e.g. if
> > they're just read-only after a certain point it might make sense
> > to wrap them in e.g. a multiprocessing.Value.
> 
> Hmm, that's quite interesting, thanks for that tidbit. We should
> consider looking into that.

Its interesting but my tests are suggesting the reference counting isn't
hurting in at least our common "execute a shell task" use case. With the
way CoW and the data store works, I can believe that.

On the other hand, the serialisation that multiprocessing puts around
its variables looked potentially problematic (and slow) last time I
looked. The manual page specifically says those structures are slow if I
remember rightly.

>  On a related note, I think we should sit
> down and basically review every use of a cache anywhere in bitbake and
> rethink their operation. We still have multiple caches stored as
> globals with no expiration policy of any sort, they accrue
> indefinitely, until process exit. As we review them, we could consider
> this as well on a case-by-case basis.

I agree we need to resolve some of the life cycle issue and remove the
globals. The ones I'm aware of do give significant speed benefits though
and I'm not sure there will be much scope for removing them :/.

Cheers,

Richard