[oe] openembedded / dpkg-buildpackage hybrid

[Chris Larson]

On Wed, Nov 24, 2010 at 2:52 AM, Esben Haabendal
<esbenhaabendal at gmail.com> wrote:
> Chris Larson <clarson at kergoth.com> writes:
>
>> ...  Right now the OE sysroot (where files are shared
>> between dependencies) is global, rather than per-recipe, and if
>> multiple versions of something get yanked into a build, they'll step
>> all over one another in that sysroot.  We've discussed possibly moving
>> to per-recipe sysroots in the past, constructing them on demand based
>> upon what that recipe requires, but we're not there yet, ...
>
> If I may be so bold, why is OE not going in that direction full speed?
> We seem to (at best) to be on a very long detour...

There are many people using OE, with widely varying priorities, and
there are *many* things that need to be fixed / improved in both
bitbake and OE.

> It "just" requires rethinking that way DEPENDS is handled, which will
> end up being much more like RDEPENDS is done now.  A recipe depends on a
> number of "items".  Items are provided by packages, and packages can
> (recursively) depend on other items.  When constructing the runqueue,
> the task that sets the stage depends on the task that packages stage
> packages on all recipes that provides any of the (recursively) needed
> packages.

I threw together a prototype of per recipe staging areas, with caching
of the areas for recipes with equivalent dependency trees at one time,
but it was suboptimal, depended on other branches that no longer
exist, and it just hasn't been a top priority for anyone to get this
to happen.  It may help if you consider that most people working on OE
just want to *use* it to get their work done.  There are very few who
delve into the core of OE, much less bitbake.

> Looking at build dependencies just makes so much more sense IMHO, and a
> lot of issues that OE has/is fighting goes away.  No more race
> situations with a common stage dir, and no more problems with -initial,
> -intermediate recipes.  It even opens up for more flexible build
> dependency handling, but that is another side of it.
>
> For anyone interested, I have something like the above (based on OE) at
> http://www.gitorious.org/~esben/oe-lite/esbens-core
>
> The entire runqueue code is rewritten from scratch, using an in-memory
> sqlite database, but the concept could just as well be done based on a
> truckload of Python dicts and lists, if this is more to ones liking ;-)
> The interesting code with regards to this subject is in lib/oelite

This sounds very interesting to me.  Personally, I think our overuse
of lists/dicts/etc is just one big code smell -- Primitive Obsession
-- but then, I'm no expert on object oriented design, or design in
general.  My python is improving, but there's still plenty of room to
go.  I'll take a look at your code.  I'm always open to seeing bitbake
improvements.  As I said above, there's a serious shortage of people
willing to improve it at this time.

> It uses standard bitbake, but OE classes are heavily modified, and a
> number of basic OE features are completely missing (but this is not
> related to the per-receipe style staging).
>
> So have a look if you like, or just skip it as it cannot easily be merged into
> OpenEmbedded as it is now.

Will do.
-- 
Christopher Larson
clarson at kergoth dot com
Founder - BitBake, OpenEmbedded, OpenZaurus
Maintainer - Tslib
Senior Software Engineer, Mentor Graphics