[oe] openembedded / dpkg-buildpackage hybrid

[Chris Larson]

On Wed, Nov 24, 2010 at 7:05 AM, Chris Larson <clarson at kergoth.com> wrote:
> On Wed, Nov 24, 2010 at 2:52 AM, Esben Haabendal
> <esbenhaabendal at gmail.com> wrote:
>> Chris Larson <clarson at kergoth.com> writes:
>>
>>> ...  Right now the OE sysroot (where files are shared
>>> between dependencies) is global, rather than per-recipe, and if
>>> multiple versions of something get yanked into a build, they'll step
>>> all over one another in that sysroot.  We've discussed possibly moving
>>> to per-recipe sysroots in the past, constructing them on demand based
>>> upon what that recipe requires, but we're not there yet, ...
>>
>> If I may be so bold, why is OE not going in that direction full speed?
>> We seem to (at best) to be on a very long detour...
>
> There are many people using OE, with widely varying priorities, and
> there are *many* things that need to be fixed / improved in both
> bitbake and OE.
>
>> It "just" requires rethinking that way DEPENDS is handled, which will
>> end up being much more like RDEPENDS is done now.  A recipe depends on a
>> number of "items".  Items are provided by packages, and packages can
>> (recursively) depend on other items.  When constructing the runqueue,
>> the task that sets the stage depends on the task that packages stage
>> packages on all recipes that provides any of the (recursively) needed
>> packages.
>
> I threw together a prototype of per recipe staging areas, with caching
> of the areas for recipes with equivalent dependency trees at one time,
> but it was suboptimal, depended on other branches that no longer
> exist, and it just hasn't been a top priority for anyone to get this
> to happen.  It may help if you consider that most people working on OE
> just want to *use* it to get their work done.  There are very few who
> delve into the core of OE, much less bitbake.
>
>> Looking at build dependencies just makes so much more sense IMHO, and a
>> lot of issues that OE has/is fighting goes away.  No more race
>> situations with a common stage dir, and no more problems with -initial,
>> -intermediate recipes.  It even opens up for more flexible build
>> dependency handling, but that is another side of it.
>>
>> For anyone interested, I have something like the above (based on OE) at
>> http://www.gitorious.org/~esben/oe-lite/esbens-core
>>
>> The entire runqueue code is rewritten from scratch, using an in-memory
>> sqlite database, but the concept could just as well be done based on a
>> truckload of Python dicts and lists, if this is more to ones liking ;-)
>> The interesting code with regards to this subject is in lib/oelite
>
> This sounds very interesting to me.  Personally, I think our overuse
> of lists/dicts/etc is just one big code smell -- Primitive Obsession
> -- but then, I'm no expert on object oriented design, or design in
> general.  My python is improving, but there's still plenty of room to
> go.  I'll take a look at your code.  I'm always open to seeing bitbake
> improvements.  As I said above, there's a serious shortage of people
> willing to improve it at this time.
>
>> It uses standard bitbake, but OE classes are heavily modified, and a
>> number of basic OE features are completely missing (but this is not
>> related to the per-receipe style staging).
>>
>> So have a look if you like, or just skip it as it cannot easily be merged into
>> OpenEmbedded as it is now.

Have you taken a look at the work Richard Purdie has done on bitbake
for Poky/Yocto?  It isn't refactoring the way you've done, but it does
rework how packaged staging is used.  It implements per-task metadata
checksumming (based upon my oe signatures code) and task level output
archiving for populate_sysroot, to be used to short-circuit its build
in subsequent runs.  Conceptually its an interesting tactic, and I
think metadata checksumming is going to be critical going forward, to
improving our build reproducibility.
-- 
Christopher Larson
clarson at kergoth dot com
Founder - BitBake, OpenEmbedded, OpenZaurus
Maintainer - Tslib
Senior Software Engineer, Mentor Graphics