[oe] OpenEmbedded and the Yocto Project

[Chris Larson]

On Fri, Nov 5, 2010 at 9:07 AM, Richard Purdie <rpurdie at rpsys.net> wrote:
>
> When it comes to using OE commercially, its actually quite hard. If you
> look at even the big companies using OE like TI, Mentor and Montavista
> to name but a few, whilst they are using it, there are significant
> changes they need to make to turn it into a commercial product. Those
> changes and stabilisation are eating a lot of the developer resources
> they put in and there is arguably a comparatively low amount of feature
> development being worked upon. If somehow we could share some of that
> effort, it would free up resources for feature development.
>

I think we can all agree with that.


> End result is that Yocto/Poky have a states release cycle, approximately
> six months. We have releases, BSPs can be made against the last release.
> When planning people can know roughly what state the tree will be in at
> a given time and whilst no timeline is perfect for everyone, its well
> defined and people can adapt to it.
>

This is good to hear, and I hope OE can adopt this as well, or,
alternatively, as has been discussed a bit, OE could become a set of layers
built upon Yocto, and share the release cycle.

Third is a QA problem and one of size and scale. Realistically, there is
> no way you can "test" OE as its just too large with too many
> combinations. On the other hand we have a pretty good known test matrix
> for Poky/Yocto as its a smaller set of data than OE. If we want to
> expand that set, we're prepared to as long as there is contribution to
> help test it too (additional autobuilder machines and QA resource).
>
> We do appreciate the value of the additional things OE contains however
> and also realise the value in additional machine support. This is why
> the Yocto project has done work to make layers work better and become
> usable for additional topic "metadata" layers and also machine support
> in the form of BSPs. This neatly allows people to take ownership of
> subsets of the metadata, be them community or commercial, a problem OE
> has long suffered from.
>

This is also good to hear, and I agree that this is a good solution -- we
actually always wanted to be able to split control and ownership in this
way, but originally determining how to combine metadata from multiple places
wasn't a priority.  Aside: the approach taken by Conary for the metadata is
a very intriguing one.

Having a known test matrix and an ability to QA the system also helps
> massively with feature development. Within Poky/Yocto we can *know*
> whether a given change causes a regression within the core without too
> much effort. I'd suggest this is much much harder to know within OE.
>

Perhaps, but I think the OE Testing effort is a step in the right direction,
so OE too will have a test matrix, just, as you say, a less complete one.


> This brings me to a forth problem which is related to some of the above
> indirectly which is that of the contributions model itself. Above, I
> mentioned we needed to ensure some kind of vetting of the license on new
> recipes and this isn't possible with the OE "push" contributions model.
> This is why Yocto/Poky have adopted a kernel like "pull" model for
> changes. If it can scale for the Linux kernel, I think it can scale for
> Poky/Yocto and could if there was a will, for OE too. Sub-maintainers
> are appointed based on trust to merge together "pull" requests and pass
> them upwards so no one person becomes overloaded.
>
> This "pull" model also makes decisions and technical direction easier,
> there has to be a lot of trust in the Linus type person to do the right
> thing. If that person screws up, the project can fork and there would
> likely be a clear winner afterwards if the decision was worth the fork
> so there is actually a strong natural pressure on that person to get
> things right. In the Yocto/Poky world, I've taken on that overall
> responsibility as I have relevant technical experience and a long
> history of leading Poky and to a large extent OE from a technical
> standpoint having contributed many of the technical advancements and
> architecture improvements that have put OE where it is today. I fully
> intend to have sub maintainers to help and I really care about their
> technical leadership, not who pays their wages.
>
> This is as good a time as any to talk about the governance of the Yocto
> Project as people look at the diagram on the website and get very
> confused. The people with the power in the model are the architect/
> maintainer groups.
>
> When it comes to technical direction this comes from the maintainers of
> the various components. They set this based on the input from the
> architect/steering group but they are the ones deciding the
> implementation details, writing code, accepting patches and developing
> the feature roadmaps and development schedules and actually doing the
> real work.
>
> One of the cornerstones of the governance is that if the steering group
> ever has to "tell" the architect what to do then the architect has
> failed at their job. This also applies to maintainers as if the
> architect has to get directly involved things aren't working correctly.
>
> So the steering group is there to suggest things that would
> help the project and steer to a degree but the architect is there to
> actually set the technical direction overall. In turn the architect is
> there to given direction to the maintainers.
>
> The real power therefore rests with the Maintainers with delegated
> authority through the Architect. The steering group gives managers at
> the companies involved a warm fuzzy feeling that they can get involved
> if things ever do look like they're going of the rails but the agreement
> is its "hands off" as much as possible. Its also worth noting the CTOs
> on the diagram are primarily about publicising and networking the
> project.
>

I really like this approach, personally.  I think OE has lost direction
somewhat since I left the project, even as it has grown in leaps and bounds,
not because of me, but because of the reduction of technical leadership in
general.  Having both working groups / steering committees to give companies
input and individual technical leaders to do day to day decision making
seems to be a good solution.  I do think we'll need a branch structure which
facilitates integration and stability testing of features before they get
pulled, along the lines of linux-next or git's next and pu branches, though,
if it doesn't have that already :)
-- 
Christopher Larson
clarson at kergoth dot com
Founder - BitBake, OpenEmbedded, OpenZaurus
Maintainer - Tslib
Senior Software Engineer, Mentor Graphics