[Openembedded-architecture] Notes from the OEDAM "Naming Table"

[Richard Purdie]

On Thu, 2016-09-08 at 16:14 -0400, Trevor Woerner wrote:
> I hesitate to weigh in because this is a big issue that can only
> really be decided by the major players in this project, and I'm well
> aware of the fact that I'm not one of those people.

I guess I probably fit that description. Keep in mind that I don't make
decisions in isolation, I take feedback, opinions and suggestions and
use that along with other things like experience to try and pick the
best outcome.

Data is therefore good and I welcome it. If decision makers lack data,
it can reflect in the decisions.

Sometimes decisions don't always go the way people want them to,
particularly if you have two groups, both wanting different things as
in that case, someone usually ends up unhappy. In that case, the best
outcome is "disagree and commit" and you move forward together,
decision made.

Where I sometimes get frustrated is if the same decisions get revisited
time and time again, particularly in the form of "disagree and not
commit". Sometimes this is because the decision is wrong and change is
needed so can sometimes a good thing. Sometimes its because there isn't
a viable alternative or that the data the decision was based on hasn't
changed. Opinion comes into it and opinions can differ.

> But on the other hand, I don't want the silence to be interpreted as
> non-interest. So instead of pestering everyone with "me too" replies,
> I'll try
> to be succinct.
> 
> On Sat 2016-09-03 @ 12:14:57 PM, Martin Jansa wrote:
> > 
> > 3) confusion between poky as a DISTRO or as a git repository,
> Ideally poky could be a DISTRO and nothing else and be named meta-
> poky and
> exist as its own, top-level layer. (or, if moving forward with the
> name "poky"
> is too confusing then pick something else and make that the de-facto
> Yocto
> reference distro).
> 
> Imagine if every DISTRO maintained its own repository with a "meta"
> subdirectory that was simply a copy of OE's meta, and a "bitbake"
> subdirectory
> that was simply a copy of the bitbake project. It wouldn't take long
> before
> someone would come along and refactor everything. Of course, over
> time, all
> these "meta" and "bitbake" subdirectories would diverge and show
> various
> levels of maintenance and become a large headache for anyone actually
> trying
> to work on products!
> 
> I guess poky, the repository, is useful because it allows the project
> to skirt
> the issue of whether to use git submodules, or repo, or whatever
> else? Are
> there other benefits? Hopefully at some point the duplication
> maintenance
> would outweigh this benefit?

To be clear, poky is autogenerated by the combo-layer tool. Its hard
for it to become out of sync and there isn't any fork or divergence or
code difference. So the world isn't as bad as the above makes out, you
don't get divergence. You are right however that combo-layer is an
alternative to submodules, repo and various other custom scripts out
there.

I do think that the varied ways you can do this is confusing to the
user base, I even started a thread about how to try and standardise.
The trouble is everyone is right, nobody wants to change and people
value the flexibility in this space.

Any one solution which OE, or the Yocto Project pick as a standard is
going to work for ~25% of the user base with the other 75% complaining
loudly about it. If you think there is frustration about poky, think
how bad it would be if we try and enforce one of these methods as a
standard!

Poky makes a convenient target here, it picked a standard and has had
the nerve to run with it. It is used as the Yocto Project test
reference (YP had to use *something*) which I agree does give it a bit
more of a focus in people's minds than was originally intended. I do
think people lose sight of the fact there is a deeper more underlying
problem.

If we had a standard way of configuring, fetching and laying out
layers, I'd probably move poky to use it. We've not found one which
enough people will agree on though.

> It seems disingenuous to be, on the one hand, teaching people that in
> order to produce an image for a device one needs to obtain the
> correct BSP layer, select a DISTRO layer, then combine that with the
> OE/meta boilerplate in order to build. Then, on the other hand,
> combining all the above into the poky repository to save everyone the
> effort. Of course, I know some of the people involved in this project
> and know that they are all very sincere people;
> therefore I know they're not intentionally trying to confuse people,
> but I could see how someone might wonder about this apparent bait-
> and-switch.

The evolution of why we end up where we do with poky is perhaps worth
revisiting. We want an great out the box experience for the Yocto
Project (as did Poky as a project before it). Do we tell the user that
a) They need to clone bitbake, clone oe-core, clone meta-yocto, clone
the docs (and make sure to get compatible pieces)? Or b) do we say
clone this poky repo? When you consider updating it also becomes even
more painful for a).

So, you either build combination repos (combo-layer does that) or you
use submodules, or you use repo, or you use custom scripts etc. The
default OE out the box experience of cloning individual components and
manually maintaining them all is pretty poor though.

Most methods all agree that to add extra layers, you add them alongside
the starting point, so at least we can agree on that much!

So the things you see here are out of a desire to try and help people.
There is however no agreement on how to standardise it and that does
create confusion.

> The poky repository is vitally important because the community has
> decided that the contents of this repository forms the baseline for
> the metrics of this project.

Poky forms the test baseline of the Yocto Project. That is a decision
the Yocto Project makes. OE has effectively chosen to go along with
that since there is no better testing that OE does which would give it
a different set of metrics. It continues to be free to do so should it
want to, I would however highlight that it isn't an easy thing to do.

>  If a BSP in the poky repository fails to build, a release is not
> made. If a build using the poky repository starts to take a
> significant amount of more time, then a fix is implemented or the
> increase justified. Releases don't go out the door until the poky
> repository is happy. But, as such, it should be viewed as an internal
> project that shouldn't ever see the light of day, should almost not
> be public, and shouldn't be referred to in the quick start guides
> etc. In this light the poky repository should be something that
> only the people responsible for making releases should know about.

Apart from the fact that the Yocto Project also wants an easy out the
box experience for users. To be frank, the OpenEmbedded project has
failed to agree on or deliver any standard way of doing that which
meets the needs of the Yocto Project quick start and gave the best user
experience.

> How often do new people to this project follow the "Getting Started
> Guide" perfectly fine, then try to build an image for some board (or
> distro) that isn't part of the poky repository and fall flat on their
> faces? In my opinion: often.

I think we can all agree on that. Is poky the problem here though?

I'd suggest that people also fall flat on their faces because layers
outside of poky have not had much testing and only work in very
specific combinations (distro X, machine Y, image Z).

>  Why is that? It's because the Getting Started Guide and the poky
> repository haven't really taught them how to find and combine layers
> on their own; 

Poky does "spoil" people in that its known to work in most
combinations. When users add in layers without that level of testing,
things break more easily as they aren't tested as often or in as many
combinations?

Killing off poky would then mean the Yocto Project doesn't have an
ability to the testing it does, or give people the out of box
experience it does. What it wouldn't do is make any of these other
layers work any better and likely OE-Core just stops working as well in
many combinations and the user experience degrades.

> which is a very important and vital skill if someone is going to be
> doing
> builds with these tools. It hasn't taught them exactly what they need
> (OE/meta, BSP, distro) to build an image.

Or put another way, the Yocto Project is selling a compatibility and
layer interoperability expectation which the quality of layers in the
wider OE ecosystem isn't able to meet?

I'd actually propose a different way forward here. The Yocto Project
hasn't made a big thing of it but the amount of automated testing
infrastructure that has been created in OE-Core is significant and
powerful. I will send a separate email about exactly how much we do
test, even before we merge commits now in many cases.

The project also has the yocto-autobuilder. Ok, everyone hates buildbot
and Jenkins is supposedly wonderful, however consider the shear amount
of testing that the Yocto Project achieves with yocto-autobuilder on a
comparatively modest hardware budget (it has some powerful machines but
not as many as you'd think). It is very easy to setup out the box and I
suspect with some polish and documentation, people could be very
productive with it.

Therefore, to address these layer quality issues I'd like to see:

a) More publicly visible automated testing of layers 

This would include build and runtime. I'd prefer it to reuse the oeqa
code already in OE-Core, I care much less if its Jenkins or buildbot
based. Obviously even build testing removes several points of failure
from new users, runtime even more so even if its just using the qemu
machines. We support kvm for qemux86* which has massively improved
speed there and meant we turned on a load more testing.

b) Help/support around creating Yocto Project Compatible v2

In the next version of this I'd like to see specific tests for checking
the interference/interaction between layers. You add a distro layer but
don't set the DISTRO? checksums shouldn't change. Ditto adding a
MACHINE layer. This would stop a significant amount of the antisocial
things people are doing

I'd also like to include some kind of test matrix in the layer which
indicates which combinations of layers/distros/machines its been tested
against and is known to support.

We can use YP Compatible as a way of rewarding people who develop
layers which do the right things.

I've simply not had time to go and do any of this but all the pieces
are ready (oe-selftest has examples of the sstate signature
comparisons).

c) Help developing the layer index

This is a key piece of making it possible for users to find layers.
Showing an indication of health and compatibility of a given layer with
other layers would be ideal.

There is also a problem of some kind of results storage here too. How
do we know performance regressed? When did test X last pass for Y.
There are pieces like buildhistory and we have performance measuring
scripts but there is no central system recording it which the layer
index can look at, or any dashboard to graph performance or report
regressions.

d) Can we unify the way people work with layers?

The last discussion I had on this topic showed people in very specific
camps with nobody willing to compromise. Is there a way we could move
forward?


In summary, I do agree there are problems for new users here. Instead
of blaming poky for it all though, I'd suggest looking at what the
Yocto Project has done and consider how we go and replicate this for
other layers in the ecosystem and have a push for better quality
together?

Also keep in mind that all the testing in the world doesn't matter if
you don't have people to look into the regressions. Martin already
tests meta-oe, how many people help with the errors/warnings he finds?

OE is attracting many new developers and making it easy for them to see
these regressions and help fix them might be a good way to grow the
project though.

Cheers,

Richard