[oe] OE recipe tree quality

[Frans Meulenbroeks]

2010/7/30 Graeme Gregory <dp at xora.org.uk>

>  On 30/07/10 09:22, Koen Kooi wrote:
> > On 30-07-10 09:21, Esben Haabendal wrote:
> > > On Thu, Jul 29, 2010 at 2:07 PM, Philip Balister
> > <philip at balister.org> wrote:
> > >>
> > >>
> > >> On 07/29/2010 05:45 AM, Koen Kooi wrote:
> > >>>
> > >>> -----BEGIN PGP SIGNED MESSAGE-----
> > >>> Hash: SHA1
> > >>>
> > >>> On 29-07-10 10:50, Frans Meulenbroeks wrote:
> > >>>>
> > >>>> Dear all,
> > >>>>
> > >>>> Given the discussions on quality that sometimes pop up (and also
> > >>>> triggered
> > >>>> by Robert's message), I decided to kick off a bitbake -k world.
> > >>>
> > >>> Could you first explain to me why 'bitbake world' is a good way to
> > >>> measure quality?
> > >>>
> > >>> I would think that building something like console-image and
> > looking at
> > >>> the following would be a much better metric:
> > >>>
> > >>> * does it build?
> > >>> * are all the rootfs types working?
> > >>> * does the image do what it is supposed to do?
> > >>> * Are all the licenses of the output packages correct?
> > >>> * Do the output packages have any spurious deps?
> > >>> * Is the content of the output packages correct?
> > >>> * Are there any known CVEs in the resulting packages?
> > >>> * Did packaged-staging do its job?
> > >>> * What kind of QA errors and warnings were raised?
> > >>> * Did all recipes pass recipe_sanity?
> > >>> * Did all recipes conform to oe-stylize.py?
> > >>>
> > >>> etc
> > >>>
> > >>> I would actually advocate removing the 'world' feature from
> bitbake/OE
> > >>> to stop people from wasting time on looking at bitbake world and have
> > >>> them fix actual problems.
> > >>
> > >> bitbake world seems to be the source of pointless listserv
> > discussions. Does
> > >> it serve any purpose?
> >
> > > Pointless or not really depends on how you look at quality.
> >
> > > If you look at it as you, Koen and other OE long-timers, yes, it looks
> > > rather pointless to have bitbake world.
> > > But for those of us who have a different view on what quality is, then
> > > bitbake world serves a purpose.
> >
> > As Thomas points out, as soon as you start blacklisting things (which
> > actually increases quality), bitbake world doesn't work anymore.
> > That alone should be enough to kill it.
> Time to jump in the cage here.
>
> "Quality" is achieved by comparing a set of known specifications against
> a known data set. In the software case this means we need a good set of
> specifications which we are testing against. We also need to know in
> detail what we are testing against this set of specifications.
>
> bitbake world meets neither of these criteria. You have no idea what
> your testing against, you also have no idea what you are submitting for
> test. A random selection of 8500 files in OE. It also doesn't take into
> account known combinations which always fail. Angstrom took care of
> these known failures by creating the concept of a blacklist.
>
> In fact if you tell me bitbake world fails, I would actually suggest
> that is a test pass, it is an expected fail.
>
> Graeme
>
> Actually I expect it to fail, but I am eager to learn which recipes fail.
And yes, it might well be that this is not a good test. If you have
suggestions for a better test that tests all our recipes feel free to speak
up.

The dependency analysis of bitbake world already showed that there are
several issues (as listed above).
E.g. there are apparently dependencies on non existing recipes.
Instead of shooting the messenger and killing the tool it would be better
to  resolve these (by either fixing them as I have seen JaMa did for some
recipes in the past) or by removing the recipes.
Keeping them around as junk does not improve things.

And failure of bitbake -k world is not necessarily bad. At least it will
give some insight in the weak spots.

Frans.