[oe] OE recipe tree quality

[Frans Meulenbroeks]

2010/7/30 Graeme Gregory <dp at xora.org.uk>

>  On 30/07/10 09:48, Frans Meulenbroeks wrote:
> > 2010/7/30 Graeme Gregory <dp at xora.org.uk>
> >
> >>  On 30/07/10 09:22, Koen Kooi wrote:
> >>> On 30-07-10 09:21, Esben Haabendal wrote:
> >>>> On Thu, Jul 29, 2010 at 2:07 PM, Philip Balister
> >>> <philip at balister.org> wrote:
> >>>>>
> >>>>> On 07/29/2010 05:45 AM, Koen Kooi wrote:
> >>>>>> -----BEGIN PGP SIGNED MESSAGE-----
> >>>>>> Hash: SHA1
> >>>>>>
> >>>>>> On 29-07-10 10:50, Frans Meulenbroeks wrote:
> >>>>>>> Dear all,
> >>>>>>>
> >>>>>>> Given the discussions on quality that sometimes pop up (and also
> >>>>>>> triggered
> >>>>>>> by Robert's message), I decided to kick off a bitbake -k world.
> >>>>>> Could you first explain to me why 'bitbake world' is a good way to
> >>>>>> measure quality?
> >>>>>>
> >>>>>> I would think that building something like console-image and
> >>> looking at
> >>>>>> the following would be a much better metric:
> >>>>>>
> >>>>>> * does it build?
> >>>>>> * are all the rootfs types working?
> >>>>>> * does the image do what it is supposed to do?
> >>>>>> * Are all the licenses of the output packages correct?
> >>>>>> * Do the output packages have any spurious deps?
> >>>>>> * Is the content of the output packages correct?
> >>>>>> * Are there any known CVEs in the resulting packages?
> >>>>>> * Did packaged-staging do its job?
> >>>>>> * What kind of QA errors and warnings were raised?
> >>>>>> * Did all recipes pass recipe_sanity?
> >>>>>> * Did all recipes conform to oe-stylize.py?
> >>>>>>
> >>>>>> etc
> >>>>>>
> >>>>>> I would actually advocate removing the 'world' feature from
> >> bitbake/OE
> >>>>>> to stop people from wasting time on looking at bitbake world and
> have
> >>>>>> them fix actual problems.
> >>>>> bitbake world seems to be the source of pointless listserv
> >>> discussions. Does
> >>>>> it serve any purpose?
> >>>> Pointless or not really depends on how you look at quality.
> >>>> If you look at it as you, Koen and other OE long-timers, yes, it looks
> >>>> rather pointless to have bitbake world.
> >>>> But for those of us who have a different view on what quality is, then
> >>>> bitbake world serves a purpose.
> >>> As Thomas points out, as soon as you start blacklisting things (which
> >>> actually increases quality), bitbake world doesn't work anymore.
> >>> That alone should be enough to kill it.
> >> Time to jump in the cage here.
> >>
> >> "Quality" is achieved by comparing a set of known specifications against
> >> a known data set. In the software case this means we need a good set of
> >> specifications which we are testing against. We also need to know in
> >> detail what we are testing against this set of specifications.
> >>
> >> bitbake world meets neither of these criteria. You have no idea what
> >> your testing against, you also have no idea what you are submitting for
> >> test. A random selection of 8500 files in OE. It also doesn't take into
> >> account known combinations which always fail. Angstrom took care of
> >> these known failures by creating the concept of a blacklist.
> >>
> >> In fact if you tell me bitbake world fails, I would actually suggest
> >> that is a test pass, it is an expected fail.
> >>
> >> Graeme
> >>
> >> Actually I expect it to fail, but I am eager to learn which recipes
> fail.
> > And yes, it might well be that this is not a good test. If you have
> > suggestions for a better test that tests all our recipes feel free to
> speak
> > up.
> >
> > The dependency analysis of bitbake world already showed that there are
> > several issues (as listed above).
> > E.g. there are apparently dependencies on non existing recipes.
> > Instead of shooting the messenger and killing the tool it would be better
> > to  resolve these (by either fixing them as I have seen JaMa did for some
> > recipes in the past) or by removing the recipes.
> > Keeping them around as junk does not improve things.
> >
> > And failure of bitbake -k world is not necessarily bad. At least it will
> > give some insight in the weak spots.
> >
> More interesting to me would be an algorithm more like.
>
> while recipe in recips/*.bb; do
>    rm -rf tmp/ ; bitbake recipe
> done
>
> with the test noting if the fail is due to a blacklisted dependency, a
> totally missing dependency (recipe been deleted sometime in past) or an
> actual compile fail due to a missing DEPENDS entry somewhere in the tree.
>
> But this is going to burn CPU time like never before.
>
> Graeme
>
>
Graeme,

Definitely an interesting test. Actually with packaged staging this might
become manageble.
I would not mind running such a test, even if it takes a week or so.

I guess it is hard to do this for older versions of recipes though.
afaik bitbake -b will not build things one depends on and bitbake recipe
will take either the pinned version or otherwise the last one a non-negative
DEFAULT_PREFERENCE.

Frans.

>
> _______________________________________________
> Openembedded-devel mailing list
> Openembedded-devel at lists.openembedded.org
> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-devel
>