[oe] OE recipe tree quality

[Graeme Gregory]

 On 30/07/10 10:07, Frans Meulenbroeks wrote:
> 2010/7/30 Graeme Gregory <dp at xora.org.uk>
>
>>  On 30/07/10 09:48, Frans Meulenbroeks wrote:
>>> 2010/7/30 Graeme Gregory <dp at xora.org.uk>
>>>
>>>>  On 30/07/10 09:22, Koen Kooi wrote:
>>>>> On 30-07-10 09:21, Esben Haabendal wrote:
>>>>>> On Thu, Jul 29, 2010 at 2:07 PM, Philip Balister
>>>>> <philip at balister.org> wrote:
>>>>>>> On 07/29/2010 05:45 AM, Koen Kooi wrote:
>>>>>>>> -----BEGIN PGP SIGNED MESSAGE-----
>>>>>>>> Hash: SHA1
>>>>>>>>
>>>>>>>> On 29-07-10 10:50, Frans Meulenbroeks wrote:
>>>>>>>>> Dear all,
>>>>>>>>>
>>>>>>>>> Given the discussions on quality that sometimes pop up (and also
>>>>>>>>> triggered
>>>>>>>>> by Robert's message), I decided to kick off a bitbake -k world.
>>>>>>>> Could you first explain to me why 'bitbake world' is a good way to
>>>>>>>> measure quality?
>>>>>>>>
>>>>>>>> I would think that building something like console-image and
>>>>> looking at
>>>>>>>> the following would be a much better metric:
>>>>>>>>
>>>>>>>> * does it build?
>>>>>>>> * are all the rootfs types working?
>>>>>>>> * does the image do what it is supposed to do?
>>>>>>>> * Are all the licenses of the output packages correct?
>>>>>>>> * Do the output packages have any spurious deps?
>>>>>>>> * Is the content of the output packages correct?
>>>>>>>> * Are there any known CVEs in the resulting packages?
>>>>>>>> * Did packaged-staging do its job?
>>>>>>>> * What kind of QA errors and warnings were raised?
>>>>>>>> * Did all recipes pass recipe_sanity?
>>>>>>>> * Did all recipes conform to oe-stylize.py?
>>>>>>>>
>>>>>>>> etc
>>>>>>>>
>>>>>>>> I would actually advocate removing the 'world' feature from
>>>> bitbake/OE
>>>>>>>> to stop people from wasting time on looking at bitbake world and
>> have
>>>>>>>> them fix actual problems.
>>>>>>> bitbake world seems to be the source of pointless listserv
>>>>> discussions. Does
>>>>>>> it serve any purpose?
>>>>>> Pointless or not really depends on how you look at quality.
>>>>>> If you look at it as you, Koen and other OE long-timers, yes, it looks
>>>>>> rather pointless to have bitbake world.
>>>>>> But for those of us who have a different view on what quality is, then
>>>>>> bitbake world serves a purpose.
>>>>> As Thomas points out, as soon as you start blacklisting things (which
>>>>> actually increases quality), bitbake world doesn't work anymore.
>>>>> That alone should be enough to kill it.
>>>> Time to jump in the cage here.
>>>>
>>>> "Quality" is achieved by comparing a set of known specifications against
>>>> a known data set. In the software case this means we need a good set of
>>>> specifications which we are testing against. We also need to know in
>>>> detail what we are testing against this set of specifications.
>>>>
>>>> bitbake world meets neither of these criteria. You have no idea what
>>>> your testing against, you also have no idea what you are submitting for
>>>> test. A random selection of 8500 files in OE. It also doesn't take into
>>>> account known combinations which always fail. Angstrom took care of
>>>> these known failures by creating the concept of a blacklist.
>>>>
>>>> In fact if you tell me bitbake world fails, I would actually suggest
>>>> that is a test pass, it is an expected fail.
>>>>
>>>> Graeme
>>>>
>>>> Actually I expect it to fail, but I am eager to learn which recipes
>> fail.
>>> And yes, it might well be that this is not a good test. If you have
>>> suggestions for a better test that tests all our recipes feel free to
>> speak
>>> up.
>>>
>>> The dependency analysis of bitbake world already showed that there are
>>> several issues (as listed above).
>>> E.g. there are apparently dependencies on non existing recipes.
>>> Instead of shooting the messenger and killing the tool it would be better
>>> to  resolve these (by either fixing them as I have seen JaMa did for some
>>> recipes in the past) or by removing the recipes.
>>> Keeping them around as junk does not improve things.
>>>
>>> And failure of bitbake -k world is not necessarily bad. At least it will
>>> give some insight in the weak spots.
>>>
>> More interesting to me would be an algorithm more like.
>>
>> while recipe in recips/*.bb; do
>>    rm -rf tmp/ ; bitbake recipe
>> done
>>
>> with the test noting if the fail is due to a blacklisted dependency, a
>> totally missing dependency (recipe been deleted sometime in past) or an
>> actual compile fail due to a missing DEPENDS entry somewhere in the tree.
>>
>> But this is going to burn CPU time like never before.
>>
>> Graeme
>>
>>
> Graeme,
>
> Definitely an interesting test. Actually with packaged staging this might
> become manageble.
> I would not mind running such a test, even if it takes a week or so.
>
> I guess it is hard to do this for older versions of recipes though.
> afaik bitbake -b will not build things one depends on and bitbake recipe
> will take either the pinned version or otherwise the last one a non-negative
> DEFAULT_PREFERENCE.
>
> Frans.
>
bitbake blah-x.x.x should work to build exact versions with dependencies
though.

G