[oe] OE recipe tree quality

[Graeme Gregory]

 On 30/07/10 09:48, Frans Meulenbroeks wrote:
> 2010/7/30 Graeme Gregory <dp at xora.org.uk>
>
>>  On 30/07/10 09:22, Koen Kooi wrote:
>>> On 30-07-10 09:21, Esben Haabendal wrote:
>>>> On Thu, Jul 29, 2010 at 2:07 PM, Philip Balister
>>> <philip at balister.org> wrote:
>>>>>
>>>>> On 07/29/2010 05:45 AM, Koen Kooi wrote:
>>>>>> -----BEGIN PGP SIGNED MESSAGE-----
>>>>>> Hash: SHA1
>>>>>>
>>>>>> On 29-07-10 10:50, Frans Meulenbroeks wrote:
>>>>>>> Dear all,
>>>>>>>
>>>>>>> Given the discussions on quality that sometimes pop up (and also
>>>>>>> triggered
>>>>>>> by Robert's message), I decided to kick off a bitbake -k world.
>>>>>> Could you first explain to me why 'bitbake world' is a good way to
>>>>>> measure quality?
>>>>>>
>>>>>> I would think that building something like console-image and
>>> looking at
>>>>>> the following would be a much better metric:
>>>>>>
>>>>>> * does it build?
>>>>>> * are all the rootfs types working?
>>>>>> * does the image do what it is supposed to do?
>>>>>> * Are all the licenses of the output packages correct?
>>>>>> * Do the output packages have any spurious deps?
>>>>>> * Is the content of the output packages correct?
>>>>>> * Are there any known CVEs in the resulting packages?
>>>>>> * Did packaged-staging do its job?
>>>>>> * What kind of QA errors and warnings were raised?
>>>>>> * Did all recipes pass recipe_sanity?
>>>>>> * Did all recipes conform to oe-stylize.py?
>>>>>>
>>>>>> etc
>>>>>>
>>>>>> I would actually advocate removing the 'world' feature from
>> bitbake/OE
>>>>>> to stop people from wasting time on looking at bitbake world and have
>>>>>> them fix actual problems.
>>>>> bitbake world seems to be the source of pointless listserv
>>> discussions. Does
>>>>> it serve any purpose?
>>>> Pointless or not really depends on how you look at quality.
>>>> If you look at it as you, Koen and other OE long-timers, yes, it looks
>>>> rather pointless to have bitbake world.
>>>> But for those of us who have a different view on what quality is, then
>>>> bitbake world serves a purpose.
>>> As Thomas points out, as soon as you start blacklisting things (which
>>> actually increases quality), bitbake world doesn't work anymore.
>>> That alone should be enough to kill it.
>> Time to jump in the cage here.
>>
>> "Quality" is achieved by comparing a set of known specifications against
>> a known data set. In the software case this means we need a good set of
>> specifications which we are testing against. We also need to know in
>> detail what we are testing against this set of specifications.
>>
>> bitbake world meets neither of these criteria. You have no idea what
>> your testing against, you also have no idea what you are submitting for
>> test. A random selection of 8500 files in OE. It also doesn't take into
>> account known combinations which always fail. Angstrom took care of
>> these known failures by creating the concept of a blacklist.
>>
>> In fact if you tell me bitbake world fails, I would actually suggest
>> that is a test pass, it is an expected fail.
>>
>> Graeme
>>
>> Actually I expect it to fail, but I am eager to learn which recipes fail.
> And yes, it might well be that this is not a good test. If you have
> suggestions for a better test that tests all our recipes feel free to speak
> up.
>
> The dependency analysis of bitbake world already showed that there are
> several issues (as listed above).
> E.g. there are apparently dependencies on non existing recipes.
> Instead of shooting the messenger and killing the tool it would be better
> to  resolve these (by either fixing them as I have seen JaMa did for some
> recipes in the past) or by removing the recipes.
> Keeping them around as junk does not improve things.
>
> And failure of bitbake -k world is not necessarily bad. At least it will
> give some insight in the weak spots.
>
More interesting to me would be an algorithm more like.

while recipe in recips/*.bb; do
    rm -rf tmp/ ; bitbake recipe
done

with the test noting if the fail is due to a blacklisted dependency, a
totally missing dependency (recipe been deleted sometime in past) or an
actual compile fail due to a missing DEPENDS entry somewhere in the tree.

But this is going to burn CPU time like never before.

Graeme