[Openembedded-architecture] LSB is dead; when shall we drop it?

[Mark Hatle]

On 4/7/17 9:05 AM, Alexander Kanavin wrote:
> On 04/07/2017 04:59 PM, Mark Hatle wrote:
> 
>> I really think this needs to be clarified.  It would be perfectly reasonable to me:
>>
>> 1) Autobuilder 'tests' LSB within the scope that is defined by the project.
>>
>> 2) Defects are opened for test failures.  (Test failures do NOT block code
>> changes, unless they are specifically related to 'LSB' work.)
>>
>> 3) People who care about the LSB will resolve the defects.  Resolution is:
>>    - release note explaining the issue/limitation with a test-case update
>>    - actual code change to resolve a problem
>>
>>
>> The LSB should not be driving the work.  The general open source community is
>> what should be driving the work forward, and the LSB work -- since it's a
>> conformance test -- should be reacting to that work.
>>
>> I would not consider the OE passing a random testing to be good or bad.  We run
>> tests to gain knowledge and decide what that knowledge means -- that is what
>> defined a pass or failure.
> 
> I don't agree with this at all. We currently do not allow autobuilder 
> tests to fail: if they fail, then the patch that is causing the failure 
> is rejected. If you change this approach and allow test failures, this 
> really opens a can of worms, and basically subverts the entire process 
> which we try to bring as close to continuous integration principles as 
> possible.
> 
> Manual runtime package testing by QA is a different story: those tests 
> can fail, and any failures are filed as bugs and dealt with after the fact.

LSB testing should not be considered gospel (it is known to be flawed).  It is
an indication of a problem, but not a clear "this is the problem".

If that is the case today, then LSB testing should be removed from the
autobuilder, and replaced with a more sane set of sanity tests (subset of the
LSB tests) for things that we do ACTUALLY care about.

--Mark

> 
> Alex
>