[OE-core] [PATCH 01/12] build-appliance: Drop test which is already automated

[Yeoh, Ee Peng]

Ok, noted, I agreed. Let me initialize the build-appliance testcases consolidation. Thank you for your inputs and feedbacks. 

-----Original Message-----
From: richard.purdie at linuxfoundation.org [mailto:richard.purdie at linuxfoundation.org] 
Sent: Wednesday, November 14, 2018 5:32 PM
To: Yeoh, Ee Peng <ee.peng.yeoh at intel.com>; openembedded-core at lists.openembedded.org
Subject: Re: [OE-core] [PATCH 01/12] build-appliance: Drop test which is already automated

On Wed, 2018-11-14 at 08:05 +0000, Yeoh, Ee Peng wrote:
> > In that case wouldn't it make more sense to combine these steps with 
> > some of the other build appliance tests? We could add an extra step 
> > where the tester builds build-appliance rather than having multiple 
> > tests which start "launch build appliance"?
> 
> In general, Pros & Cons for merging test cases, Pros: reduced the 
> number of test cases & making it simpler for reading, Cons: loss of 
> lower level tracking on the test status.
> Example: currently we have multiple test cases for build-appliance 
> where each test cases testing the ability to use build-appliance- 
> image to bitbake different type of images (recently we detect that 
> build-appliance-image was able to build core-image-minimal & core- 
> image-sato-sdk but it failed to build the build-appliance-image), if 
> we merge multiple test cases into general 
> test_build_appliance_can_build_images, then we will loss the 
> visibilities on which type of images failed when this test failed, 
> unless we looked into the test log if test log were generated.

If a step fails, QA will write a bug report for the failure. I'd have thought it would be quite clear in this case what the failing step was?
QA would also always include a failing build log in such a case which should also make the problem quite clear.

Even if the step was "bitbake core-image-minimal core-image-sato build- appliance-image -k", the logs should still clearly show what it was that failed?

Cheers,

Richard