[OE-core] [PATCH 01/12] build-appliance: Drop test which is already automated

[Yeoh, Ee Peng]

-----Original Message-----
From: richard.purdie at linuxfoundation.org [mailto:richard.purdie at linuxfoundation.org] 
Sent: Monday, November 12, 2018 7:44 PM
To: Yeoh, Ee Peng <ee.peng.yeoh at intel.com>; openembedded-core at lists.openembedded.org
Subject: Re: [OE-core] [PATCH 01/12] build-appliance: Drop test which is already automated

On Mon, 2018-11-12 at 01:41 +0000, Yeoh, Ee Peng wrote:
> I realize the execution action inside "build-appliance.build- 
> appliance.Bitbake_build-appliance-image" was not complete. Actually 
> this particular test was testing that using build-appliance-image and 
> vmware, one can bitbake build-applianc-image. This was different 
> compare the automated tests inside AB where it test that bitbake can 
> build-appliance-image.
> 
> The correct steps for "build-appliance.build-appliance.Bitbake_build-
> appliance-image" shall be below. Allow us to submit a patch to add the 
> missing detail for this test.
> 
> Steps to Reproduce :
> 1. Get build appliance image from following link
> 
https://autobuilder.yocto.io/pub/releases/yocto-2.6.rc1/build-appliance/Yocto_Build_Appliance.zip
> 
> 2. boot the build appliance image in vmware.
> 
> 3. Set MACHINE to qemux86 and add the following line to 
> conf/local.conf :
> SRCREV_pn-build-appliance-image = "$
> 
> {AUTOREV}
> "
> 4. Run "bitbake build-appliance-image"
> 
> I am sorry for the confusion it created.

In that case wouldn't it make more sense to combine these steps with some of the other build appliance tests? We could add an extra step where the tester builds build-appliance rather than having multiple tests which start "launch build appliance"?

In general, Pros & Cons for merging test cases, Pros: reduced the number of test cases & making it simpler for reading, Cons: loss of lower level tracking on the test status. 
Example: currently we have multiple test cases for build-appliance where each test cases testing the ability to use build-appliance-image to bitbake different type of images (recently we detect that build-appliance-image was able to build core-image-minimal & core-image-sato-sdk but it failed to build the build-appliance-image), if we merge multiple test cases into general test_build_appliance_can_build_images, then we will loss the visibilities on which type of images failed when this test failed, unless we looked into the test log if test log were generated.  

Cheers,

Richard