[OE-core] oe-selftest proof of concept

[Stoicescu, CorneliuX]

Thank you for the reply, Paul. I will make sure to make the most of your input!

> -----Original Message-----
> From: Paul Eggleton [mailto:paul.eggleton at linux.intel.com]
> Sent: Thursday, October 31, 2013 12:34 PM
> To: Stoicescu, CorneliuX
> Cc: openembedded-core at lists.openembedded.org
> Subject: Re: [OE-core] oe-selftest proof of concept
> 
> Hi Corneliu,
> 
> This is a well-rounded proposal, thanks. Some comments below.
> 
> On Thursday 31 October 2013 07:53:48 Stoicescu, CorneliuX wrote:
> > This e-mail was originally sent to the Yocto mailing list in the form
> > of 2 e-mails. As per Paul's and Richard's request, I am re-sending and
> > moving the conversation here. Feel free to add any input :) .
> >
> > NOTE: I also made a small syntax correction in the example at the end
> > of the e-mail for oeSelfTest.var_append() .
> >
> > After a chat with Richard and Stefan, I came up with an outline of how
> > the oe-selftest
> > feature(https://bugzilla.yoctoproject.org/show_bug.cgi?id=4740
> > ) should look like. I made a summary of my proposal below. Please feel
> > free to add your thoughts!
> >
> > There will be a new script introduced(similar to bitbake-selftest)
> > that will use python unit test to execute tests. Name has not been
> > decided but it can be "oe-selftest". Running this script will not compromise
> poky in any way.
> > Initially, the script does not need any preparation in order to be
> > run. If this changes in the future, the user will be prompted upon
> > execution with the pre-required tasks. Oe-selftest can be used
> > together with the automated runtime tests if necessary.
> >
> > The following types of tests are targeted for the initial implementation:
> > - testing the functionality of scripts in poky/scripts (such as:
> > bitbake-layers, yocto-bsp, yocto-kernel, yocto-layers) - testing of
> > the 'bitbake' command and its output (this includes output data
> > validation such as the sstate-cache/ and tmp/ directories)
> 
> It could be considered a separate exercise, but I'd like us to test the
> installation and usage of the SDK installer as well. Initially this would be fairly
> straightforward - install the SDK to a non-standard location, fetch some
> nominated piece of source code and try to build it using the installed SDK.
> (One issue springs to mind though - unless we take steps to guard against it,
> this won't be able to pick up problems where the SDK contains references to
> files within TMPDIR, since those will still be valid on the build host).
> 
> Later we'd want to be able to test the executables produced using the SDK
> on the target; however that would presumably necessitate some integration
> with the runtime tests and that could be complicated.
> 

We will be able to add such functionality with ease once we are able to run bitbake commands in a shared resources multi-build environment. It would help though to keep in mind when designing the execute method that ${BUILDDIR} should actually be more like ${WORKDIR} and we should be able to create more types of build environments(not just poky/build/ types).

> > In order to achieve the above, the script will need to:
> >
> > -          manipulate configuration files and variables
> >
> > -          issue commands in a certain order
> >
> > -          analyze command output
> >
> > -          analyze output files and directories
> >
> > -          manage shared resources such as sstate-cache
> >
> > Oe-selftest will use Python Unit Testing as follows:
> >                 - we will use what has already been done with the 'unittest'
> > and 'oeqa' classes. This includes 'decorators' and methods like
> > 'skipModules'. - the tests configuration files will be similar to
> > those of the automated runtime tests.
> > - because we are using the same framework and structure, the output
> > will also be similar to that of the automated runtime tests
> > - the build environments used will be set up in the setUpClass() or
> > setUpModule() methods
> >    * local.conf may be modified here to use shared resources like
> > sstate-cache and downloads directory with other build environments -
> > the build environments used will be removed in the
> > tearDownClass() or tearDownModule() methods * this ensures no residue
> > is left over that we are not aware of
> > - we will add a new class
> > oeqa.oeSelfTest that contains the following base methods:
> >
> > 1) A few methods to manipulate variables in poky in a standardized way:
> >                 -
> > oeSelfTest.var_rewrite(filepath_relative_to_poky_location,
> > variable_name, new_variable_value) * resets a variable to the
> > specified value(may be similar to how HOB does to local.conf and
> > bblayers.conf) -
> > oeSelfTest.var_append(filepath_relative_to_poky_location,
> > variable_name,
> > appended_variable_value) * appends data to a variable
> >                 -
> > oeSelfTest.var_remove(filepath_relative_to_poky_location,
> > variable_name, remove_variable_value) * removes data from a variable
> >
> > 2) We need to be able to execute tasks in a multiple build directories
> > environment.
> > - oeSelfTest.execute(command, build_dir)
> >                                 * this method executes the command 'command'
> > from 'build_dir' * for each execution, the environment is sourced
> > (similar to autobuilder) (if the build directory does not exist, the
> > command will fail. This needs to be taken care of in the setUpClass()
> > method) * it returns the exit code and the output of the command
> >
> > Additional information/notes:
> > - class and method naming and structure is not final and open for
> > suggestions
> > - poky under git is required for the script to work; we will use git
> > to reset modified files to default.
> 
> Actually I think we should avoid modifying existing files if at all possible -
> instead we should add an additional layer on top to make changes, using
> bbappends / overlayed recipes as necessary. There are several reasons for
> this:
> 
> * Most of the time this is the approach users should be using when they
> make customisations, so it's what we ought to concentrate on testing.
> 
> * It preserves the ability to run the tests with uncommitted changes, which
> would be useful during development.
> 
> * Last but certainly not least, the tests need to work also in separate OE-
> Core + BitBake  configuration, since that's how many in the community use
> the build system. If we don't try to manage the repository this should just
> work without extra effort.
> 

I also think a meta-selftest layer would be helpful and make the oe-selftest functionality self-contained. The git functionality will be there anyway if special situations arise though. 

> > - a policy needs to be set in place(after experience has been gained
> > from using the tool) for working with a shared sstate-cache directory
> > and multiple build environments at a time.
> 
> For tests where it matters, we'll have to ensure the sstate-cache only
> contains what we need it to contain. I suspect this will mean we'll need to
> have the script set SSTATE_DIR and SSTATE_MIRRORS so that any user
> configuration of these doesn't interfere with the test.
> 

We can do this when setting up the build environment. This can be done either by exporting variables(straying from the normal use pattern) or setting them in local.conf(following the normal use pattern). I believe we should not stray too far from the normal use pattern because the results might not be so relevant(though having both or a combination may be a good settle). We need to design the 'execute', 'var_append' and 'var_rewrite' methods to follow normal user patterns in order to get the best data possible. 

> > - coding style is very important in order to get a reasonable
> > execution time. At the moment this is not a priority though.
> 
> I guess this is perhaps implicit; but we'll want to have a means of easily
> selecting which tests to run (as we currently do with the runtime testing).
> 
> > - no strategy has been put in place yet in order to deal with multiple
> > poky environments
>
 > What exactly do you mean by this?

There are some(few) test cases that require having multiple poky directories at the same time (example: the 'bitbake-prserv' tool) . I cannot say of if the future these types of tests will become more "popular".

> 
> > I made an example of how we can use the functionality described to
> > conduct 3 of our test cases:
> > https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=90
> > https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=93
> > https://bugzilla.yoctoproject.org/tr_show_case.cgi?case_id=83
> >
> > As mentioned, the structure is very similar to the automated runtime
> > tests configuration files. NOTE: ${POKYDIR} does not exist but I use
> > it to refer to the location of the poky directory.
> >
> > "
> > import unittest
> > import os
> > from oeqa import oeSelfTest, skipModule from oeqa.utils.decorators
> > import *
> >
> > def setUpModule():
> > #             Here we set up the environment for a build directory
> >
> > class bitbake_layers(oeSelfTest):
> >
> >                 @classmethod
> >                 def setUpClass(self):
> > #                             Here we set up special settings for the build
> > environment, like shared sstate-cache directory
> >
> >                 def test_show_overlayed(self):
> >                                 oeSelfTest.execute("cp -R
> > ${POKYDIR}/meta/recipes-graphics/clutter
> > ${POKYDIR}/meta/recipes-core/") (status, output) =
> > oeSelfTest.execute("bitbake-layers show_overlayed | grep clutter",
> > build_dir) oeSelfTest.execute("rm -R
> > ${POKYDIR}/meta/recipes-core/clutter") self.assertEqual(status, 0,
> > msg="Could not find overlayed recipe")
> >
> >                 def test_show_appends(self):
> >                                 oeSelfTest.execute("echo "" >
> > ${POKYDIR}/meta-yocto/recipes-core/busybox/linux-
> yocto_3.10.bbappend"/
> > ") (status, output) = oeSelfTest.execute("bitbake_layers show_appends
> > | grep
> > \"/meta-yocto/recipes-core/busybox/linux-yocto_3.10.bbappend\"",
> > build_dir) self.assertEqual(status, 0, msg=" 'bitbake_layers
> > show_appends' Could not find the bbappend file")
> >
> >                 def test_show_layers(self):
> >                                 (ls_status) = oeSelfTest.execute("ls
> > ${POKYDIR} | grep meta-intel") if ls_status == 1:
> >                                                 (git_status) =
> > oeSelfTest.execute("git clone git://git.yoctoproject.org/meta-intel
> > ${POKYDIR}/")
> > oeSelfTest.var_append("${BUILDDIR}/conf.bblayers.conf",
> > "BBLAYERS", "${POKYDIR}/meta-intel")
> >
> > (status, output) =
> > oeSelfTest.execute("bitbake-layers show_layers | grep meta-intel",
> > build_dir)
> 
> I know the test case mentions it explicitly, but we don't actually need meta-
> intel to check this functionality, any layer will work. We probably ought to be
> creating a meta-selftest layer (or similar) within OE-Core that we could use
> for this purpose and the addition of bbappends / additional configuration.
> This avoids the need for something like POKYDIR as well.
> 
> Cheers,
> Paul
> 
> --
> 
> Paul Eggleton
> Intel Open Source Technology Centre

Regards,
Corneliu
Romania Yocto QA