[OE-core] [PATCH] selftest/manifest.py: Test to verify rootfs manifest

[Paul Eggleton]

On Thursday 27 August 2015 10:50:51 Benjamin Esquivel wrote:
> Hey Paul, thanks for looking at this, check the comments below.
> 
> On Thu, 2015-08-27 at 09:34 +0100, Paul Eggleton wrote:
> > Hi Benjamin / Mariano,
> > 
> > On Wednesday 26 August 2015 12:26:02 Benjamin Esquivel wrote:
> > > Adding a new test to verify if the packages in the
> > > manifest files actually exists in pkgdata.
> > > 
> > >  -adding a setUpClass for when more tests get created here
> > >  -check for the paths and fail gracefully if not there
> > >  -skip the test when there are no manifest files to check
> > >  -debug prints for failure analysis
> > > 
> > > [YOCTO#8028]
> > > 
> > > Signed-off-by: Benjamin Esquivel <benjamin.esquivel at linux.intel.com
> > > 
> > > Signed-off-by: Mariano Lopez <mariano.lopez at linux.intel.com>
> > > ---
> > > 
> > >  meta/lib/oeqa/selftest/manifest.py | 52
> > > 
> > > ++++++++++++++++++++++++++++++++++++++ 1 file changed, 52
> > > insertions(+)
> > > 
> > >  create mode 100644 meta/lib/oeqa/selftest/manifest.py
> > > 
> > > diff --git a/meta/lib/oeqa/selftest/manifest.py
> > > b/meta/lib/oeqa/selftest/manifest.py new file mode 100644
> > > index 0000000..c2bc945
> > > --- /dev/null
> > > +++ b/meta/lib/oeqa/selftest/manifest.py
> > > @@ -0,0 +1,52 @@
> > > +import unittest
> > > +import os
> > > +from glob import glob
> > > +
> > > +from oeqa.selftest.base import oeSelfTest
> > > +from oeqa.utils.commands import get_bb_var
> > > +from oeqa.utils.decorators import testcase
> > > +
> > > +class VerifyManifest(oeSelfTest):
> > > +    '''Tests for the manifest files and contents of an image'''
> > > +
> > > +    @classmethod
> > > +    def setUpClass(self):
> > > +
> > > +        # get directory locations from variable values and check
> > > them
> > > +        self.deploy_dir_image = get_bb_var('DEPLOY_DIR_IMAGE');
> > > +        if not self.deploy_dir_image \
> > > +            or not os.path.isdir(self.deploy_dir_image):
> > > +            raise unittest.SkipTest("{}: DEPLOY_DIR_IMAGE does not
> > > exist:
> > > {}"\ +                    .format("VerifyManifest",
> > > self.deploy_dir_image))
> > > +
> > > +        self.pkgdata_dir = get_bb_var('PKGDATA_DIR');
> > > +        if not self.pkgdata_dir \
> > > +            or not os.path.isdir(self.pkgdata_dir):
> > > +            raise unittest.SkipTest("{}: PKGDATA_DIR does not
> > > exist: {}"\
> > > +                    .format("VerifyManifest", self.pkgdata_dir))
> > > +
> > > +        # get the manifest files
> > > +        # no need to try: since glob would return an empty list if
> > > +        # the path is non-existant
> > > +        self.manifest_files = glob("%s/*.manifest" %
> > > self.deploy_dir_image)
> > > +        self.log.debug("manifest files:
> > > {}".format(self.manifest_files)) +
> > > 
> > >        if not self.manifest_files:
> > > +            raise unittest.SkipTest("{}: No manifest files found
> > > in: {}"\
> > > +                .format("VerifyManifest", self.deploy_dir_image))
> > > +
> > > +
> > > +    def test_manifest_entries(self):
> > > +        '''Verifying the manifest entries as packages '''
> > > +        testname = self.id().split('.')[-1]
> > > +        rundir = os.path.join(self.pkgdata_dir, "runtime-reverse")
> > > +
> > > +        errmsg = "ERROR: Package %s is in manifest but not in
> > > pkgdata"
> > > +        for manifest in self.manifest_files:
> > > +            with open(manifest, "r") as mfile:
> > > +                for manifest_entries in mfile:
> > > +                    pkg = manifest_entries.split()[0]
> > > +                    pkgfile = os.path.join(rundir, pkg)
> > > +                    self.log.debug("{}: looking for {}"\
> > > +                            .format(testname, pkgfile))
> > > +                    self.assertTrue(os.path.isfile(pkgfile),
> > > +                            errmsg % pkg)
> > 
> > Shouldn't this test actually be building an image (e.g. using
> > bitbake('core-image-minimal') ) in order to ensure there's a manifest
> > file for
> > it to look at? Then you can make the test fail rather than skipping
> > if the file
> > isn't there or doesn't contain a small set of packages we know should
> > be
> > listed.
> 
> The reason the actual build of an image was not included in the test is
> because it is abstracted from it. That's why it contains checks for
> when the manifests are not there it skips the test instead of failing.
> 
> This would give you the chance to write some level of abstraction that
> is able to have test subjects (i.e. subject A: core-image-minimal with
> systemd and ipk). Otherwise you're locking the test to a single
> scenario which is usually the most common. All the corner cases are
> hidden from that scope of testing.
>
> On the other hand, if we had that design I'm talking about we could
> easily have a default behavior that does exactly what you're proposing.
> 
> What do you think?

We want oe-selftest to be able to objectively test that expected functionality 
works when run unattended (as it is on the autobuilder), so if there are 
multiple scenarios to test then we probably need to be explicitly setting 
those up and testing them here rather than expecting the user to set them up 
beforehand. If that adds time to the oe-selftest run, well, so be it.

> > Also while you're adding this could you add a similar test function
> > for the
> > SDK (both host + target manifests).
> 
> I believe we could add many more tests that's why I started splitting
> common tasks to the setup function of the test.
> 
> I'll add this in the next patch version.

Great, thanks.

Cheers,
Paul

-- 

Paul Eggleton
Intel Open Source Technology Centre