[oe-commits] [openembedded-core] 18/19: ptest: add an option to enforce a failure (e.g. ptests are required to pass)

[git at git.openembedded.org]

This is an automated email from the git hooks/post-receive script.

rpurdie pushed a commit to branch master
in repository openembedded-core.

commit 102afcc7ea7d7673183f272a29ab233084ad168b
Author: Alexander Kanavin <alex.kanavin at gmail.com>
AuthorDate: Sun Feb 23 14:52:31 2020 +0100

    ptest: add an option to enforce a failure (e.g. ptests are required to pass)
    
    This can be useful with a more specific, targeted and robust set of ptest
    packages; the benefit is that ptest regressions are caught as they happen
    and can be more easily traced to changes that caused them.
    
    The existing AB ptest image continues to be expected to fail, my observation
    of the AB runs is that the full set of ptests is not robust enough
    (particularly around socket/network related tests) and sporadically fails
    in random places. This can probably be addressed by making ptests exclusive
    to a worker (e.g. there is no other workload happening at the same time as
    ptests).
    
    Signed-off-by: Alexander Kanavin <alex.kanavin at gmail.com>
    Signed-off-by: Richard Purdie <richard.purdie at linuxfoundation.org>
---
 meta/lib/oeqa/runtime/cases/ptest.py                  | 15 ++++++++++++++-
 meta/recipes-sato/images/core-image-sato-sdk-ptest.bb |  3 +++
 2 files changed, 17 insertions(+), 1 deletion(-)

diff --git a/meta/lib/oeqa/runtime/cases/ptest.py b/meta/lib/oeqa/runtime/cases/ptest.py
index eb284df..5626f70 100644
--- a/meta/lib/oeqa/runtime/cases/ptest.py
+++ b/meta/lib/oeqa/runtime/cases/ptest.py
@@ -18,7 +18,20 @@ class PtestRunnerTest(OERuntimeTestCase):
     @OETestDepends(['ssh.SSHTest.test_ssh'])
     @OEHasPackage(['ptest-runner'])
     @unittest.expectedFailure
-    def test_ptestrunner(self):
+    def test_ptestrunner_expectfail(self):
+        if not self.td.get('PTEST_EXPECT_FAILURE'):
+            self.skipTest('Cannot run ptests with @expectedFailure as ptests are required to pass')
+        self.do_ptestrunner()
+
+    @skipIfNotFeature('ptest', 'Test requires ptest to be in DISTRO_FEATURES')
+    @OETestDepends(['ssh.SSHTest.test_ssh'])
+    @OEHasPackage(['ptest-runner'])
+    def test_ptestrunner_expectsuccess(self):
+        if self.td.get('PTEST_EXPECT_FAILURE'):
+            self.skipTest('Cannot run ptests without @expectedFailure as ptests are expected to fail')
+        self.do_ptestrunner()
+
+    def do_ptestrunner(self):
         status, output = self.target.run('which ptest-runner', 0)
         if status != 0:
             self.skipTest("No -ptest packages are installed in the image")
diff --git a/meta/recipes-sato/images/core-image-sato-sdk-ptest.bb b/meta/recipes-sato/images/core-image-sato-sdk-ptest.bb
index 58c257c..bf749ac 100644
--- a/meta/recipes-sato/images/core-image-sato-sdk-ptest.bb
+++ b/meta/recipes-sato/images/core-image-sato-sdk-ptest.bb
@@ -18,3 +18,6 @@ IMAGE_ROOTFS_EXTRA_SPACE = "1124288"
 # ptests need more memory than standard to avoid the OOM killer
 # also lttng-tools needs /tmp that has at least 1G
 QB_MEM = "-m 2048"
+
+# Sadly at the moment the full set of ptests is not robust enough and sporadically fails in random places
+PTEST_EXPECT_FAILURE = "1"

-- 
To stop receiving notification emails like this one, please contact
the administrator of this repository.