[OE-core] Discussion about architecure of ptest-runner patch

[Tudor Florea]

Hi,

> -----Original Message-----
> From: openembedded-core-bounces at lists.openembedded.org
> [mailto:openembedded-core-bounces at lists.openembedded.org] On Behalf
> Of Markus Boos
> Sent: Friday, October 24, 2014 12:37
> To: 'openembedded-core at lists.openembedded.org'
> Subject: [OE-core] Discussion about architecure of ptest-runner patch
> 
> Hi
> 
> We patched the ptest-runner script to deliver a xUnit file as a rough
> overview/indicator over the package tests.
> Having such a report allows us to use the file in Jenkins and in our ALM tool to
> display the results of the ptest run.
> 
> Before we send the patch upstream I like to start the discussion about the
> architecture of the patch because I think it's worth to extend the ptest-
> runner to deliver the results in readable text format (xUnit, JSON, html table,
> csv,  whateverformat).
> I think of something similar like the automated deployment mechanism
> where yocto delivers the base class and you can write your own extension
> with this class.

The concept of ptest is to run the tests using the minimum possible appropriate resource of a target to be tested.
As per wiki page (https://wiki.yoctoproject.org/wiki/Ptest):
"One major point of ptest is to consolidate the output format of all tests into a single common format. The format selected is the automake "simple test" format
result: testname"
As per automake manual (http://www.gnu.org/software/automake/manual/automake.html#Simple-Tests).
"The possible results (whose meanings should be clear from the previous Generalities about Testing) are PASS, FAIL, SKIP, XFAIL, XPASS and ERROR"
While I acknowledge that an automated mechanism to process and display the test results would be a nice piece of work for an automated testing framework,  I don't think that this belongs to ptest (ptest-runner).
The processing of the test results of an embedded tested device should not be done by the embedded device itself (inside ptest-runner in particular) but rather by the testing infrastructure (Jenkins? autobuild?) that eventually spawn ptest-runner.

The next steps to improve ptest is to provide parallelism (that is to run parallel-tests for each feature tested and  to run ptest for many feature in parallel). The work to be done to process the results (e.g. for a readable text format)  should take into consideration this aspect.

Regards,
  Tudor.

> 
> What is the best way to add such a feature?
> 
> Simply extend the ptest-runner script with the preferred output formatter?
> 
> Or
> 
> Write a bitbake class with an interface where you can attach your own
> preferred output formatter?
> 
> I'm sure there are other solutions possible as well. If you have one in mind,
> please share it.
> 
> Thank you.
> 
> Best regards
> Markus
> --
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core at lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-core