[OE-core] Discussion about architecure of ptest-runner patch

[Markus Boos]

Hi Tudor

Thank you for your explanation.

We made a small sed/awk script to format the googletest output to the ptest format.
We published it on the yocto automated-testing list: https://lists.yoctoproject.org/pipermail/automated-testing/2014-November/000091.html

BR
Markus




> -----Ursprüngliche Nachricht-----
> Von: Tudor Florea [mailto:Tudor.Florea at enea.com]
> Gesendet: Samstag, 25. Oktober 2014 02:57
> An: Boos Markus; 'openembedded-core at lists.openembedded.org'
> Betreff: RE: Discussion about architecure of ptest-runner patch
> 
> Hi,
> 
> > -----Original Message-----
> > From: openembedded-core-bounces at lists.openembedded.org
> > [mailto:openembedded-core-bounces at lists.openembedded.org] On Behalf Of
> > Markus Boos
> > Sent: Friday, October 24, 2014 12:37
> > To: 'openembedded-core at lists.openembedded.org'
> > Subject: [OE-core] Discussion about architecure of ptest-runner patch
> >
> > Hi
> >
> > We patched the ptest-runner script to deliver a xUnit file as a rough
> > overview/indicator over the package tests.
> > Having such a report allows us to use the file in Jenkins and in our
> > ALM tool to display the results of the ptest run.
> >
> > Before we send the patch upstream I like to start the discussion about
> > the architecture of the patch because I think it's worth to extend the
> > ptest- runner to deliver the results in readable text format (xUnit,
> > JSON, html table, csv,  whateverformat).
> > I think of something similar like the automated deployment mechanism
> > where yocto delivers the base class and you can write your own
> > extension with this class.
> 
> The concept of ptest is to run the tests using the minimum possible appropriate
> resource of a target to be tested.
> As per wiki page (https://wiki.yoctoproject.org/wiki/Ptest):
> "One major point of ptest is to consolidate the output format of all tests into a single
> common format. The format selected is the automake "simple test" format
> result: testname"
> As per automake manual
> (http://www.gnu.org/software/automake/manual/automake.html#Simple-Tests).
> "The possible results (whose meanings should be clear from the previous
> Generalities about Testing) are PASS, FAIL, SKIP, XFAIL, XPASS and ERROR"
> While I acknowledge that an automated mechanism to process and display the test
> results would be a nice piece of work for an automated testing framework,  I don't
> think that this belongs to ptest (ptest-runner).
> The processing of the test results of an embedded tested device should not be done
> by the embedded device itself (inside ptest-runner in particular) but rather by the
> testing infrastructure (Jenkins? autobuild?) that eventually spawn ptest-runner.
> 
> The next steps to improve ptest is to provide parallelism (that is to run parallel-tests
> for each feature tested and  to run ptest for many feature in parallel). The work to be
> done to process the results (e.g. for a readable text format)  should take into
> consideration this aspect.
> 
> Regards,
>   Tudor.
> 
> >
> > What is the best way to add such a feature?
> >
> > Simply extend the ptest-runner script with the preferred output formatter?
> >
> > Or
> >
> > Write a bitbake class with an interface where you can attach your own
> > preferred output formatter?
> >
> > I'm sure there are other solutions possible as well. If you have one
> > in mind, please share it.
> >
> > Thank you.
> >
> > Best regards
> > Markus
> > --
> > _______________________________________________
> > Openembedded-core mailing list
> > Openembedded-core at lists.openembedded.org
> > http://lists.openembedded.org/mailman/listinfo/openembedded-core