<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 14 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:Wingdings;
panose-1:5 0 0 0 0 0 0 0 0 0;}
@font-face
{font-family:Wingdings;
panose-1:5 0 0 0 0 0 0 0 0 0;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman","serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p
{mso-style-priority:99;
mso-margin-top-alt:auto;
margin-right:0in;
mso-margin-bottom-alt:auto;
margin-left:0in;
font-size:12.0pt;
font-family:"Times New Roman","serif";}
span.EmailStyle18
{mso-style-type:personal-reply;
font-family:"Calibri","sans-serif";
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri","sans-serif";}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Hello Eric,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">I am happy to see you are also working on this. We already have 2 features that should be implemented in 1.7 related to this that should help along with integrating
ptest with automated tests. These features use the current ‘testimage’ framework that runs runtime tests on both QEMU and hardware targets. More information can be found in the Yocto manual and some Yocto wiki pages:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><a href="http://www.yoctoproject.org/docs/current/mega-manual/mega-manual.html#performing-automated-runtime-testing">http://www.yoctoproject.org/docs/current/mega-manual/mega-manual.html#performing-automated-runtime-testing</a>
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><a href="https://wiki.yoctoproject.org/wiki/Image_tests">https://wiki.yoctoproject.org/wiki/Image_tests</a><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><a href="https://wiki.yoctoproject.org/wiki/Test-remote-image">https://wiki.yoctoproject.org/wiki/Test-remote-image</a><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Here are the feature requests themselves:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><a href="https://bugzilla.yoctoproject.org/show_bug.cgi?id=6335">https://bugzilla.yoctoproject.org/show_bug.cgi?id=6335</a><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><a href="https://bugzilla.yoctoproject.org/show_bug.cgi?id=4114">https://bugzilla.yoctoproject.org/show_bug.cgi?id=4114</a><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">The parsing of the log files is an important part of this integration and maybe we can work together on this!
</span><span style="font-size:11.0pt;font-family:Wingdings;color:#1F497D">J</span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"> I myself just started to work on these features for the 1.7 M3 milestone and came across your
email while digging the opemembedded mailing list for a starting point.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Regards,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Corneliu<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Yocto QA<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<div style="border:none;border-left:solid blue 1.5pt;padding:0in 0in 0in 4.0pt">
<div>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">From:</span></b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif""> openembedded-core-bounces@lists.openembedded.org [mailto:openembedded-core-bounces@lists.openembedded.org]
<b>On Behalf Of </b>Eric Yu<br>
<b>Sent:</b> Thursday, July 24, 2014 7:40 PM<br>
<b>To:</b> openembedded-core@lists.openembedded.org<br>
<b>Cc:</b> Alejandro del Castillo; Bill Pittman<br>
<b>Subject:</b> [OE-core] ptest formatting<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial","sans-serif"">Hello,<br>
My name is Eric Yu, and I am an intern at National Instruments for this summer. The project I am currently working on involves integrating ptest with our current automated testing framework to help test our open-embedded distributions. On the Yocto Project
wiki, it states that one major point of ptest is to consolidate the output of different package tests in a common “<result>: <testname>” format. I was hoping that this common format would allow ptest results to be easily machine parsed and integrated with
our current testing framework. However, after enabling and running the ptests, I discovered that the formatting of the results was not as friendly to automation as I had hoped.</span>
<o:p></o:p></p>
<p><span style="font-size:10.0pt;font-family:"Arial","sans-serif""> It appears that each separate package prints out its own errors, warnings, and other output along with these test results, burying the common output in lots of other text. Also, one package
(gdk-pixbuff) used “FAILED” when reporting results rather than the expected “FAIL”.
</span><o:p></o:p></p>
<p><span style="font-size:10.0pt;font-family:"Arial","sans-serif"">In the bash ptests, several tests print warnings saying to ignore failures where the output differs only by whitespace. This seems to be bad practice for test writing and is not friendly to
automated analysis of test results. </span><o:p></o:p></p>
<p><span style="font-size:10.0pt;font-family:"Arial","sans-serif"">At the conclusion of each ptest, some packages give a summary of how many tests were passed, skipped, and failed, while others do not. I find that having these summaries gives a useful overview
of how the test went overall and is a good reference in case some tests fail to produce the common “<result>: <testname>” output.</span>
<o:p></o:p></p>
<p><span style="font-size:10.0pt;font-family:"Arial","sans-serif"">I understand that much of this is due to the fact that separate developers write the tests for different packages, but it would be beneficial if ptest was friendlier to automated parsing and
analysis of test results. Currently, I have addressed some of these obstacles by writing a simple script that parses the output of each ptest and only outputs only the “<result>: <testname>” results while accounting for both “FAIL” and “FAILED”. The script
keeps a running count of how many tests were reported as failed, skipped or passed, and at the conclusion of each ptest, the script prints a summary including the number of tests passed, skipped, and failed along with a total number of tests run. While this
works with the current set of ptests, as more and more packages add ptest functionality, this script may not scale well if more inconsistencies in formatting are introduced. Therefore, I believe it would be a good idea to enforce a more consistent formatting
of ptest results to assist in the use of ptest for automated testing. Are there any plans to further consolidate the ptest result format such that it is more accessible for automated testing?</span>
<o:p></o:p></p>
</div>
</div>
</body>
</html>