<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman","serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:#954F72;
text-decoration:underline;}
p.MsoPlainText, li.MsoPlainText, div.MsoPlainText
{mso-style-priority:99;
mso-style-link:"Plain Text Char";
margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif";}
p
{mso-style-priority:99;
mso-margin-top-alt:auto;
margin-right:0in;
mso-margin-bottom-alt:auto;
margin-left:0in;
font-size:12.0pt;
font-family:"Times New Roman","serif";}
span.EmailStyle18
{mso-style-type:personal;
font-family:"Calibri","sans-serif";
color:#1F497D;}
span.PlainTextChar
{mso-style-name:"Plain Text Char";
mso-style-priority:99;
mso-style-link:"Plain Text";
font-family:"Calibri","sans-serif";}
span.EmailStyle21
{mso-style-type:personal-compose;
font-family:"Calibri","sans-serif";
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;}
@page WordSection1
{size:8.5in 11.0in;
margin:70.85pt 70.85pt 70.85pt 70.85pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="#0563C1" vlink="#954F72">
<div class="WordSection1">
<p class="MsoPlainText">Hi,<o:p></o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText">On 7/24/2014 19:40, Eric Yu wrote:<o:p></o:p></p>
<p class="MsoPlainText">> Hello,<o:p></o:p></p>
<p class="MsoPlainText">> My name is Eric Yu, and I am an intern at National Instruments for this summer. The project I am currently
<o:p></o:p></p>
<p class="MsoPlainText">> working on involves integrating ptest with our current automated testing framework to help test our open-<o:p></o:p></p>
<p class="MsoPlainText">> embedded distributions. On the Yocto Project wiki, it states that one major point of ptest is to consolidate the
<o:p></o:p></p>
<p class="MsoPlainText">> output of different package tests in a common “<result>: <testname>” format. I was hoping that this common
<o:p></o:p></p>
<p class="MsoPlainText">> format would allow ptest results to be easily machine parsed and integrated with our current testing framework.
<o:p></o:p></p>
<p class="MsoPlainText">> However, after enabling and running the ptests, I discovered that the formatting of the results was not as
<o:p></o:p></p>
<p class="MsoPlainText">> friendly to automation as I had hoped.<o:p></o:p></p>
<p class="MsoPlainText">><o:p> </o:p></p>
<p class="MsoPlainText">> It appears that each separate package prints out its own errors, warnings, and other output along with these
<o:p></o:p></p>
<p class="MsoPlainText">> test results, burying the common output in lots of other text.
<o:p></o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText">Indeed (one of) the purpose of ptest is to facilitate automation by using the common <result>: <testname> format.<o:p></o:p></p>
<p class="MsoPlainText">However, ptest should not filter the output of a package test (so that the automation could be done easier).<o:p></o:p></p>
<p class="MsoPlainText">That output might be useful in later investigation about why a test failed and further improving the package (testing).<o:p></o:p></p>
<p class="MsoPlainText">Is should be easier to implement this filter in your automation scripts.<o:p></o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText">><o:p> </o:p></p>
<p class="MsoPlainText">> Also, one package (gdk-pixbuff) used “FAILED” when reporting results rather than the expected “FAIL”.
<o:p></o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText">This sounds like a bug. You can help us sending a patch for this. :)<o:p></o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText">><o:p> </o:p></p>
<p class="MsoPlainText">> In the bash ptests, several tests print warnings saying to ignore failures where the output differs only by
<o:p></o:p></p>
<p class="MsoPlainText">> whitespace. This seems to be bad practice for test writing and is not friendly to automated analysis of test
<o:p></o:p></p>
<p class="MsoPlainText">> results. <o:p></o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText">This is a good example to support my statement above: While automation scripts may easily
<o:p></o:p></p>
<p class="MsoPlainText">add some PASS, FAIL, SKIP, XFAIL, XPASS and ERROR entries into a database, those (additional) warning are helpful
<o:p></o:p></p>
<p class="MsoPlainText">for people interested in improving bash (an bash testing), hence the output should be kept along with the results.<o:p></o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText">><o:p> </o:p></p>
<p class="MsoPlainText">> At the conclusion of each ptest, some packages give a summary of how many tests were passed, skipped, and
<o:p></o:p></p>
<p class="MsoPlainText">> failed, while others do not. I find that having these summaries gives a useful overview of how the test went
<o:p></o:p></p>
<p class="MsoPlainText">> overall and is a good reference in case some tests fail to produce the common “<result>: <testname>” output.<o:p></o:p></p>
<p class="MsoPlainText">><o:p> </o:p></p>
<p class="MsoPlainText">> I understand that much of this is due to the fact that separate developers write the tests for different packages,
<o:p></o:p></p>
<p class="MsoPlainText">> but it would be beneficial if ptest was friendlier to automated parsing and analysis of test results. Currently, I <o:p></o:p></p>
<p class="MsoPlainText">> have addressed some of these obstacles by writing a simple script that parses the output of each ptest and only
<o:p></o:p></p>
<p class="MsoPlainText">> outputs only the “<result>: <testname>” results while accounting for both “FAIL” and “FAILED”. The script keeps
<o:p></o:p></p>
<p class="MsoPlainText">> a running count of how many tests were reported as failed, skipped or passed, and at the conclusion of each
<o:p></o:p></p>
<p class="MsoPlainText">> ptest, the script prints a summary including the number of tests passed, skipped, and failed along with a total
<o:p></o:p></p>
<p class="MsoPlainText">> number of tests run. While this works with the current set of ptests, as more and more packages add ptest
<o:p></o:p></p>
<p class="MsoPlainText">> functionality, this script may not scale well if more inconsistencies in formatting are introduced. Therefore, I
<o:p></o:p></p>
<p class="MsoPlainText">> believe it would be a good idea to enforce a more consistent formatting of ptest results to assist in the use of
<o:p></o:p></p>
<p class="MsoPlainText">> ptest for automated testing. Are there any plans to further consolidate the ptest result format such that it is
<o:p></o:p></p>
<p class="MsoPlainText">> more accessible for automated testing?<o:p></o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText">I hope I have answered at least partially to your questions above.<o:p></o:p></p>
<p class="MsoPlainText">Kind regards,<o:p></o:p></p>
<p class="MsoPlainText"> Tudor.<o:p></o:p></p>
</div>
</body>
</html>