[OE-core] [PATCH] libevent: add granularity to ptest

[Trevor Gamblin]

On 9/6/19 10:26 AM, Richard Purdie wrote:

> On Fri, 2019-09-06 at 10:04 -0400, Trevor Gamblin wrote:
>> From: Trevor Gamblin <trevor.gamblin at windriver.com>
>>
>> The libevent ptest used to report only a global pass or a fail
>> result.
>> Count individual PASS, FAIL, SKIP results. The SKIP results now
>> include tests that are disabled in the libevent code.
>>
>> Signed-off-by: Trevor Gamblin <trevor.gamblin at windriver.com>
>> ---
>>   .../libevent/libevent/run-ptest               | 34 ++++++++++++-----
>> --
>>   1 file changed, 22 insertions(+), 12 deletions(-)
>>
>> diff --git a/meta/recipes-support/libevent/libevent/run-ptest
>> b/meta/recipes-support/libevent/libevent/run-ptest
>> index 0241851c70..b7d945246f 100644
>> --- a/meta/recipes-support/libevent/libevent/run-ptest
>> +++ b/meta/recipes-support/libevent/libevent/run-ptest
>> @@ -1,18 +1,28 @@
>>   #!/bin/sh
>>   
>> -fail=0
>> +# run-ptest - 'ptest' test infrastructure shell script that
>> +#   wraps the libevent test scripts
>> +#
>> +# Trevor Gamblin <trevor.gamblin at windriver.com>
>> +###############################################################
>> +LIBEVENTLIB=/usr/lib/libevent
>> +LOG="${LIBEVENTLIB}/ptest/libevent_ptest_$(date +%Y%m%d-%H%M%S).log"
>> +
>> +cd ${LIBEVENTLIB}/ptest
>> +
>>   for test in ./test/*
>>   do
>> -	$test
>> -	if [ $? -ne 0 ]
>> -	then
>> -		fail=1
>> -	fi
>> +    $test 2>&1|tee -a ${LOG}
>>   done
>>   
>> -if [ $fail -eq 0 ]
>> -then
>> -	echo "PASS: libevent"
>> -else
>> -	echo "FAIL: libevent"
>> -fi
>> +passed=`grep OK ${LOG}|wc -l`
>> +failed=`grep FAILED ${LOG}|wc -l`
>> +skipped=`grep -E 'DISABLED|SKIPPED' ${LOG}|wc -l`
>> +all=$((passed + failed + skipped))
>> +
>> +(   echo "=== Test Summary ==="
>> +    echo "TOTAL: ${all}"
>> +    echo "PASSED: ${passed}"
>> +    echo "FAILED: ${failed}"
>> +    echo "SKIPPED: ${skipped}"
>> +) | tee -a ${LOG}
> Does this work correctly such that ptest-runner picks up the individual
> test pass/fail results? I thought the tests would need individual
> pass/fail lines for that to work?
>
> Cheers,
>
> Richard
>
>
They do have individual pass/fail results in the ptest-runner output, 
but those don't match up with the pass/fail usually provided by ptest, 
so the log is parsed for OK/FAILED/SKIPPED/DISABLED and translated. 
Example output:

|write_cb: write 12
write_cb: write -1
=== Test Summary ===
TOTAL: 316
PASSED: 300
FAILED: 0
SKIPPED: 16
DURATION: 87
END: /usr/lib/libevent/ptest
2019-09-06T14:50
STOP: ptest-runner|

Compare with the last log from autobuilder 
(https://autobuilder.yocto.io/pub/non-release/20190829-10/testresults/qemux86-64-ptest/libevent.log):

|write_cb: write 12||
||write_cb: write -1||
||FAIL: libevent||
||DURATION: 116|

||I'm working on a v2 to address multilib - I'll add the example output 
to the commit as well.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openembedded.org/pipermail/openembedded-core/attachments/20190906/420e28a8/attachment.html>