[OE-core] [PATCH v3 0/3] ptest-runner 2.0

[Tudor Florea]

Hi Alimon
See inline.
Tudor.

On 06/01/2016 18:07, Aníbal Limón wrote:
> Hi again Tudor,
> 
> Comments below,
> 
> alimon
> 
> 
> 
> On 01/06/2016 09:46 AM, Aníbal Limón wrote:
>> Hi Todor,
>> 
>> Comments below, alimon
>> 
>> 
>> On 01/05/2016 06:13 PM, Tudor Florea wrote:
>>> Hi Anibal, Please see my comments inline. Regards, Tudor.
>>> 
>>> On 04/01/2016 20:29, Aníbal Limón wrote:
>>>> The whole set is sent to two ML for reviewing purposes the
>>>> first two patches are for oe-core and the last one for poky.
>>>> 
>>>> The first two commits (oe-core) adds a new ptest-runner
>>>> written in python the sh one is conserved to use in tiny
>>>> systems.
>>>> 
>>>> What's new in ptest runner:
>>>> 
>>>> - Monitor/timeout stdout, stderr of the test suite to avoid
>>>> block indefinetly.
>>> This is definitely something useful.
>>>> - Add option for change ptest root directory. - Add option
>>>> for list available tests. - Add option for only run certain
>>>> tests.
>>>> 
>>>> The last commit (meta-yocto) sets ptest-runner to 1.0 in
>>>> poky-tiny systems due to python dependency.
>>> I think adding python dependency is a shift in a wrong
>>> direction for for ptest-runner. The assumption that most
>>> embedded devices (tiny or not) have python might not be
>>> correct. The alternative of using the old version of
>>> ptest-runner only complicate things. I do think this kind of
>>> work is really useful but this should be done into a testing
>>> framework that runs outside of the DUT.
>> 
>> The original problem is the current sh ptest-runner blocks
>> indefinitely when a package ptest blocks (we don't control this),
>> now python-ptest is blocking the ptest-runner, see [1]. The way
>> to solve this is adding timeout in some place.
>> 
>> I know that bash support timeout option but this option works
>> over the whole process making us to define timeouts too big or
>> per machine/device also tiny systems mainly don't use bash.
>> That's the reason for i decide to write python version of
>> ptest-runner that applies a timeout over the output of the
>> program give us better control/granularity over the running
>> ptest.
>> 
>> 
>> If you now other way to solve the problem please tell us.
We may have to use our own version of timeout as we cannot rely on
bash either. A good starting point is here:
http://www.pixelbeat.org/scripts/timeout

I'm thinking this is a good opportunity to improve ptest-runner to run
the package tests in parallel. That is, we could spawn a shell per
each run-ptest (wrapper) that have the output redirected to its own
file  (e.g. /var/log/ptest/{package}/output) and then we only timeout
monitoring the output file.
In future we may eventually mimic the logic of automake parallel test
harness
(https://www.gnu.org/software/automake/manual/html_node/Parallel-Test-Harness.html)

> 
> I was thinking and one possible solution for this will be implement
> in C the ptest-runner with the same features of python version, any
> comment?
That would be great!
> 
> 
>> 
>> 
>> [1] https://bugzilla.yoctoproject.org/show_bug.cgi?id=8021
>> 
>> 
>>>> 
>>>> The changes are available in the git repository at:
>>>> 
>>>> git://git.yoctoproject.org/poky-contrib alimon/ptest-runner 
>>>> http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=alimon/ptest-runner
>>>>
>>>>
>>>> 
Aníbal Limón (3): ptest-runner: Add version 2.0 re-implementation
>>>> in python. ptest-runner: Add a recipe for install
>>>> ptest-runner 2.0. conf/distro/poky-tiny.conf: Add default
>>>> install of ptest-runner 1.0
>>>> 
>>>> meta-yocto/conf/distro/poky-tiny.conf              |   4 + 
>>>> .../ptest-runner/files/ptest-runner_2.0.py         | 162 
>>>> +++++++++++++++++++++ .../ptest-runner/ptest-runner_2.0.bb |
>>>> 27 ++++ 3 files changed, 193 insertions(+) create mode
>>>> 100755 
>>>> meta/recipes-support/ptest-runner/files/ptest-runner_2.0.py
>>>> create mode 100644
>>>> meta/recipes-support/ptest-runner/ptest-runner_2.0.bb
>>>>