[OE-core] Automated testing on real hardware

[Nicolas Dechesne]

Paul,


On Fri, Nov 29, 2013 at 4:58 PM, Paul Eggleton <
paul.eggleton at linux.intel.com> wrote:

> LAVA
> -----
>
> A number of people had suggested I look at LAVA [1]. It is split into
> different
> components for each function, some of which should be usable independently
> so
> you don't have to use the entire framework. Many of the components could be
> useful assuming they are independent, but in terms of being able to
> actually
> run our tests, the one that stands out as being most relevant is lava-
> dispatcher. This is the component that deploys images, and boots the
> hardware
> in order to run the tests.
>
> I've looked at lava-dispatcher a couple of times; firstly at the code, and
> yesterday I looked at actually running it. Initially it looked very
> promising
> - reasonably licensed, written in Python, has some rudimentary support for
> OE
> images. However while looking at it I found the following:
>
> * It requires root to run, since it mounts the images temporarily and
> modifies
> the contents. We've done quite a lot to avoid having to run as root in OE,
> so
> this is a hard sell.
>

i have forwarded this email to the relevant people in Linaro working on
LAVA. I hope to be able to bring more information about that, as I am not
involved with LAVA, just a 'user' of it. By 'root' here, you mean on the
server side, e.g. the LAVA instance, which will typically not be the
'builder', or do you want to couple build and test on the same 'server'?
LAVA jobs are generally submitted by a CI instance (Jenkins in our case).


> * It expects images to be created by linaro-media-create, which in turn
> requires an "hwpack" [2]. The hwpack concept seems primarily geared to
> Ubuntu
> and distributions like it where you'd have a generic base image upon which
> you'd install some board-specific packages in order to make it work on the
> board. OE doesn't work that way - we just build images specific to the
> board,
> which makes this mechanism completely superfluous for our usage; but the
> tool
> appears to require it.
>

that is no longer the case. we do have support for 'native' OE images, and
we've had that for a while. That is what I am using for my project at
Linaro. None of our projects/jobs deliverable are 'public' at the moment,
so you won't find them, but i can confirm that what we deploy for our jobs
is the output of OE (e.g. what is in the deploy/image folder). hwpack was
something mostly meant to make it simpler to work with Ubuntu and Android
root FS, and is not suited for OE.


>
> * There is a general lack of abstraction and far too much hardcoding. For
> example, even at the most abstract level, the "Target" class (which is the
> base class for defining what to do for different types of targets) has
> hardcoded
> deploy_linaro / deploy_android functions:
>
>
> https://git.linaro.org/gitweb?p=lava/lava-dispatcher.git;a=blob;f=lava_dispatcher/device/target.py
>
> * It's not clear to me how well it will work across different host
> distributions that we want to support; the main LAVA installer quits if it
> isn't run on Ubuntu. For convenience I happened to be using an Ubuntu VM,
> and
> I wasn't running the full installer so I avoided this check altogether. It
> just concerns me that such a check would even be there.
>

I agree that this is a problem and we should fix. Our IT infrastructure is
largely based on Ubuntu servers, but that should not affect what we create
that much.


>
> Of course, none of these problems are impossible to overcome, if we're
> prepared to put a significant amount of engineering into resolving them. In
> terms of enabling the Yocto Project QA team to test images on real
> hardware in
> the 1.6 development cycle however, I don't believe this is going to be
> deliverable.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openembedded.org/pipermail/openembedded-core/attachments/20131129/234a0fb4/attachment-0002.html>