[OE-core] [RFC] Report build

[João Henrique Ferreira de Freitas]

Hi,

I would like to share an implementation that I've used with
great success to solve the following situation:

We have a farm server that Jenkins CI has the control. All the
nights a bunch of devel, snapshot and stable builds are fired.
Each machine receives a job that the aim is to build image, SDK,
packages and wic images. After that, all the artifacts are collected
and send to a consolidation server which my users could download.

I believed that many YP users has the very similar settings. So I wanted
to create a solution which is integrated with YP.

This solution involves three things:

* new class to collect the artifacts
* script to send the collected artifacts
* web interface to summaries the results

So I implemented a class called report-build. Once properly setup, it
will collect the artifacts writing in a json file. The meaning of
artifact is: images, packages, sdks, manifest, deploy kernel and bootloader.
After that, all artifact is collected using another script called 
send-build-report. This scripts has
the responsibility to control and send the artifacts (until now using rsync)
to consolidation server.

The consolidation server is a machine running an instance of 
build-report-web that
receives the json payload, creates all the database entry and filesystem 
places where
the artifacts will be placed.

On Jenkins I've created Freestyle jobs that  is setup scripts to build 
the artifacts and
configures the variables that build-report class use:

BUILD_REPORT_DISTRO ?= "${DISTRO}"
BUILD_REPORT_DISTRO_VERSION ?= "${DISTRO_VERSION}"

BUILD_REPORT_JOB_RETENTION ?= "1"
BUILD_REPORT_JOB_ID ?= "job-${DATE}"
BUILD_REPORT_DIR ?= "${LOG_DIR}/build-report"
BUILD_REPORT_SERVER ?= "http://127.0.0.1:8000/Builds"
BUILD_REPORT_USER ?= ""
BUILD_REPORT_EMAIL ?= ""
BUILD_REPORT_TYPE ?= "snapshot"
BUILD_REPORT_DEPLOY_DIR = "${DEPLOY_DIR}"

When the Jenkins runs the job, it only executes the script and 
report-build is in charge to
sends the generated artifacts.

So, the users goes to build-report-web tool and inspects and downloads 
the results.

Here is the very basic web interface:

A home page showing all builds: http://imgur.com/LTqRewq

A detail page to get files (image, sdk, packages): http://imgur.com/gNOtoLO

There are some drawbacks like: the solution uses rsync over ssh to send 
files and
the user must setup ssh keys between build machines and consolidation 
server. But
this is done only once, to setup the initial environment.

This work was based on ideas from:
- report-error
- Error reporting tool

Currently this implementation works using daisy and master. On my 
production environment
I need to use daisy but I am doing the same using master on my free time.


Implementation links:

http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/log/?h=joaohf/master-build-report-tool
http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/log/?h=joaohf/build-report-web

Thanks