Difference between revisions of "Testing:TestBuilder"

From Openembedded.org
Jump to: navigation, search
m (TestBuilder)
m (How to setup and use)
 
(8 intermediate revisions by the same user not shown)
Line 1: Line 1:
 
= TestBuilder =
 
= TestBuilder =
 
Here is script collections I using for doing OpenEmbedded test-builds.
 
Here is script collections I using for doing OpenEmbedded test-builds.
 +
It is on OE tree now (contrib/testing/testbuilder). You can view it online here: http://cgit.openembedded.org/cgit.cgi/openembedded/tree/contrib/testing/testbuilder
  
 +
== How to setup and use ==
  
== Config file ==
+
I assume below that you wish to install TestBuilder into `~/testbuilder` directory.
<pre>
 
# Testbuilder config
 
# For first time setup look into end of file
 
# for "Misc dirs" and "Setup options"
 
  
# You can prefix any word in LISTS with '!' do disable it
+
1. Copy the files from OE `contrib/testing/testbuilder` directory to any place you wish to use (e.g. `~/testbuilder`) and go to that directory.
  
## DISTROS: Distributions to build
+
2. Run the `setup-testbuilder` script with an OE branch and BitBake branch you wish to use. E.g.:
DISTROS="angstrom_2008 minimal"
+
  ./setup-testbuilder "testing-next" "1.12"
# OE name of distribution (if needed)
 
DISTRO_NAME_angstrom_2008="angstrom-2008.1"
 
  
## BUILD_SETS: build-sets, should have unique names
+
It will clone the OE and BitBake trees and prepare the file `build/conf/local.conf` for you. You can review `local.conf` afterward and add some settings there.
# Build all machines in one TMPDIR
 
BUILD_SETS_minimal="minimal"
 
# Catch-all variable (will catch angstrom distro)
 
BUILD_SETS="armv4 armv5 armv7 mips32 i686"
 
  
## MACHINES: sets to build in one TMPDIR
+
3. Create `testbuilder.conf` according to your needs. You can use `testbuilder.conf.sample` as reference.
# sets for angstrom distro
 
MACHINES_armv4="collie"
 
MACHINES_armv5="tosa akita qemuarm"
 
MACHINES_armv7="efikamx"
 
MACHINES_mips32="ben-nanonote"
 
MACHINES_i686="qemux86"
 
  
# Catch-all (will catch 'minimal' group)
+
4. Create your build configs in `testbuilder.d` directory. You can use provided sample config files as reference.
MACHINES="qemux86 qemuarm"
 
  
## Or you can build every machine for angstrom-2008.1 in separate TMPDIRs
+
5. Run `./testbuilder` and wait for the build results. Check the `logs` subdir for log files (`tail logs/B.*` is useful)
#BUILD_SETS_angstrom_2008=""
 
#MACHINES="tosa collie akita !efikamx !ben-nanonote !qemux86 !qemuarm"
 
  
## IMAGES: what images to build.
+
You can run a single configured build (e.g. "testing").
# Build console-image for 'minimal' group machines
 
IMAGES_minimal="console-image"
 
# Build console-image for set MACHINE_armv7
 
IMAGES_armv7="console-image x11-image"
 
# Catch-all (will catch all other machines)
 
IMAGES="console-image x11-image opie-image"
 
  
## STAGES: What to do with images:
+
  ./testbuilder -B ./testbuilder.d/testing.conf
# update_tree - update git tree before builds
 
# build_clean - do clean builds
 
# build_incremental - do incremental builds
 
STAGES="update_tree build_clean build_incremental"
 
  
## Other options
+
Or you can use TestBuilder to build a specific distro/machine/image just like bitbake (e.g. angstrom-2008.1/qemuarm/console-image).
  
# Number of parallel bitbake threads
+
  ./testbuilder -D angstrom-2008.1 -M qemuarm console-image
BB_NUMBER_THREADS=4
 
# Number of parallel make threads inside one bitbake thread (-j)
 
MAKE_NUMBER_THREADS=2
 
  
# Name for oestats-client
+
If you wish to only setup an environment and run BitBake by hand you can use the option `-S`.
TB_BUILDER="Jay7-tb"
 
  
# Misc dirs
+
  ./testbuilder -D angstrom-2008.1 -M qemuarm -S
TB_BASE=`pwd`
 
BB_DIR="${TB_BASE}/bitbake"
 
OE_DIR="${TB_BASE}/openembedded"
 
DL_DIR="${TB_BASE}/sources"
 
BLD_DIR="${TB_BASE}/build"
 
  
# testbuilder log file
+
Look ./testbuilder -h for full list of possible options.
TB_LOG="${TB_BASE}/tb.log"
 
 
 
## Setup options
 
 
 
# Which bitbake version to use
 
# use branch
 
BB_BRANCH="1.10" # 1.8, 1.10, master
 
# or use specified release tag
 
#BB_TAG="1.10.1" # 1.8.18
 
 
 
# Which OE branch to checkout
 
OE_BRANCH="testing-next"
 
</pre>
 
 
 
 
 
== TestBuilder setup script ==
 
<pre>
 
#!/bin/sh
 
 
 
# TestBuilder setup script for OpenEmbedded
 
# Copyright (c) 2010 Yuri Bushmelev <jay4mail@gmail.com>
 
#
 
# This program is free software; you can redistribute it and/or modify
 
# it under the terms of the GNU General Public License as published by
 
# the Free Software Foundation; either version 2, or (at your option)
 
# any later version.
 
#
 
# This program is distributed in the hope that it will be useful,
 
# but WITHOUT ANY WARRANTY; without even the implied warranty of
 
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
 
# GNU General Public License for more details.
 
 
 
TB_DIR=`dirname $0`
 
CFG_FILE="${TB_DIR}/testbuilder.conf"
 
 
 
. "${CFG_FILE}"
 
 
 
cd "${TB_BASE}"
 
 
 
## Checkout bitbake
 
echo "= Fetching bitbake"
 
git clone -n git://git.openembedded.org/bitbake "${BB_DIR}"
 
cd "${BB_DIR}"
 
 
 
if [ -n "${BB_BRANCH}" ]; then
 
git checkout -b "${BB_BRANCH}" "origin/${BB_BRANCH}"
 
elif [ -n "${BB_TAG}" ]; then
 
git checkout -b "${BB_TAG}" "${BB_TAG}"
 
else
 
echo "You should specify BB_BRANCH or BB_TAG in ${CFG_FILE}"
 
fi
 
 
 
echo "= Fetching OE"
 
cd "${TB_BASE}"
 
 
 
git clone -n git://git.openembedded.org/openembedded "${OE_DIR}"
 
cd "${OE_DIR}"
 
 
 
if [ -n "${OE_BRANCH}" ]; then
 
git checkout -b "${OE_BRANCH}" "origin/${OE_BRANCH}"
 
else
 
echo "You should specify OE_BRANCH in ${CFG_FILE}"
 
fi
 
 
 
cd "${TB_BASE}"
 
mkdir -p "${BLD_DIR}/conf"
 
 
 
echo "= Creating bitbake config"
 
cat >> "${BLD_DIR}/conf/local.conf" << "EOF"
 
# Testbuilder bitbake local configuration file
 
 
 
DISTRO ?= "${@bb.fatal('Set DISTRO in your config')}"
 
MACHINE ?= "${@bb.fatal('Set MACHINE in your config')}"
 
 
 
DL_DIR ?= "${@bb.fatal('Set DL_DIR in your config')}"
 
TMPDIR = "${TMP_DIR}"
 
 
 
BBFILES ?= "${OE_DIR}/recipes/*/*.bb"
 
 
 
# ENABLE_BINARY_LOCALE_GENERATION = "0"
 
# GLIBC_GENERATE_LOCALES = "en_US.UTF-8 en_GB.UTF-8 de_DE.UTF-8 ru_RU.UTF-8"
 
# IMAGE_LINGUAS="en-us en-gb ru-ru"
 
 
 
# jffs2, tar(.gz|bz2), cpio(.gz), cramfs, ext2(.gz), ext3(.gz)
 
# squashfs, squashfs-lzma
 
IMAGE_FSTYPES = "jffs2 tar.gz"
 
 
 
# CCACHE = "${@bb.which(bb.data.getVar('PATH', d, 1), 'ccache') and 'ccache '}"
 
PARALLEL_MAKE = "-j ${MAKE_NUMBER_THREADS}"
 
BB_NUMBER_THREADS ?= "${@bb.fatal('Set BB_NUMBER_THREADS in your config')}"
 
 
 
INHERIT += "rm_work"
 
#INHERIT += "devshell"
 
#TERMCMD = ${SCREEN_TERMCMD}
 
 
 
BBINCLUDELOGS = "yes"
 
 
 
# OE stats client (make troubleshooting easier)
 
INHERIT += "oestats-client"
 
OESTATS_BUILDER = "${TB_BUILDER}"
 
OESTATS_SERVER = "tinderbox.openembedded.net"
 
 
 
EOF
 
 
 
echo "= All is done. Run ${TB_DIR}/testbuilder now"
 
</pre>
 
 
 
 
 
== TestBuilder itself ==
 
<pre>
 
#!/bin/sh
 
 
 
# TestBuilder for OpenEmbedded
 
# Copyright (c) 2010 Yuri Bushmelev <jay4mail@gmail.com>
 
#
 
# This program is free software; you can redistribute it and/or modify
 
# it under the terms of the GNU General Public License as published by
 
# the Free Software Foundation; either version 2, or (at your option)
 
# any later version.
 
#
 
# This program is distributed in the hope that it will be useful,
 
# but WITHOUT ANY WARRANTY; without even the implied warranty of
 
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
 
# GNU General Public License for more details.
 
 
 
 
 
TB_DIR=`dirname $0`
 
cd "${TB_DIR}"
 
 
 
CFG_FILE="${TB_DIR}/testbuilder.conf"
 
. "${CFG_FILE}"
 
 
 
# Sanity cleanup
 
ORIG_PATH=${PATH}
 
ORIG_LD_LIBRARY_PATH=${LD_LIBRARY_PATH}
 
ORIG_LANG=${LANG}
 
export ORIG_PATH ORIG_LD_LIBRARY_PATH ORIG_LANG
 
 
 
# clean LD_LIBRARY_PATH
 
unset LD_LIBRARY_PATH
 
 
 
# clean locale
 
#LANG=C
 
 
 
# include bitbake into PATH
 
PATH=${BB_DIR}/bin:${PATH}
 
 
 
BBPATH="${BLD_DIR}:${OE_DIR}"
 
 
 
#export LD_LIBRARY_PATH LANG PATH
 
export LD_LIBRARY_PATH LANG
 
 
 
BB_ENV_EXTRAWHITE="MACHINE DISTRO TB_BUILDER OE_DIR TMP_DIR DL_DIR MAKE_NUMBER_THREADS BB_NUMBER_THREADS"
 
export  BB_ENV_EXTRAWHITE BBPATH TB_BUILDER OE_DIR TMP_DIR DL_DIR MAKE_NUMBER_THREADS BB_NUMBER_THREADS
 
 
 
 
 
### Functions
 
 
 
# log message
 
tb_log() {
 
local _dt=`date "+%F %X (%s)"`
 
echo "[${_dt}] $@" >&2
 
echo "[${_dt}] $@" >> $TB_LOG
 
}
 
 
 
# Do build of image-set $bbimages for machine-set $MLIST in $TMP_DIR
 
# tb_build_machines "${TMP_DIR}" "$MLIST" $bbimages
 
tb_build_machines() {
 
local _TMP_DIR=$1; shift
 
local _MLIST=$1; shift
 
local _bbimages=$@
 
local _machine
 
local _rc
 
 
for _machine in $_MLIST; do
 
[ "${_machine}" != "${_machine#!}" ] && continue
 
 
 
tb_log "${_machine} build started, images: ${_bbimages}"
 
 
 
export MACHINE=$_machine
 
export TMP_DIR=${_TMP_DIR}
 
bitbake $_bbimages
 
#echo $_bbimages
 
_rc=$?
 
tb_log "${_machine} build finished. Exit code:${_rc}."
 
done
 
}
 
 
 
 
 
# update git tree
 
tb_update_tree() {
 
local _branch=`git --git-dir=openembedded/.git branch | awk '/^*/{print $2;}'`
 
 
 
# check current branch
 
if [ "$_branch" != "testing-next" ]; then
 
tb_log "Current branch (${_branch}) is not 'testing-next'!"
 
#return 1
 
fi
 
 
 
# pull
 
tb_log "Updating git tree ${OE_DIR}"
 
git --git-dir="${OE_DIR}/.git" pull
 
}
 
 
 
### Main code
 
 
 
#mv -f $TB_LOG "${TB_LOG}.old" 2>/dev/null
 
 
 
DO_BUILD_CLEAN=''
 
DO_BUILD_INCREMENTAL=''
 
for stage in $STAGES; do
 
[ "${stage}" != "${stage#!}" ] && continue
 
case ${stage} in
 
'build_clean')
 
DO_BUILD_CLEAN=y
 
;;
 
'build_incremental')
 
DO_BUILD_INCREMENTAL=y
 
;;
 
'update_tree')
 
tb_update_tree
 
;;
 
*)
 
tb_log "Unknown stage ${stage}."
 
;;
 
esac
 
done
 
 
 
for distro in $DISTROS; do
 
[ "${distro}" != "${distro#!}" ] && continue
 
 
 
# Get real distro name
 
eval DISTRO="\$DISTRO_NAME_${distro}"
 
[ -z "${DISTRO}" ] && DISTRO=${distro}
 
export DISTRO
 
 
 
# Get groups list
 
eval BSLIST="\$BUILD_SETS_${distro}"
 
[ -z "${BSLIST}" ] && BSLIST=${BUILD_SETS}
 
 
 
# Empty $BUILD_SETS is exception - we should use $MACHINES
 
# to build every machine in separate TMPDIRs
 
if [ -z "${BSLIST}" ]; then
 
# Build every machine as separate group
 
for machine in $MACHINES; do
 
[ "${machine}" != "${machine#!}" ] && continue
 
eval "MACHINE_${machine}=${machine}"
 
BSLIST="${BSLIST} ${machine}"
 
done
 
fi
 
 
 
tb_log "= Processing distro ${DISTRO}, groups: ${BSLIST}"
 
 
 
for group in $BSLIST; do
 
[ "${group}" != "${group#!}" ] && continue
 
 
 
eval MLIST="\$MACHINES_${group}"
 
[ -z "${MLIST}" ] && MLIST=${MACHINES}
 
 
 
tb_log "== Processing group ${group}, machines: ${MLIST}"
 
 
 
CLN_TMP_DIR="${BLD_DIR}/tmp"
 
INC_TMP_DIR="${BLD_DIR}/${distro}/${group}"
 
 
 
eval ILIST="\$IMAGES_${group}"
 
[ -z "${ILIST}" ] && ILIST=${IMAGES}
 
 
 
# collect images to build in $bbimages
 
bbimages=''
 
for image in $ILIST; do
 
[ "${image}" != "${image#!}" ] && continue
 
bbimages="${bbimages} ${image}"
 
done
 
 
 
# If there is no previous incremental builddir and we are requested
 
# to do both incremental and clean builds
 
# then skip clean build and do only incremental because it will be clean build then :)
 
if [ "${DO_BUILD_INCREMENTAL}" = 'y' \
 
-a "${DO_BUILD_CLEAN}" = 'y' \
 
-a ! -d "${INC_TMP_DIR}/work" \
 
]; then
 
tb_log "Will do incremental build instead of clean to populate TMPDIR"
 
DO_BUILD_CLEAN='skip-once'
 
fi
 
 
 
## Do clean builds
 
case "${DO_BUILD_CLEAN}" in
 
'y')
 
mkdir -p "${CLN_TMP_DIR}"
 
 
tb_log "== Cleaning ${CLN_TMP_DIR}"
 
rm -rf ${CLN_TMP_DIR}/*
 
 
tb_log "== Starting clean builds for machine-set {$MLIST}"
 
tb_build_machines "${CLN_TMP_DIR}" "$MLIST" $bbimages
 
 
 
tb_log "== Cleaning ${CLN_TMP_DIR} again"
 
rm -rf ${CLN_TMP_DIR}/*
 
;;
 
'skip-once')
 
# Re-enable skipped clean build
 
DO_BUILD_CLEAN=y
 
;;
 
esac
 
 
 
## Do incremental build
 
case "${DO_BUILD_INCREMENTAL}" in
 
'y')
 
# Switch tmpdir to archive
 
mkdir -p "${INC_TMP_DIR}"
 
 
 
tb_log "== Starting incremental builds for machine-set {$MLIST}"
 
tb_build_machines "${INC_TMP_DIR}" "$MLIST" $bbimages
 
;;
 
esac
 
done
 
done
 
tb_log "All done."
 
</pre>
 

Latest revision as of 15:06, 5 March 2011

TestBuilder

Here is script collections I using for doing OpenEmbedded test-builds. It is on OE tree now (contrib/testing/testbuilder). You can view it online here: http://cgit.openembedded.org/cgit.cgi/openembedded/tree/contrib/testing/testbuilder

How to setup and use

I assume below that you wish to install TestBuilder into `~/testbuilder` directory.

1. Copy the files from OE `contrib/testing/testbuilder` directory to any place you wish to use (e.g. `~/testbuilder`) and go to that directory.

2. Run the `setup-testbuilder` script with an OE branch and BitBake branch you wish to use. E.g.:

 ./setup-testbuilder "testing-next" "1.12"

It will clone the OE and BitBake trees and prepare the file `build/conf/local.conf` for you. You can review `local.conf` afterward and add some settings there.

3. Create `testbuilder.conf` according to your needs. You can use `testbuilder.conf.sample` as reference.

4. Create your build configs in `testbuilder.d` directory. You can use provided sample config files as reference.

5. Run `./testbuilder` and wait for the build results. Check the `logs` subdir for log files (`tail logs/B.*` is useful)

You can run a single configured build (e.g. "testing").

 ./testbuilder -B ./testbuilder.d/testing.conf

Or you can use TestBuilder to build a specific distro/machine/image just like bitbake (e.g. angstrom-2008.1/qemuarm/console-image).

 ./testbuilder -D angstrom-2008.1 -M qemuarm console-image

If you wish to only setup an environment and run BitBake by hand you can use the option `-S`.

 ./testbuilder -D angstrom-2008.1 -M qemuarm -S

Look ./testbuilder -h for full list of possible options.