Please note that User Registration has been temporarily disabled due to a recent increase in automated registrations. If anyone needs an account, please request one here: RequestAccount. Thanks for your patience!

Difference between revisions of "Testing:TestBuilder"

From Openembedded.org
Jump to: navigation, search
(Config file)
m (How to setup and use)
 
(6 intermediate revisions by one user not shown)
Line 1: Line 1:
 
= TestBuilder =
 
= TestBuilder =
 
Here is script collections I using for doing OpenEmbedded test-builds.
 
Here is script collections I using for doing OpenEmbedded test-builds.
You can try to get archive here: http://jay-home.ath.cx/oe/testbuilder-0.1.tar.gz
+
It is on OE tree now (contrib/testing/testbuilder). You can view it online here: http://cgit.openembedded.org/cgit.cgi/openembedded/tree/contrib/testing/testbuilder
  
 +
== How to setup and use ==
  
== Config file ==
+
I assume below that you wish to install TestBuilder into `~/testbuilder` directory.
<pre>
+
# Testbuilder config
+
# For first time setup look into end of file
+
# for "Misc dirs" and "Setup options"
+
  
# You can prefix any word in LISTS with '!' do disable it
+
1. Copy the files from OE `contrib/testing/testbuilder` directory to any place you wish to use (e.g. `~/testbuilder`) and go to that directory.
  
## DISTROS: Distributions to build
+
2. Run the `setup-testbuilder` script with an OE branch and BitBake branch you wish to use. E.g.:
DISTROS="angstrom_2008 minimal"
+
  ./setup-testbuilder "testing-next" "1.12"
# OE name of distribution (if needed)
+
DISTRO_NAME_angstrom_2008="angstrom-2008.1"
+
  
## BUILD_SETS: build-sets, should have unique names
+
It will clone the OE and BitBake trees and prepare the file `build/conf/local.conf` for you. You can review `local.conf` afterward and add some settings there.
# Build all machines in one TMPDIR
+
BUILD_SETS_minimal="minimal"
+
# Catch-all variable (will catch angstrom distro)
+
BUILD_SETS="armv4 armv5 armv7 mips32 i686"
+
  
## MACHINES: sets to build in one TMPDIR
+
3. Create `testbuilder.conf` according to your needs. You can use `testbuilder.conf.sample` as reference.
# sets for angstrom distro
+
MACHINES_armv4="collie"
+
MACHINES_armv5="tosa akita qemuarm"
+
MACHINES_armv7="efikamx"
+
MACHINES_mips32="ben-nanonote"
+
MACHINES_i686="qemux86"
+
  
# Catch-all (will catch 'minimal' group)
+
4. Create your build configs in `testbuilder.d` directory. You can use provided sample config files as reference.
MACHINES="qemux86 qemuarm"
+
  
## Or you can build every machine for angstrom-2008.1 in separate TMPDIRs
+
5. Run `./testbuilder` and wait for the build results. Check the `logs` subdir for log files (`tail logs/B.*` is useful)
#BUILD_SETS_angstrom_2008=""
+
#MACHINES="tosa collie akita !efikamx !ben-nanonote !qemux86 !qemuarm"
+
  
## IMAGES: what images to build.
+
You can run a single configured build (e.g. "testing").
# Build console-image for 'minimal' group machines
+
IMAGES_minimal="console-image"
+
# Build console-image for set MACHINE_armv7
+
IMAGES_armv7="console-image x11-image"
+
# Catch-all (will catch all other machines)
+
IMAGES="console-image x11-image opie-image"
+
  
## STAGES: What to do with images:
+
  ./testbuilder -B ./testbuilder.d/testing.conf
# update_tree - update git tree before builds
+
# build_clean - do clean builds
+
# build_incremental - do incremental builds
+
STAGES="update_tree build_clean build_incremental"
+
  
## Other options
+
Or you can use TestBuilder to build a specific distro/machine/image just like bitbake (e.g. angstrom-2008.1/qemuarm/console-image).
  
# Number of parallel bitbake threads
+
  ./testbuilder -D angstrom-2008.1 -M qemuarm console-image
BB_NUMBER_THREADS=4
+
# Number of parallel make threads inside one bitbake thread (-j)
+
MAKE_NUMBER_THREADS=2
+
  
# Name for oestats-client
+
If you wish to only setup an environment and run BitBake by hand you can use the option `-S`.
TB_BUILDER="username-testing-next"
+
  
# Misc dirs
+
  ./testbuilder -D angstrom-2008.1 -M qemuarm -S
TB_BASE=`pwd`
+
BB_DIR="${TB_BASE}/bitbake"
+
OE_DIR="${TB_BASE}/openembedded"
+
DL_DIR="${TB_BASE}/sources"
+
BLD_DIR="${TB_BASE}/build"
+
  
# testbuilder log file
+
Look ./testbuilder -h for full list of possible options.
TB_LOG="${TB_BASE}/tb.log"
+
 
+
## Setup options
+
 
+
# Which bitbake version to use
+
# use branch
+
BB_BRANCH="1.10" # 1.8, 1.10, master
+
# or use specified release tag
+
#BB_TAG="1.10.1" # 1.8.18
+
 
+
# Which OE branch to checkout
+
OE_BRANCH="testing-next"
+
</pre>
+
 
+
== TestBuilder setup script ==
+
<pre>
+
#!/bin/sh
+
 
+
# TestBuilder setup script for OpenEmbedded
+
# Copyright (c) 2010 Yuri Bushmelev <jay4mail@gmail.com>
+
#
+
# This program is free software; you can redistribute it and/or modify
+
# it under the terms of the GNU General Public License as published by
+
# the Free Software Foundation; either version 2, or (at your option)
+
# any later version.
+
#
+
# This program is distributed in the hope that it will be useful,
+
# but WITHOUT ANY WARRANTY; without even the implied warranty of
+
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+
# GNU General Public License for more details.
+
 
+
TB_DIR=`dirname $0`
+
CFG_FILE="${TB_DIR}/testbuilder.conf"
+
 
+
. "${CFG_FILE}"
+
 
+
cd "${TB_BASE}"
+
 
+
## Checkout bitbake
+
echo "= Fetching bitbake"
+
git clone -n git://git.openembedded.org/bitbake "${BB_DIR}"
+
cd "${BB_DIR}"
+
 
+
if [ -n "${BB_BRANCH}" ]; then
+
git checkout -b "${BB_BRANCH}" "origin/${BB_BRANCH}"
+
elif [ -n "${BB_TAG}" ]; then
+
git checkout -b "${BB_TAG}" "${BB_TAG}"
+
else
+
echo "You should specify BB_BRANCH or BB_TAG in ${CFG_FILE}"
+
fi
+
 
+
echo "= Fetching OE"
+
cd "${TB_BASE}"
+
 
+
git clone -n git://git.openembedded.org/openembedded "${OE_DIR}"
+
cd "${OE_DIR}"
+
 
+
if [ -n "${OE_BRANCH}" ]; then
+
git checkout -b "${OE_BRANCH}" "origin/${OE_BRANCH}"
+
else
+
echo "You should specify OE_BRANCH in ${CFG_FILE}"
+
fi
+
 
+
cd "${TB_BASE}"
+
mkdir -p "${BLD_DIR}/conf"
+
 
+
echo "= Creating bitbake config"
+
cat >> "${BLD_DIR}/conf/local.conf" << "EOF"
+
# Testbuilder bitbake local configuration file
+
 
+
DISTRO ?= "${@bb.fatal('Set DISTRO in your config')}"
+
MACHINE ?= "${@bb.fatal('Set MACHINE in your config')}"
+
 
+
DL_DIR ?= "${@bb.fatal('Set DL_DIR in your config')}"
+
TMPDIR = "${TMP_DIR}"
+
 
+
BBFILES ?= "${OE_DIR}/recipes/*/*.bb"
+
 
+
# ENABLE_BINARY_LOCALE_GENERATION = "0"
+
# GLIBC_GENERATE_LOCALES = "en_US.UTF-8 en_GB.UTF-8 de_DE.UTF-8 ru_RU.UTF-8"
+
# IMAGE_LINGUAS="en-us en-gb ru-ru"
+
 
+
# jffs2, tar(.gz|bz2), cpio(.gz), cramfs, ext2(.gz), ext3(.gz)
+
# squashfs, squashfs-lzma
+
IMAGE_FSTYPES = "jffs2 tar.gz"
+
 
+
# CCACHE = "${@bb.which(bb.data.getVar('PATH', d, 1), 'ccache') and 'ccache '}"
+
PARALLEL_MAKE = "-j ${MAKE_NUMBER_THREADS}"
+
BB_NUMBER_THREADS ?= "${@bb.fatal('Set BB_NUMBER_THREADS in your config')}"
+
 
+
INHERIT += "rm_work"
+
#INHERIT += "devshell"
+
#TERMCMD = ${SCREEN_TERMCMD}
+
 
+
BBINCLUDELOGS = "yes"
+
 
+
# OE stats client (make troubleshooting easier)
+
INHERIT += "oestats-client"
+
OESTATS_BUILDER = "${TB_BUILDER}"
+
OESTATS_SERVER = "tinderbox.openembedded.net"
+
 
+
EOF
+
 
+
echo "= All is done. Run ${TB_DIR}/testbuilder now"
+
</pre>
+
 
+
 
+
== TestBuilder itself ==
+
<pre>
+
#!/bin/sh
+
 
+
# TestBuilder for OpenEmbedded
+
# Copyright (c) 2010 Yuri Bushmelev <jay4mail@gmail.com>
+
#
+
# This program is free software; you can redistribute it and/or modify
+
# it under the terms of the GNU General Public License as published by
+
# the Free Software Foundation; either version 2, or (at your option)
+
# any later version.
+
#
+
# This program is distributed in the hope that it will be useful,
+
# but WITHOUT ANY WARRANTY; without even the implied warranty of
+
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+
# GNU General Public License for more details.
+
 
+
 
+
TB_DIR=`dirname $0`
+
cd "${TB_DIR}"
+
 
+
CFG_FILE="${TB_DIR}/testbuilder.conf"
+
. "${CFG_FILE}"
+
 
+
# Sanity cleanup
+
ORIG_PATH=${PATH}
+
ORIG_LD_LIBRARY_PATH=${LD_LIBRARY_PATH}
+
ORIG_LANG=${LANG}
+
export ORIG_PATH ORIG_LD_LIBRARY_PATH ORIG_LANG
+
 
+
# clean LD_LIBRARY_PATH
+
unset LD_LIBRARY_PATH
+
 
+
# clean locale
+
#LANG=C
+
 
+
# include bitbake into PATH
+
PATH=${BB_DIR}/bin:${PATH}
+
 
+
BBPATH="${BLD_DIR}:${OE_DIR}"
+
 
+
#export LD_LIBRARY_PATH LANG PATH
+
export LD_LIBRARY_PATH LANG
+
 
+
BB_ENV_EXTRAWHITE="MACHINE DISTRO TB_BUILDER OE_DIR TMP_DIR DL_DIR MAKE_NUMBER_THREADS BB_NUMBER_THREADS"
+
export  BB_ENV_EXTRAWHITE BBPATH TB_BUILDER OE_DIR TMP_DIR DL_DIR MAKE_NUMBER_THREADS BB_NUMBER_THREADS
+
 
+
 
+
### Functions
+
 
+
# log message
+
tb_log() {
+
local _dt=`date "+%F %X (%s)"`
+
echo "[${_dt}] $@" >&2
+
echo "[${_dt}] $@" >> $TB_LOG
+
}
+
 
+
# Do build of image-set $bbimages for machine-set $MLIST in $TMP_DIR
+
# tb_build_machines "${TMP_DIR}" "$MLIST" $bbimages
+
tb_build_machines() {
+
local _TMP_DIR=$1; shift
+
local _MLIST=$1; shift
+
local _bbimages=$@
+
local _machine
+
local _rc
+
+
for _machine in $_MLIST; do
+
[ "${_machine}" != "${_machine#!}" ] && continue
+
 
+
tb_log "${_machine} build started, images: ${_bbimages}"
+
 
+
export MACHINE=$_machine
+
export TMP_DIR=${_TMP_DIR}
+
bitbake $_bbimages
+
#echo $_bbimages
+
_rc=$?
+
tb_log "${_machine} build finished. Exit code:${_rc}."
+
done
+
}
+
 
+
 
+
# update git tree
+
tb_update_tree() {
+
local _branch=`git --git-dir=openembedded/.git branch | awk '/^*/{print $2;}'`
+
 
+
# check current branch
+
if [ "$_branch" != "testing-next" ]; then
+
tb_log "Current branch (${_branch}) is not 'testing-next'!"
+
#return 1
+
fi
+
 
+
# pull
+
tb_log "Updating git tree ${OE_DIR}"
+
git --git-dir="${OE_DIR}/.git" pull
+
}
+
 
+
### Main code
+
 
+
#mv -f $TB_LOG "${TB_LOG}.old" 2>/dev/null
+
 
+
DO_BUILD_CLEAN=''
+
DO_BUILD_INCREMENTAL=''
+
for stage in $STAGES; do
+
[ "${stage}" != "${stage#!}" ] && continue
+
case ${stage} in
+
'build_clean')
+
DO_BUILD_CLEAN=y
+
;;
+
'build_incremental')
+
DO_BUILD_INCREMENTAL=y
+
;;
+
'update_tree')
+
tb_update_tree
+
;;
+
*)
+
tb_log "Unknown stage ${stage}."
+
;;
+
esac
+
done
+
 
+
for distro in $DISTROS; do
+
[ "${distro}" != "${distro#!}" ] && continue
+
 
+
# Get real distro name
+
eval DISTRO="\$DISTRO_NAME_${distro}"
+
[ -z "${DISTRO}" ] && DISTRO=${distro}
+
export DISTRO
+
 
+
# Get groups list
+
eval BSLIST="\$BUILD_SETS_${distro}"
+
[ -z "${BSLIST}" ] && BSLIST=${BUILD_SETS}
+
 
+
# Empty $BUILD_SETS is exception - we should use $MACHINES
+
# to build every machine in separate TMPDIRs
+
if [ -z "${BSLIST}" ]; then
+
# Build every machine as separate group
+
for machine in $MACHINES; do
+
[ "${machine}" != "${machine#!}" ] && continue
+
eval "MACHINE_${machine}=${machine}"
+
BSLIST="${BSLIST} ${machine}"
+
done
+
fi
+
 
+
tb_log "= Processing distro ${DISTRO}, groups: ${BSLIST}"
+
 
+
for group in $BSLIST; do
+
[ "${group}" != "${group#!}" ] && continue
+
 
+
eval MLIST="\$MACHINES_${group}"
+
[ -z "${MLIST}" ] && MLIST=${MACHINES}
+
 
+
tb_log "== Processing group ${group}, machines: ${MLIST}"
+
 
+
CLN_TMP_DIR="${BLD_DIR}/tmp"
+
INC_TMP_DIR="${BLD_DIR}/${distro}/${group}"
+
 
+
eval ILIST="\$IMAGES_${group}"
+
[ -z "${ILIST}" ] && ILIST=${IMAGES}
+
 
+
# collect images to build in $bbimages
+
bbimages=''
+
for image in $ILIST; do
+
[ "${image}" != "${image#!}" ] && continue
+
bbimages="${bbimages} ${image}"
+
done
+
 
+
# If there is no previous incremental builddir and we are requested
+
# to do both incremental and clean builds
+
# then skip clean build and do only incremental because it will be clean build then :)
+
if [ "${DO_BUILD_INCREMENTAL}" = 'y' \
+
-a "${DO_BUILD_CLEAN}" = 'y' \
+
-a ! -d "${INC_TMP_DIR}/work" \
+
]; then
+
tb_log "Will do incremental build instead of clean to populate TMPDIR"
+
DO_BUILD_CLEAN='skip-once'
+
fi
+
 
+
## Do clean builds
+
case "${DO_BUILD_CLEAN}" in
+
'y')
+
mkdir -p "${CLN_TMP_DIR}"
+
+
tb_log "== Cleaning ${CLN_TMP_DIR}"
+
rm -rf ${CLN_TMP_DIR}/*
+
+
tb_log "== Starting clean builds for machine-set {$MLIST}"
+
tb_build_machines "${CLN_TMP_DIR}" "$MLIST" $bbimages
+
 
+
tb_log "== Cleaning ${CLN_TMP_DIR} again"
+
rm -rf ${CLN_TMP_DIR}/*
+
;;
+
'skip-once')
+
# Re-enable skipped clean build
+
DO_BUILD_CLEAN=y
+
;;
+
esac
+
 
+
## Do incremental build
+
case "${DO_BUILD_INCREMENTAL}" in
+
'y')
+
# Switch tmpdir to archive
+
mkdir -p "${INC_TMP_DIR}"
+
 
+
tb_log "== Starting incremental builds for machine-set {$MLIST}"
+
tb_build_machines "${INC_TMP_DIR}" "$MLIST" $bbimages
+
;;
+
esac
+
done
+
done
+
tb_log "All done."
+
</pre>
+

Latest revision as of 15:06, 5 March 2011

[edit] TestBuilder

Here is script collections I using for doing OpenEmbedded test-builds. It is on OE tree now (contrib/testing/testbuilder). You can view it online here: http://cgit.openembedded.org/cgit.cgi/openembedded/tree/contrib/testing/testbuilder

[edit] How to setup and use

I assume below that you wish to install TestBuilder into `~/testbuilder` directory.

1. Copy the files from OE `contrib/testing/testbuilder` directory to any place you wish to use (e.g. `~/testbuilder`) and go to that directory.

2. Run the `setup-testbuilder` script with an OE branch and BitBake branch you wish to use. E.g.:

 ./setup-testbuilder "testing-next" "1.12"

It will clone the OE and BitBake trees and prepare the file `build/conf/local.conf` for you. You can review `local.conf` afterward and add some settings there.

3. Create `testbuilder.conf` according to your needs. You can use `testbuilder.conf.sample` as reference.

4. Create your build configs in `testbuilder.d` directory. You can use provided sample config files as reference.

5. Run `./testbuilder` and wait for the build results. Check the `logs` subdir for log files (`tail logs/B.*` is useful)

You can run a single configured build (e.g. "testing").

 ./testbuilder -B ./testbuilder.d/testing.conf

Or you can use TestBuilder to build a specific distro/machine/image just like bitbake (e.g. angstrom-2008.1/qemuarm/console-image).

 ./testbuilder -D angstrom-2008.1 -M qemuarm console-image

If you wish to only setup an environment and run BitBake by hand you can use the option `-S`.

 ./testbuilder -D angstrom-2008.1 -M qemuarm -S

Look ./testbuilder -h for full list of possible options.

Personal tools
Namespaces

Variants
Actions
Navigation
Categories
OE services
Toolbox