[oe] Eliminating dependency race-conditions (was Re: [PATCH] net-snmp: disable libnl use)

[Graham Gower]

On 03/15/2011 07:38 PM, Esben Haabendal wrote:
> On Fri, 2011-03-11 at 12:04 +0100, Steffen Sledz wrote:
>> which occurred sometimes depending on build order (not in clean
>> package only builds).
> 
> I would like to raise awareness of the underlying problem here.
> 
> The current dependency/staging model of OE basically has this feature
> that a build can be influenced not only by it's own dependencies, but
> also what has been build before it (or not).
> 
> I strongly believe that this has to be fixed on the architectural level,
> and not just on a case-by-case level as is currently needed.
> 
> I haven't received much feedback on the preivous posting about the
> per-recipe staging principle implemented in OE-lite, but I decided to
> take this opportunity to re-iterate the fact that the OE-lite
> implementation of staging and build dependencies eliminates this
> problem.
> 
> I am still very much interested in discussing how to move this
> technology from OE-lite to OE, but as it impacts all recipe metadata
> (build dependencies has to be redefined), OE community at a large really
> needs to value the benefits of solving this problem.
> 
> Best regards,
> Esben
> 
> 
> _______________________________________________
> Openembedded-devel mailing list
> Openembedded-devel at lists.openembedded.org
> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-devel


I did some work last year to fix some of the low hanging fruit (on a
case by case basis).
My methodology was to do two runs of bitbake -k world, then compare
the output. In case anyone is interested, I used the script below to
identify the problematic recipes. Last time I ran it was in November,
by which time the output had reduced to a list of 5-10 items most of
the time.


#!/bin/sh
#
# Quick and dirty script to determine which tasks fail in one world.log,
# where they succeeded in another. world.log should be generated with, e.g.
# $ bitbake -k world | tee world.log
#
# If there are too many failures, this script will take too long as grep -f
# is slooow.

if [ $# != 2 ]; then
	echo "usage: $0 world.log.1 world.log.2"
	exit 1
fi

tasks="do_setscene do_fetch do_distribute_sources do_unpack do_prepsources
	do_patch do_configure do_qa_configure do_compile do_install
	do_populate_sysroot do_package do_package_write_ipk
	do_package_write do_package_stage do_package_stage_all do_build
	do_qa_staging"

tmp1=`mktemp /tmp/world_regress.XXXXXX`
tmp2=`mktemp /tmp/world_regress.XXXXXX`

for task in $tasks; do
	grep "task $task: Failed" $2 \
		| sed -e 's/^.*NOTE: package //' \
			-e 's/task \(.*\): Failed/\1/' \
		> $tmp1

	grep "task $task: Succeeded" $1 \
		| sed -e 's/^.*NOTE: package //' \
			-e 's/task \(.*\): Succeeded/\1/' \
		> $tmp2

	grep -f $tmp1 $tmp2 \
		| sort -n
done

rm -f $tmp1 $tmp2