[bitbake-devel] Random Bitbake build failures

[Nair, Sandeep (Contractor)]

Hi Christopher,

Thanks for the details. Please find the additional details based on your feedback

>>remote side? bitbake builds have never been supported on nfs.

We mount remote folders and bitbake sees these as a regular folder. Since this can be treated as a regular folder, we may not need nfs support in bitbake.

>> But yes, generally the real task runs if the setscene fails.

We do the real task when sstate cache fetch fails and we are trying to identify the reason why bitbake returns 1 (which happens when error count != 0) even when real task succeeds.  We see that Bitbake returns 1 in this case which causes random build failure even when the real task was performed successfully.

Thanks,
Sandeep


From: kergoth at gmail.com [mailto:kergoth at gmail.com] On Behalf Of Christopher Larson
Sent: Saturday, June 04, 2016 5:52 PM
To: Nair, Sandeep (Contractor) <Sandeep_Nair2 at cable.comcast.com>
Cc: bitbake-devel at lists.openembedded.org
Subject: Re: [bitbake-devel] Random Bitbake build failures


On Sat, Jun 4, 2016 at 11:09 AM, Nair, Sandeep (Contractor) <Sandeep_Nair2 at cable.comcast.com<mailto:Sandeep_Nair2 at cable.comcast.com>> wrote:
We are facing random failures with Bitbake builds. Bitbake is returning an error even when the operation succeeded on the successful retry.

Eg:

ERROR: Error executing a python function in /mnt/jenkins/workspace/PACEXI3V2-Yocto-Build/meta-rdk-comcast/recipes-extended/netzyn/netzyn_git.bb<http://netzyn_git.bb>:
Exception: IOError: [Errno 116] Stale file handle

WARNING: Logfile for failed setscene task is /mnt/jenkins/workspace/PACEXI3V2-Yocto-Build/build-pacexi3v2/tmp/work/mips32el-rdk-linux/netzyn/1.99+gitAUTOINC+c3752ca038_c7068967a1-r0/temp/log.do_package_setscene.8946
WARNING: Setscene task 1277 (/mnt/jenkins/workspace/PACEXI3V2-Yocto-Build/meta-rdk-comcast/recipes-extended/netzyn/netzyn_git.bb<http://netzyn_git.bb>, do_package_setscene) failed with exit code '1' - real task will be run instead

“Setscene” is a bb attempt to use sstate cache.

It says about real task will run instead. Does that mean it tried build locally (it should)?

In all stale cases it fails in this code:

File: ‘…./openembedded-core/bitbake/lib/bb/utils.py', lineno: 467, function: md5_file
     0463:        import md5
     0464:        m = md5.new()
     0465:
     0466:    with open(filename, "rb") as f:
*** 0467:        for line in f:
     0468:            m.update(line)
     0469:    return m.hexdigest()
     0470:

As can be seen it tries use handle it was given but the file is not valid on remote side already. So system lib returns stale handle.

remote side? bitbake builds have never been supported on nfs. But yes, generally the real task runs if the setscene fails.
--
Christopher Larson
clarson at kergoth dot com
Founder - BitBake, OpenEmbedded, OpenZaurus
Maintainer - Tslib
Senior Software Engineer, Mentor Graphics
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openembedded.org/pipermail/bitbake-devel/attachments/20160614/869b404d/attachment-0002.html>