[OE-core] [PATCH 0/5] refactor the archive*.bbcalss
Martin Jansa
martin.jansa at gmail.com
Fri Jan 31 15:57:40 UTC 2014
On Wed, Jan 22, 2014 at 06:13:37PM +0800, Robert Yang wrote:
>
> On 01/22/2014 10:59 AM, Robert Yang wrote:
> >
> >
> > On 01/21/2014 08:37 PM, Martin Jansa wrote:
> >> On Tue, Jan 21, 2014 at 06:41:04PM +0800, Robert Yang wrote:
> >>>
> >>> Hi Martin,
> >>>
>
> I've talked with Martin, the difference is that he tried it on dylan,
> this is only for master, I'd like to backport if for dora, we may also
> backport it for dylan, but this may need more discussions.
Have you tried multi-machine build as suggested? In master I see the
same behavior, these tasks are reexecuted after MACHINE switch causing
not only rebuild of everything (probably caused by combination with
rm_work), but also warnings like this:
NOTE: recipe gcc-cross-4.8.2-r0: task do_deploy_archives: Started
WARNING: The recipe gcc-cross is trying to install files into a shared
area when those files already exist. Those files and their manifest
location are:
/OE/build/oe-core/tmp-eglibc/deploy/sources/i586-oe-linux/gcc-cross-4.8.2-r0/gcc-cross-4.8.2-r0-log.tar.gz
Matched in manifest-x86_64_qemux86-gcc-cross.deploy_archives
/OE/build/oe-core/tmp-eglibc/deploy/sources/i586-oe-linux/gcc-cross-4.8.2-r0/gcc-cross-4.8.2-r0-showdata.dump
Matched in manifest-x86_64_qemux86-gcc-cross.deploy_archives
/OE/build/oe-core/tmp-eglibc/deploy/sources/i586-oe-linux/gcc-cross-4.8.2-r0/gcc-cross-4.8.2-r0-recipe.tar.gz
Matched in manifest-x86_64_qemux86-gcc-cross.deploy_archives
Please verify which package should provide the above files.
NOTE: recipe gcc-cross-4.8.2-r0: task do_deploy_archives: Succeeded
and in some cases enabling archiver (or changing its configuration)
causes not only archiver-related tasks to be executed (do_dumpdata,
do_ar_recipe, do_ar_log, do_deploy_archives) but triggers also another
do_patch task in already patched ${S} (probably because of changed
prefunc signature), causing do_patch errors.
(I haven't found simple reproducer, maybe its related
to cache as well, because repeating the failing setup in the same tmpdir
doesn't work - but also doesn't work when I delete cache and
tmp-eglibc/cache between steps).
Interestingly bitbake -S doesn't show different signatures when compared
with:
oe-core/scripts/sstate-diff-machines.sh --tmpdir=tmp-eglibc --targets="libffi" --machines="qemux86 qemux86copy"
you can cherry-pick qemux86copy test MACHINE from:
http://git.openembedded.org/openembedded-core-contrib/log/?h=jansa/tune2-test
I'm testing it on libffi recipe, because it has extra patch in SRC_URI
and exists also in -native version which is even faster to build.
OE qemux86@ ~/build/oe-core $ bitbake libffi | tee log.qemux86.prepatch
NOTE: Started PRServer with DBfile: /OE/build/oe-core/cache/prserv.sqlite3, IP: 127.0.0.1, PORT: 47743, PID: 8217
Parsing recipes...WARNING: Unable to get checksum for cogl-1.0 SRC_URI entry kms-include-stddef.h-before-drm.h.patch: file could not be found
done.
Parsing of 847 .bb files complete (0 cached, 847 parsed). 1203 targets, 35 skipped, 0 masked, 0 errors.
NOTE: Resolving any missing task queue dependencies
Build Configuration:
BB_VERSION = "1.21.1"
BUILD_SYS = "x86_64-linux"
NATIVELSBSTRING = "Gentoo-2.2"
TARGET_SYS = "i586-oe-linux"
MACHINE = "qemux86"
DISTRO = "nodistro"
DISTRO_VERSION = "nodistro.0"
TUNE_FEATURES = "m32 i586"
TARGET_FPU = ""
meta = "jansa/test:9d7f81bc7257b9e5f902daa11ce63c84046256d2"
NOTE: Preparing runqueue
NOTE: Executing SetScene Tasks
NOTE: Running setscene task 121 of 128 (/OE/build/oe-core/openembedded-core/meta/recipes-gnome/libffi/libffi_3.0.13.bb, do_populate_lic_setscene)
NOTE: Running setscene task 122 of 128 (/OE/build/oe-core/openembedded-core/meta/recipes-devtools/libtool/libtool-cross_2.4.2.bb, do_populate_lic_setscene)
NOTE: recipe libffi-3.0.13-r0: task do_populate_lic_setscene: Started
NOTE: recipe libtool-cross-2.4.2-r6.1: task do_populate_lic_setscene: Started
....
NOTE: Running task 725 of 728 (ID: 17, /OE/build/oe-core/openembedded-core/meta/recipes-gnome/libffi/libffi_3.0.13.bb, do_deploy_archives)
NOTE: recipe libffi-3.0.13-r0: task do_deploy_archives: Started
NOTE: recipe libffi-3.0.13-r0: task do_deploy_archives: Succeeded
NOTE: Running noexec task 726 of 728 (ID: 11, /OE/build/oe-core/openembedded-core/meta/recipes-gnome/libffi/libffi_3.0.13.bb, do_build)
NOTE: Running task 727 of 728 (ID: 12, /OE/build/oe-core/openembedded-core/meta/recipes-gnome/libffi/libffi_3.0.13.bb, do_rm_work)
NOTE: recipe libffi-3.0.13-r0: task do_rm_work: Started
NOTE: recipe libffi-3.0.13-r0: task do_rm_work: Succeeded
NOTE: Running task 728 of 728 (ID: 13, /OE/build/oe-core/openembedded-core/meta/recipes-gnome/libffi/libffi_3.0.13.bb, do_rm_work_all)
NOTE: recipe libffi-3.0.13-r0: task do_rm_work_all: Started
NOTE: recipe libffi-3.0.13-r0: task do_rm_work_all: Succeeded
NOTE: Tasks Summary: Attempted 728 tasks of which 74 didn't need to be rerun and all succeeded.
NOTE: Writing buildhistory
Summary: There was 1 WARNING message shown.
OE qemux86@ ~/build/oe-core $ export MACHINE=qemux86copy
OE qemux86copy@ ~/build/oe-core $ bitbake libffi | tee log.qemux86.prepatch2
NOTE: Started PRServer with DBfile: /OE/build/oe-core/cache/prserv.sqlite3, IP: 127.0.0.1, PORT: 55093, PID: 7136
Parsing recipes...WARNING: Unable to get checksum for cogl-1.0 SRC_URI entry kms-include-stddef.h-before-drm.h.patch: file could not be found
done.
Parsing of 847 .bb files complete (0 cached, 847 parsed). 1203 targets, 35 skipped, 0 masked, 0 errors.
NOTE: Resolving any missing task queue dependencies
Build Configuration:
BB_VERSION = "1.21.1"
BUILD_SYS = "x86_64-linux"
NATIVELSBSTRING = "Gentoo-2.2"
TARGET_SYS = "i586-oe-linux"
MACHINE = "qemux86copy"
DISTRO = "nodistro"
DISTRO_VERSION = "nodistro.0"
TUNE_FEATURES = "m32 i586"
TARGET_FPU = ""
meta = "jansa/test:9d7f81bc7257b9e5f902daa11ce63c84046256d2"
NOTE: Preparing runqueue
NOTE: Executing SetScene Tasks
NOTE: Running setscene task 99 of 128 (/OE/build/oe-core/openembedded-core/meta/recipes-gnome/libffi/libffi_3.0.13.bb, do_populate_sysroot_setscene)
NOTE: Running setscene task 100 of 128 (/OE/build/oe-core/openembedded-core/meta/recipes-gnome/libffi/libffi_3.0.13.bb, do_package_write_ipk_setscene)
NOTE: recipe libffi-3.0.13-r0: task do_populate_sysroot_setscene: Started
NOTE: recipe libffi-3.0.13-r0: task do_package_write_ipk_setscene: Started
NOTE: recipe libffi-3.0.13-r0: task do_populate_sysroot_setscene: Succeeded
NOTE: Running setscene task 101 of 128 (/OE/build/oe-core/openembedded-core/meta/recipes-devtools/libtool/libtool-cross_2.4.2.bb, do_package_write_ipk_setscene)
...
NOTE: Running task 725 of 728 (ID: 17, /OE/build/oe-core/openembedded-core/meta/recipes-gnome/libffi/libffi_3.0.13.bb, do_deploy_archives)
NOTE: recipe libffi-3.0.13-r0: task do_deploy_archives: Started
NOTE: recipe libffi-3.0.13-r0: task do_deploy_archives: Succeeded
NOTE: Running noexec task 726 of 728 (ID: 11, /OE/build/oe-core/openembedded-core/meta/recipes-gnome/libffi/libffi_3.0.13.bb, do_build)
NOTE: Running task 727 of 728 (ID: 12, /OE/build/oe-core/openembedded-core/meta/recipes-gnome/libffi/libffi_3.0.13.bb, do_rm_work)
NOTE: recipe libffi-3.0.13-r0: task do_rm_work: Started
NOTE: recipe libffi-3.0.13-r0: task do_rm_work: Succeeded
NOTE: Running task 728 of 728 (ID: 13, /OE/build/oe-core/openembedded-core/meta/recipes-gnome/libffi/libffi_3.0.13.bb, do_rm_work_all)
NOTE: recipe libffi-3.0.13-r0: task do_rm_work_all: Started
NOTE: recipe libffi-3.0.13-r0: task do_rm_work_all: Succeeded
NOTE: Tasks Summary: Attempted 728 tasks of which 110 didn't need to be rerun and all succeeded.
NOTE: Writing buildhistory
Summary: There were 4 WARNING messages shown.
Normally this should reuse everything from sstate and build only kernel and very few MACHINE_ARCH recipes.
conf/local.conf:
INHERIT += "rm_work"
INHERIT += "archiver"
ARCHIVER_MODE[src] = "prepatch"
ARCHIVER_MODE[diff] = "1"
ARCHIVER_MODE[diff-exclude] ?= ".pc autom4te.cache patches no yes"
ARCHIVER_MODE[dumpdata] = "1"
ARCHIVER_MODE[recipe] = "1"
ARCHIVER_MODE[log] = "1"
> >>> I've tested the archiver.bbclass with rm_work.bbclasss again, it works for me,
> >>> it seems that you only downloaded the archiver.bbclass was not enough, would
> >>> you please try this PULL? It is based on oe-core and on oe-core-contrib now:-)
> >>> And if the error occurs, would you please show the configuration and error
> >>> logs ?
> >>
> >> I've tried to enable "everything" to test it properly, so my config
> >> looks like this (BTW: it would be nice to show in local.conf.sample
> >> what are the default values):
> >>
> >> # Create source archive for complience with the licenses
> >> # Enable all to properly test newer archiver backported from oe-core/master
> >> INHERIT_DISTRO += "archiver"
> >
> > I changed it to INHERIT += "archiver" since the INHERIT_DISTRO will override
> > this:
> >
> > meta/conf/distro/defaultsetup.conf:INHERIT_DISTRO ?= "debian devshell sstate
> > license"
> >
> > Everything works fine to me, I'm afraid that set INHERIT_DISTRO += "archiver"
> > may override other INHERIT_DISTRO's settings.
> >
> > I also added INHERIT += "rm_work"
> >
> > And please see my comments below.
> >
> >> ARCHIVER_MODE[src] = "prepatch"
> >> ARCHIVER_MODE[diff] = "1"
> >> ARCHIVER_MODE[diff-exclude] ?= ".pc autom4te.cache patches"
> >> ARCHIVER_MODE[dumpdata] = "1"
> >> ARCHIVER_MODE[recipe] = "1"
> >> ARCHIVER_MODE[log] = "1"
> >>
> >> And error in do_patch:
> >>
> >> NOTE: recipe perf-1.0-r8: task do_patch: Started
> >> ERROR: Error executing a python function in
> >> /OE/openembedded-core/meta/recipes-kernel/perf/perf.bb:
> >> OSError: [Errno 2] No such file or directory:
> >> '/OE/tmp-eglibc/sysroots/qemux86/usr/src'
> >
> > The perf's ${S} is sysroots/qemux86/usr/src/kernel, but the problem is that why
> > the /OE/tmp-eglibc/sysroots/qemux86/usr/src doesn't exist ? The rm_work doesn't
> > remove it, no matter how many times I tried, it still works.
> >
> > So, please:
> >
> > 1) Check whether your other INHERIT_DISTRO settings have been overrided or not,
> > and try to use INHERIT rather than INHERIT_DISTRO ?
> >
> > 2) If it still fails, would you please provide more info ? For example, how to
> > reproduce it, exactly ?
> >
> > 3) It would be great if we can talk oneline, how about gtalk ? I will try to
> > reach you there.
> >
> > 4) It would be great if others can help me to test it.
> >
> > // Robert
> >
> >>
> >> ERROR: The stack trace of python calls that resulted in this exception/failure
> >> was:
> >> ERROR: File "do_ar_prepatch", line 6, in <module>
> >> ERROR:
> >> ERROR: File "do_ar_prepatch", line 3, in do_ar_prepatch
> >> ERROR:
> >> ERROR: File "archiver.bbclass", line 17, in create_tarball
> >> ERROR:
> >> ERROR: The code that was being executed was:
> >> ERROR: 0002:def do_ar_prepatch(d):
> >> ERROR: 0003: create_tarball(d, d.getVar('S', True), 'prepatch')
> >> ERROR: 0004:
> >> ERROR: 0005:
> >> ERROR: *** 0006:do_ar_prepatch(d)
> >> ERROR: 0007:
> >> ERROR: [From file: 'do_ar_prepatch', lineno: 6, function: <module>]
> >> ERROR: 0001:
> >> ERROR: 0002:def do_ar_prepatch(d):
> >> ERROR: *** 0003: create_tarball(d, d.getVar('S', True), 'prepatch')
> >> ERROR: 0004:
> >> ERROR: 0005:
> >> ERROR: 0006:do_ar_prepatch(d)
> >> ERROR: 0007:
> >> ERROR: [From file: 'do_ar_prepatch', lineno: 3, function: do_ar_prepatch]
> >> ERROR: 0013: srcdir = srcdir.rstrip('/')
> >> ERROR: 0014: dirname = os.path.dirname(srcdir)
> >> ERROR: 0015: basename = os.path.basename(srcdir)
> >> ERROR: 0016: olddir = os.getcwd()
> >> ERROR: *** 0017: os.chdir(dirname)
> >> ERROR: 0018: bb.note("Creating %s" % tarname)
> >> ERROR: 0019: if (len(os.listdir(basename))) != 0:
> >> ERROR: 0020: tar = tarfile.open(tarname, "w:gz")
> >> ERROR: 0021: tar.add(basename)
> >> ERROR: [From file: 'archiver.bbclass', lineno: 17, function: create_tarball]
> >> ERROR: Function failed: do_ar_prepatch
> >> ERROR: Logfile of failure stored in:
> >> /OE/tmp-eglibc/work/i586-oe-linux/perf/1.0-r8/temp/log.do_patch.28364
> >> NOTE: recipe perf-1.0-r8: task do_patch: Failed
> >> ERROR: Task 20159 (/OE/openembedded-core/meta/recipes-kernel/perf/perf.bb,
> >> do_patch) failed with exit code '1'
> >>
> >>> git://git.openembedded.org/openembedded-core-contrib rbt/archiver
> >>>
> >>> // Robert
> >>>
> >>> On 01/18/2014 08:26 PM, Martin Jansa wrote:
> >>>> On Thu, Jan 16, 2014 at 05:33:20PM +0800, Robert Yang wrote:
> >>>>>
> >>>>>
> >>>>> On 01/16/2014 05:25 PM, Martin Jansa wrote:
> >>>>>> On Thu, Jan 16, 2014 at 10:14:30AM +0800, Robert Yang wrote:
> >>>>>>>
> >>>>>>>
> >>>>>>> On 01/16/2014 12:55 AM, Martin Jansa wrote:
> >>>>>>>> On Tue, Jan 14, 2014 at 12:20:14AM +0800, Robert Yang wrote:
> >>>>>>>>> * The archive*.bbclass didn't work, and there were a few problems, for
> >>>>>>>>> example:
> >>>>>>>>> 1) There were a few duplicated code
> >>>>>>>>> 2) There was no src_dir.org (or orig), but the diff command still
> >>>>>>>>> use
> >>>>>>>>> it, and it is not easy to fix this issue if we don't change a lot
> >>>>>>>>> of the code.
> >>>>>>>>> 3) It didn't archive the source for the native or gcc
> >>>>>>>>> 4) The work flow is not very well
> >>>>>>>>> 5) The "subprocess.call('fakeroot cp xxxx'" should be removed
> >>>>>>>>> 6) And others ...
> >>>>>>>>>
> >>>>>>>>> * So that we have to refactor it, the benefits are:
> >>>>>>>>> 1) Fix the problems and make it work well.
> >>>>>>>>> 2) Reduce more than 400 lines in total.
> >>>>>>>>> 3) Make it easy to use.
> >>>>>>>>>
> >>>>>>>>> // Robert
> >>>>>>>>>
> >>>>>>>>> The following changes since commit
> >>>>>>>>> 1b636173ca88e5ccca1992f9a12367a1189fa674:
> >>>>>>>>>
> >>>>>>>>> bitbake: toaster: Toaster GUI, generic search, filter and order
> >>>>>>>>> (2014-01-10 15:20:26 +0000)
> >>>>>>>>>
> >>>>>>>>> are available in the git repository at:
> >>>>>>>>>
> >>>>>>>>> git://git.pokylinux.org/poky-contrib rbt/archiver
> >>>>>>>>> http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=rbt/archiver
> >>>>>>>>
> >>>>>>>> Can you send pull-request for oe-core?
> >>>>>>>>
> >>>>>>>
> >>>>>>> Here it is:
> >>>>>>> git://git.pokylinux.org/poky-contrib rbt/ar_oe
> >>>>>>> http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/log/?h=rbt/ar_oe
> >>>>>>
> >>>>>> This still looks like poky-contrib (so it will download whole
> >>>>>> poky-contrib when I add it as remote to oe-core repo).
> >>>>>>
> >>>>>
> >>>>> Sorry, I can't create repo on oe-contrib, seems that I lost my key.
> >>>>
> >>>> That's sad, you should send new one to khem to replace it, before
> >>>> someone else finds your old key.
> >>>>
> >>>>> How about:
> >>>>>
> >>>>> git fetch git://git.pokylinux.org/poky-contrib rbt/ar_oe
> >>>>
> >>>> That still fetches whole poky repo I'm not interested in.
> >>>>
> >>>> So I've downloaded just archiver.bbclass manually and tested it on my
> >>>> use-case, I can confirm that it doesn't remove required file anymore,
> >>>> but it seems that there is something terribly broken when used in
> >>>> combination with rm_work or multimachine builds.
> >>>>
> >>>> I haven't debugged it yet, but 2nd build (in the same directory) with
> >>>> this new archiver included almost every recipe failed in do_patch task.
> >>>>
> >>>>>>> // Robert
> >>>>>>>
> >>>>>>>>> Robert Yang (5):
> >>>>>>>>> classes/archive*.bbclass: remove archive-*-source.bbclass
> >>>>>>>>> archiver.bbclass: refactor it
> >>>>>>>>> package_rpm.bbclass: archive the source to srpm package
> >>>>>>>>> archiver.bbclass: move a few code to copyleft_compliance.bbclass
> >>>>>>>>> local.conf.sample.extended: update for the archiver
> >>>>>>>>>
> >>>>>>>>> meta-yocto/conf/local.conf.sample.extended | 72 +--
> >>>>>>>>> meta/classes/archive-configured-source.bbclass | 65 ---
> >>>>>>>>> meta/classes/archive-original-source.bbclass | 65 ---
> >>>>>>>>> meta/classes/archive-patched-source.bbclass | 65 ---
> >>>>>>>>> meta/classes/archiver.bbclass | 723
> >>>>>>>>> ++++++++-----------------
> >>>>>>>>> meta/classes/copyleft_compliance.bbclass | 55 +-
> >>>>>>>>> meta/classes/package_rpm.bbclass | 31 +-
> >>>>>>>>> 7 files changed, 316 insertions(+), 760 deletions(-)
> >>>>>>>>> delete mode 100644 meta/classes/archive-configured-source.bbclass
> >>>>>>>>> delete mode 100644 meta/classes/archive-original-source.bbclass
> >>>>>>>>> delete mode 100644 meta/classes/archive-patched-source.bbclass
> >>>>>>>>>
> >>>>>>>>> --
> >>>>>>>>> 1.8.3.1
> >>>>>>>>>
> >>>>>>>>> _______________________________________________
> >>>>>>>>> Openembedded-core mailing list
> >>>>>>>>> Openembedded-core at lists.openembedded.org
> >>>>>>>>> http://lists.openembedded.org/mailman/listinfo/openembedded-core
> >>>>>>>>
> >>>>>>
> >>>>
> >>
> > _______________________________________________
> > Openembedded-core mailing list
> > Openembedded-core at lists.openembedded.org
> > http://lists.openembedded.org/mailman/listinfo/openembedded-core
> >
> >
--
Martin 'JaMa' Jansa jabber: Martin.Jansa at gmail.com
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 198 bytes
Desc: Digital signature
URL: <http://lists.openembedded.org/pipermail/openembedded-core/attachments/20140131/36afe43e/attachment-0002.sig>
More information about the Openembedded-core
mailing list