[bitbake-devel] RFC: Override expansion - changing the behaviour
Richard Purdie
richard.purdie at linuxfoundation.org
Tue May 31 23:00:04 UTC 2011
Currently if you do:
OVERRIDES = "z"
DEPENDS_prepend = "a "
DEPENDS = "b"
DEPENDS_z = "c"
d.update_data()
d.getVar("DEPENDS") gives "a c"
d.update_data()
d.getVar("DEPENDS") gives "c"
I'm proposing we should change the bitbake behaviour such that at the
time bitbake expands the DEPENDS_z override, it removes "DEPENDS_z" from
the data store.
In the above example this would mean that it wouldn't matter how often
you call d.update_data(), you'd always get "a c" back.
This is a fairly major change in behaviour so the question is what is
the likely impact to existing systems. To check into that I've devised a
patch which parses the metadata and saves out the datastore as parsed.
If the data is already present it will list the differenced between it
and the existing data. I've then assumed the metadata in OE-Core is
representative and made some tests on it.
The baseline data (ignoring the __base_depends and TIME variables) was:
$ touch conf/local.conf
$ bitbake core-image-sato -g
[...]
$ touch conf/local.conf
$ bitbake core-image-sato -g
NOTE: Out of date cache found, rebuilding...
NOTE: Variable BB_BASEHASH_task-do_bootimg(content) changed from 8abe544bb45386aa3228c53d29965efc to 9918e6d60bba2bead7079c02ff07f460
NOTE: Variable BB_BASEHASH_task-do_bootimg(content) changed from 7ec98c48f134c67b8ad93837dbb89025 to ac42a8ade2b7361ca6049f338323efee
NOTE: Variable BB_BASEHASH_task-do_populate_sdk(content) changed from 3f2c6e5f8ebb90bfe4e0b01d3f5b9bfe to 2a08f65f0dd7933d88e1e353ba0e263c
NOTE: Variable BB_BASEHASH_task-do_bootdirectdisk(content) changed from 906e35362d949ee08d53522cd3c91c86 to ed100e159e205d25848b1a139686d9c7
NOTE: Variable BB_BASEHASH_task-do_bootimg(content) changed from 105e1321d452e9d62616e21ca0382ba4 to 7296313feba540ddb155c9216deef95a
NOTE: Variable BB_BASEHASH_task-do_bootdirectdisk(content) changed from 8a376303f61fb09f7a23378f9a688edc to 0c5c44baf48b3cdf129d7b722f1b9509
NOTE: Variable BB_BASEHASH_task-do_bootimg(content) changed from 0cafcf23b8a6382d4d60febf0824042f to db0f70b813d513b80d8fb5a740d3e6f0
NOTE: Variable BB_BASEHASH_task-do_populate_sdk(content) changed from 06a328a672551fa5a6baf1666005c74d to 6a0a23e0234919b1b55bc633919ee737
NOTE: Variable BB_BASEHASH_task-do_generate_content(content) changed from 53c5fcc503372e0455c6436aaa3603a8 to 9b976b9dcea65ca1d7813f26ff74e641
NOTE: Variable BB_BASEHASH_task-do_populate_sdk(content) changed from 21f3d18c425404d4b9dda29dea79246f to c405f07f9fc4f2c7c8c7f7a623d3760b
NOTE: Variable BB_BASEHASH_task-do_populate_sdk(content) changed from ea9b9e2ec8d4b024bfd2ac2f887d60a0 to ac55c4e9c074a8941150fb112ef81ef5
NOTE: Variable BB_BASEHASH_task-do_bootdirectdisk(content) changed from ee3f87895e9e26b9543f9158c1405bcb to 82ac6687e9ee0393ecee345feaac3574
NOTE: Variable BB_BASEHASH_task-do_bootimg(content) changed from 4777ac9660c0230a72942facf8225d01 to bee704c11cf4d680a9a92a2bfa323533
Parsing of 783 .bb files complete (0 cached, 783 parsed). 1001 targets, 13 skipped, 0 masked, 0 errors.
(I didn't enable filename output on this but its clear its from tasks
with DATE/TIME dependencies so these changes aren't a surprise).
I then applied my change in behaviour to bitbake and reparsed.
For the comparison, I've excluded anything from this list where a
variable changed contents which had the name *virtclass-*. The reasoning
is that we don't have any place in the metadata where we directly
dereference the full named version of these variables. This makes the
list a little more readable:
$ touch conf/local.conf
$ time bitbake core-image-sato -g
NOTE: Out of date cache found, rebuilding...
NOTE: Variable ASNEEDED_pn-rpm(content) changed from to None (/tmp/bbsave/rpm_5.4.0.bb)
NOTE: Variable ASNEEDED_pn-distcc(content) changed from to None (/tmp/bbsave/distcc_2.18.3.bb)
NOTE: Variable DESKTOPINSTALL_libc-glibc(content) changed from install -d ${D}${datadir}/distcc/
install -m 0644 ${WORKDIR}/distccmon-gnome.desktop ${D}${datadir}/distcc/
to None (/tmp/bbsave/distcc_2.18.3.bb)
NOTE: Flag func in variable DESKTOPINSTALL_libc-glibc disappeared (/tmp/bbsave/distcc_2.18.3.bb)
NOTE: Flag func in variable do_install_virtclass-native disappeared (/tmp/bbsave/binutils_2.21.bbnative)
NOTE: Flag func in variable do_install_virtclass-native disappeared (/tmp/bbsave/binutils-crosssdk_2.21.bbnative)
NOTE: Flag func in variable do_install_virtclass-native disappeared (/tmp/bbsave/binutils-cross-canadian_2.21.bbnative)
NOTE: Flag func in variable do_install_virtclass-native disappeared (/tmp/bbsave/binutils-cross_2.21.bbnative)
NOTE: Variable BB_BASEHASH_task-do_buildall(content) changed from 01be347d9d39d35b93982691de7578b5 to e8a39cd6c321c9114ea9ae6853e75cd7 (/tmp/bbsave/intltool_0.40.6.bbnative)
NOTE: Flag export in variable PERL_virtclass-native disappeared (/tmp/bbsave/intltool_0.40.6.bbnative)
NOTE: Variable BB_BASEHASH_task-do_package_write(content) changed from 01be347d9d39d35b93982691de7578b5 to e8a39cd6c321c9114ea9ae6853e75cd7 (/tmp/bbsave/intltool_0.40.6.bbnative)
NOTE: Variable BB_BASEHASH_task-do_package_write_rpm_setscene(content) changed from a4f69a004a31cd3cf529c978342c3423 to 4f995188b8e3ba19ee35f9b455c8cdbc (/tmp/bbsave/intltool_0.40.6.bbnative)
NOTE: Variable BB_BASEHASH_task-do_populate_lic_setscene(content) changed from a4f69a004a31cd3cf529c978342c3423 to 4f995188b8e3ba19ee35f9b455c8cdbc (/tmp/bbsave/intltool_0.40.6.bbnative)
NOTE: Variable BB_BASEHASH_task-do_devshell(content) changed from e03946d56f12d74acd57c1629642621a to 7a1261fcd11991b07c9942f890ae99db (/tmp/bbsave/intltool_0.40.6.bbnative)
NOTE: Variable BB_BASEHASH_task-do_populate_sysroot(content) changed from d605ca766df88f8c9a2e2b46166fe379 to c8fa2aa352640dbe2c28744a87cafbcd (/tmp/bbsave/intltool_0.40.6.bbnative)
NOTE: Variable BB_BASEHASH_task-do_configure(content) changed from aa58885014f2884d2c75da6c8e4db7af to 8c82c55e641d0de452e8e4b2e33bd985 (/tmp/bbsave/intltool_0.40.6.bbnative)
NOTE: Variable BB_BASEHASH_task-do_checkuriall(content) changed from 01be347d9d39d35b93982691de7578b5 to e8a39cd6c321c9114ea9ae6853e75cd7 (/tmp/bbsave/intltool_0.40.6.bbnative)
NOTE: Variable BB_BASEHASH_task-do_compile(content) changed from 7f78bd1073cbba067c5b075f03533c4a to 0848e8774d954485fd9bdbb297a2f3ce (/tmp/bbsave/intltool_0.40.6.bbnative)
NOTE: Variable BB_BASEHASH_task-do_populate_sysroot_setscene(content) changed from a4f69a004a31cd3cf529c978342c3423 to 4f995188b8e3ba19ee35f9b455c8cdbc (/tmp/bbsave/intltool_0.40.6.bbnative)
NOTE: Variable BB_BASEHASH_task-do_fetchall(content) changed from 01be347d9d39d35b93982691de7578b5 to e8a39cd6c321c9114ea9ae6853e75cd7 (/tmp/bbsave/intltool_0.40.6.bbnative)
NOTE: Variable BB_BASEHASH_task-do_install(content) changed from 73199b10f19780e409eaad2589e01e06 to c0652a84492ab99c89004ef042ad5f5b (/tmp/bbsave/intltool_0.40.6.bbnative)
NOTE: Variable BB_BASEHASH_task-do_build(content) changed from 01be347d9d39d35b93982691de7578b5 to e8a39cd6c321c9114ea9ae6853e75cd7 (/tmp/bbsave/intltool_0.40.6.bbnative)
NOTE: Variable BB_BASEHASH_task-do_package_setscene(content) changed from a4f69a004a31cd3cf529c978342c3423 to 4f995188b8e3ba19ee35f9b455c8cdbc (/tmp/bbsave/intltool_0.40.6.bbnative)
NOTE: Variable RDEPENDS_task-core-lsb-graphic-add_qemux86(content) changed from libqtopengl4 to None (/tmp/bbsave/task-core-lsb.bb)
NOTE: Variable BB_BASEHASH_task-do_bootimg(content) changed from 8abe544bb45386aa3228c53d29965efc to c1a65834157b5ed43c9c8220d608fea2 (/tmp/bbsave/core-image-lsb-live.bb)
NOTE: Variable BB_BASEHASH_task-do_bootimg(content) changed from 7ec98c48f134c67b8ad93837dbb89025 to 20c10600593d2871d100c9cdacd2dce5 (/tmp/bbsave/core-image-lsb-sdk-live.bb)
NOTE: Flag func in variable do_install_virtclass-native disappeared (/tmp/bbsave/ghostscript_9.02.bbnative)
NOTE: Flag func in variable do_configure_virtclass-native disappeared (/tmp/bbsave/ghostscript_9.02.bbnative)
NOTE: Flag func in variable do_compile_virtclass-native disappeared (/tmp/bbsave/ghostscript_9.02.bbnative)
NOTE: Flag python in variable do_patch_virtclass-native disappeared (/tmp/bbsave/ghostscript_9.02.bbnative)
NOTE: Flag func in variable do_patch_virtclass-native disappeared (/tmp/bbsave/ghostscript_9.02.bbnative)
NOTE: Variable LSOF_OS_libc-glibc(content) changed from linux to None (/tmp/bbsave/lsof_4.84.bb)
NOTE: Variable ASNEEDED_pn-icu(content) changed from to None (/tmp/bbsave/icu_3.6.bb)
NOTE: Flag func in variable do_install_virtclass-native disappeared (/tmp/bbsave/shared-mime-info_0.90.bbnative)
NOTE: Flag func in variable do_configure_virtclass-native disappeared (/tmp/bbsave/freetype_2.4.4.bbnative)
NOTE: Variable EXCLUDE_FROM_WORLD_pn-clutter-box2d(content) changed from 1 to None (/tmp/bbsave/clutter-box2d_git.bb)
NOTE: Variable TARGET_CC_ARCH_arm_pn-mesa-xlib(content) changed from -march=i586 to None (/tmp/bbsave/mesa-xlib_7.10.2.bb)
NOTE: Variable RRECOMMENDS_gstreamer_qemux86(content) changed from kernel-module-snd-ens1370 kernel-module-snd-rawmidi to None (/tmp/bbsave/gstreamer_0.10.32.bb)
NOTE: Variable ASNEEDED_pn-pulseaudio(content) changed from to None (/tmp/bbsave/pulseaudio_0.9.15.bb)
NOTE: Variable ASNEEDED_pn-pulseaudio(content) changed from to None (/tmp/bbsave/pulseaudio_0.9.22.bb)
NOTE: Variable KMACHINE_qemux86(content) changed from yocto/standard/common-pc/base to None (/tmp/bbsave/linux-yocto_git.bb)
NOTE: Variable SRCREV_machine_qemux86(content) changed from b2afed7ecd54918b73db7f30062aaaf02fdaba54 to None (/tmp/bbsave/linux-yocto_git.bb)
NOTE: Variable QT_GLFLAGS_qemux86(content) changed from -opengl to None (/tmp/bbsave/qt4-x11-free_4.7.3.bb)
NOTE: Variable BB_BASEHASH_task-do_populate_sdk(content) changed from 3f2c6e5f8ebb90bfe4e0b01d3f5b9bfe to 79ae63b96596a285366bee5e1ff0376c (/tmp/bbsave/meta-toolchain-qte.bb)
NOTE: Variable EXCLUDE_FROM_WORLD_pn-eds-tools(content) changed from 1 to None (/tmp/bbsave/eds-tools_bzr.bb)
NOTE: Variable ASNEEDED_pn-puzzles(content) changed from to None (/tmp/bbsave/puzzles_r9173.bb)
NOTE: Variable BB_BASEHASH_task-do_bootimg(content) changed from 105e1321d452e9d62616e21ca0382ba4 to d8b545984abbae73cc72f73d18246270 (/tmp/bbsave/core-image-sato-live.bb)
NOTE: Variable BB_BASEHASH_task-do_bootdirectdisk(content) changed from 8a376303f61fb09f7a23378f9a688edc to e535f7b5db8ab168ef0fd9b9147b27c3 (/tmp/bbsave/core-image-sato-sdk-directdisk.bb)
NOTE: Variable BB_BASEHASH_task-do_bootimg(content) changed from 0cafcf23b8a6382d4d60febf0824042f to df9504d5e2bcb882bdee203c8d09ff1a (/tmp/bbsave/core-image-sato-sdk-live.bb)
NOTE: Variable BB_BASEHASH_task-do_bootdirectdisk(content) changed from 906e35362d949ee08d53522cd3c91c86 to 1a2dfc525a0e1622de09fe09c13a6639 (/tmp/bbsave/core-image-sato-directdisk.bb)
NOTE: Variable OWL_poky(content) changed from --enable-owl to None (/tmp/bbsave/contacts_git.bb)
NOTE: Variable OWL_poky(content) changed from --enable-owl --disable-dnd to None (/tmp/bbsave/dates_git.bb)
NOTE: Variable OWL_poky(content) changed from --enable-owl to None (/tmp/bbsave/contacts_0.9.bb)
NOTE: Variable OWL_poky(content) changed from --with-owl to None (/tmp/bbsave/tasks_0.19.bb)
NOTE: Variable OWL_poky(content) changed from --enable-owl to None (/tmp/bbsave/tasks_git.bb)
NOTE: Variable OWL_poky(content) changed from --enable-owl --disable-dnd to None (/tmp/bbsave/dates_0.4.8.bb)
NOTE: Variable ASNEEDED_pn-pciutils(content) changed from to None (/tmp/bbsave/pciutils_3.1.7.bb)
NOTE: Variable EXCLUDE_FROM_WORLD_pn-gobject-introspection(content) changed from 1 to None (/tmp/bbsave/gobject-introspection_git.bb)
NOTE: Variable SDK-EXTRAS_qemux86(content) changed from valgrind lttng-ust-dev to None (/tmp/bbsave/task-core-standalone-gmae-sdk-target.bb)
NOTE: Variable SDK-EXTRAS_qemux86(content) changed from valgrind lttng-ust-dev to None (/tmp/bbsave/task-core-sdk-gmae.bb)
NOTE: Variable RRECOMMENDS_gtk+_libc-glibc(content) changed from ${GTKGLIBC_RRECOMMENDS} to None (/tmp/bbsave/gtk+_2.12.7.bb)
NOTE: Variable RRECOMMENDS_gtk+_libc-glibc(content) changed from ${GTKGLIBC_RRECOMMENDS} to None (/tmp/bbsave/gtk+_2.22.1.bb)
NOTE: Variable RRECOMMENDS_gtk+_libc-glibc(content) changed from ${GTKGLIBC_RRECOMMENDS} to None (/tmp/bbsave/gtk+_2.16.6.bb)
NOTE: Variable ENABLE_BINARY_LOCALE_GENERATION_pn-glibc-nativesdk(content) changed from 0 to None (/tmp/bbsave/glibc_2.10.1.bbnativesdk)
NOTE: Flag func in variable do_install_virtclass-native disappeared (/tmp/bbsave/dbus_1.4.1.bbnative)
NOTE: Variable ENABLE_BINARY_LOCALE_GENERATION_pn-eglibc-nativesdk(content) changed from 0 to None (/tmp/bbsave/eglibc_2.12.bbnativesdk)
NOTE: Variable ENABLE_BINARY_LOCALE_GENERATION_pn-eglibc-nativesdk(content) changed from 0 to None (/tmp/bbsave/eglibc_2.13.bbnativesdk)
NOTE: Variable BB_BASEHASH_task-do_populate_sdk(content) changed from 06a328a672551fa5a6baf1666005c74d to 7b324767972020cf7fbbfcec7234709a (/tmp/bbsave/external-python-tarball.bb)
NOTE: Variable BB_BASEHASH_task-do_generate_content(content) changed from 53c5fcc503372e0455c6436aaa3603a8 to 7ebaac1a1815be948aeda09aa08da0f7 (/tmp/bbsave/meta-environment.bb)
NOTE: Variable BB_BASEHASH_task-do_populate_sdk(content) changed from 21f3d18c425404d4b9dda29dea79246f to 377c0d96af0bc090f654dfa321134130 (/tmp/bbsave/meta-toolchain.bb)
NOTE: Variable BB_BASEHASH_task-do_populate_sdk(content) changed from ea9b9e2ec8d4b024bfd2ac2f887d60a0 to 3efc642a10f2a012e7412b4750bdeabc (/tmp/bbsave/meta-toolchain-gmae.bb)
NOTE: Variable BB_BASEHASH_task-do_bootdirectdisk(content) changed from ee3f87895e9e26b9543f9158c1405bcb to e9354cb677fac53fe0924bbf7f0752f3 (/tmp/bbsave/core-image-minimal-directdisk.bb)
NOTE: Variable BB_BASEHASH_task-do_bootimg(content) changed from 4777ac9660c0230a72942facf8225d01 to 9bc113fdffb796c9749a50e86a52d6d4 (/tmp/bbsave/core-image-minimal-live.bb)
NOTE: Variable PACKAGE_ARCH_qemux86(content) changed from ${MACHINE_ARCH} to None (/tmp/bbsave/netbase_4.45.bb)
NOTE: Variable ASNEEDED_pn-console-tools(content) changed from to None (/tmp/bbsave/console-tools_0.3.2.bb)
Parsing of 783 .bb files complete (0 cached, 783 parsed). 1001 targets, 13 skipped, 0 masked, 0 errors.
Of the things in this list, many are overrides which get expanded and
removed from the datastore that aren't virtclass-* ones (or are flag
changes rather than variable contents). This is expected and those can
be discounted from the list of changes.
The changes that would worry me are variables themselves without
overrides (there don't appear to be any) and the BASEHASH* changes. The
task hashes are designed to represent all the inputs to tasks. If this
change has no impact all the hashes would be expected to be the same.
Some BASEHASH changes are expected so discounting the DATE/TIME issues
we'd expect (as shown by the control) the rest are intltool-native
changes. There, an export PERL_virtclass-native variable is playing
havok.
The crazy thing is that its the "PERL_virtclass-native" variable that
was getting exported and is no longer. The code doesn't "export PERL" as
you'd expect before or after this change. I'm therefore concluding that
this change is harmless (or more correct) even if the BASEHASH does
change.
I've tested a build with this change applied and after I fixed a bug in
the PREFERRED_VERSION handling code and an issue with allarch.bbclass
unrelated to this code change the build was successful.
So the question is should we make this change?
It would:
* make things work more as expected
* possibly make the parser slightly faster
* make the data store slightly smaller
* allow simplification of native.bbclass and BBCLASSEXTEND usage
with the last point being the primary driver for fixing this as it would
avoid some horrible hacks.
I've attached two patches, my instrumentation and the override behaviour
change. These need some cleanup tweaks but illustrate the change I'm
proposing. The if len > 0 test was due to some weirdness I noticed where
we're caching empty overrides (variables ending in "_" ?).
Cheers,
Richard
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0001-bitbake-Add-ability-to-save-out-parsed-recipes-and-t.patch
Type: text/x-patch
Size: 4046 bytes
Desc: not available
URL: <http://lists.openembedded.org/pipermail/bitbake-devel/attachments/20110601/1f95cfb3/attachment-0002.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0002-bitbake-data_smart-Change-overrides-behaviour-to-rem.patch
Type: text/x-patch
Size: 2517 bytes
Desc: not available
URL: <http://lists.openembedded.org/pipermail/bitbake-devel/attachments/20110601/1f95cfb3/attachment-0003.bin>
More information about the bitbake-devel
mailing list