[Openembedded-architecture] Package postinstall dependencies - Introduction of PACKAGE_WRITE_DEPS
Mark Hatle
mark.hatle at windriver.com
Wed Jan 18 15:24:03 UTC 2017
On 1/18/17 8:23 AM, Richard Purdie wrote:
> One problem we're seeing with recipe specific sysroots (rss) is that
> the dependencies package postinstalls have is badly represented.
>
> For example, dbus adds a user and this needs tools from shadow-native.
>
> If you run a do_rootfs build from an empty TMPDIR and have a populated
> sstate cache, how does the system know it needs to install shadow-
> native from sstate so that dbus' postinsts aren't deferred to on
> target?
I have seen other systems declare these in different ways. Things like:
RPROVIDES_pn += "user(name)"
RREQUIRES_pn += "user(name)"
Automation can declare provides and dependencies users that are need. But of
course there will always be manual users/groups that are needed as well.
Theoretically we should already be doing some of this via dependencies today --
but it can be a complex problem.
Of course this doesn't handle other things postinsts may be needed for...
> Currently its a bit of mess/hack. There is a list of hardcoded special
> cases in sstate.bbclass but its incomplete. There are also lists of
> special dependencies in the image.bbclass rootfs code to pull in things
> like depmodwrapper-cross and ldconfig.
>
> Some things are detected at build time like ldconfig dependencies so
> there isn't much that can be done for them but most dependencies like
> depmod, useradd, gtk-icon-cache, gdk-pixbuf, systemd and so on are
> known in advance.
>
> With rss, the problem gets more complicated since not only do we need
> to ensure the tools get extracted from sstate but that they must get
> into the image recipe's own native sysroot.
>
> I've spent quite some time pondering how to do this. A normal DEPENDS
> means something quite different so we need some kind of new markup.
> Adding new kinds of dependencies to bitbake doesn't seem attractive
> though.
>
> I'm going to propose a new PACKAGE_WRITE_DEPS variable which gets added
> to the [depends] flag of the do_package_write_XXX tasks.
>
> What this means is that the dependency is indicated in the task graph
> and its done in a generic way which the code can identify and act upon.
> These tasks don't currently have much in the way of dependencies other
> than tools to actually build packages.
>
> The PACKAGE_WRITE_DEPS can either be in addition to DEPENDS if the
> recipe needs these tools at build time, or if its only needed for the
> postinst, PACKAGE_WRITE_DEPS can be sufficient and the DEPENDS can be
> removed.
I'm a little worried this is making things more complicated then they may need
to be.
What is the downside of simply specifying those items in the existing DEPENDS as
we have it today? (Performance, they have to be installed before the package
can begin to build?)
The name itself doesn't make me understand (without reading your explanation)
why it's required. That also makes me suspect it might be difficult (over time)
to explain to people when they need this behavior and why.
--Mark
> I've had a go at a quick proof of concept patch below. This includes
> removing the hardcoded pieces from sstate.bbclass but I'm still in the
> process of testing this.
>
> Right now, I wanted to put this proposal out there and see if people
> were ok with the general idea. There are perhaps some tweaks that could
> be made to the final implementation. I am conscious the M2 deadline is
> looming and if we want rss in this release, we need to be thinking
> about merging code soon though.
>
> Cheers,
>
> Richard
>
>
> From c8c37b97427e37852f2a6efc14b5337422489116 Mon Sep 17 00:00:00 2001
> From: Richard Purdie <richard.purdie at linuxfoundation.org>
> Date: Wed, 18 Jan 2017 12:33:16 +0000
> Subject: PACKAGE_WRITE_DEPS first pass
>
> Signed-off-by: Richard Purdie <richard.purdie at linuxfoundation.org>
>
> diff --git a/meta/classes/gtk-icon-cache.bbclass b/meta/classes/gtk-icon-cache.bbclass
> index c5d8d7c..1e75bde 100644
> --- a/meta/classes/gtk-icon-cache.bbclass
> +++ b/meta/classes/gtk-icon-cache.bbclass
> @@ -2,6 +2,8 @@ FILES_${PN} += "${datadir}/icons/hicolor"
>
> DEPENDS += "${@['hicolor-icon-theme', '']['${BPN}' == 'hicolor-icon-theme']} gtk-icon-utils-native"
>
> +PACKAGE_WRITE_DEPS += "gtk-icon-utils-native:do_populate_sysroot gdk-pixbuf-native:do_populate_sysroot"
> +
> gtk_icon_cache_postinst() {
> if [ "x$D" != "x" ]; then
> $INTERCEPT_DIR/postinst_intercept update_icon_cache ${PKG} \
> diff --git a/meta/classes/kernel-module-split.bbclass b/meta/classes/kernel-module-split.bbclass
> index efe1b42..c7f4880 100644
> --- a/meta/classes/kernel-module-split.bbclass
> +++ b/meta/classes/kernel-module-split.bbclass
> @@ -22,6 +22,8 @@ if [ x"$D" = "x" ]; then
> fi
> }
>
> +PACKAGE_WRITE_DEPS += "kmod-native:do_populate_sysroot depmodwrapper-cross:do_populate_sysroot"
> +
> do_install_append() {
> install -d ${D}${sysconfdir}/modules-load.d/ ${D}${sysconfdir}/modprobe.d/
> }
> diff --git a/meta/classes/package.bbclass b/meta/classes/package.bbclass
> index 568b85c..0a56d1c 100644
> --- a/meta/classes/package.bbclass
> +++ b/meta/classes/package.bbclass
> @@ -54,6 +54,8 @@ ALL_MULTILIB_PACKAGE_ARCHS = "${@all_multilib_tune_values(d, 'PACKAGE_ARCHS')}"
> # rpm is used for the per-file dependency identification
> PACKAGE_DEPENDS += "rpm-native"
>
> +PACKAGE_WRITE_DEPS ??= ""
> +
> def legitimize_package_name(s):
> """
> Make sure package names are legitimate strings
> diff --git a/meta/classes/package_deb.bbclass b/meta/classes/package_deb.bbclass
> index 68eca61..d4f9262 100644
> --- a/meta/classes/package_deb.bbclass
> +++ b/meta/classes/package_deb.bbclass
> @@ -351,6 +351,7 @@ python do_package_write_deb () {
> do_package_write_deb[dirs] = "${PKGWRITEDIRDEB}"
> do_package_write_deb[cleandirs] = "${PKGWRITEDIRDEB}"
> do_package_write_deb[umask] = "022"
> +do_package_write_deb[depends] += "${PACKAGE_WRITE_DEPS}"
> addtask package_write_deb after do_packagedata do_package
>
>
> diff --git a/meta/classes/package_ipk.bbclass b/meta/classes/package_ipk.bbclass
> index 7018a60..9def81b 100644
> --- a/meta/classes/package_ipk.bbclass
> +++ b/meta/classes/package_ipk.bbclass
> @@ -291,6 +291,7 @@ python do_package_write_ipk () {
> do_package_write_ipk[dirs] = "${PKGWRITEDIRIPK}"
> do_package_write_ipk[cleandirs] = "${PKGWRITEDIRIPK}"
> do_package_write_ipk[umask] = "022"
> +do_package_write_ipk[depends] += "${PACKAGE_WRITE_DEPS}"
> addtask package_write_ipk after do_packagedata do_package
>
> PACKAGEINDEXDEPS += "opkg-utils-native:do_populate_sysroot"
> diff --git a/meta/classes/package_rpm.bbclass b/meta/classes/package_rpm.bbclass
> index b9f049e..d07ac51 100644
> --- a/meta/classes/package_rpm.bbclass
> +++ b/meta/classes/package_rpm.bbclass
> @@ -772,6 +772,7 @@ python do_package_write_rpm () {
> do_package_write_rpm[dirs] = "${PKGWRITEDIRRPM}"
> do_package_write_rpm[cleandirs] = "${PKGWRITEDIRRPM}"
> do_package_write_rpm[umask] = "022"
> +do_package_write_rpm[depends] += "${PACKAGE_WRITE_DEPS}"
> addtask package_write_rpm after do_packagedata do_package
>
> PACKAGEINDEXDEPS += "rpm-native:do_populate_sysroot"
> diff --git a/meta/classes/pixbufcache.bbclass b/meta/classes/pixbufcache.bbclass
> index 0295a66..3f5b6ef 100644
> --- a/meta/classes/pixbufcache.bbclass
> +++ b/meta/classes/pixbufcache.bbclass
> @@ -8,7 +8,7 @@ inherit qemu
>
> PIXBUF_PACKAGES ??= "${PN}"
>
> -do_package_write_rpm[depends] += "qemu-native:do_populate_sysroot gdk-pixbuf-native:do_populate_sysroot"
> +PACKAGE_WRITE_DEPS += "qemu-native:do_populate_sysroot gdk-pixbuf-native:do_populate_sysroot"
>
> pixbufcache_common() {
> if [ "x$D" != "x" ]; then
> diff --git a/meta/classes/sstate.bbclass b/meta/classes/sstate.bbclass
> index e84e2d9..9500303 100644
> --- a/meta/classes/sstate.bbclass
> +++ b/meta/classes/sstate.bbclass
> @@ -938,17 +938,14 @@ BB_SETSCENE_DEPVALID = "setscene_depvalid"
> def setscene_depvalid(task, taskdependees, notneeded, d):
> # taskdependees is a dict of tasks which depend on task, each being a 3 item list of [PN, TASKNAME, FILENAME]
> # task is included in taskdependees too
> + # Return - False - We need this dependency
> + # - True - We can skip this dependency
>
> bb.debug(2, "Considering setscene task: %s" % (str(taskdependees[task])))
>
> def isNativeCross(x):
> return x.endswith("-native") or "-cross-" in x or "-crosssdk" in x or x.endswith("-cross")
>
> - def isPostInstDep(x):
> - if x in ["qemu-native", "gdk-pixbuf-native", "qemuwrapper-cross", "depmodwrapper-cross", "systemd-systemctl-native", "gtk-icon-utils-native", "ca-certificates-native"]:
> - return True
> - return False
> -
> # We only need to trigger populate_lic through direct dependencies
> if taskdependees[task][1] == "do_populate_lic":
> return True
> @@ -970,10 +967,11 @@ def setscene_depvalid(task, taskdependees, notneeded, d):
> # do_package_write_* and do_package doesn't need do_package
> if taskdependees[task][1] == "do_package" and taskdependees[dep][1] in ['do_package', 'do_package_write_deb', 'do_package_write_ipk', 'do_package_write_rpm', 'do_packagedata', 'do_package_qa']:
> continue
> - # do_package_write_* and do_package doesn't need do_populate_sysroot, unless is a postinstall dependency
> - if taskdependees[task][1] == "do_populate_sysroot" and taskdependees[dep][1] in ['do_package', 'do_package_write_deb', 'do_package_write_ipk', 'do_package_write_rpm', 'do_packagedata', 'do_package_qa']:
> - if isPostInstDep(taskdependees[task][0]) and taskdependees[dep][1] in ['do_package_write_deb', 'do_package_write_ipk', 'do_package_write_rpm']:
> - return False
> + # do_package_write_* need do_populate_sysroot as they're mainly postinstall dependencies
> + if taskdependees[task][1] == "do_populate_sysroot" and taskdependees[dep][1] in ['do_package_write_deb', 'do_package_write_ipk', 'do_package_write_rpm']:
> + return False
> + # do_package/packagedata/package_qa don't need do_populate_sysroot
> + if taskdependees[task][1] == "do_populate_sysroot" and taskdependees[dep][1] in ['do_package', 'do_packagedata', 'do_package_qa']:
> continue
> # Native/Cross packages don't exist and are noexec anyway
> if isNativeCross(taskdependees[dep][0]) and taskdependees[dep][1] in ['do_package_write_deb', 'do_package_write_ipk', 'do_package_write_rpm', 'do_packagedata', 'do_package', 'do_package_qa']:
> diff --git a/meta/classes/systemd.bbclass b/meta/classes/systemd.bbclass
> index 0b1c228..3e33456 100644
> --- a/meta/classes/systemd.bbclass
> +++ b/meta/classes/systemd.bbclass
> @@ -17,6 +17,7 @@ python __anonymous() {
> # files.
> if bb.utils.contains('DISTRO_FEATURES', 'systemd', True, False, d):
> d.appendVar("DEPENDS", " systemd-systemctl-native")
> + d.appendVar("PACKAGE_WRITE_DEPS", " systemd-systemctl-native:do_populate_sysroot")
> if not bb.utils.contains('DISTRO_FEATURES', 'sysvinit', True, False, d):
> d.setVar("INHIBIT_UPDATERCD_BBCLASS", "1")
> }
> diff --git a/meta/classes/useradd.bbclass b/meta/classes/useradd.bbclass
> index 5bf81c4..7fdfca5 100644
> --- a/meta/classes/useradd.bbclass
> +++ b/meta/classes/useradd.bbclass
> @@ -4,6 +4,7 @@ inherit useradd_base
> # target sysroot, and shadow -native and -sysroot provide the utilities
> # and support files needed to add and modify user and group accounts
> DEPENDS_append_class-target = " base-files shadow-native shadow-sysroot shadow base-passwd"
> +PACKAGE_WRITE_DEPS += "shadow-native:do_populate_sysroot"
>
> # This preinstall function can be run in four different contexts:
> #
> diff --git a/meta/recipes-bsp/keymaps/keymaps_1.0.bb b/meta/recipes-bsp/keymaps/keymaps_1.0.bb
> index 5793a76..adea233 100644
> --- a/meta/recipes-bsp/keymaps/keymaps_1.0.bb
> +++ b/meta/recipes-bsp/keymaps/keymaps_1.0.bb
> @@ -37,7 +37,7 @@ do_install () {
> fi
> }
>
> -DEPENDS_append = " ${@bb.utils.contains('DISTRO_FEATURES','systemd','systemd-systemctl-native','',d)}"
> +PACKAGE_WRITE_DEPS_append = " ${@bb.utils.contains('DISTRO_FEATURES','systemd','systemd-systemctl-native:do_populate_sysroot','',d)}"
> pkg_postinst_${PN} () {
> if ${@bb.utils.contains('DISTRO_FEATURES','systemd sysvinit','true','false',d)}; then
> if [ -n "$D" ]; then
> diff --git a/meta/recipes-bsp/v86d/v86d_0.1.10.bb b/meta/recipes-bsp/v86d/v86d_0.1.10.bb
> index 1046d63..8bd3700 100644
> --- a/meta/recipes-bsp/v86d/v86d_0.1.10.bb
> +++ b/meta/recipes-bsp/v86d/v86d_0.1.10.bb
> @@ -60,7 +60,7 @@ python __anonymous() {
>
> inherit update-rc.d
>
> -DEPENDS_append = " ${@bb.utils.contains('DISTRO_FEATURES','systemd','systemd-systemctl-native','',d)}"
> +PACKAGE_WRITE_DEPS_append = " ${@bb.utils.contains('DISTRO_FEATURES','systemd','systemd-systemctl-native:do_populate_sysroot','',d)}"
> pkg_postinst_${PN} () {
> if ${@bb.utils.contains('DISTRO_FEATURES','systemd sysvinit','true','false',d)}; then
> if [ -n "$D" ]; then
> diff --git a/meta/recipes-core/initscripts/initscripts_1.0.bb b/meta/recipes-core/initscripts/initscripts_1.0.bb
> index 8f110b0..836e9eb 100644
> --- a/meta/recipes-core/initscripts/initscripts_1.0.bb
> +++ b/meta/recipes-core/initscripts/initscripts_1.0.bb
> @@ -44,7 +44,7 @@ KERNEL_VERSION = ""
>
> inherit update-alternatives
> DEPENDS_append = " update-rc.d-native"
> -DEPENDS_append = " ${@bb.utils.contains('DISTRO_FEATURES','systemd','systemd-systemctl-native','',d)}"
> +PACKAGE_WRITE_DEPS_append = " ${@bb.utils.contains('DISTRO_FEATURES','systemd','systemd-systemctl-native:do_populate_sysroot','',d)}"
>
> PACKAGES =+ "${PN}-functions"
> RDEPENDS_${PN} = "${PN}-functions \
> diff --git a/meta/recipes-core/psplash/psplash_git.bb b/meta/recipes-core/psplash/psplash_git.bb
> index b0d6bb4..335593d 100644
> --- a/meta/recipes-core/psplash/psplash_git.bb
> +++ b/meta/recipes-core/psplash/psplash_git.bb
> @@ -110,7 +110,7 @@ FILES_${PN} += "/mnt/.psplash"
> INITSCRIPT_NAME = "psplash.sh"
> INITSCRIPT_PARAMS = "start 0 S . stop 20 0 1 6 ."
>
> -DEPENDS_append = " ${@bb.utils.contains('DISTRO_FEATURES','systemd','systemd-systemctl-native','',d)}"
> +PACKAGE_WRITE_DEPS_append = " ${@bb.utils.contains('DISTRO_FEATURES','systemd','systemd-systemctl-native:do_populate_sysroot','',d)}"
> pkg_postinst_${PN} () {
> if ${@bb.utils.contains('DISTRO_FEATURES','systemd','true','false',d)}; then
> if [ -n "$D" ]; then
> diff --git a/meta/recipes-core/systemd/systemd-compat-units.bb b/meta/recipes-core/systemd/systemd-compat-units.bb
> index 0f0876b..7655246 100644
> --- a/meta/recipes-core/systemd/systemd-compat-units.bb
> +++ b/meta/recipes-core/systemd/systemd-compat-units.bb
> @@ -4,7 +4,7 @@ LICENSE = "MIT"
>
> PR = "r29"
>
> -DEPENDS = "systemd-systemctl-native"
> +PACKAGE_WRITE_DEPS += "systemd-systemctl-native:do_populate_sysroot"
>
> S = "${WORKDIR}"
>
> diff --git a/meta/recipes-kernel/modutils-initscripts/modutils-initscripts.bb b/meta/recipes-kernel/modutils-initscripts/modutils-initscripts.bb
> index db670cf..9b602a5 100644
> --- a/meta/recipes-kernel/modutils-initscripts/modutils-initscripts.bb
> +++ b/meta/recipes-kernel/modutils-initscripts/modutils-initscripts.bb
> @@ -22,7 +22,7 @@ do_install () {
> install -m 0755 ${WORKDIR}/modutils.sh ${D}${sysconfdir}/init.d/
> }
>
> -DEPENDS_append = " ${@bb.utils.contains('DISTRO_FEATURES','systemd','systemd-systemctl-native','',d)}"
> +PACKAGE_WRITE_DEPS_append = " ${@bb.utils.contains('DISTRO_FEATURES','systemd','systemd-systemctl-native:do_populate_sysroot','',d)}"
> pkg_postinst_${PN} () {
> if ${@bb.utils.contains('DISTRO_FEATURES','systemd','true','false',d)}; then
> if [ -n "$D" ]; then
> diff --git a/meta/recipes-sato/matchbox-sato/matchbox-session-sato_0.1.bb b/meta/recipes-sato/matchbox-sato/matchbox-session-sato_0.1.bb
> index 42c742f..bf0de36 100644
> --- a/meta/recipes-sato/matchbox-sato/matchbox-session-sato_0.1.bb
> +++ b/meta/recipes-sato/matchbox-sato/matchbox-session-sato_0.1.bb
> @@ -6,7 +6,6 @@ LICENSE = "GPLv2.0+"
> LIC_FILES_CHKSUM = "file://session;endline=3;md5=f8a5c5b9c279e52dc094d10e11c2be63"
>
> SECTION = "x11"
> -DEPENDS = "gconf-native"
> RDEPENDS_${PN} = "formfactor matchbox-theme-sato matchbox-panel-2 matchbox-desktop matchbox-session gconf"
> PR = "r30"
>
> @@ -43,6 +42,7 @@ do_install() {
> chmod +x ${D}/${sysconfdir}/matchbox/session
> }
>
> +PACKAGE_WRITE_DEPS += "gconf-native:do_populate_sysroot"
> pkg_postinst_${PN} () {
> set_value() {
> #type, name, value
> diff --git a/meta/recipes-support/ca-certificates/ca-certificates_20161130.bb b/meta/recipes-support/ca-certificates/ca-certificates_20161130.bb
> index e0b2e41..1c70408 100644
> --- a/meta/recipes-support/ca-certificates/ca-certificates_20161130.bb
> +++ b/meta/recipes-support/ca-certificates/ca-certificates_20161130.bb
> @@ -11,6 +11,7 @@ LIC_FILES_CHKSUM = "file://debian/copyright;md5=e7358b9541ccf3029e9705ed8de57968
> DEPENDS = "ca-certificates-native"
> DEPENDS_class-native = "openssl-native"
> DEPENDS_class-nativesdk = "ca-certificates-native openssl-native"
> +PACKAGE_WRITE_DEPS += "ca-certificates-native:do_populate_sysroot"
>
> SRCREV = "61b70a1007dc269d56881a0d480fc841daacc77c"
>
>
>
> _______________________________________________
> Openembedded-architecture mailing list
> Openembedded-architecture at lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-architecture
>
More information about the Openembedded-architecture
mailing list