[OE-core] [RFC][PATCH] bitbake.conf, kernel*.bbclass: include IMAGE_VERSION_SUFFIX only in the _LINK_NAME variables and change it to hard link
Martin Jansa
martin.jansa at gmail.com
Thu Sep 27 12:13:16 UTC 2018
The reason is that for our builds we set the IMAGE_VERSION_SUFFIX based on
jenkins job number which creates them (or you can
imagine IMAGE_VERSION_SUFFIX being set to something like "release-2.6").
With current metadata we have 2 options:
1) keep IMAGE_VERSION_SUFFIX as is, but don't vardepexclude the build
number (DATETIME in the default value)
But this causes e.g. kernel.do_deploy signature to be changed and whole
do_deploy task re-executed (instead of reused from sstate if it's the same
except the output files names), which in turn causes whole kernel to be
rebuilt in every build, because to run do_deploy we need do_install etc
(and these builds are usually using rm_work and TMPDIR is empty before each
job).
2) leave IMAGE_VERSION_SUFFIX as is, but rename the artifacts after the
build
When the image artifact says "image-release-2.6.img" and kernel image from
the same build says "uImage-release-2.5.bin" (because kernel wasn't
modified at all since previous release), people will be confused.
This is what we were seeing in CI which was usually sending just the image,
but then for some new MACHINEs we needed to send kernel artifact separately
and sometimes the artifact wasn't found with expected name.
Then we changed the build number (aka IMAGE_VERSION_SUFFIX) to be included
in do_deploy signatures which lead to 1) and kernel rebuilds every time
To solve both, we've added webos_deploy_fixup tasks (in many places) which
just adds the hardlinks with expected filenames (so that other systems just
using the artifacts doesn't need to know anything about internal issues
with sstate reuse, nor needing external script to rename things after the
build).
Hardlinks are better than symlinks in this case, because you don't want to
have symlink
image-release-2.5.img -> image.img
still in the deploy directory after you do another build which creates new
image.img and new symlink
image-release-2.6.img -> image.img
If someone downloads image-release-2.5.img "too late" he will get 2.6
version instead.
Hardlinks don't have this issue and yes the need to check the inode numbers
might be confusing to some, but when do you really care where does it
point? In our use-case we always care only about the *"release-2.5"* and
the artifacts without the version string are considered as intermediate
files created by the build or "whatever version was built last in this
build directory", pretty much the same as version-less symlinks in current
build (which are easy to resolve when looking at the filesystem, but
exposing those symlinks for download e.g. over HTTP hides where it points
completely.
I'm not sure I understand your concern about bypassing sstate control, we
still want to reuse kernel.do_deploy and image.do_image_complete. Just with
additional very fast intentionally not-covered-by-sstate task before
do_build to create consistent names.
On Thu, Sep 27, 2018 at 1:15 PM Richard Purdie <
richard.purdie at linuxfoundation.org> wrote:
> Hi Martin,
>
> In the commit message you say a lot about what you've changed but not
> so much about why the changes are important and the advantages they
> bring.
>
> There are tradeoffs, for example symlinks make it clear which artefact
> they're really pointing at, hardlinks hide that fact, you need to go
> and look at inode numbers to figure out which of several artefacts one
> might be pointing at. I'm not sure that is an improvement.
>
> I'm also slightly concerned that we need to bypass sstate control, the
> whole intent there was to ensure that output is reproducible and
> consistently restored. Adding in a new task means do_build target usage
> will work but any of the code that has do_deploy as a dependency (e.g.
> an recrdep) will now also need to consider do_deploy_links. For that
> reason alone, I'm not sure we can do this.
>
> Cheers,
>
> Richard
>
> On Thu, 2018-09-27 at 09:11 +0000, Martin Jansa wrote:
> > * just RFC, the part for images isn't finished yet and there is
> > still some issue with DATETIME when kernel artifacts are used
> > from sstate, this is just to validate the idea behind
> > [YOCTO #12937] before finishing the implementation (it's already
> > finished and used by various LGE builds, but having simpler
> > way of doing it directly in oe-core mighe be useful for others).
> >
> > * move IMAGE_VERSION_SUFFIX from _NAME variables to _LINK_NAME
> > that way e.g. kernel.do_deploy can be reused from sstate to
> > provide "version-less" artifacts and then very fast
> > do_deploy_links task just adds links with consistent suffixes
> > * create hard links instead of symlinks, so that whatever version
> > the filename says is really there
> > * some IMAGE_FSTYPES might need the "version-less" IMAGE_NAME file
> > to be removed first or they might either append or update the
> > content of the image instead of creating new image file from
> > scratch - I have seen this only with one proprietary format we
> > generate with our own tool, so hopefully this isn't very common
> > * this is basically the mechanism are using in webOS with
> > WEBOS_IMAGE_NAME_SUFFIX which is for official builds set from
> > jenkins job and then all artifacts (images as well as corresponding
> > kernel files) have the same version string, the implementation
> > "from outside" is a bit tricky as shown in webOS OSE, the variable
> > modifications:
> > https://github.com/webosose/meta-webosose/blob/a35e81622aae1066591e
> > 44a132d01297ff478248/meta-webos/conf/distro/include/webos.inc#L65
> > and then various bbclasses to hook do_webos_deploy_fixup task
> > creating
> > the hardlinks for possible artifacts:
> > https://github.com/webosose/meta-webosose/blob/a35e81622aae1066591e
> > 44a132d01297ff478248/meta-webos/classes/webos_deploy.bbclass
> > https://github.com/webosose/meta-webosose/blob/a35e81622aae1066591e
> > 44a132d01297ff478248/meta-webos/classes/kernel.bbclass
> > https://github.com/webosose/meta-webosose/blob/a35e81622aae1066591e
> > 44a132d01297ff478248/meta-webos/classes/image.bbclass
> > so hopefully with all these changes in oe-core other project can
> > achieve the same just by setting one variable IMAGE_VERSION_SUFFIX
> >
> > [YOCTO #12937]
> >
> > Signed-off-by: Martin Jansa <Martin.Jansa at gmail.com>
> > ---
> > meta/classes/kernel-artifact-names.bbclass | 4 +--
> > meta/classes/kernel.bbclass | 40 ++++++++++++++++--
> > ----
> > meta/conf/bitbake.conf | 4 +--
> > 3 files changed, 33 insertions(+), 15 deletions(-)
> >
> > diff --git a/meta/classes/kernel-artifact-names.bbclass
> > b/meta/classes/kernel-artifact-names.bbclass
> > index bbeecba7bd..84ec193b5a 100644
> > --- a/meta/classes/kernel-artifact-names.bbclass
> > +++ b/meta/classes/kernel-artifact-names.bbclass
> > @@ -1,5 +1,5 @@
> > -KERNEL_ARTIFACT_NAME ?= "${PKGE}-${PKGV}-${PKGR}-
> > ${MACHINE}${IMAGE_VERSION_SUFFIX}"
> > -KERNEL_ARTIFACT_LINK_NAME ?= "${MACHINE}"
> > +KERNEL_ARTIFACT_NAME ?= "${PKGE}-${PKGV}-${PKGR}-${MACHINE}"
> > +KERNEL_ARTIFACT_LINK_NAME ?=
> > "${KERNEL_ARTIFACT_NAME}${IMAGE_VERSION_SUFFIX}"
> >
> > KERNEL_IMAGE_NAME ?= "${KERNEL_ARTIFACT_NAME}"
> > KERNEL_IMAGE_LINK_NAME ?= "${KERNEL_ARTIFACT_LINK_NAME}"
> > diff --git a/meta/classes/kernel.bbclass
> > b/meta/classes/kernel.bbclass
> > index d0fbbd1989..7aaebb56b4 100644
> > --- a/meta/classes/kernel.bbclass
> > +++ b/meta/classes/kernel.bbclass
> > @@ -667,25 +667,18 @@ kernel_do_deploy() {
> > fi
> >
> > for imageType in ${KERNEL_IMAGETYPES} ; do
> > - base_name=${imageType}-${KERNEL_IMAGE_NAME}
> > - install -m 0644 ${KERNEL_OUTPUT_DIR}/${imageType}
> > $deployDir/${base_name}.bin
> > - symlink_name=${imageType}-${KERNEL_IMAGE_LINK_NAME}
> > - ln -sf ${base_name}.bin
> > $deployDir/${symlink_name}.bin
> > - ln -sf ${base_name}.bin $deployDir/${imageType}
> > + install -m 0644 ${KERNEL_OUTPUT_DIR}/${imageType}
> > $deployDir/${imageType}-${KERNEL_IMAGE_NAME}.bin
> > + ln -sf ${imageType}-${KERNEL_IMAGE_NAME}.bin
> > $deployDir/${imageType}
> > done
> >
> > if [ ${MODULE_TARBALL_DEPLOY} = "1" ] && (grep -q -i -e
> > '^CONFIG_MODULES=y$' .config); then
> > mkdir -p ${D}${root_prefix}/lib
> > tar -cvzf $deployDir/modules-
> > ${MODULE_TARBALL_NAME}.tgz -C ${D}${root_prefix} lib
> > - ln -sf modules-${MODULE_TARBALL_NAME}.tgz
> > $deployDir/modules-${MODULE_TARBALL_LINK_NAME}.tgz
> > fi
> >
> > if [ ! -z "${INITRAMFS_IMAGE}" -a
> > x"${INITRAMFS_IMAGE_BUNDLE}" = x1 ]; then
> > for imageType in ${KERNEL_IMAGETYPES} ; do
> > - initramfs_base_name=${imageType}-
> > ${INITRAMFS_NAME}
> > - initramfs_symlink_name=${imageType}-
> > ${INITRAMFS_LINK_NAME}
> > - install -m 0644
> > ${KERNEL_OUTPUT_DIR}/${imageType}.initramfs
> > $deployDir/${initramfs_base_name}.bin
> > - ln -sf ${initramfs_base_name}.bin
> > $deployDir/${initramfs_symlink_name}.bin
> > + install -m 0644
> > ${KERNEL_OUTPUT_DIR}/${imageType}.initramfs $deployDir/${imageType}-
> > ${INITRAMFS_NAME}.bin
> > done
> > fi
> > }
> > @@ -695,7 +688,32 @@ do_deploy[prefuncs] += "package_get_auto_pr"
> >
> > addtask deploy after do_populate_sysroot do_packagedata
> >
> > -EXPORT_FUNCTIONS do_deploy
> > +kernel_do_deploy_links() {
> > + deployDir="${DEPLOY_DIR_IMAGE}"
> > + if [ -n "${KERNEL_DEPLOYSUBDIR}" ]; then
> > + deployDir="${DEPLOY_DIR_IMAGE}/${KERNEL_DEPLOYSUBDIR
> > }"
> > + mkdir "$deployDir"
> > + fi
> > +
> > + for imageType in ${KERNEL_IMAGETYPES} ; do
> > + ln -vf $deployDir/${imageType}-
> > ${KERNEL_IMAGE_NAME}.bin $deployDir/${imageType}-
> > ${KERNEL_IMAGE_LINK_NAME}.bin
> > + done
> > +
> > + if [ ${MODULE_TARBALL_DEPLOY} = "1" ] && (grep -q -i -e
> > '^CONFIG_MODULES=y$' .config); then
> > + ln -vf $deployDir/modules-${MODULE_TARBALL_NAME}.tgz
> > $deployDir/modules-${MODULE_TARBALL_LINK_NAME}.tgz
> > + fi
> > +
> > + if [ ! -z "${INITRAMFS_IMAGE}" -a
> > "${INITRAMFS_IMAGE_BUNDLE}" = "1" ]; then
> > + for imageType in ${KERNEL_IMAGETYPES} ; do
> > + ln -vf ${imageType}-${INITRAMFS_NAME}.bin
> > ${imageType}-${INITRAMFS_LINK_NAME}.bin
> > + done
> > + fi
> > +}
> > +do_deploy_links[prefuncs] += "package_get_auto_pr"
> > +
> > +addtask deploy_links after do_deploy before do_build
> > +
> > +EXPORT_FUNCTIONS do_deploy do_deploy_links
> >
> > # Add using Device Tree support
> > inherit kernel-devicetree
> > diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
> > index 93aee1ae46..017cb163ba 100644
> > --- a/meta/conf/bitbake.conf
> > +++ b/meta/conf/bitbake.conf
> > @@ -447,8 +447,8 @@ IMAGE_ROOTFS = "${WORKDIR}/rootfs"
> > IMAGE_BASENAME = "${PN}"
> > IMAGE_VERSION_SUFFIX = "-${DATETIME}"
> > IMAGE_VERSION_SUFFIX[vardepsexclude] += "DATETIME"
> > -IMAGE_NAME = "${IMAGE_BASENAME}-${MACHINE}${IMAGE_VERSION_SUFFIX}"
> > -IMAGE_LINK_NAME = "${IMAGE_BASENAME}-${MACHINE}"
> > +IMAGE_NAME = "${IMAGE_BASENAME}-${MACHINE}"
> > +IMAGE_LINK_NAME = "${IMAGE_NAME}${IMAGE_VERSION_SUFFIX}"
> >
> > # This option allows for a percentage overage of the actual image
> > size rather than a
> > # fixed extra space, this is space needed for initial startup and
> > basic operations.
> > --
> > 2.17.1
> >
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openembedded.org/pipermail/openembedded-core/attachments/20180927/7c8a8491/attachment-0002.html>
More information about the Openembedded-core
mailing list