[OE-core] [RFC] Source packages
Andre McCurdy
armccurdy at gmail.com
Wed Dec 9 17:56:39 UTC 2015
Hi Haris,
On Tue, Dec 8, 2015 at 12:53 PM, Haris Okanovic <haris.okanovic at ni.com> wrote:
> This change enables creation of ${PN}-src packages which provide recipe
> source code on target machines. A distribution might use this facility
> to provide a feed of source packages for user reference and
> debugging/development activities.
>
> The packaged source code is copy of SRC_URI files + a manifest declaring
> the contents in order of appearance in SRC_URI. For example, it might
> include a tarball + patch files, which could be used to reconstruct the
> state of a recipe's working dir after do_patch() by unpacking the
> archive and applying patches in order.
>
> At the moment, unpacking need to be done by hand. However, RPM and Dpkg
> can both do this automatically upon install when the source is packaged
> in a src.rpm or dsc file, respectively, as opposed to a binary archive
> (rpm or deb). A good future improvement might be to provide proper
> src.rpm's or .dsc's for distributions using those package manager.
>
> I don't believe opkg can distinguish between binary and source archives
> at the moment, so this may be the best we can do at the moment for IPK
> distributions.
>
>
> Classic OE provided a similar facility for IPKs only with
> sourceipk.bbclass [1]. It was dropped at some point in modern OE and I'm
> not familiar enough with OE's history pre Yocto to track down exactly
> why. I'm curious if this was this a deliberate change or just an
> accidental regression, and the reasons it was removed in the former
> case.
>
> One significant difference between the classic sourceipk.bbclass and my
> proposal is what gets packaged: sourceipk simply archived the entire
> ${S} dir after do_patch() but before do_configure() pollutes it. I opted
> not to go this route because it's seems at odds with RPM and Dpkg, both
> of which use the aforementioned tarball+patch file scheme. This may
> allow us to re-use the staging logic should we implement proper
> src.rpm's and dsc's in the future.
>
>
> Not all recipes have useful source to package: For example,
> linux-libc-headers build from the kernel source, which is is already
> provided by the kernel recipe. base-files just provides initial system
> configuration, so it's source package provides no additional info than
> it's binary.
>
> I couldn't find a clever way to filter out these edge cases, other than
> disabling source packages inside those recipes which would require a
> lengthy review of OE meta layers.
>
> I imagine many OE distro also target small, resource constrained,
> systems, that couldn't benefit from source packaging.
>
> Therefore I've made this an opt-in feature for individual distro
> maintainers to enable as needed. It can be enabled per recipe via the
> ENABLE_SRC_INSTALL_${PN} var or globally via ENABLE_SRC_INSTALL from
> distribution confs.
>
>
> === Implementation details ===
>
> [PATCH 1/3] package.bbclass/package.py: Add do_install_source() task
>
> Add do_install_source() task to stage a recipe's SRC_URI files under
> SRC_D.
>
> Dependencies:
> After do_fetch() to ensure SRC_URI files are downloaded
> After do_install() because it resets ${D} that's also used by this task
> Before do_package() to stage files before writing installers
>
> No-ops unless ENABLE_SRC_INSTALL_${PN} = 1, which needs to be set in
> distro config or recipes wanting to use this facility.
>
> [PATCH 2/3] documentation.conf: Document do_install_source()
>
> [PATCH 3/3] bitbake.conf: Define source package, disabled by default
>
> Add ${PN}-src to PACKAGES with a default set of FILES_{PV}-src
> per FHS [2].
>
> Define a staging directory SRC_D under ${D}.
>
> Define ENABLE_SRC_INSTALL to enable source staging via package.bbclass's
> do_install_source() task. It's disabled by default, but may be
> overridden by distro configs or individual recipes as needed.
>
>
> === References ===
>
> [1] https://github.com/openembedded/openembedded/blob/master/classes/sourceipk.bbclass
> [2] https://refspecs.linuxfoundation.org/FHS_3.0/fhs-3.0.html#usrsrcSourceCode
>
>
> Comments, opinions, and other meandering thoughts welcome....
>
> Thanks,
> Haris
> ---
> meta/classes/package.bbclass | 5 +++
> meta/conf/bitbake.conf | 9 ++++-
> meta/conf/documentation.conf | 1 +
> meta/lib/oe/package.py | 96 ++++++++++++++++++++++++++++++++++++++++++++
> 4 files changed, 110 insertions(+), 1 deletion(-)
>
> diff --git a/meta/classes/package.bbclass b/meta/classes/package.bbclass
> index d731757..98f01e5 100644
> --- a/meta/classes/package.bbclass
> +++ b/meta/classes/package.bbclass
> @@ -2079,3 +2079,8 @@ def mapping_rename_hook(d):
> runtime_mapping_rename("RRECOMMENDS", pkg, d)
> runtime_mapping_rename("RSUGGESTS", pkg, d)
>
> +addtask do_install_source after do_fetch after do_install before do_package
> +
> +python do_install_source () {
> + oe.package.do_install_source(d)
> +}
> diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
> index 06971da..431865d 100644
> --- a/meta/conf/bitbake.conf
> +++ b/meta/conf/bitbake.conf
> @@ -36,6 +36,7 @@ export systemd_unitdir = "${nonarch_base_libdir}/systemd"
> export systemd_system_unitdir = "${nonarch_base_libdir}/systemd/system"
> export nonarch_libdir = "${exec_prefix}/lib"
> export systemd_user_unitdir = "${nonarch_libdir}/systemd/user"
> +export srcdir = "${prefix}/src"
I wonder if globally exporting a "srcdir" environment variable is the
correct thing to do. If it's not expected to be recognised or used
directly by any package's "make install", etc, then perhaps it should
just be a bitbake internal variable to avoid any concerns about
namespace pollution.
(AFAIK, variables like "prefix" are only exported to support legacy
packages which rely on environment variables + "make -e" to ensure
that "make install", etc, do the right thing. For autotools packages,
for example, prefix etc are all passed directly on the configure
command line so exporting them too is mostly redundant. If we succeed
in getting rid of "make -e" then I guess eventually none of these
directory variables will need to be exported any more?).
> # Architecture dependent paths
> export bindir = "${exec_prefix}/bin"
> @@ -267,7 +268,7 @@ SOLIBSDEV = ".so"
> SOLIBSDEV_darwin = ".dylibbroken"
>
> PACKAGE_BEFORE_PN ?= ""
> -PACKAGES = "${PN}-dbg ${PN}-staticdev ${PN}-dev ${PN}-doc ${PN}-locale ${PACKAGE_BEFORE_PN} ${PN}"
> +PACKAGES = "${PN}-src ${PN}-dbg ${PN}-staticdev ${PN}-dev ${PN}-doc ${PN}-locale ${PACKAGE_BEFORE_PN} ${PN}"
> PACKAGES_DYNAMIC = "^${PN}-locale-.*"
> FILES = ""
>
> @@ -313,6 +314,12 @@ ALLOW_EMPTY_${PN}-dbg = "1"
>
> FILES_${PN}-locale = "${datadir}/locale"
>
> +FILES_${PN}-src = "${srcdir}/${PN}"
> +SRC_D = "${D}${srcdir}/${PN}"
> +
> +ENABLE_SRC_INSTALL = "0"
> +ENABLE_SRC_INSTALL_${PN} = "${ENABLE_SRC_INSTALL}"
> +
> # File manifest
>
> FILE_DIRNAME = "${@os.path.dirname(d.getVar('FILE', False))}"
> diff --git a/meta/conf/documentation.conf b/meta/conf/documentation.conf
> index 845559a..0df8a2f 100644
> --- a/meta/conf/documentation.conf
> +++ b/meta/conf/documentation.conf
> @@ -26,6 +26,7 @@ do_fetchall[doc] = "Fetches all remote sources required to build a target"
> do_generate_qt_config_file[doc] = "Writes a qt.conf file for building a Qt-based application"
> do_install[doc] = "Copies files from the compilation directory to a holding area"
> do_install_ptest_base[doc] = "Copies the runtime test suite files from the compilation directory to a holding area"
> +do_install_source[doc] = "Stages source code for packaging"
> do_kernel_checkout[doc] = "Checks out source/meta branches for a linux-yocto style kernel"
> do_kernel_configcheck[doc] = "Validates the kernel configuration for a linux-yocto style kernel"
> do_kernel_configme[doc] = "Assembles the kernel configuration for a linux-yocto style kernel"
> diff --git a/meta/lib/oe/package.py b/meta/lib/oe/package.py
> index ea6feaa..f980f25 100644
> --- a/meta/lib/oe/package.py
> +++ b/meta/lib/oe/package.py
> @@ -123,3 +123,99 @@ def read_shlib_providers(d):
> shlib_provider[s[0]] = {}
> shlib_provider[s[0]][s[1]] = (dep_pkg, s[2])
> return shlib_provider
> +
> +def archive_dir(dirPath, archivePath):
> + ''' Create tar.bz2 archive at archivePath from dirPath '''
> + import os, oe, bb
> +
> + arDir = os.path.dirname(dirPath)
> + arName = os.path.basename(dirPath)
> +
> + cmd = 'tar -c -j -f \"%s\" -C \"%s\" -p \"%s\"' % (archivePath, arDir, arName)
Perhaps try to make use of pbzip2 instead of "tar -j" to create
tar.bz2 files. See:
http://git.openembedded.org/openembedded-core/commit/?id=670f5cda06070ae888d17cca7a07aa74d751c2f7
> + (retval, output) = oe.utils.getstatusoutput(cmd)
> + if retval:
> + bb.fatal('Failed to archive %s --> %s: %s %s' % (dirPath, archivePath, cmd, output))
> +
> +def do_install_source(d):
> + ''' Stage recipe's source for packaging '''
> + import os, oe, bb
> +
> + pn = d.getVar("PN", True)
> +
> + if d.getVar("ENABLE_SRC_INSTALL_%s" % pn, True) != "1":
> + return
> +
> + packages = (d.getVar("PACKAGES") or "").split()
> + if ("%s-src" % pn) not in packages:
> + # Some recipes redefine PACKAGES without ${PN}-src. Don't stage
> + # anything in this case to avoid installed-vs-shipped warning.
> + return
> +
> + urls = (d.getVar('SRC_URI', True) or "").split()
> + if len(urls) == 0:
> + return
> +
> + workdir = d.getVar('WORKDIR', True)
> +
> + # TODO rm_work() should clean this up
> + unpackTempDir = os.path.join(workdir, 'install-source-unpack-temp')
> + if os.path.exists(unpackTempDir):
> + bb.utils.remove(unpackTempDir, recurse=True)
> + os.makedirs(unpackTempDir, 0755)
> +
> + src_d = d.getVar("SRC_D", True)
> + if os.path.exists(src_d):
> + bb.warn("SRC_D already exist. Removing.")
> + bb.utils.remove(src_d, recurse=True)
> + os.makedirs(src_d, 0755)
> +
> + fetcher = bb.fetch2.Fetch(urls, d)
> +
> + fileManif = []
> + for url in urls:
> + urlScheme = bb.fetch2.decodeurl(url)[0]
> + srcPath = fetcher.localpath(url)
> + srcName = os.path.basename(srcPath)
> +
> + dstName = srcName
> + if os.path.isdir(srcPath):
> + dstName += '.tar.bz2'
> +
> + dstPath = os.path.join(src_d, dstName)
> +
> + # fetch() doesn't retrieve any actual files from git:// URLs,
> + # so we do an additional unpack() step to get something useful
> + # for these.
> + # TODO: May need to pre-process other revision control schemes
> + if urlScheme == 'git':
> + unpackPath = os.path.join(unpackTempDir, srcName)
> + if os.path.exists(unpackPath):
> + bb.utils.remove(unpackPath, recurse=True)
> + os.makedirs(unpackPath, 0755)
> +
> + fetcher.unpack(unpackPath, [url])
> +
> + # unpack() puts actual source in a 'git' subdir
> + srcPath = os.path.join(unpackPath, 'git')
> +
> + if os.path.exists(dstPath):
> + bb.warn('Duplicate file %s in SRC_URI. Overwriting.' % dstName)
> + bb.utils.remove(dstPath, recurse=True)
> +
> + if not dstName in fileManif:
> + fileManif.append(dstName)
> +
> + if os.path.isdir(srcPath):
> + archive_dir(srcPath, dstPath)
> + else:
> + bb.utils.copyfile(srcPath, dstPath)
> +
> + manifFilePath = os.path.join(src_d, 'manifest')
> + if os.path.exists(manifFilePath):
> + bb.warn('manifest file found in SRC_URI. Overwriting.')
> + bb.utils.remove(manifFilePath, recurse=True)
> +
> + with open(manifFilePath, 'wb') as manif:
> + for fname in fileManif:
> + manif.write(fname)
> + manif.write('\n')
> --
> 2.6.2
>
> --
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core at lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-core
More information about the Openembedded-core
mailing list