[OE-core] [PATCH 3/3] package_ipk: Parallelise ipk creation

[Khem Raj]

On Wed, Jun 14, 2017 at 6:42 AM, Richard Purdie
<richard.purdie at linuxfoundation.org> wrote:
> Allow the creation of ipks to happen in parallel, making best use of resources
> on multiprocessor systems.
>
> Signed-off-by: Richard Purdie <richard.purdie at linuxfoundation.org>
> ---
>  meta/classes/package_ipk.bbclass | 19 +++++++++++++++++--
>  1 file changed, 17 insertions(+), 2 deletions(-)
>
> diff --git a/meta/classes/package_ipk.bbclass b/meta/classes/package_ipk.bbclass
> index d2ce3b3..282d212 100644
> --- a/meta/classes/package_ipk.bbclass
> +++ b/meta/classes/package_ipk.bbclass
> @@ -17,6 +17,8 @@ OPKG_ARGS += "${@['', '--add-exclude ' + ' --add-exclude '.join((d.getVar('PACKA
>  OPKGLIBDIR = "${localstatedir}/lib"
>
>  python do_package_ipk () {
> +    from multiprocessing import Process
> +
>      oldcwd = os.getcwd()
>
>      workdir = d.getVar('WORKDIR')
> @@ -37,11 +39,24 @@ python do_package_ipk () {
>      if os.access(os.path.join(tmpdir, "stamps", "IPK_PACKAGE_INDEX_CLEAN"), os.R_OK):
>          os.unlink(os.path.join(tmpdir, "stamps", "IPK_PACKAGE_INDEX_CLEAN"))
>
> -    for pkg in packages.split():
> -        ipk_write_pkg(pkg, d)
> +    max_process = int(d.getVar("BB_NUMBER_THREADS") or os.cpu_count() or 1)
> +    launched = []
> +    pkgs = packages.split()
> +    while pkgs:
> +        if len(launched) < max_process:
> +            p = Process(target=ipk_write_pkg, args=(pkgs.pop(), d))
> +            p.start()
> +            launched.append(p)
> +        for q in launched:
> +            # The finished processes are joined when calling is_alive()
> +            if not q.is_alive():
> +                launched.remove(q)
> +    for p in launched:
> +        p.join()
>
>      os.chdir(oldcwd)
>  }
> +do_package_ipk[vardepsexclude] = "BB_NUMBER_THREADS"
>
>  def ipk_write_pkg(pkg, d):
>      import re, copy

on a 44 core system, this patch bails out like below. Once I revert
this it works all fine.

ERROR: westeros-wpe-image-1.0-r0 do_rootfs: Unable to install
packages. Command
'/mnt/a/oe/build/tmp/work/raspberrypi3-bec-linux-musleabi/westeros-wpe-image/1.0-r0/recipe-sysroot-native/usr/bin/opkg
--volatile-cache -f /mnt/a/oe/build/tmp
/work/raspberrypi3-bec-linux-musleabi/westeros-wpe-image/1.0-r0/opkg.conf
-t /mnt/a/oe/build/tmp/work/raspberrypi3-bec-linux-musleabi/westeros-wpe-image/1.0-r0/temp/ipktemp/
-o /mnt/a/oe/build/tmp/work/raspberrypi3-bec-linux-musleabi/west
eros-wpe-image/1.0-r0/rootfs  --force_postinstall
--prefer-arch-to-version   install 96boards-tools kernel-modules opkg
packagegroup-core-boot packagegroup-core-ssh-openssh
packagegroup-ml-wpe packagegroup-westeros psplash-raspberrypi run
-postinsts runit runit-serialgetty' returned 255:
Collected errors:
 * opkg_prepare_url_for_install: Couldn't find anything to satisfy
'96boards-tools'.
 * rm_r: Failed to open dir
/mnt/a/oe/build/tmp/work/raspberrypi3-bec-linux-musleabi/westeros-wpe-image/1.0-r0/temp/ipktemp//opkg-J9PI7V:
No such file or directory.

ERROR: westeros-wpe-image-1.0-r0 do_rootfs: Function failed: do_rootfs