[OE-core] [PATCH 1/2] nopackages.bbclass: improve performance for cleansstate

[Richard Purdie]

On Thu, 2018-11-22 at 19:51 +0800, Robert Yang wrote:
> Leave the task in SSTATETASKS doesn't make sense when it is removed,
> so also
> remove it from SSTATETASKS.
> 
> This can improve the performance a lot for "bitbake <recipe-
> native/cross/crosssdk>
> -ccleansstate" when there are a lot of sstate files.
> 
> For example:
> * Before
>   $ bitbake quilt-native -ccleansstate
>   - Check log.do_cleansstate:
>   Removing /sstate-cache/*/*/sstate:quilt-native:x86_64-
> linux:0.65:r0:x86_64:3:*_package.tgz*
>   Removing /sstate-cache/*/*/sstate:quilt-native:x86_64-
> linux:0.65:r0:x86_64:3:*_package_qa.tgz*
>   Removing /sstate-cache/*/*/sstate:quilt-native:x86_64-
> linux:0.65:r0:x86_64:3:*_package_write_rpm.tgz*
>   Removing /sstate-cache/*/*/sstate:quilt-native:x86_64-
> linux:0.65:r0:x86_64:3:*_packagedata.tgz*
>   Removing /sstate-cache/*/sstate:quilt-
> native::0.65:r0::3:*_populate_lic.tgz*
>   Removing /sstate-cache/*/*/sstate:quilt-native:x86_64-
> linux:0.65:r0:x86_64:3:*_populate_sysroot.tgz*
> 
>   There are no package tasks for quilt-native, so the first 4 lines
> doesn't
>   make any sense, but the glob pattern "sstate-cache/*/*" is very
> time
>   consuming when there are no disk caches. E.g., I have more than
> 600,000
>   sstate files:
>   - Without disk caches
>   # echo 3 >/proc/sys/vm/drop_caches
>   $ time python3 -c 'import glob; glob.glob("/sstate-
> cache/*/*/sstate:quilt-native:x86_64-
> linux:0.65:r0:x86_64:3:*_package.tgz*")'
>     real    4m32.583s
>     user    0m5.768s
>     sys     0m12.892s
> 
>   - With disk caches (e.g., run it in the second time)
>   $ time python3 -c 'import glob; glob.glob("/sstate-
> cache/*/*/sstate:quilt-native:x86_64-
> linux:0.65:r0:x86_64:3:*_package.tgz*")'
>     real    0m5.128s
>     user    0m2.772s
>     sys     0m2.308s
> 
>   So the 4 removing *package* commands cost more than 20s or 272s in
> theory.
> 
> * After
>   $ bitbake quilt-native -ccleansstate
>   - Check log.do_cleansstate:
>   Removing /sstate-cache/*/sstate:quilt-
> native::0.65:r0::3:*_populate_lic.tgz*
>   Removing /sstate-cache/*/*/sstate:quilt-native:x86_64-
> linux:0.65:r0:x86_64:3:*_populate_sysroot.tgz*
> 
>   Now the package tasks are gone when cleanssate for native recipes,
> which
>   saved 20s or 272s in theory.
> 
> Signed-off-by: Robert Yang <liezhi.yang at windriver.com>
> ---
>  meta/classes/nopackages.bbclass | 35 +++++++++++++++++++++++------
> ------
>  1 file changed, 23 insertions(+), 12 deletions(-)
> 
> diff --git a/meta/classes/nopackages.bbclass
> b/meta/classes/nopackages.bbclass
> index 559f507..66e4936 100644
> --- a/meta/classes/nopackages.bbclass
> +++ b/meta/classes/nopackages.bbclass
> @@ -1,12 +1,23 @@
> -deltask do_package
> -deltask do_package_write_rpm
> -deltask do_package_write_ipk
> -deltask do_package_write_deb
> -deltask do_package_qa
> -deltask do_packagedata
> -deltask do_package_setscene
> -deltask do_package_write_rpm_setscene
> -deltask do_package_write_ipk_setscene
> -deltask do_package_write_deb_setscene
> -deltask do_package_qa_setscene
> -deltask do_packagedata_setscene
> +NO_PACKAGES_TASKS = " \
> +    do_package \
> +    do_package_write_rpm \
> +    do_package_write_ipk \
> +    do_package_write_deb \
> +    do_package_qa \
> +    do_packagedata \
> +"
> +
> +python() {
> +    sstatetasks = (d.getVar('SSTATETASKS') or '').split()
> +    for task in d.getVar('NO_PACKAGES_TASKS').split():
> +        setscene_task = '%s_setscene' % task
> +        tasks = (task, setscene_task)
> +        for t in tasks:
> +            bb.build.deltask(t, d)
> +
> +        if task in sstatetasks:
> +            bb.debug(2, 'Removing sstate task %s from SSTATETASKS' %
> task)
> +            sstatetasks.remove(task)
> +    d.setVar('SSTATETASKS', ' '.join(sstatetasks))
> +
> +}

I really don't want to turn more comparatively readable classes into a
mess of anonymous python if we can help it.

Can we not just teach cleansstate to check if a task exists before
trying to delete the sstate? It should be possible with
d.getVarFlag(taskname, "task")?

Cheers,

Richard