[OE-core] [PATCH v3] package.bbclass: allow using EXCLUDE_FROM_SHLIBS for subpackages

Mon Jun 11 12:20:49 UTC 2018

Ping again.

Thank you,
Andrii

On 04/12/2018 05:42 PM, Andrii Bordunov wrote:
> Another ping.
> 
> Any comments? Or could it be just applied as is?
> 
> 
> Thank you,
> Andrii
> 
> On 03/13/2018 07:25 PM, Andrii Bordunov wrote:
>> Ping
>>
>> Rebased patch to latest master.
>>
>> Regarding special value for the variable, it could be done with 
>> PRIVATE_LIBS to mean all libraries, I think. PRIVATE_LIBS could be set 
>> per-package already.
>> Would it be a better option?
>> However, EXCLUDE_FROM_SHLIBS seems to be a more proper thing to use.
>>
>>
>> Thank you,
>> Andrii
>>
>> On 03/16/2017 05:07 PM, Andrii Bordunov wrote:
>>> Hi Richard,
>>>
>>> Sorry, my Python knowledge is quite basic. I'm sure it's possible to 
>>> have a (global?) EXCLUDE_FROM_SHLIBS list holding subpackage names, 
>>> but how it could lead to performance improvement? It would also need 
>>> some check for every subpackage.
>>> Could you please give more details?
>>> And what do you mean under "all packages"? The intent is like 
>>> opposite, to split package operations, not to gather.
>>>
>>> As for False parameter to getVar, OK, I just used existing 
>>> EXCLUDE_FROM_SHLIBS example (seems fixed now).
>>>
>>> In the meantime tried to make a quick and dirty profiling like 
>>> following:
>>>
>>>    bitbake -c package glibc
>>>    bitbake -c clean glibc
>>>    python -m cProfile -s cumtime 
>>> /mnt/src/oe/distro/bitbake/bin/bitbake -c package glibc > result.txt
>>>
>>> Result is below. If got it right, there are 3833 getVar calls with 
>>> total time 0.025 second (including subcalls).
>>> This gives ~6.5 microsecond duration of one call (incl subcalls). Not 
>>> a lot.
>>> This is only a partial rebuild of one package of course.
>>> Also tried to do all of this with and without my patch, but getVar 
>>> calls number stays exactly the same (w/o modifying glibc recipe, but 
>>> anyway).
>>>
>>>    NOTE: Tasks Summary: Attempted 314 tasks of which 314 didn't need 
>>> to be rerun and all succeeded.
>>>             672869 function calls (667640 primitive calls) in 12.912 
>>> seconds
>>>       Ordered by: cumulative time
>>>       ncalls  tottime  percall  cumtime  percall 
>>> filename:lineno(function)
>>>            1    0.001    0.001   12.913   12.913 bitbake:25(<module>)
>>>            1    0.001    0.001   12.082   12.082 
>>> main.py:320(bitbake_main)
>>>            1    0.005    0.005   11.494   11.494 knotty.py:253(main)
>>>          250   11.476    0.046   11.476    0.046 {method 'poll' of 
>>> '_multiprocessing.Connection' objects}
>>>    ...
>>>    4481/4046    0.007    0.000    0.026    0.000 
>>> data_smart.py:666(getVarFlag)
>>>    ...
>>>    3833/3719    0.002    0.000    0.025    0.000 
>>> data_smart.py:569(getVar)
>>>    ...
>>>         1338    0.002    0.000    0.003    0.000 
>>> data_smart.py:805(getVarFlags)
>>>
>>>
>>>
>>> Thank you,
>>> Andrii
>>>
>>> On 16.11.16 12:47, Richard Purdie wrote:
>>>> On Mon, 2016-10-10 at 20:02 +0300, Andrii Bordunov wrote:
>>>>> Some packages containing shared libraries might be registered
>>>>> as shlib providers when they shouldn't (for example, the lib is for
>>>>> their private use and must not generate any dependency).
>>>>>
>>>>> EXCLUDE_FROM_SHLIBS is targeted at that, but it could be set
>>>>> for entire recipe only.
>>>>>
>>>>> This patch expands EXCLUDE_FROM_SHLIBS usage, so now it's possible
>>>>> to set it in a style similar with RDEPENDS. For example:
>>>>>  EXCLUDE_FROM_SHLIBS_${PN}-ptest = "1"
>>>>>
>>>>> Signed-off-by: Andrii Bordunov <aborduno at cisco.com>
>>>>> ---
>>>>>  meta/classes/package.bbclass | 12 ++++++++++--
>>>>>  1 file changed, 10 insertions(+), 2 deletions(-)
>>>>
>>>> My main concern here is actually performance. For a package with many
>>>> sub packages you just added many calls to getVar when that getVar is
>>>> extremely unlikely to find any value. Whilst not hugely slow, that
>>>> operation isn't trivial and hard to fix later.
>>>>
>>>> I'm wondering if we could support the syntax
>>>>
>>>> EXCLUDE_FROM_SHLIBS = "${PN}-ptest"
>>>>
>>>> and then have "1", used in a couple of places as the special value to
>>>> mean all packages?
>>>>
>>>> Also please use False, not 0 as the parameter to getVar.
>>>>
>>>> Cheers,
>>>>
>>>> Richard
> 
> -- 
> 
>  From 4a33d384a72a351bad5a98e46a7c9c1c9755635e Mon Sep 17 00:00:00 2001
> From: Andrii Bordunov <aborduno at cisco.com>
> Date: Mon, 12 Mar 2018 08:13:44 -0700
> Subject: [PATCH v3] Allow EXCLUDE_FROM_SHLIBS to be specified for
> subpackages
> 
> Some subpackages containing shared libraries might be registered
> as shlib providers when they shouldn't (the lib is for their private use
> and must not generate any dependency).
> 
> EXCLUDE_FROM_SHLIBS is targeted at that, but it could be set
> for entire recipe only.
> 
> This patch expands EXCLUDE_FROM_SHLIBS usage, so now it's possible
> to set it like following:
> EXCLUDE_FROM_SHLIBS_${PN}-ptest = "1"
> 
> Signed-off-by: Andrii Bordunov <aborduno at cisco.com>
> ---
>    meta/classes/package.bbclass | 11 +++++++++--
>    1 file changed, 9 insertions(+), 2 deletions(-)
> 
> diff --git a/meta/classes/package.bbclass b/meta/classes/package.bbclass
> index 317c775..5d49d41 100644
> --- a/meta/classes/package.bbclass
> +++ b/meta/classes/package.bbclass
> @@ -1571,6 +1571,13 @@ python package_do_shlibs() {
> 
>        workdir = d.getVar('WORKDIR')
> 
> +    shlib_pkgs = []
> +    for pkg in packages.split():
> +        if d.getVar('EXCLUDE_FROM_SHLIBS_' + pkg, False):
> +            bb.note("not generating shlibs for %s" % pkg)
> +        else:
> +            shlib_pkgs.append(pkg)
> +
>        ver = d.getVar('PKGV')
>        if not ver:
>            msg = "PKGV not defined"
> @@ -1698,7 +1705,7 @@ python package_do_shlibs() {
>        needed = {}
>        shlib_provider = oe.package.read_shlib_providers(d)
> 
> -    for pkg in packages.split():
> +    for pkg in shlib_pkgs:
>            private_libs = d.getVar('PRIVATE_LIBS_' + pkg) or
> d.getVar('PRIVATE_LIBS') or ""
>            private_libs = private_libs.split()
>            needs_ldconfig = False
> @@ -1770,7 +1777,7 @@ python package_do_shlibs() {
> 
>        libsearchpath = [d.getVar('libdir'), d.getVar('base_libdir')]
> 
> -    for pkg in packages.split():
> +    for pkg in shlib_pkgs:
>            bb.debug(2, "calculating shlib requirements for %s" % pkg)
> 
>            private_libs = d.getVar('PRIVATE_LIBS_' + pkg) or
> d.getVar('PRIVATE_LIBS') or ""