[OE-core] [PATCH 1/2] nopackages.bbclass: improve performance for cleansstate

[Yang, Liezhi]

Sent from mobile phone

> 在 2018年11月22日，20:15，Richard Purdie <richard.purdie at linuxfoundation.org> 写道：
> 
>> On Thu, 2018-11-22 at 19:51 +0800, Robert Yang wrote:
>> Leave the task in SSTATETASKS doesn't make sense when it is removed,
>> so also
>> remove it from SSTATETASKS.
>> 
>> This can improve the performance a lot for "bitbake <recipe-
>> native/cross/crosssdk>
>> -ccleansstate" when there are a lot of sstate files.
>> 
>> For example:
>> * Before
>>  $ bitbake quilt-native -ccleansstate
>>  - Check log.do_cleansstate:
>>  Removing /sstate-cache/*/*/sstate:quilt-native:x86_64-
>> linux:0.65:r0:x86_64:3:*_package.tgz*
>>  Removing /sstate-cache/*/*/sstate:quilt-native:x86_64-
>> linux:0.65:r0:x86_64:3:*_package_qa.tgz*
>>  Removing /sstate-cache/*/*/sstate:quilt-native:x86_64-
>> linux:0.65:r0:x86_64:3:*_package_write_rpm.tgz*
>>  Removing /sstate-cache/*/*/sstate:quilt-native:x86_64-
>> linux:0.65:r0:x86_64:3:*_packagedata.tgz*
>>  Removing /sstate-cache/*/sstate:quilt-
>> native::0.65:r0::3:*_populate_lic.tgz*
>>  Removing /sstate-cache/*/*/sstate:quilt-native:x86_64-
>> linux:0.65:r0:x86_64:3:*_populate_sysroot.tgz*
>> 
>>  There are no package tasks for quilt-native, so the first 4 lines
>> doesn't
>>  make any sense, but the glob pattern "sstate-cache/*/*" is very
>> time
>>  consuming when there are no disk caches. E.g., I have more than
>> 600,000
>>  sstate files:
>>  - Without disk caches
>>  # echo 3 >/proc/sys/vm/drop_caches
>>  $ time python3 -c 'import glob; glob.glob("/sstate-
>> cache/*/*/sstate:quilt-native:x86_64-
>> linux:0.65:r0:x86_64:3:*_package.tgz*")'
>>    real    4m32.583s
>>    user    0m5.768s
>>    sys     0m12.892s
>> 
>>  - With disk caches (e.g., run it in the second time)
>>  $ time python3 -c 'import glob; glob.glob("/sstate-
>> cache/*/*/sstate:quilt-native:x86_64-
>> linux:0.65:r0:x86_64:3:*_package.tgz*")'
>>    real    0m5.128s
>>    user    0m2.772s
>>    sys     0m2.308s
>> 
>>  So the 4 removing *package* commands cost more than 20s or 272s in
>> theory.
>> 
>> * After
>>  $ bitbake quilt-native -ccleansstate
>>  - Check log.do_cleansstate:
>>  Removing /sstate-cache/*/sstate:quilt-
>> native::0.65:r0::3:*_populate_lic.tgz*
>>  Removing /sstate-cache/*/*/sstate:quilt-native:x86_64-
>> linux:0.65:r0:x86_64:3:*_populate_sysroot.tgz*
>> 
>>  Now the package tasks are gone when cleanssate for native recipes,
>> which
>>  saved 20s or 272s in theory.
>> 
>> Signed-off-by: Robert Yang <liezhi.yang at windriver.com>
>> ---
>> meta/classes/nopackages.bbclass | 35 +++++++++++++++++++++++------
>> ------
>> 1 file changed, 23 insertions(+), 12 deletions(-)
>> 
>> diff --git a/meta/classes/nopackages.bbclass
>> b/meta/classes/nopackages.bbclass
>> index 559f507..66e4936 100644
>> --- a/meta/classes/nopackages.bbclass
>> +++ b/meta/classes/nopackages.bbclass
>> @@ -1,12 +1,23 @@
>> -deltask do_package
>> -deltask do_package_write_rpm
>> -deltask do_package_write_ipk
>> -deltask do_package_write_deb
>> -deltask do_package_qa
>> -deltask do_packagedata
>> -deltask do_package_setscene
>> -deltask do_package_write_rpm_setscene
>> -deltask do_package_write_ipk_setscene
>> -deltask do_package_write_deb_setscene
>> -deltask do_package_qa_setscene
>> -deltask do_packagedata_setscene
>> +NO_PACKAGES_TASKS = " \
>> +    do_package \
>> +    do_package_write_rpm \
>> +    do_package_write_ipk \
>> +    do_package_write_deb \
>> +    do_package_qa \
>> +    do_packagedata \
>> +"
>> +
>> +python() {
>> +    sstatetasks = (d.getVar('SSTATETASKS') or '').split()
>> +    for task in d.getVar('NO_PACKAGES_TASKS').split():
>> +        setscene_task = '%s_setscene' % task
>> +        tasks = (task, setscene_task)
>> +        for t in tasks:
>> +            bb.build.deltask(t, d)
>> +
>> +        if task in sstatetasks:
>> +            bb.debug(2, 'Removing sstate task %s from SSTATETASKS' %
>> task)
>> +            sstatetasks.remove(task)
>> +    d.setVar('SSTATETASKS', ' '.join(sstatetasks))
>> +
>> +}
> 
> I really don't want to turn more comparatively readable classes into a
> mess of anonymous python if we can help it.
> 
> Can we not just teach cleansstate to check if a task exists before
> trying to delete the sstate? It should be possible with
> d.getVarFlag(taskname, "task")?

Thanks, yes, this is my first thought, but I only knew __BBTASKS which is an internal var of bitbake, I will use d.getVarFlag(taskname, "task") to send a v2.

// Robert


> 
> Cheers,
> 
> Richard
>