[OE-core] [PATCH] rpm: run binary package generation via thread pools

[Alexander Kanavin]

On 06/13/2017 07:15 PM, Leonardo Sandoval wrote:
> On Mon, 2017-06-12 at 14:29 -0500, Leonardo Sandoval wrote:
>> On Mon, 2017-06-12 at 17:58 +0300, Alexander Kanavin wrote:
>>> This greatly reduces build times when there is a large amount of small
>>> rpm packages to produce. The patches are rather invasive,
>>> and so will be submitted upstream.
>>>
>>
>> What is the buildstat value (those from /proc/[PID]) you think this
>> patch would make a considerable performance burst?
>>
>> These are the top most consuming recipes just for the task of interested
>> (do_package_write_rpm)
>>
>> without this patch:
>>
>> do_package_write_rpm glibc-locale-2.25-r0 268.87 seconds
>> do_package_write_rpm perl-5.24.1-r0 85.87 seconds
>> do_package_write_rpm python3-3.5.3-r1.0 38.01 seconds
>> do_package_write_rpm gtk+3-3.22.8-r0 30.10 seconds
>> do_package_write_rpm libxml2-2.9.4-r0 25.97 seconds
>> do_package_write_rpm glibc-2.25-r0 25.67 seconds
>> do_package_write_rpm db-1_5.3.28-r1 23.80 seconds
>> do_package_write_rpm binutils-2.28-r0 22.92 seconds
>> do_package_write_rpm util-linux-2.29.1-r0 20.86 seconds
>> do_package_write_rpm mesa-2_17.0.6-r0 20.32 seconds
>>
>> with this patch:
>>
>>
>> do_package_write_rpm perl-5.24.1-r0 68.77 seconds
>> do_package_write_rpm glibc-locale-2.25-r0 53.59 seconds
>> do_package_write_rpm python3-3.5.3-r1.0 30.88 seconds
>> do_package_write_rpm libxml2-2.9.4-r0 30.51 seconds
>> do_package_write_rpm glibc-2.25-r0 29.99 seconds
>> do_package_write_rpm glib-2.0-1_2.52.2-r0 29.24 seconds
>> do_package_write_rpm db-1_5.3.28-r1 24.47 seconds
>> do_package_write_rpm coreutils-8.27-r0 23.93 seconds
>> do_package_write_rpm gettext-0.19.8.1-r0 23.88 seconds
>> do_package_write_rpm gtk+3-3.22.15-r0 23.48 seconds
>>
>> times are not wall-times, these are times coming from the bitbake
>> scheduler but I believe this provide us some insight.
>
> Just one minor correction after briefly discussing with RP: these are
> indeed wall times (real times), which is the delta between the starting
> and finishing times for the relevant tasks.

Have you ran these all at once? If so, then the wall times aren't very 
meaningful: if the CPU is already saturated by having a lot of bitbake 
tasks running, then adding further threads to these tasks isn't going to 
make anything finish faster than before.


Alex