[OE-core] [PATCH 1/2] kernel: stop using -exec rm for deleting files

[Paul Barker]

On 5 February 2014 00:35, Bruce Ashfield <bruce.ashfield at gmail.com> wrote:
> On Tue, Feb 4, 2014 at 6:26 PM, Richard Purdie
> <richard.purdie at linuxfoundation.org> wrote:
>> On Tue, 2014-02-04 at 12:34 -0500, Bruce Ashfield wrote:
>>> Removing files from the source tree via find, exec and rm is not the
>>> most efficient operation, due to (among other things) the many forked
>>> processes.
>>>
>>> If we use -delete, it saves a significant amount of time. But -delete
>>> does not work with -prune (since it forces -depth). To maintain the
>>> lib, tools and scripts source files, we can hide them temporarily,
>>> skip their hidden directories and then finally restore them.
>>>
>>> Time for install before this change:
>>>
>>>  real    2m48.563s
>>>  user    0m35.220s
>>>  sys     0m33.036s
>>>
>>> Time for install after this change:
>>>
>>>  real    1m21.301s
>>>  user    0m33.160s
>>>  sys     0m28.388s
>>>
>>> We could further speed this up by using inline perl to delete the files,
>>> but that complexity is avoided for now.
>>>
>>> Signed-off-by: Bruce Ashfield <bruce.ashfield at windriver.com>
>>> ---
>>>  meta/classes/kernel.bbclass | 18 +++++++++++++++---
>>>  1 file changed, 15 insertions(+), 3 deletions(-)
>>>
>>> diff --git a/meta/classes/kernel.bbclass b/meta/classes/kernel.bbclass
>>> index 51626b03f824..b76a65699755 100644
>>> --- a/meta/classes/kernel.bbclass
>>> +++ b/meta/classes/kernel.bbclass
>>> @@ -260,9 +260,21 @@ kernel_do_install() {
>>>       # we clean the scripts dir while leaving the generated config
>>>       # and include files.
>>>       #
>>> -     oe_runmake -C $kerneldir CC="${KERNEL_CC}" LD="${KERNEL_LD}" clean
>>> -     make -C $kerneldir _mrproper_scripts
>>> -     find $kerneldir -path $kerneldir/lib -prune -o -path $kerneldir/tools -prune -o -path $kerneldir/scripts -prune -o -name "*.[csS]" -exec rm '{}' \;
>>
>> If we wanted to keep the single expression like this, we could do
>> something like:
>>
>> find $kerneldir -path $kerneldir/lib -prune -o -path $kerneldir/tools -prune -o -path $kerneldir/scripts -prune -o -name "*.[csS]" -print0 | xargs -0 rm -f
>>
>> ?
>>
>> I don't have strong feelings about it, I'm just curious if you tried it
>> really. The xargs should batch things up enough that the fork overhead
>> is much less.
>
> I did try xargs, and a few other things in between. I had a 7 hour plane flight
> to look into this (and packaging overhead .. that's next) and I tried
> a few other
> techniques. As I hinted in my commit message, the fastest is to actually
> locate the files with perl and calling unlink() directly .. but that's
> even more of
> a departure from what we had before :)
>
> find, prune and friends get so unreadable, that when I found the faster way, I
> tossed them in the bin and didn't look back!
>

How does """find $kerneldir -not -regex
$kerneldir'/\(lib\|tools\|scripts)/.*' -type f -name "*.[csS]"
-delete""" stack up?

-- 
Paul Barker

Email: paul at paulbarker.me.uk
http://www.paulbarker.me.uk