[OE-core] [PATCH 1/2] kernel: stop using -exec rm for deleting files

[Bruce Ashfield]

On Tue, Feb 4, 2014 at 8:36 PM, Paul Barker <paul at paulbarker.me.uk> wrote:
> On 5 February 2014 00:35, Bruce Ashfield <bruce.ashfield at gmail.com> wrote:
>> On Tue, Feb 4, 2014 at 6:26 PM, Richard Purdie
>> <richard.purdie at linuxfoundation.org> wrote:
>>> On Tue, 2014-02-04 at 12:34 -0500, Bruce Ashfield wrote:
>>>> Removing files from the source tree via find, exec and rm is not the
>>>> most efficient operation, due to (among other things) the many forked
>>>> processes.
>>>>
>>>> If we use -delete, it saves a significant amount of time. But -delete
>>>> does not work with -prune (since it forces -depth). To maintain the
>>>> lib, tools and scripts source files, we can hide them temporarily,
>>>> skip their hidden directories and then finally restore them.
>>>>
>>>> Time for install before this change:
>>>>
>>>>  real    2m48.563s
>>>>  user    0m35.220s
>>>>  sys     0m33.036s
>>>>
>>>> Time for install after this change:
>>>>
>>>>  real    1m21.301s
>>>>  user    0m33.160s
>>>>  sys     0m28.388s
>>>>
>>>> We could further speed this up by using inline perl to delete the files,
>>>> but that complexity is avoided for now.
>>>>
>>>> Signed-off-by: Bruce Ashfield <bruce.ashfield at windriver.com>
>>>> ---
>>>>  meta/classes/kernel.bbclass | 18 +++++++++++++++---
>>>>  1 file changed, 15 insertions(+), 3 deletions(-)
>>>>
>>>> diff --git a/meta/classes/kernel.bbclass b/meta/classes/kernel.bbclass
>>>> index 51626b03f824..b76a65699755 100644
>>>> --- a/meta/classes/kernel.bbclass
>>>> +++ b/meta/classes/kernel.bbclass
>>>> @@ -260,9 +260,21 @@ kernel_do_install() {
>>>>       # we clean the scripts dir while leaving the generated config
>>>>       # and include files.
>>>>       #
>>>> -     oe_runmake -C $kerneldir CC="${KERNEL_CC}" LD="${KERNEL_LD}" clean
>>>> -     make -C $kerneldir _mrproper_scripts
>>>> -     find $kerneldir -path $kerneldir/lib -prune -o -path $kerneldir/tools -prune -o -path $kerneldir/scripts -prune -o -name "*.[csS]" -exec rm '{}' \;
>>>
>>> If we wanted to keep the single expression like this, we could do
>>> something like:
>>>
>>> find $kerneldir -path $kerneldir/lib -prune -o -path $kerneldir/tools -prune -o -path $kerneldir/scripts -prune -o -name "*.[csS]" -print0 | xargs -0 rm -f
>>>
>>> ?
>>>
>>> I don't have strong feelings about it, I'm just curious if you tried it
>>> really. The xargs should batch things up enough that the fork overhead
>>> is much less.
>>
>> I did try xargs, and a few other things in between. I had a 7 hour plane flight
>> to look into this (and packaging overhead .. that's next) and I tried
>> a few other
>> techniques. As I hinted in my commit message, the fastest is to actually
>> locate the files with perl and calling unlink() directly .. but that's
>> even more of
>> a departure from what we had before :)
>>
>> find, prune and friends get so unreadable, that when I found the faster way, I
>> tossed them in the bin and didn't look back!
>>
>
> How does """find $kerneldir -not -regex
> $kerneldir'/\(lib\|tools\|scripts)/.*' -type f -name "*.[csS]"
> -delete""" stack up?

timewise it is similar, but I could have cooked that up myself .. I just don't
like incantations of find and its arcane options.

Bruce

>
> --
> Paul Barker
>
> Email: paul at paulbarker.me.uk
> http://www.paulbarker.me.uk



-- 
"Thou shalt not follow the NULL pointer, for chaos and madness await
thee at its end"