[OE-core] [PATCH] lib/oe/path: Fix performance issue got copyhardlinktree()

[Andrea Adami]

On Sat, Nov 9, 2013 at 11:52 PM, Richard Purdie
<richard.purdie at linuxfoundation.org> wrote:
> On Sat, 2013-11-09 at 21:43 +0100, Andrea Adami wrote:
>> On Fri, Nov 8, 2013 at 4:42 PM, Bruce Ashfield
>> <bruce.ashfield at windriver.com> wrote:
>> > On 13-11-08 10:19 AM, Richard Purdie wrote:
>> >>
>> >> With the directory copy was added to avoid race issues, it wasn't noticed
>> >> that
>> >> tar was recursing the directories and copying files too. This is
>> >> completely
>> >> crazy when we hardlink those files in the next command.
>> >>
>> >> Resolve the issue by telling tar not to recurse. This gives a significant
>> >> performance boost to various parts of the system (do_package for
>> >> linux-yocto
>> >> 256s -> 178s for example).
>> >
>> >
>> > Every second makes a difference in this beast .. when adding up a
>> > zillion package runs a week. :)
>> >
>> > Cheers,
>> >
>> > Bruce
>> >
>> >
>> >>
>> >> Signed-off-by: Richard Purdie <richard.purdie at linuxfoundation.org>
>> >> ---
>> >> diff --git a/meta/lib/oe/path.py b/meta/lib/oe/path.py
>> >> index 1310e38..d0588ba 100644
>> >> --- a/meta/lib/oe/path.py
>> >> +++ b/meta/lib/oe/path.py
>> >> @@ -93,7 +93,7 @@ def copyhardlinktree(src, dst):
>> >>       if (os.stat(src).st_dev ==  os.stat(dst).st_dev):
>> >>           # Need to copy directories only with tar first since cp will
>> >> error if two
>> >>           # writers try and create a directory at the same time
>> >> -        cmd = 'cd %s; find . -type d -print | tar -cf - -C %s -p
>> >> --files-from - | tar -xf - -C %s' % (src, src, dst)
>> >> +        cmd = 'cd %s; find . -type d -print | tar -cf - -C %s -p
>> >> --files-from - --no-recursion | tar -xf - -C %s' % (src, src, dst)
>> >>           check_output(cmd, shell=True, stderr=subprocess.STDOUT)
>> >>           if os.path.isdir(src):
>> >>               src = src + "/*"
>> >>
>> >>
>> >
>> > _______________________________________________
>> > Openembedded-core mailing list
>> > Openembedded-core at lists.openembedded.org
>> > http://lists.openembedded.org/mailman/listinfo/openembedded-core
>>
>> After this patch do_populate_sysroot fails:
>
> This should fix it:
>
> http://git.yoctoproject.org/cgit.cgi/poky-contrib/commit/?h=rpurdie/t2&id=5237c8e2c1780983c000247ffa0c890b4bb5980a
>
> Cheers,
>
> Richard
>

Yes, thx for the magic fix.
Now with all the last patches my results are much better:

do_install:
Elapsed time: 83.77 seconds CPU usage: 3.0% (worst before 131s)
(avg before:103 seconds CPU usage: 2.9%)

do_populate_sysroot:
Elapsed time: 16.12 seconds CPU usage: 18.5%
(avg before: 95 seconds CPU usage: 6.5%)

Cheers

Andrea