[OE-core] [PATCH] lib/oe/path: Fix performance issue got copyhardlinktree()

[Andrea Adami]

On Fri, Nov 8, 2013 at 4:42 PM, Bruce Ashfield
<bruce.ashfield at windriver.com> wrote:
> On 13-11-08 10:19 AM, Richard Purdie wrote:
>>
>> With the directory copy was added to avoid race issues, it wasn't noticed
>> that
>> tar was recursing the directories and copying files too. This is
>> completely
>> crazy when we hardlink those files in the next command.
>>
>> Resolve the issue by telling tar not to recurse. This gives a significant
>> performance boost to various parts of the system (do_package for
>> linux-yocto
>> 256s -> 178s for example).
>
>
> Every second makes a difference in this beast .. when adding up a
> zillion package runs a week. :)
>
> Cheers,
>
> Bruce
>
>
>>
>> Signed-off-by: Richard Purdie <richard.purdie at linuxfoundation.org>
>> ---
>> diff --git a/meta/lib/oe/path.py b/meta/lib/oe/path.py
>> index 1310e38..d0588ba 100644
>> --- a/meta/lib/oe/path.py
>> +++ b/meta/lib/oe/path.py
>> @@ -93,7 +93,7 @@ def copyhardlinktree(src, dst):
>>       if (os.stat(src).st_dev ==  os.stat(dst).st_dev):
>>           # Need to copy directories only with tar first since cp will
>> error if two
>>           # writers try and create a directory at the same time
>> -        cmd = 'cd %s; find . -type d -print | tar -cf - -C %s -p
>> --files-from - | tar -xf - -C %s' % (src, src, dst)
>> +        cmd = 'cd %s; find . -type d -print | tar -cf - -C %s -p
>> --files-from - --no-recursion | tar -xf - -C %s' % (src, src, dst)
>>           check_output(cmd, shell=True, stderr=subprocess.STDOUT)
>>           if os.path.isdir(src):
>>               src = src + "/*"
>>
>>
>
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core at lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-core

After this patch do_populate_sysroot fails:

ERROR: Function failed: kernelscripts_sstate_postinst (log file is
located at /oe/oe-core/build/tmp-eglibc/work/collie-oe-linux-gnueabi/linux-yocto/3.10.17+gitAUTOINC+f1c9080cd2_c03195ed6e-r0/temp/log.do_populate_sysroot.26363)
ERROR: Logfile of failure stored in:
/oe/oe-core/build/tmp-eglibc/work/collie-oe-linux-gnueabi/linux-yocto/3.10.17+gitAUTOINC+f1c9080cd2_c03195ed6e-r0/temp/log.do_populate_sysroot.26363
Log data follows:
| DEBUG: Executing python function sstate_task_prefunc
| DEBUG: Python function sstate_task_prefunc finished
| DEBUG: Executing python function do_populate_sysroot
| DEBUG: Executing python function sysroot_stage_all
| DEBUG: Python function sysroot_stage_all finished
| DEBUG: Python function do_populate_sysroot finished
| DEBUG: Executing python function do_qa_staging
| NOTE: QA checking staging
| DEBUG: Python function do_qa_staging finished
| DEBUG: Executing python function sstate_task_postfunc
| DEBUG: Staging files from
/oe/oe-core/build/tmp-eglibc/work/collie-oe-linux-gnueabi/linux-yocto/3.10.17+gitAUTOINC+f1c9080cd2_c03195ed6e-r0/sysroot-destdir
to /oe/oe-core/build/tmp-eglibc/sysroots/collie
| DEBUG: Executing shell function kernelscripts_sstate_postinst
| NOTE: make scripts
|   HOSTCC  scripts/basic/fixdep
|   HOSTCC  scripts/kconfig/conf.o
|   SHIPPED scripts/kconfig/zconf.tab.c
|   SHIPPED scripts/kconfig/zconf.lex.c
|   SHIPPED scripts/kconfig/zconf.hash.c
|   HOSTCC  scripts/kconfig/zconf.tab.o
|   HOSTLD  scripts/kconfig/conf
| scripts/kconfig/conf --silentoldconfig Kconfig
| ***
| *** Configuration file ".config" not found!
| ***
| *** Please run some configurator (e.g. "make oldconfig" or
| *** "make menuconfig" or "make xconfig").
| ***
| make[2]: *** [silentoldconfig] Error 1
| make[1]: *** [silentoldconfig] Error 2
| make: *** No rule to make target `include/config/auto.conf', needed
by `scripts'.  Stop.
| ERROR: oe_runmake failed
| WARNING: /oe/oe-core/build/tmp-eglibc/work/collie-oe-linux-gnueabi/linux-yocto/3.10.17+gitAUTOINC+f1c9080cd2_c03195ed6e-r0/temp/run.kernelscripts_sstate_postinst.26363:1
exit 1 from
|   [ "populate_sysroot" = "populate_sysroot" -o "populate_sysroot" =
"populate_sysroot_setscene" ]
| DEBUG: Python function sstate_task_postfunc finished
| ERROR: Function failed: kernelscripts_sstate_postinst (log file is
located at /oe/oe-core/build/tmp-eglibc/work/collie-oe-linux-gnueabi/linux-yocto/3.10.17+gitAUTOINC+f1c9080cd2_c03195ed6e-r0/temp/log.do_populate_sysroot.26363)
ERROR: Task 5 (/oe/oe-core/meta/recipes-kernel/linux/linux-yocto_3.10.bb,
do_populate_sysroot) failed with exit code '1'


Cheers

Andrea