[OE-core] OSError: [Errno 7] Argument list too long in sstate_install

[Martin Jansa]

When building bigger world-image I always get this error from
populate_lic. There is debug output to show what copytree/Popen/execute_child 
was calling and it looks OK to me.

Could it be caused by pseudo or something like that? do_populate_lic works fine in other recipes.

Running the same cmdline in shell works fine and running it from small .py file also works fine:
$ PYTHONPATH="openembedded-core/meta/lib/:bitbake/lib:bitbake/lib/bb" ./copytree.test.py
copytree cmd 'tar -cf - -C /home/jenkins/oe/shr-core-branches/shr-core/tmp-eglibc/work/qemuarm-oe-linux-gnueabi/world-image/1.0-r0/license-destdir -ps . | tar -xf - -C /home/jenkins/oe/shr-core-branches/shr-core/tmp-eglibc/deploy/licenses'
Calling Popen '('tar -cf - -C /home/jenkins/oe/shr-core-branches/shr-core/tmp-eglibc/work/qemuarm-oe-linux-gnueabi/world-image/1.0-r0/license-destdir -ps . | tar -xf - -C /home/jenkins/oe/shr-core-branches/shr-core/tmp-eglibc/deploy/licenses',)' '{'shell': True, 'stderr': -2}'
execute_child(args, executable, preexec_fn, close_fds,
cwd, env, universal_newlines,
startupinfo, creationflags, shell,
p2cread, p2cwrite,
c2pread, c2pwrite,
errread, errwrite)
execute_child(tar -cf - -C /home/jenkins/oe/shr-core-branches/shr-core/tmp-eglibc/work/qemuarm-oe-linux-gnueabi/world-image/1.0-r0/license-destdir -ps . | tar -xf - -C /home/jenkins/oe/shr-core-branches/shr-core/tmp-eglibc/deploy/licenses, None, None, False,
None, None, False,
None, 0,
True,
None, None,
3, 4,
None, 4)

$ cat copytree.test.py 
#!/usr/bin/env python
import bb
import oe.path
src='/home/jenkins/oe/shr-core-branches/shr-core/tmp-eglibc/work/qemuarm-oe-linux-gnueabi/world-image/1.0-r0/license-destdir'
dst='/home/jenkins/oe/shr-core-branches/shr-core/tmp-eglibc/deploy/licenses'
oe.path.copytree(src, dst)

src has only few files, dst has a lot of directories:

$ find /home/jenkins/oe/shr-core-branches/shr-core/tmp-eglibc/work/qemuarm-oe-linux-gnueabi/world-image/1.0-r0/license-destdir
/home/jenkins/oe/shr-core-branches/shr-core/tmp-eglibc/work/qemuarm-oe-linux-gnueabi/world-image/1.0-r0/license-destdir
/home/jenkins/oe/shr-core-branches/shr-core/tmp-eglibc/work/qemuarm-oe-linux-gnueabi/world-image/1.0-r0/license-destdir/world-image
/home/jenkins/oe/shr-core-branches/shr-core/tmp-eglibc/work/qemuarm-oe-linux-gnueabi/world-image/1.0-r0/license-destdir/world-image/generic_MIT
/home/jenkins/oe/shr-core-branches/shr-core/tmp-eglibc/work/qemuarm-oe-linux-gnueabi/world-image/1.0-r0/license-destdir/world-image/COPYING.MIT
/home/jenkins/oe/shr-core-branches/shr-core/tmp-eglibc/work/qemuarm-oe-linux-gnueabi/world-image/1.0-r0/license-destdir/world-image/LICENSE

$ ls /home/jenkins/oe/shr-core-branches/shr-core/tmp-eglibc/deploy/licenses -1 | wc -l
1696

Complete error as shown in work/qemuarm-oe-linux-gnueabi/world-image/1.0-r0/temp/log.do_populate_lic:

DEBUG: Executing python function sstate_task_prefunc
DEBUG: Removing manifest: /home/jenkins/oe/shr-core-branches/shr-core/tmp-eglibc/deploy/licenses/world-image/
DEBUG: Python function sstate_task_prefunc finished
DEBUG: Executing python function do_populate_lic
DEBUG: Python function do_populate_lic finished
DEBUG: Executing python function sstate_task_postfunc
DEBUG: Staging files from /home/jenkins/oe/shr-core-branches/shr-core/tmp-eglibc/work/qemuarm-oe-linux-gnueabi/world-image/1.0-r0/license-destdir to /home/jenkins/oe/shr-core-branches/shr-core/tmp-eglibc/deploy/licenses
DEBUG: Staging files from /home/jenkins/oe/shr-core-branches/shr-core/tmp-eglibc/work/qemuarm-oe-linux-gnueabi/world-image/1.0-r0/license-destdir to /home/jenkins/oe/shr-core-branches/shr-core/tmp-eglibc/deploy/licenses
copytree cmd 'tar -cf - -C /home/jenkins/oe/shr-core-branches/shr-core/tmp-eglibc/work/qemuarm-oe-linux-gnueabi/world-image/1.0-r0/license-destdir -ps . | tar -xf - -C /home/jenkins/oe/shr-core-branches/shr-core/tmp-eglibc/deploy/licenses'
Calling Popen '('tar -cf - -C /home/jenkins/oe/shr-core-branches/shr-core/tmp-eglibc/work/qemuarm-oe-linux-gnueabi/world-image/1.0-r0/license-destdir -ps . | tar -xf - -C /home/jenkins/oe/shr-core-branches/shr-core/tmp-eglibc/deploy/licenses',)' '{'shell': True, 'stderr': -2}'
execute_child(args, executable, preexec_fn, close_fds,
cwd, env, universal_newlines,
startupinfo, creationflags, shell,
p2cread, p2cwrite,
c2pread, c2pwrite,
errread, errwrite)
execute_child(tar -cf - -C /home/jenkins/oe/shr-core-branches/shr-core/tmp-eglibc/work/qemuarm-oe-linux-gnueabi/world-image/1.0-r0/license-destdir -ps . | tar -xf - -C /home/jenkins/oe/shr-core-branches/shr-core/tmp-eglibc/deploy/licenses, None, None, False,
None, None, False,
None, 0,
True,
None, None,
20, 21,
None, 21)
ERROR: Error executing a python function in /home/jenkins/oe/shr-core-branches/shr-core/openembedded-core/meta/recipes-core/images/world-image.bb:
OSError: [Errno 7] Argument list too long

ERROR: The stack trace of python calls that resulted in this exception/failure was:
ERROR:   File "sstate_task_postfunc", line 12, in <module>
ERROR:
ERROR:   File "sstate_task_postfunc", line 4, in sstate_task_postfunc
ERROR:
ERROR:   File "sstate.bbclass", line 76, in sstate_install
ERROR:
ERROR:   File "/home/jenkins/oe/shr-core-branches/shr-core/openembedded-core/meta/lib/oe/path.py", line 85, in copytree
ERROR:     check_output(cmd, shell=True, stderr=subprocess.STDOUT)
ERROR:
ERROR:   File "/home/jenkins/oe/shr-core-branches/shr-core/openembedded-core/meta/lib/oe/path.py", line 141, in check_output
ERROR:     process = subprocess.Popen(stdout=subprocess.PIPE, *popenargs, **kwargs)
ERROR:
ERROR:   File "/usr/lib/python2.7/subprocess.py", line 679, in __init__
ERROR:     errread, errwrite)
ERROR:
ERROR:   File "/usr/lib/python2.7/subprocess.py", line 1249, in _execute_child
ERROR:     raise child_exception
ERROR:
ERROR: The code that was being executed was:
ERROR:      0008:            bb.build.exec_func(intercept, d)
ERROR:      0009:        sstate_package(shared_state, d)
ERROR:      0010:
ERROR:      0011:
ERROR:  *** 0012:sstate_task_postfunc(d)
ERROR:      0013:
ERROR: [From file: 'sstate_task_postfunc', lineno: 12, function: <module>]
ERROR:      0001:
ERROR:      0002:def sstate_task_postfunc(d):
ERROR:      0003:    shared_state = sstate_state_fromvars(d)
ERROR:  *** 0004:    sstate_install(shared_state, d)
ERROR:      0005:    enabled = d.getVar('SSTATE_CREATE_PKG', True)
ERROR:      0006:    if enabled and enabled == "1":
ERROR:      0007:        for intercept in shared_state['interceptfuncs']:
ERROR:      0008:            bb.build.exec_func(intercept, d)
ERROR: [From file: 'sstate_task_postfunc', lineno: 4, function: sstate_task_postfunc]
ERROR:      0072:    f.close()
ERROR:      0073:
ERROR:      0074:    # Run the actual file install
ERROR:      0075:    for state in ss['dirs']:
ERROR:  *** 0076:        oe.path.copytree(state[1], state[2])
ERROR:      0077:
ERROR:      0078:    for postinst in (d.getVar('SSTATEPOSTINSTFUNCS', True) or '').split():
ERROR:      0079:        bb.build.exec_func(postinst, d)
ERROR:      0080:
ERROR: [From file: 'sstate.bbclass', lineno: 76, function: sstate_install]
DEBUG: Python function sstate_task_postfunc finished
ERROR: Function failed: sstate_task_postfunc


-- 
Martin 'JaMa' Jansa     jabber: Martin.Jansa at gmail.com
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 198 bytes
Desc: Digital signature
URL: <http://lists.openembedded.org/pipermail/openembedded-core/attachments/20130129/7d374df9/attachment-0002.sig>