[OE-core] [PATCH 1/1] package_manager.py: fix for "Argument list too long"

[Robert Yang]

Hi Christopher,

On 10/16/2016 12:26 PM, Christopher Larson wrote:
>
> On Fri, Oct 14, 2016 at 3:18 AM, Robert Yang <liezhi.yang at windriver.com
> <mailto:liezhi.yang at windriver.com>> wrote:
>
>     On 10/14/2016 05:58 PM, Robert Yang wrote:
>
>
>
>         On 10/13/2016 06:39 PM, Burton, Ross wrote:
>
>
>             On 13 October 2016 at 10:45, Robert Yang <liezhi.yang at windriver.com
>             <mailto:liezhi.yang at windriver.com>
>             <mailto:liezhi.yang at windriver.com
>             <mailto:liezhi.yang at windriver.com>>> wrote:
>
>                 This is because "copyhardlinktree(deploy_arch_dir,
>             arch_channel)" does:
>                 "cp -afl deploy_arch_dir/* arch_channel", while the
>             deploy_arch_dir/* is
>                 expanded to "deploy_arch_dir/pkg1 deploy_arch_dir/pkg2
>                 deploy_arch_dir/pkg3 ..." which causes the "Argument list too long",
>                 change cwd to deploy_arch_dir can avoid the error.
>
>
>             Would it be better to change the implementation of copyhardlinktree
>             so that it
>             *can't* have this problem?
>
>
>         Good idea, thanks, updated in the repo:
>
>           git://git.openembedded.org/openembedded-core-contrib
>         <http://git.openembedded.org/openembedded-core-contrib> rbt/long
>
>         http://cgit.openembedded.org/cgit.cgi/openembedded-core-contrib/log/?h=rbt/long
>         <http://cgit.openembedded.org/cgit.cgi/openembedded-core-contrib/log/?h=rbt/long>
>
>
>     Sorry, this patch is not what I wanted to paste, it should be:
>
>     diff --git a/meta/lib/oe/path.py b/meta/lib/oe/path.py
>     index 06a5af2..1d9cca5 100644
>     --- a/meta/lib/oe/path.py
>     +++ b/meta/lib/oe/path.py
>     @@ -80,15 +80,20 @@ def copyhardlinktree(src, dst):
>              cmd = "cd %s; find . -type d -print | tar --xattrs
>     --xattrs-include='*' -cf - -C %s -p --no-recursion --files-from - | tar
>     --xattrs --xattrs-include='*' -xf - -C %s" % (src, src, dst)
>              subprocess.check_output(cmd, shell=True, stderr=subprocess.STDOUT)
>              source = ''
>     +        oldcwd = os.getcwd()
>              if os.path.isdir(src):
>     +            dst = os.path.realpath(dst)
>     +            # chdir() to src to avoid "Argument list too long" error
>     +            os.chdir(src)
>                  import glob
>     -            if len(glob.glob('%s/.??*' % src)) > 0:
>     -                source = '%s/.??* ' % src
>     -            source = source + '%s/*' % src
>     +            if len(glob.glob('./.??*')) > 0:
>     +                source = './.??* '
>     +            source += './*'
>              else:
>                  source = src
>              cmd = 'cp -afl --preserve=xattr %s %s' % (source, dst)
>              subprocess.check_output(cmd, shell=True, stderr=subprocess.STDOUT)
>     +        os.chdir(oldcwd)
>
>
> Question, why not just pass cwd=src in the check_output() call, rather than
> changing it in the parent python process?

src can be a file or directory, so we can't simply use cwd=src, I will use
os.getcwd() as cwd when it is not a directory. Updated patch in the repo.

   git://git.openembedded.org/openembedded-core-contrib rbt/long

Author: Robert Yang <liezhi.yang at windriver.com>
Date:   Thu Oct 13 01:28:41 2016 -0700

     oe/path.py: fix for "Argument list too long"

     Fixed when len(TMPDIR) = 410:
     $ bitbake core-image-sato-sdk
     [snip]
     Subprocess output:
     /bin/sh: /bin/cp: Argument list too long

     ERROR: core-image-sato-sdk-1.0-r0 do_rootfs: Function failed: do_rootfs
     [snip]

     This is because "copyhardlinktree(src, dst)" does "cp -afl src/* dst",
     while src/* is expanded to "src/file1 src/file2, src/file3..." which
     causes the "Argument list too long", use ./* as src and change cwd in
     subprocess.check_output() to fix the problem.

     Signed-off-by: Robert Yang <liezhi.yang at windriver.com>

diff --git a/meta/lib/oe/path.py b/meta/lib/oe/path.py
index 06a5af2..ed7fd1e 100644
--- a/meta/lib/oe/path.py
+++ b/meta/lib/oe/path.py
@@ -83,12 +83,14 @@ def copyhardlinktree(src, dst):
          if os.path.isdir(src):
              import glob
              if len(glob.glob('%s/.??*' % src)) > 0:
-                source = '%s/.??* ' % src
-            source = source + '%s/*' % src
+                source = './.??* '
+            source += './*'
+            s_dir = src
          else:
              source = src
-        cmd = 'cp -afl --preserve=xattr %s %s' % (source, dst)
-        subprocess.check_output(cmd, shell=True, stderr=subprocess.STDOUT)
+            s_dir = os.getcwd()
+        cmd = 'cp -afl --preserve=xattr %s %s' % (source, os.path.realpath(dst))
+        subprocess.check_output(cmd, shell=True, cwd=s_dir, 
stderr=subprocess.STDOUT)
      else:
          copytree(src, dst)


// Robert

> --
> Christopher Larson
> clarson at kergoth dot com
> Founder - BitBake, OpenEmbedded, OpenZaurus
> Maintainer - Tslib
> Senior Software Engineer, Mentor Graphics