[Openembedded-users] Ar file handling.

Wed Mar 31 06:17:42 UTC 2010

I ran into problem that is similar to one in the list archives from Josef
Wolf.  Here is what I found as the root cause. Still looking for a good way
to fix it.

This issue appears when the user's UID or GID has 6 or more digits, or >
99999.  In my case, some people built without error, where others failed
with a ValueError. For people on a corporate NIS, its common for UIDs to
exceed 99999.

The ipkg/debian package format uses 'ar' archives. The 'ar' header uses a 5
character ASCII digit (plus null termination) to store the UID and GID and
gets overrun when > 99999.  When overrun, to arfile.py it looks like two of
the header fields have merged and descriptor[5] returns the next field of
'`' which is an invalid int.

struct ar_hdr in /usr/include/ar.h:
 char ar_name[16];           /* Member file name, sometimes / terminated. */
 char ar_uid[6], ar_gid[6];  /* User and group IDs, in ASCII decimal.  */

I would also be worried about overflowing ar_name[16], which would be seen
as the same ValueError symptom.

+ ipkg-make-index -r /x/y/z/tmp/deploy/ipk/all/Packages -p
/x/y/z/tmp/deploy/ipk/all/Packages -l
/x/y/z/tmp/deploy/ipk/all/Packages.filelist -m /x/y/z/tmp/deploy/ipk/all/
/x/y/z/tmp/staging/x86_64-linux/usr/bin/ipkg.py:39: DeprecationWarning: the
md5 module is deprecated; use hashlib instead
  import md5
Traceback (most recent call last):
  File "/x/y/z/tmp/staging/x86_64-linux/usr/bin/ipkg-make-index", line 114,
in <module>
    pkg = ipkg.Package(filename)
  File "/x/y/z/tmp/staging/x86_64-linux/usr/bin/ipkg.py", line 161, in
__init__
    tarStream = ar.open("control.tar.gz")
  File "/x/y/z/tmp/staging/x86_64-linux/usr/bin/arfile.py", line 59, in open
    f = self._scan(fname)
  File "/x/y/z/tmp/staging/x86_64-linux/usr/bin/arfile.py", line 79, in
_scan
    size = int(descriptor[5])
ValueError: invalid literal for int() with base 10: '`'

Possible Solution:
Hard code the uid/gid to a constant one. I dont see where the uid/gid in the
archive header is actually used.

-Blaine

Wolf, Josef wrote:

> On suse-11.0, OE fails to build images because ar file
> extraction fails with "AR member not found: control.tar.gz"
> error message.  I have found a fix for this problem in the
> mailing list archives (attached below).
> It looks like this fix has not made its way into the stable
> branch yet.
> Is this by intent (maybe this should be fixed upstream?) or
> was it simply forgotten?

Since there were no replies to this topic so far, I have
decided to turn this into a little exercise of how to
create my own overlay of the official OE tree.

So I have created my own hierarchy and put my patch into it:

   $ cat local/packages/ipkg-utils/ipkg-utils/ipkg-py-arfile.patch
   --- ipkg-utils/arfile.py.orig        2008-09-17 12:00:57.000000000
+0200
   +++ ipkg-utils/arfile.py     2008-09-17 12:01:13.000000000 +0200
   @@ -86,6 +86,8 @@
                    return FileSection(self.f, self.f.tell(), size)

                # Skip data and loop
   +            if size % 2:
   +                size = size + 1
                data = self.f.seek(size, 1)
    #            print hex(f.tell())

Then I copied the original bitbake file to my overlay and added my
patch to the file list:

   $ diff -u
org.openembedded.stable/packages/ipkg-utils/ipkg-utils_1.6+cvs20050404.b
b \

local/packages/ipkg-utils/ipkg-utils_1.6+cvs20050404.bb

   ---
org.openembedded.stable/packages/ipkg-utils/ipkg-utils_1.6+cvs20050404.b
b       2008-09-16 12:22:35.000000000 +0200
   +++ local/packages/ipkg-utils/ipkg-utils_1.6+cvs20050404.bb
2008-09-17 11:54:14.000000000 +0200
   @@ -11,6 +11,7 @@
               file://ipkg-utils-fix.patch;patch=1 \
               file://ipkg-py-sane-vercompare.patch;patch=1 \
               file://ipkg-py-tarfile.patch;patch=1 \
   +           file://ipkg-py-arfile.patch;patch=1 \
               file://ipkg-make-index-track-stamps.patch;patch=1 \
               file://fields_tweaks.patch;patch=1 "

Then I rebuild.  But now it fails with

   NOTE: package helloworld-image-1.0-r0: task do_rootfs: started
   ERROR: function do_rootfs failed
   ERROR: log data follows
(/m/a/tmp/builds/openembedded1/projects/1sbt/tmp/work/at91sam9260ek-angs
trom-linux-gnueabi/helloworld-image-1.0-r0/temp/log.do_rootfs.5744)
   | + rm -rf /m/a/tmp/builds/openembedded1/projects/1sbt/tmp/rootfs
   | + mkdir -p /m/a/tmp/builds/openembedded1/projects/1sbt/tmp/rootfs
   | + '[' 0 '!=' 1 ']'
   | + for devtable in
/m/a/tmp/builds/openembedded1/org.openembedded.stable/files/device_table
-minimal.txt
   | + makedevs -r
/m/a/tmp/builds/openembedded1/projects/1sbt/tmp/rootfs -D
/m/a/tmp/builds/openembedded1/org.openembedded.stable/files/device_table
-minimal.txt
   | + rootfs_ipk_do_rootfs
   | + set -x
   | + package_update_index_ipk
   | + set -x
   | + ipkgarchs='all any noarch arm armv4t armv5te at91sam9260ek'
   | + '[' '!' -z '' ']'
   | + touch
/m/a/tmp/builds/openembedded1/projects/1sbt/tmp/deploy/glibc/ipk/Package
s
   | + ipkg-make-index -r
/m/a/tmp/builds/openembedded1/projects/1sbt/tmp/deploy/glibc/ipk/Package
s -p
/m/a/tmp/builds/openembedded1/projects/1sbt/tmp/deploy/glibc/ipk/Package
s -l
/m/a/tmp/builds/openembedded1/projects/1sbt/tmp/deploy/glibc/ipk/Package
s.filelist -m
/m/a/tmp/builds/openembedded1/projects/1sbt/tmp/deploy/glibc/ipk
   |
   | + for arch in '$ipkgarchs'
   | + '[' -e
/m/a/tmp/builds/openembedded1/projects/1sbt/tmp/deploy/glibc/ipk/all/
']'
   | + '[' -e
/m/a/tmp/builds/openembedded1/projects/1sbt/tmp/deploy/glibc/ipk/i686-al
l-sdk/ ']'
   | + for arch in '$ipkgarchs'
   | + '[' -e
/m/a/tmp/builds/openembedded1/projects/1sbt/tmp/deploy/glibc/ipk/any/
']'
   | + '[' -e
/m/a/tmp/builds/openembedded1/projects/1sbt/tmp/deploy/glibc/ipk/i686-an
y-sdk/ ']'
   | + for arch in '$ipkgarchs'
   | + '[' -e
/m/a/tmp/builds/openembedded1/projects/1sbt/tmp/deploy/glibc/ipk/noarch/
']'
   | + '[' -e
/m/a/tmp/builds/openembedded1/projects/1sbt/tmp/deploy/glibc/ipk/i686-no
arch-sdk/ ']'
   | + for arch in '$ipkgarchs'
   | + '[' -e
/m/a/tmp/builds/openembedded1/projects/1sbt/tmp/deploy/glibc/ipk/arm/
']'
   | + '[' -e
/m/a/tmp/builds/openembedded1/projects/1sbt/tmp/deploy/glibc/ipk/i686-ar
m-sdk/ ']'
   | + for arch in '$ipkgarchs'
   | + '[' -e
/m/a/tmp/builds/openembedded1/projects/1sbt/tmp/deploy/glibc/ipk/armv4t/
']'
   | + '[' -e
/m/a/tmp/builds/openembedded1/projects/1sbt/tmp/deploy/glibc/ipk/i686-ar
mv4t-sdk/ ']'
   | + for arch in '$ipkgarchs'
   | + '[' -e
/m/a/tmp/builds/openembedded1/projects/1sbt/tmp/deploy/glibc/ipk/armv5te
/ ']'
   | + touch
/m/a/tmp/builds/openembedded1/projects/1sbt/tmp/deploy/glibc/ipk/armv5te
/Packages
   | + ipkg-make-index -r
/m/a/tmp/builds/openembedded1/projects/1sbt/tmp/deploy/glibc/ipk/armv5te
/Packages -p
/m/a/tmp/builds/openembedded1/projects/1sbt/tmp/deploy/glibc/ipk/armv5te
/Packages -l
/m/a/tmp/builds/openembedded1/projects/1sbt/tmp/deploy/glibc/ipk/armv5te
/Packages.filelist -m
/m/a/tmp/builds/openembedded1/projects/1sbt/tmp/deploy/glibc/ipk/armv5te
/
   | debian-binary/  1221654949  1000  1000  100644  4         `
   |
   | data.tar.gz/    1221654949  1000  1000  100644  1805      `
   |
   | `control.tar.gz/ 1221654949  1000  1000  100644  587       `
   |
   | `
   | Traceback (most recent call last):
   |   File
"/m/a/tmp/builds/openembedded1/projects/1sbt/tmp/staging/i686-linux/bin/
ipkg-make-index", line 114, in <module>
   |     pkg = ipkg.Package(filename)
   |   File
"/m/a/tmp/builds/openembedded1/projects/1sbt/tmp/staging/i686-linux/bin/
ipkg.py", line 162, in __init__
   |     tarStream = ar.open("control.tar.gz")
   |   File
"/m/a/tmp/builds/openembedded1/projects/1sbt/tmp/staging/i686-linux/bin/
arfile.py", line 59, in open
   |     f = self._scan(fname)
   |   File
"/m/a/tmp/builds/openembedded1/projects/1sbt/tmp/staging/i686-linux/bin/
arfile.py", line 80, in _scan
   |     size = int(descriptor[5])
   | IndexError: list index out of range
   NOTE: Task failed:
/m/a/tmp/builds/openembedded1/projects/1sbt/tmp/work/at91sam9260ek-angst
rom-linux-gnueabi/helloworld-image-1.0-r0/temp/log.do_rootfs.5744
   NOTE: package helloworld-image-1.0-r0: task do_rootfs: failed
   ERROR: TaskFailed event exception, aborting
   NOTE: package helloworld-image-1.0: failed
   ERROR: Build of
/m/a/tmp/builds/openembedded1/org.openembedded.stable/packages/images/he
lloworld-image.bb do_rootfs failed
   ERROR: Task 9
(/m/a/tmp/builds/openembedded1/org.openembedded.stable/packages/images/h
elloworld-image.bb, do_rootfs) failed
   NOTE: Tasks Summary: Attempted 546 tasks of which 546 didn't need to
be rerun and 1 failed.
   ERROR:
'/m/a/tmp/builds/openembedded1/org.openembedded.stable/packages/images/h
elloworld-image.bb' failed
   NOTE: build 200809171544: completed

Any ideas?