[OE-core] glibc binary reproducibility

Wed Oct 3 02:37:45 UTC 2018

On 03/10/18 05:58, Richard Purdie wrote:

> On Thu, 2018-09-27 at 12:18 +1200, Douglas Royds wrote:
>> When I build glibc in two different places, I get non-reproducible
>> results - a 4-byte difference:
>>> $ cmp -l ~/workspace/upstream[12]/build/tmp/work/armv5e-tait-linux-
>>> gnueabi/glibc/2.28-r0/package/lib/libc-2.28.so
>>> 1259181  71 172
>>> 1259182  27 304
>>> 1259183 152  77
>>> 1259184 363 243
>> These 4 bytes are the checksum that objcopy --add-gnu-debuglink adds
>> to the binary when the .debug info is split out at do_package time,
>> see package.bbclass +416
>> So why is the debug info different? We add -fdebug-prefix-map to our
>> CFLAGS, which ensures that all our intra-component debug paths are
>> prefixed with /usr/src/debug/glibc/, for instance, but this isn't
>> working in this case. The difference is happening in csu/crt1.o,
>> which is being linked into libc.so (and others):
>>> $ ../recipe-sysroot-native/usr/bin/arm-tait-linux-gnueabi/arm-tait-
>>> linux-gnueabi-objdump -t csu/crt1.o
>>> ...
>>> 00000000 l    df *ABS*    00000000
>>> /home/douglas/workspace/upstream1/build/tmp/work/armv5e-tait-linux-
>>> gnueabi/glibc/2.28-r0/build-arm-tait-linux-gnueabi/csu/abi-note.o
>>> 00000000 l    df *ABS*    00000000
>>> /home/douglas/workspace/upstream1/build/tmp/work/armv5e-tait-linux-
>>> gnueabi/glibc/2.28-r0/build-arm-tait-linux-gnueabi/csu/start.o
>>   This abi-note.o symbol is finding its way into libc.so:
>>> $ ../recipe-sysroot-native/usr/bin/arm-tait-linux-gnueabi/arm-tait-
>>> linux-gnueabi-objdump -t libc.so
>>> ...
>>> 00000000 l    df *ABS*    00000000
>>> /home/douglas/workspace/upstream1/build/tmp/work/armv5e-tait-linux-
>>> gnueabi/glibc/2.28-r0/build-arm-tait-linux-gnueabi/csu/abi-note.o

No it isn't, it turns out. The non-reproducibility of crt1.o and friends 
is a problem, as they get copied into the recipe-sysroot of all other 
components, causing the 780ish different packages observed by Juro in 
https://bugzilla.yoctoproject.org/show_bug.cgi?id=12743

>> There is a work-around: turn off the debug packaging:
>>> INHIBIT_PACKAGE_DEBUG_SPLIT_pn-glibc = "1"
>> I don't have a solution for this. Suggestions?
> I did some digging.
>
> I tried what I suggested using relative paths:
>
> http://git.yoctoproject.org/cgit.cgi/poky-contrib/commit/?h=rpurdie/t222&id=22189be4bf508851950f72654870c4eebd4b73d9
>
> and whilst it helps remove some references, there is a much wider
> problem. I therefore went and had a look at the linker itself to
> understand why its injecting this path. I found this code:
>
> http://git.yoctoproject.org/cgit.cgi/poky-contrib/commit/?h=rpurdie/t222&id=f9aca51204990fbdbdfa7442f1dcc938e97bf782
>
> which if disabled as per this hack, makes the binaries reproducible and
> drops the problematic references.
>
> We're swapping some misleading debug output for reproducibility with
> that hack.
>
> At this point we probably need to talk to some toolchain people about
> what 'real' fixes may be possible...

Excellent. Your patch in 22189be is much tidier than my own Makefile 
hack that I was too embarassed to publish on this list (it may or may 
not have involved sedding a binary file).

It turns out that we have two binary difference issues: The crt1.o (and 
friends) one discussed above, and a similar problem linking all the 
shared libs, eg. libm.so

    $
    ../recipe-sysroot-native/usr/bin/arm-tait-linux-gnueabi/arm-tait-linux-gnueabi-objdump
    -t math/libm.so | grep armv5e
    00000000 l    df *ABS*    00000000
    /home/douglas/workspace/upstream1/build/tmp/work/armv5e-tait-linux-gnueabi/glibc/2.28-r0/build-arm-tait-linux-gnueabi/csu/abi-note.o
    00000000 l    df *ABS*    00000000
    /home/douglas/workspace/upstream1/build/tmp/work/armv5e-tait-linux-gnueabi/glibc/2.28-r0/build-arm-tait-linux-gnueabi/csu/crti.o
    00000000 l    df *ABS*    00000000
    /home/douglas/workspace/upstream1/build/tmp/work/armv5e-tait-linux-gnueabi/glibc/2.28-r0/build-arm-tait-linux-gnueabi/csu/crtn.o

 From temp/log.do_compile:

    arm-tait-linux-gnueabi-gcc
         -march=armv5e -marm
         ...
    -B/home/douglas/workspace/upstream1/build/tmp/work/armv5e-tait-linux-gnueabi/glibc/2.28-r0/build-arm-tait-linux-gnueabi/csu/
         ...
         -o
    /home/douglas/workspace/upstream1/build/tmp/work/armv5e-tait-linux-gnueabi/glibc/2.28-r0/build-arm-tait-linux-gnueabi/math/libm.so
         ...
    /home/douglas/workspace/upstream1/build/tmp/work/armv5e-tait-linux-gnueabi/glibc/2.28-r0/build-arm-tait-linux-gnueabi/csu/abi-note.o
         ...

This link command ignores -fdebug-prefix-map even if we add it (ld 
doesn't accept it). The fully-qualified paths to crti.o and crtn.o come 
from the -B option. If we use relative paths to these files and to 
abi-note.o, the problem goes away:

    $ cd git/math/
    $ arm-tait-linux-gnueabi-gcc \
         -march=armv5e -marm \
         ...
         -B../../build-arm-tait-linux-gnueabi/csu/ \
         ...
         -o
    /home/douglas/workspace/upstream1/build/tmp/work/armv5e-tait-linux-gnueabi/glibc/2.28-r0/build-arm-tait-linux-gnueabi/math/libm.so
    \
         ...
    ../../build-arm-tait-linux-gnueabi/csu/abi-note.o \
         ...

    $ cd ../../build-arm-tait-linux-gnueabi/
    $
    ../recipe-sysroot-native/usr/bin/arm-tait-linux-gnueabi/arm-tait-linux-gnueabi-objdump
    -t math/libm.so | grep csu
    00000000 l    df *ABS*    00000000
    ../../build-arm-tait-linux-gnueabi/csu/abi-note.o
    00000000 l    df *ABS*    00000000
    ../../build-arm-tait-linux-gnueabi/csu/crti.o
    00000000 l    df *ABS*    00000000
    ../../build-arm-tait-linux-gnueabi/csu/crtn.o

Make is invoked from the build-arm-tait-linux-gnueabi/ directory, whose 
Makefile recursively invokes the Makefile in ../git/, passing in a 
fully-qualified "objdir":

    all .DEFAULT:
             $(MAKE) -r PARALLELMFLAGS="$(PARALLELMFLAGS)" -C $(srcdir)
    *objdir=`pwd`* $@

Unfortunately, the makefiles are written on the assumption that the 
objdir is fully-qualified, so simply hacking this line doesn't work.

The -B path comes from git/Makerules +612, and the fully-qualified path 
to abi-note.o from line +665, in both cases via the csu-objpfx variable.

    $(extra-B-$(@F:lib%.so=%).so) -B$(csu-objpfx) \
    ...

    $(csu-objpfx)abi-note.o $(build-shlib-objlist)

I added "realpath" to the HOSTTOOLS, and precomputed a relative path to 
the csu/ dir in git/Makerules.

    csu-objpfx-relative = $(shell realpath --relative-to=`pwd`
    $(csu-objpfx))/

Seems to do the trick. I'll send a patch(ish) separately.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openembedded.org/pipermail/openembedded-core/attachments/20181003/98e1e502/attachment-0002.html>