[OE-core] glibc binary reproducibility

[Richard Purdie]

On Thu, 2018-09-27 at 12:18 +1200, Douglas Royds wrote:
> When I build glibc in two different places, I get non-reproducible
> results - a 4-byte difference:
> > $ cmp -l ~/workspace/upstream[12]/build/tmp/work/armv5e-tait-linux-
> > gnueabi/glibc/2.28-r0/package/lib/libc-2.28.so
> > 1259181  71 172
> > 1259182  27 304
> > 1259183 152  77
> > 1259184 363 243
> 
> These 4 bytes are the checksum that objcopy --add-gnu-debuglink adds
> to the binary when the .debug info is split out at do_package time,
> see package.bbclass +416
> So why is the debug info different? We add -fdebug-prefix-map to our
> CFLAGS, which ensures that all our intra-component debug paths are
> prefixed with /usr/src/debug/glibc/, for instance, but this isn't
> working in this case. The difference is happening in csu/crt1.o,
> which is being linked into libc.so (and others):
> > $ ../recipe-sysroot-native/usr/bin/arm-tait-linux-gnueabi/arm-tait-
> > linux-gnueabi-objdump -t csu/crt1.o
> > ...
> > 00000000 l    df *ABS*    00000000
> > /home/douglas/workspace/upstream1/build/tmp/work/armv5e-tait-linux-
> > gnueabi/glibc/2.28-r0/build-arm-tait-linux-gnueabi/csu/abi-note.o
> > 00000000 l    df *ABS*    00000000
> > /home/douglas/workspace/upstream1/build/tmp/work/armv5e-tait-linux-
> > gnueabi/glibc/2.28-r0/build-arm-tait-linux-gnueabi/csu/start.o
>  This abi-note.o symbol is finding its way into libc.so: 
> > $ ../recipe-sysroot-native/usr/bin/arm-tait-linux-gnueabi/arm-tait-
> > linux-gnueabi-objdump -t libc.so
> > ...
> > 00000000 l    df *ABS*    00000000             
> > /home/douglas/workspace/upstream1/build/tmp/work/armv5e-tait-linux-
> > gnueabi/glibc/2.28-r0/build-arm-tait-linux-gnueabi/csu/abi-note.o
>  So where does csu/crt1.o come from?
> > 
> > $ arm-tait-linux-gnueabi-gcc  -march=armv5te -marm -mcpu=arm926ej-s 
> > --sysroot=/home/douglas/workspace/upstream1/build/tmp/work/armv5e-
> > tait-linux-gnueabi/glibc/2.28-r0/recipe-sysroot -nostdlib
> > -nostartfiles -r -o
> > /home/douglas/workspace/upstream1/build/tmp/work/armv5e-tait-linux-
> > gnueabi/glibc/2.28-r0/build-arm-tait-linux-gnueabi/csu/crt1.o
> > /home/douglas/workspace/upstream1/build/tmp/work/armv5e-tait-linux-
> > gnueabi/glibc/2.28-r0/build-arm-tait-linux-gnueabi/csu/start.o
> > /home/douglas/workspace/upstream1/build/tmp/work/armv5e-tait-linux-
> > gnueabi/glibc/2.28-r0/build-arm-tait-linux-gnueabi/csu/abi-note.o
> > /home/douglas/workspace/upstream1/build/tmp/work/armv5e-tait-linux-
> > gnueabi/glibc/2.28-r0/build-arm-tait-linux-gnueabi/csu/init.o
> > /home/douglas/workspace/upstream1/build/tmp/work/armv5e-tait-linux-
> > gnueabi/glibc/2.28-r0/build-arm-tait-linux-gnueabi/csu/static-
> > reloc.o
> 
> Note the -r = --relocatable, an ld option, which, "Generate[s]
> relocatable output---i.e., generate[s] an output file that can in
> turn serve as input to ld.  This is often called partial linking",
> ie. the glibc build is just putting these .o files together for later
> convenience.
> Regrettably, this command both ignores -fdebug-prefix-map (which ld
> doesn't accept) and puts the fully-qualified path to some of the
> input .o files in the resulting crt1.o. At package splitdebuginfo()
> time, although the fully-qualified path info is split off into the
> .debug files, a (relative) path to the .debug files plus a checksum
> is tacked onto libc.so by objcopy --add-gnu-debuglink ... and the
> checksum depends on the path to the build.
> There is a work-around: turn off the debug packaging:
> > INHIBIT_PACKAGE_DEBUG_SPLIT_pn-glibc = "1"
> 
> I don't have a solution for this. Suggestions?

I did some digging.

I tried what I suggested using relative paths:

http://git.yoctoproject.org/cgit.cgi/poky-contrib/commit/?h=rpurdie/t222&id=22189be4bf508851950f72654870c4eebd4b73d9

and whilst it helps remove some references, there is a much wider
problem. I therefore went and had a look at the linker itself to
understand why its injecting this path. I found this code:

http://git.yoctoproject.org/cgit.cgi/poky-contrib/commit/?h=rpurdie/t222&id=f9aca51204990fbdbdfa7442f1dcc938e97bf782

which if disabled as per this hack, makes the binaries reproducible and
drops the problematic references.

We're swapping some misleading debug output for reproducibility with
that hack.

At this point we probably need to talk to some toolchain people about
what 'real' fixes may be possible...

Cheers,

Richard