[OE-core] Mis-generation of shell script (run.do_install)?

[Jason Andryuk]

On Wed, Jan 16, 2019 at 3:28 PM Richard Purdie
<richard.purdie at linuxfoundation.org> wrote:
> The data is in the codeparser cache which is first populated at parse
> time so its enough just to parse the machine+recipe in question, not
> build it. I think that explains the answer to a ew of your questions
> above.

Yes, thanks.

> Sorry for asking so many questions btw, I'd just really love to be able
> to reproduce this issue! Thanks for trying to help answer them too!
>
> Is the bitbake-cookerdeamon.log file still there for this build (in the
> top level build directory)?

I don't seem to have this file in any of my OpenXT builds.

I still have the "bad" bb_codeparser.dat file.  It is 30MB whereas the
new one is only 6.5MB.  I thought it may be excessively large, but I
actually have an 80MB one in a different build directory.

Anyway, it has 4 different entries that look like core2-32
python-async do_install()-s

3c6fe664c51d2f793f8fd0eb103d68cb - reproduces currently
3df9018676de219bb3e46e88eea09c98 - one matching binutils core2-64 do_install
382871fb17743ba9635d7efc4db7d993
ee6850bdcf70ba63dea37e09c78c599f

They all have
frozenset({'[', 'mv', 'test',
'/home/build/openxt-compartments/build/tmp-glibc/work/core2-32-oe-linux/python-async/0.6.2-r0/recipe-sysroot-native/usr/bin/python-native/python',
'sed', 'install', 'bbfatal_log', 'find', 'rm', 'rmdir'})

Eyeballing distutils_do_install, I don't see what could produce so
many variations.

Going into the new, clean build container, I can see those last two
hashes with different entries:
>>> d['ee6850bdcf70ba63dea37e09c78c599f']
frozenset({'tr', 'rm', 'sed', 'ln', 'cd', 'oe_multilib_header',
'autotools_do_install', 'echo', 'basename', 'install'})
>>> d['382871fb17743ba9635d7efc4db7d993']
frozenset({'tr', 'rm', 'sed', 'ln', 'cd', 'oe_multilib_header',
'autotools_do_install', 'echo', 'basename', 'install'})

and the expected core2-32 python-async do_install
>>> d['3c6fe664c51d2f793f8fd0eb103d68cb']
frozenset({'bbfatal_log', 'rm', 'test', 'sed', '[', 'rmdir',
'/home/build/openxt-compartments/build/tmp-glibc/work/core2-32-oe-linux/python-async/0.6.2-r0/recipe-sysroot-native/usr/bin/python-native/python',
'find', 'install', 'mv'})

I've only run one core2-32 build in the fresh container, so no 64bit
binutils at the original collision 3df9018676de219bb3e46e88eea09c98

Ok, I hacked up a script to check two bb_codeparser.dat files for
collisions.  Compare the current one with the "bad" one:
$ ./pickle-cmp.py cache/bb_codeparser.dat cache/bb_codeparser.dat.old-bad-one
Collision ee6850bdcf70ba63dea37e09c78c599f
frozenset({'echo', 'rm', 'autotools_do_install', 'tr',
'oe_multilib_header', 'cd', 'basename', 'sed', 'ln', 'install'})
frozenset({'find', 'test', 'rm', 'bbfatal_log', '[', 'sed', 'mv',
'/home/build/openxt-compartments/build/tmp-glibc/work/core2-32-oe-linux/python-async/0.6.2-r0/recipe-sysroot-native/usr/bin/python-native/python',
'rmdir', 'install'})
Collision 382871fb17743ba9635d7efc4db7d993
frozenset({'echo', 'rm', 'autotools_do_install', 'tr',
'oe_multilib_header', 'cd', 'basename', 'sed', 'ln', 'install'})
frozenset({'find', 'test', 'rm', 'bbfatal_log', '[', 'sed', 'mv',
'/home/build/openxt-compartments/build/tmp-glibc/work/core2-32-oe-linux/python-async/0.6.2-r0/recipe-sysroot-native/usr/bin/python-native/python',
'rmdir', 'install'})
Collision 5254083eac08e32fc68bc9421d7df287
frozenset({'autotools_do_install', 'rm', 'sed', 'touch', 'install'})
frozenset({'/etc/init.d/xenclient-boot-sound', 'true', ':', '['})
Collision d0701fd5c05175aeafc06d8ce34d3532
frozenset({'create-cracklib-dict', 'autotools_do_install'})
frozenset({'/etc/init.d/gateone', 'true', ':', '['})
Collision ec332415bd96520823ba383494e7a9a7
frozenset({'ln', 'popd', ':', 'pushd'})
frozenset({'DEPLOY_DIR', 'useradd_preinst', 'perform_useradd', 'PKGD',
'PKGDEST', 'pkg_preinst', 'MLPREFIX', 'perform_groupadd', 'PN',
'perform_groupmems', 'PACKAGES', 'NOAUTOPACKAGEDEBUG',
'USERADD_PACKAGES', 'WORKDIR'})
Collision 3df9018676de219bb3e46e88eea09c98
frozenset({'echo', 'rm', 'autotools_do_install', 'tr',
'oe_multilib_header', 'cd', 'basename', 'sed', 'ln', 'install'})
frozenset({'find', 'test', 'rm', 'bbfatal_log', '[', 'sed', 'mv',
'/home/build/openxt-compartments/build/tmp-glibc/work/core2-32-oe-linux/python-async/0.6.2-r0/recipe-sysroot-native/usr/bin/python-native/python',
'rmdir', 'install'})
Collision 0aa15eb469ad8854cda0b0675217b8f6
frozenset({'find', 'test', 'rm', 'bbfatal_log', '[', 'sed', 'mv',
'/home/build/openxt-compartments/build/tmp-glibc/work/core2-32-oe-linux/python-mock/2.0.0-r0/recipe-sysroot-native/usr/bin/python-native/python',
'rmdir', 'install'})
frozenset({'oe_runmake', 'find', 'true', 'test', 'echo', 'chmod',
'rm', 'mkdir', '[', 'oe_multilib_header', 'cd', 'lnr', 'basename',
'continue', 'mv', 'ln', 'local', 'install'})

Compare the current one with the fresh one from the other container (build4):
$ ./pickle-cmp.py cache/bb_codeparser.dat build4-codeparser.dat
Collision d0701fd5c05175aeafc06d8ce34d3532
frozenset({'create-cracklib-dict', 'autotools_do_install'})
frozenset({'[', ':', '/etc/init.d/gateone', 'true'})
Collision 5254083eac08e32fc68bc9421d7df287
frozenset({'touch', 'install', 'sed', 'autotools_do_install', 'rm'})
frozenset({'[', '/etc/init.d/xenclient-boot-sound', ':', 'true'})

I figured I can reproduce the hash collisions for
d0701fd5c05175aeafc06d8ce34d3532, but I cannot.

gateone update-rc.d updatercd_prerm matches d0701fd5c05175aeafc06d8ce34d3532
Below, it should be a tab before /etc/init.d/gateone , but I can't
insert that because of gmail)
'''
# Begin section update-rc.d
if true && [ -z "$D" -a -x "/etc/init.d/gateone" ]; then
        /etc/init.d/gateone stop || :
fi
# End section update-rc.d
'''

cracklib do_install is only two lines, but I cannot reproduce.
(Below, it should be a tab before create-cracklib-dict)
$ sed -n '/^do_install/,/^}/p'
tmp-glibc/work/core2-32-oe-linux/cracklib/2.9.5-r0/temp/run.do_install
| head -n -7 | tail -n 2 | tee /dev/stderr | md5sum
    autotools_do_install
       create-cracklib-dict -o
/home/build/openxt-compartments/build/tmp-glibc/work/core2-32-oe-linux/cracklib/2.9.5-r0/image/usr/share/cracklib/pw_dict
/home/build/openxt-compartments/build/tmp-glibc/work/core2-32-oe-linux/cracklib/2.9.5-r0/image/usr/share/cracklib/cracklib-small
8ca3daacb394a11f38c851a8d71ed4de  -

#current
$ ./i-m-in-a-pickle.py cache/bb_codeparser.dat cracklib
3933316
'8ca3daacb394a11f38c851a8d71ed4de': frozenset({'create-cracklib-dict',
'autotools_do_install'})
5050048
'b50633e8f154817d7cc3ec94c6405379': frozenset({'create-cracklib-dict',
'autotools_do_install'})
5641024
'd0701fd5c05175aeafc06d8ce34d3532': frozenset({'create-cracklib-dict',
'autotools_do_install'})

#old bad
$ ./i-m-in-a-pickle.py cache/bb_codeparser.dat.old cracklib
13503349
'8ca3daacb394a11f38c851a8d71ed4de': frozenset({'autotools_do_install',
'create-cracklib-dict'})
39348111
'b50633e8f154817d7cc3ec94c6405379': frozenset({'autotools_do_install',
'create-cracklib-dict'})

#fresh
$ ./i-m-in-a-pickle.py build4-codeparser.dat cracklib
2085662
'8ca3daacb394a11f38c851a8d71ed4de': frozenset({'create-cracklib-dict',
'autotools_do_install'}),

b50633e8f154817d7cc3ec94c6405379 is the core2-64 cracklib do_install
$ sed -n '/^do_install/,/^}/p'
tmp-glibc/work/core2-64-oe-linux/cracklib/2.9.5-r0/temp/run.do_install
| head -n -7 | tail -n 2 | tee /dev/stderr | md5sum
    autotools_do_install
    create-cracklib-dict -o
/home/build/openxt-compartments/build/tmp-glibc/work/core2-64-oe-linux/cracklib/2.9.5-r0/image/usr/share/cracklib/pw_dict
/home/build/openxt-compartments/build/tmp-glibc/work/core2-64-oe-linux/cracklib/2.9.5-r0/image/usr/share/cracklib/cracklib-small
b50633e8f154817d7cc3ec94c6405379  -

But where did d0701fd5c05175aeafc06d8ce34d3532 come from?

When pickling the cache for writing to disk, could a dangling
pointer/index/reference be left such that a given hash's frozenset
entry changes?

An aside - entries include the standard linux utilities & shell
builtins, but those are always available.  If a given shell cache
entry only relies on those utilities, it could collide and still run.
It's only an issue when a shell function or special utility is needed.

Regards,
Jason