[OE-core] Mis-generation of shell script (run.do_install)?

[Jason Andryuk]

On Sat, Dec 15, 2018, 5:51 AM <richard.purdie at linuxfoundation.org wrote:
>
> On Fri, 2018-12-14 at 14:30 -0500, Jason Andryuk wrote:
> > I wasn't sure how to tie into the pysh stuff, but that got me poking
> > around in bitbake/lib/bb/codeparser.py .  Adding debug messages to
> > parse_shell(), I see that do_install is found in the CodeParserCache,
> > bb_codeparser.dat, but the returned `execs` do not include
> > autotools_do_install.  Strangely, it includes a path to python -
> > ...core2-32-oe-linux/python-async/0.6.2-r0/recipe-sysroot-
> > native/usr/bin/python-native/python.
> > It looks like `execs` could be for `distutils_do_install`.  And
> > again,
> > strangely, python-async is not in my tmp-glibc.  It must have been
> > built at some point which left the entry in bb_codeparser.dat.
> >
> > I built python-async, but its distutils_do_install hash value does
> > not
> > match the one in the cache.
> >
> > Moving cache/bb_codeparser.dat out of the way, bitbake complains:
> > ERROR: When reparsing
> > /home/build/openxt/build/repos/openembedded-core/meta/recipes-
> > devtools/binutils/binutils_2.28.bb.do_install,
> > the basehash value changed from 80812e0772cf901b51790c205564070d to
> > 493152cd3740c5420d0bf7a5d09df001. The metadata is not deterministic
> > and this needs to be fixed.
> >
> > `cleanall` does not clear out the message, but the package builds.
>
> Its a little hard to make sense of this. If you move the cache out the
> way it should simply get regenerated. It is long lived so things from
> old builds in there is expected.
>
> Were you able to isolate this into a smaller test case someone else
> could reproduce?

As far as I can tell, pysh is working properly - it's just the
bb_codeparser.dat which is returning the incorrect shellCacheLine
entry.  It seems like I have an md5 collision between a pyro core2-64
binutils do_install and core2-32 python-async distutils_do_install in
the shellCacheLine.  python-async's entry got in first, so that's why
binutils run.do_install doesn't include autotools_do_install - the
shellCacheLine `execs` entry doesn't include it.  Or somehow the
`bb_codeparser.dat` file was corrupted to have an incorrect `execs`
for the binutils do_install hash.

I briefly tried to reproduce the python-async distutils_do_install
with the same hash, but could not get it to match.  Also I tried to
manually unpickle bb_codeparser.dat, but it threw a stack underflow
error - maybe I just didn't have all the necessary imports?

I'm not sure where the basehash/"metadata is not deterministic"
message comes from.  I am using two different x86-64 machines types
that both fall back to the core2-64 binutils.  Could that be an issue?

Regards,
Jason