[bitbake-devel] [1.44 02/18] runqueue/siggen: Allow handling of equivalent hashes

[Richard Purdie]

On Mon, 2019-12-23 at 08:25 +0000, Peter Kjellerstedt wrote:
> > -----Original Message-----
> > From: bitbake-devel-bounces at lists.openembedded.org <bitbake-devel-
> > bounces at lists.openembedded.org> On Behalf Of Armin Kuster
> > Sent: den 23 december 2019 05:50
> > To: bitbake-devel at lists.openembedded.org
> > Subject: [bitbake-devel] [1.44 02/18] runqueue/siggen: Allow handling of
> > equivalent hashes
> > 
> > From: Richard Purdie <richard.purdie at linuxfoundation.org>
> > 
> > Based on the hashserv's new ability to accept hash mappings, update
> > runqueue
> > to use this through a helper function in siggen.
> > 
> > This addresses problems with meta-extsdk-toolchain and its dependency on
> > gdb-cross which caused errors when building eSDK. See the previous commit
> > for more details.
> > 
> > Signed-off-by: Richard Purdie <richard.purdie at linuxfoundation.org>
> > (cherry picked from commit 39098b4ba2133f4d9229a0aa4fcf4c3e1291286a)
> > ---
> >  lib/bb/runqueue.py | 31 +++++++++++++++++++------------
> >  lib/bb/siggen.py   | 26 ++++++++++++++++++++++++++
> >  2 files changed, 45 insertions(+), 12 deletions(-)
> > 
> > diff --git a/lib/bb/runqueue.py b/lib/bb/runqueue.py
> > index bd7f03f..a869ba5 100644
> > --- a/lib/bb/runqueue.py
> > +++ b/lib/bb/runqueue.py
> > @@ -2283,12 +2283,26 @@ class RunQueueExecute:
> >                          for dep in self.rqdata.runtaskentries[tid].depends:
> >                              procdep.append(dep)
> >                          orighash = self.rqdata.runtaskentries[tid].hash
> > -                        self.rqdata.runtaskentries[tid].hash = bb.parse.siggen.get_taskhash(tid, procdep, self.rqdata.dataCaches[mc_from_tid(tid)])
> > +                        newhash = bb.parse.siggen.get_taskhash(tid, procdep, self.rqdata.dataCaches[mc_from_tid(tid)])
> >                          origuni = self.rqdata.runtaskentries[tid].unihash
> > -                        self.rqdata.runtaskentries[tid].unihash = bb.parse.siggen.get_unihash(tid)
> > -                        logger.debug(1, "Task %s hash changes: %s->%s %s->%s" % (tid, orighash, self.rqdata.runtaskentries[tid].hash, origuni, self.rqdata.runtaskentries[tid].unihash))
> > +                        newuni = bb.parse.siggen.get_unihash(tid)
> > +                        # FIXME, need to check it can come from sstate at all for determinism?
> > +                        remapped = False
> > +                        if newuni == origuni:
> > +                            # Nothing to do, we match, skip code below
> > +                            remapped = True
> > +                        elif tid in self.scenequeue_covered or tid in self.sq_live:
> > +                            # Already ran this setscene task or it running. Report the new taskhash
> > +                            remapped = bb.parse.siggen.report_unihash_equiv(tid, newhash, origuni, newuni, self.rqdata.dataCaches)
> > +                            logger.info("Already covered setscene for %s so ignoring rehash (remap)" % (tid))
> 
> Do we really want this level of logging? I have not commented on it for 
> master (yet), as I expect it is invaluable for RP & co that are currently 
> working hard on the hash server, but for me as common user of bitbake, 
> this is just noise (and a lot of it). I have a local patch that changes 
> this to logger.debug(1, ...) together with another logger.info("Task %s 
> unihash changed to %s" ...) a few lines earlier to get the cooker UI 
> reasonably back to usable. Also see below.

This is a tough one. I appreciate the console is getting noisy,
equally, there are some significant problems with hashequiv and without
this logging we're even more in the dark about what went wrong :(.

We're struggling to figure out what is wrong already and I'd much
rather have logs I can look at and have some idea of what happened
rather than shrugging and saying "dunno".

Whilst right now I'm using this to debug the issues the autobuilder is
showing, if other users run into issues we'll need the same logs. If
someone sends me a log with these entries missing, I will close the bug
as unfixable, I really don't have any other choice.

Fundamentally, we need to sort this out differently so that bitbake
does write a decent log somewhere about what happened even if its not
on the console.

We also need to ensure it *is* on the console on the autobuilder since
those are the logs which are preserved.

I do agree with Armin that we should keep master and zeus with the same
behaviour as to do otherwise will just create more of a mess.

I will likely reduce the noise eventually but only when we're not
hitting issues.

Cheers,

Richard