[OE-core] [PATCH] Checksums for local files now stored using partial recipe path

[Martin Jansa]

On Wed, Jun 19, 2013 at 04:45:55PM +0100, Paul Eggleton wrote:
> On Wednesday 19 June 2013 16:24:53 Paul Eggleton wrote:
> > Hi Jate,
> > 
> > On Wednesday 19 June 2013 11:08:10 Jate Sujjavanich wrote:
> > > This allows sstate-cache to be shared between builds in different
> > > directories.
> > > 
> > > Differences in the full path were triggering a false positive when there
> > > were actually no changes.
> > > 
> > > Signed-off-by: Jate Sujjavanich <jate.sujjavanich at myfuelmaster.com>
> > > ---
> > > 
> > >  bitbake/lib/bb/fetch2/__init__.py |   14 +++++++++-----
> > >  bitbake/lib/bb/siggen.py          |    3 ++-
> > >  2 files changed, 11 insertions(+), 6 deletions(-)
> > > 
> > > diff --git a/bitbake/lib/bb/fetch2/__init__.py
> > > b/bitbake/lib/bb/fetch2/__init__.py index dd1cc93..7ab44d7 100644
> > > --- a/bitbake/lib/bb/fetch2/__init__.py
> > > +++ b/bitbake/lib/bb/fetch2/__init__.py
> > > 
> > > @@ -900,8 +900,7 @@ def get_checksum_file_list(d):
> > >      return " ".join(filelist)
> > > 
> > > -
> > > -def get_file_checksums(filelist, pn):
> > > 
> > > +def get_file_checksums(filelist, pn, topdir):
> > >      """Get a list of the checksums for a list of local files
> > >      
> > >      Returns the checksums for a list of local files, caching the results
> > >      as
> > > 
> > > @@ -917,7 +916,12 @@ def get_file_checksums(filelist, pn): bb.warn("Unable
> > > to get checksum for %s SRC_URI entry %s: %s" % (pn, os.path.basename(f),
> > > e)) return None
> > > 
> > >          return checksum
> > > 
> > > +
> > > +    (recipe_root, _) = os.path.split(topdir)
> > > 
> > > +    def remove_recipe_parent(data):
> > > +        return data.replace(recipe_root, '').strip('/')
> > > +
> > > 
> > >      checksums = []
> > >      
> > >      for pth in filelist.split():
> > >          checksum = None
> > > 
> > > @@ -927,7 +931,7 @@ def get_file_checksums(filelist, pn):
> > >              for f in glob.glob(pth):
> > >                  checksum = checksum_file(f)
> > > 
> > >                  if checksum:
> > > -                    checksums.append((f, checksum))
> > > +                    checksums.append((remove_recipe_parent(f),
> > > + checksum))
> > > 
> > >          elif os.path.isdir(pth):
> > >              # Handle directories
> > > 
> > >              for root, dirs, files in os.walk(pth):
> > > @@ -935,12 +939,12 @@ def get_file_checksums(filelist, pn):
> > >                      fullpth = os.path.join(root, name)
> > >                      checksum = checksum_file(fullpth)
> > > 
> > >                      if checksum:
> > > -                        checksums.append((fullpth, checksum))
> > > +
> > > + checksums.append((remove_recipe_parent(fullpth), checksum))
> > > 
> > >          else:
> > >              checksum = checksum_file(pth)
> > >          
> > >          if checksum:
> > > -            checksums.append((pth, checksum))
> > > +            checksums.append((remove_recipe_parent(pth), checksum))
> > > 
> > >      checksums.sort(key=operator.itemgetter(1))
> > >      return checksums
> > > 
> > > diff --git a/bitbake/lib/bb/siggen.py b/bitbake/lib/bb/siggen.py index
> > > 8861337..c64acfe 100644 --- a/bitbake/lib/bb/siggen.py
> > > +++ b/bitbake/lib/bb/siggen.py
> > > 
> > > @@ -74,6 +74,7 @@ class SignatureGeneratorBasic(SignatureGenerator):
> > >          self.pkgnameextract = re.compile("(?P<fn>.*)\..*")
> > >          self.basewhitelist = set((data.getVar("BB_HASHBASE_WHITELIST",
> > > 
> > > True) or "").split()) self.taskwhitelist = None
> > > +        self.topdir = data.getVar("TOPDIR", True)
> > > 
> > >          self.init_rundepcheck(data)
> > >      
> > >      def init_rundepcheck(self, data):
> > > @@ -187,7 +188,7 @@ class SignatureGeneratorBasic(SignatureGenerator):
> > >              self.runtaskdeps[k].append(dep)
> > >          
> > >          if task in dataCache.file_checksums[fn]:
> > > -            checksums =
> > > bb.fetch2.get_file_checksums(dataCache.file_checksums[fn][task],
> > > recipename) +            checksums =
> > > + bb.fetch2.get_file_checksums(dataCache.file_checksums[fn][task],
> > > + recipename, self.topdir)
> > > 
> > >              for (f,cs) in checksums:
> > >                  self.file_checksum_values[k][f] = cs
> > >                  data = data + cs
> > 
> > Good catch! The only thing is, this will not help for files within different
> > layers which may not be underneath TOPDIR; I think we'll need a function
> > that determines which layer the file is under (longest path match from
> > data.getVar('BBLAYERS', True).split()) and then take that path off the
> > beginning.
> > 
> > Additionally, this is a patch against bitbake so it will need to go to the
> > bitbake-devel at lists.openembedded.org mailing list.
> 
> Actually, looking more closely at this I'm not sure how the full path to the 
> file would be getting into the signature - looking at lib/bb/siggen.py it 
> should only be adding the file checksum value to the signature data and not the 
> path. I did a quick test with master by moving some files referred to in 
> SRC_URI to a different valid location (thus changing their full path), cleaning 
> the recipe and then building it again, and the output was restored from sstate 
> rather than rebuilding.
> 
> Can you explain how you came to the conclusion that this was why the checksums 
> were different on different machines?

I sometimes compare signatures between two hosts with different TOPDIR
and I also haven't seen this issue. I'm using sstate-diff-machines.sh script.

-- 
Martin 'JaMa' Jansa     jabber: Martin.Jansa at gmail.com
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 198 bytes
Desc: Digital signature
URL: <http://lists.openembedded.org/pipermail/openembedded-core/attachments/20130619/baf5c553/attachment-0002.sig>