[oe] Checksums in Bitbake

[Richard Purdie]

On Wed, 2010-03-24 at 21:17 +0100, Frans Meulenbroeks wrote:
> 2010/3/24 Chris Larson <clarson at kergoth.com>:
> > On Wed, Mar 24, 2010 at 8:15 AM, Frans Meulenbroeks <
> > fransmeulenbroeks at gmail.com> wrote:
> >
> >> Interesting ideas.
> >> I need to let this digest a little  bit.
> >>
> >> Some initial thougths
> >> The checksum should also depend on the checksum of the underlying
> >> packages. E.g. if A depends on B and the checksum of B changes it
> >> should trigger a rebuild of A.
> >>
> >
> > I don't think this is a very good idea, personally.  As an option, perhaps,
> > but we do things the way we do for a reason, just because a dep of mine is
> > rebuilt doesn't automatically require that I be rebuilt.  I'd suggest moving
> > to an alternative which encodes the library ABI and incorporates that into
> > the hashes of things that depend upon it, but we can certainly do what you
> > want as an optional feature.
> 
> If a dep is rebuild there is a reason for it (bug fix, packaging
> changed, changes in exported files etc etc).
> This might impact the using recipe.
> If baking a file does not result in a rebuild when a dependency is
> changed, probably a warning should be given.
> 
> Encoding the library ABI is only part of the job. You'd also have to
> take the .h files a package exports into account as constants in it
> could be changed.
> And even the using package could change its behaviour (e.g. because
> configure runs differently).
> Note also that if we abandon PR we do not really have an easy
> mechanism to force recompilation of a package (if a depenency changed
> and we want to force a rebuild).

Its worth noting you can enable this now with BB_STAMP_POLICY. I see
something similar being the likely outcome with checksums. Some people
will want the full dependency tree, some people won't. We can support
both just as we do now.

> > Global variables should absolutely be included, imo.  The reason for going
> > with a blacklist rather than a whitelist approach is to, as richard says,
> > make it less error prone.  It ensures that the failure mode is something
> > being rebuilt, rather than using possibly incorrect binaries.  I'd rather it
> > take a bit longer to build than result in questionable output.  If
> > calculating the checksum time becomes a concern, which I doubt, you could
> > hash the configuration metadata at ConfigParsed time and incorporate that
> > hash into the hash generated of the recipe.  This could increase the
> > likelihood of collisions, but I'm not too worried.  Let's get things
> > working, and determine the bottlenecks at that point.
> 
> Agree, but as changes in vars are less likely we could consider having
> something to DISTRO_PR.
> My nightmare is that if I am going to build console-image (about 3000
> tasks) that it goes to check 3000 times if my TMPDIR is not changed.
> Some caching will definitely be needed

The checksums are likely to be constructed at parsing time. Compared to
the cost of building the datastore I'm hopeful the cost of building the
checksum will be low. The current caching algorithms will rebuild the
checksums when needed just fine.

Cheers,

Richard