[OE-core] BB_SIGNATURE_HANDLER = "basichash" unusable strict?

[Martin Jansa]

On Wed, Nov 09, 2011 at 10:32:18AM +0000, Richard Purdie wrote:
> On Tue, 2011-11-08 at 15:37 +0100, Martin Jansa wrote:
> > Today I've started build from scratch and dediced to give basichash a try (as it is supposed to become default IIRC):
> > 
> > So after cleaning tmpdir, sstate cache, pseudo I've started clean build..
> > 
> > 1) bitbake -k gcc-cross | tee -a log.${MACHINE}; 
> > 2) bitbake -k virtual/kernel | tee -a log.${MACHINE}; 
> > 3) bitbake -k core-image-core | tee -a log.${MACHINE}; 
> > 4) bitbake -k shr-lite-image  | tee -a log.${MACHINE}; 
> > 
> > But then I've noticed that after successfull build of gcc-cross in step 1 it started another gcc-* build in step 2..
> [...]
> > Ah.. yes I did 2 small patches to libxml2 and openssl between step 1 and step2:
> > http://patchwork.openembedded.org/patch/14521/
> > http://patchwork.openembedded.org/patch/14519/
> > 
> > But do we want to rebuild everything after every change small like this?
> 
> The biggest problem we have here is deciding when to rebuild and when
> not to. Can you define when this should/shouldn't happen?
> 
> > Or is it configuration issue or just bug in sstate implementation?
> 
> I think its behaving as currently configured. Whether that configuration
> is right/wrong and what it should be is the question. If we can define
> the configuration, we can then work out how to implement it which is a
> separate issue.
> 
> > Btw libxml2 isn't first difference.. I can dig more..
> [...]
> > I have few extra patches in my branch so for this particular test case you also need ie
> > http://patchwork.openembedded.org/patch/13699/
> > 
> > But it shouldn't be hard to find similar issue for any other dependency tree (ie with git-native instead of subversion-native).
> 
> The situation is currently configurable through:
> 
> BB_HASHTASK_WHITELIST ?= "(.*-cross$|.*-native$|.*-cross-initial$|.*-cross-intermediate$|^virtual:native:.*|^virtual:nativesdk:.*)"
> 
> however I have to admit looking at the bitbake code handling this its
> not that simple.
> 
> The code only triggers for recipes which are not matched by the
> whitelist. For those not matching, it iterates through their
> dependencies and removes anything that matches the expression.
> 
> So effectively it only modified target recipes, removes dependencies
> matching the above expressions.
> 
> This isn't an easy problem and this is reminding me I wanted to revisit
> this code. I think we actually need some kind of double expression to
> match a regexp against like:
> 
> <depender>___<depend>
> 
> So we could then do:
> 
> REGEXP_NONNATIVE = "(.*-cross|.*-native|.*-cross-initial|.*-cross-intermediate|virtual:native:.*|virtual:nativesdk:.*)"
> 
> BB_HASHTASK_WHITELIST ?= "^.(?!${REGEXP_NONNATIVE})___${REGEXP_NONNATIVE}$"
> 
> which would function as above but move more of the control into the code.
> 
> I was then trying to come up with a further example to extend this but
> its not scaling. So lets throw away the idea of using regexps and use
> python. Coding off the top of my head, we could have something like:
> 
> def filter_dep(depender, depend):
>     # Return True if we should keep the dependency, False to drop it
>     def isNative(x):
>         return x.startswith("virtual:native:") or x.endswith("-native")
>     def isCross(x):
>         return x.endswith("-cross") or x.endswith("-cross-initial") or x.endswith("-cross-intermediate")
>     def isNativeSDK(x):
>         return x.startswith("virtual:nativesdk:")
> 
>     if isNative(depender) or isCross(depender) or isNativeSDK(depender):
>         return True
> 
>     # Only target packages beyond here
> 
>     if isNative(depend) or isCross(depend) or isNativeSDK(depend):
>         return False
> 
>     return True
> 
> which would then be easy to extend to for example ensure the python
> dependency on python-native is kept.
> 
> The siggen code was designed to be a plugin so changing it to the above
> form isn't the problem. The real problem is deciding what the policy it
> implements should be.
> 
> So to go back to the original question, out of those changes you made,
> which ones would you expect to change the hash and which ones would you
> not expect to see changes for?

I have talked with kergoth on IRC yesterday and he had very nice remark:

16:40:50 < kergoth_> JaMa: heh, the biggest weakness of the sstate
signature bits, in my opinion, is that it only tracks inputs, not
outputs. If task A depends on B, and the metadata input to B changes,
then A will be rebuilt, even if the *output* of B didn't change as a 
result of the change to its metadata.

And with this idea applied on those 2 changes I think that PR change in
libxml2 should of course invalidate checksum for 
sstate-libxml2-native-x86_64-linux-2.7.8-r*populate-sysroot.tgz.siginfo
and probably wont hurt so much when neon-native is also rebuilt, but then 
if the output of neon build is the same with new sstate checksum as it was 
with older one (I know it's hard to detect ie if some file in build has 
"generation timestamp inside"), then we won't continue to rebuild
subversion, gcc, ... all (just because neon was rebuilt due to libxml2 PR 
change which didn't influence neon output).

The same with openssl PR change.. which can cause python-native rebuild,
but as long as python-native build output is "the same" we don't need to
rebuild everything which (even transitively) depends on python-native.

Regards,
-- 
Martin 'JaMa' Jansa     jabber: Martin.Jansa at gmail.com
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 198 bytes
Desc: Digital signature
URL: <http://lists.openembedded.org/pipermail/openembedded-core/attachments/20111109/88a80ed5/attachment-0002.sig>