[bitbake-devel] [PATCH] runqueue: Address issues with incomplete sstate sets

[Richard Purdie]

The first part of the sstate code checks en-mass whether given checksums
are available. The next part of the code then either triggers those
setscene tasks either running them or skipping them if they've been
covered by others.

The problems was that this second part would always skip a task if it
was unavailable in the first part, even if it would have otherwise been
covered by other tasks.

This mean the mere presence of an artefact (or lack of presence) could
cause a different build failure.

The issue reproduces if you run a build and populate an sstate feed, then
run a second build off that feed, then run a third build off the sstate
feed of the second build (which is reduced in size).

The fix is rather than immediately skipping tasks if the checksum is
unavailable, create a list of missing tasks, then, if that task cannot
be covered by others we can skip it later. The deferral makes the
behaviour the same even when the cache is "incomplete".

[YOCTO #6081]

Signed-off-by: Richard Purdie <richard.purdie at linuxfoundation.org>
---
diff --git a/bitbake/lib/bb/runqueue.py b/bitbake/lib/bb/runqueue.py
index 1a19677..6372b65 100644
--- a/bitbake/lib/bb/runqueue.py
+++ b/bitbake/lib/bb/runqueue.py
@@ -1779,6 +1779,7 @@ class RunQueueExecuteScenequeue(RunQueueExecute):
             if len(self.sq_revdeps[task]) == 0:
                 self.runq_buildable[task] = 1
 
+        self.outrightfail = []
         if self.rq.hashvalidate:
             sq_hash = []
             sq_hashfn = []
@@ -1829,7 +1830,7 @@ class RunQueueExecuteScenequeue(RunQueueExecute):
                     realtask = self.rqdata.runq_setscene[task]
                     logger.debug(2, 'No package found, so skipping setscene task %s',
                                  self.rqdata.get_user_idstring(realtask))
-                    self.task_failoutright(task)
+                    self.outrightfail.append(task)
 
         logger.info('Executing SetScene Tasks')
 
@@ -1914,6 +1915,9 @@ class RunQueueExecuteScenequeue(RunQueueExecute):
                             self.task_skip(nexttask)
                             self.scenequeue_notneeded.add(nexttask)
                             return True
+                    if nexttask in self.outrightfail:
+                        self.task_failoutright(nexttask)
+                        return True
                     task = nexttask
                     break
         if task is not None: