[bitbake-devel] [PATCH 2/2] runqueue: Introduce load balanced task spawning

[Andreas Müller]

To get most out of build host, bitbake now detects if the CPU workload is low.
If so, additional tasks are spawned. Maximum 'dynamic' tasks are set by
BB_NUMBER_THREADS_LOW_CPU.

So now user can set a range for the count of tasks:

Min: BB_NUMBER_THREADS
Min: BB_NUMBER_THREADS_LOW_CPU

in which bitbake can operate on demand.

Some numbers for 6 core AMD bulldozer 12GB RAM / build image from scratch with
3104 tasks / PARALLEL_MAKE = "-j6" / PARALLEL_MAKEINST="-j6":

Before the patch (same as BB_NUMBER_THREADS_LOW_CPU = "0" or not set):

 BB_NUMBER_THREADS | Build time [s]
------------------------------------
 2                 | 156m48.741s
------------------------------------
 3                 | 126m27.426s
------------------------------------
 4                 | 114m30.560s  <-- winner (as suggested in doc!)
------------------------------------
 5                 | 117m2.679s
------------------------------------
 6                 | 116m37.515s
------------------------------------
 8                 | 116m37.515s
------------------------------------
 10                | 118m18.441s
------------------------------------
 12                | 117m38.264s

With the patch applied and BB_NUMBER_THREADS_LOW_CPU = "20" (as written in docs
for max thread count)

 BB_NUMBER_THREADS | Build time [s]
------------------------------------
 3                 | 114m48.105s
------------------------------------
 4                 | 113m26.502s
------------------------------------

Some additional notes:

+ Although not tested I expect better enhancement for setscene sessions
+ At times back when do_package_qa was a dependency for do_rootfs, the
  performance win would have been more significant: For the image tested only
  very few do_package_qa were performed while do_rootfs. The static threads = 4
  winner had more of them and would have waited longer - sigh :)
+ It's more fun to watch bitbake at work torturing CPU. If you want to do so
  and use gnome's system monitor, be aware that CPU History is delayed for
  2-3s. I was sometimes wondering 'why more task's now?
- For building image from scratch the performace win is somewhat dissapointing ~1%
- Patch creates a dependecy on psutils
- time.monotonic is not yet used. It was introduced in python 3.3 (2012) and is
  considered supported in all environments (whatever that means) since 3.5
  (2015).

Signed-off-by: Andreas Müller <schnitzeltony at gmail.com>
---
 lib/bb/runqueue.py | 18 ++++++++++++++++++
 1 file changed, 18 insertions(+)

diff --git a/lib/bb/runqueue.py b/lib/bb/runqueue.py
index 7095ea5a..2690c2a2 100644
--- a/lib/bb/runqueue.py
+++ b/lib/bb/runqueue.py
@@ -37,6 +37,8 @@ from bb import monitordisk
 import subprocess
 import pickle
 from multiprocessing import Process
+import psutil
+import time
 
 bblogger = logging.getLogger("BitBake")
 logger = logging.getLogger("BitBake.RunQueue")
@@ -1668,6 +1670,7 @@ class RunQueueExecute:
         self.rqdata = rq.rqdata
 
         self.number_tasks = int(self.cfgData.getVar("BB_NUMBER_THREADS") or 1)
+        self.number_tasks_low_cpu = int(self.cfgData.getVar("BB_NUMBER_THREADS_LOW_CPU") or 0)
         self.scheduler = self.cfgData.getVar("BB_SCHEDULER") or "speed"
 
         self.runq_buildable = set()
@@ -1679,6 +1682,8 @@ class RunQueueExecute:
         self.failed_tids = []
 
         self.stampcache = {}
+        self.last_cpu_percent = psutil.cpu_percent()
+        self.last_cpu_percent_time = time.monotonic()
 
         for mc in rq.worker:
             rq.worker[mc].pipe.setrunqueueexec(self)
@@ -1687,6 +1692,8 @@ class RunQueueExecute:
 
         if self.number_tasks <= 0:
              bb.fatal("Invalid BB_NUMBER_THREADS %s" % self.number_tasks)
+        if self.number_tasks_low_cpu < 0:
+             bb.fatal("Invalid BB_NUMBER_THREADS_LOW_CPU %s" % self.number_tasks_low_cpu)
 
     def runqueue_process_waitpid(self, task, status):
 
@@ -1756,6 +1763,17 @@ class RunQueueExecute:
 
     def can_start_task(self):
         can_start = self.stats.active < self.number_tasks
+        # Can we inject extra tasks for low workload?
+        if not can_start and self.number_tasks_low_cpu > 0:
+            _time = time.monotonic()
+            # avoid workload inaccuray
+            if _time - self.last_cpu_percent_time >= 0.1:
+                cpu_percent = psutil.cpu_percent()
+                self.last_cpu_percent = cpu_percent
+                self.last_cpu_percent_time = _time
+                if cpu_percent < 90 and self.stats.active < self.number_tasks_low_cpu:
+                    can_start = True
+
         return can_start
 
 class RunQueueExecuteDummy(RunQueueExecute):
-- 
2.14.4