[bitbake-devel] [PATCH 1/3] bitbake-worker: Further IO performance tweaks

[Richard Purdie]

Looking further at the CPU loads on systems running large numbers of tasks,
the following things helps performance:

* Loop on waitpid until there are no processes still waiting
* Using select to wait for the cooker pipe to be writable before writing
  avoiding pointless 100% cpu usage
* Only reading from worker pipes that select highlights are readable

Signed-off-by: Richard Purdie <richard.purdie at linuxfoundation.org>
---
 bitbake/bin/bitbake-worker | 12 ++++++++----
 1 file changed, 8 insertions(+), 4 deletions(-)

diff --git a/bitbake/bin/bitbake-worker b/bitbake/bin/bitbake-worker
index a70791e..3d00c9e 100755
--- a/bitbake/bin/bitbake-worker
+++ b/bitbake/bin/bitbake-worker
@@ -97,6 +97,7 @@ def worker_flush(worker_queue):
             pass
         while (worker_queue_int or not worker_queue.empty()):
             try:
+                (_, ready, _) = select.select([], [worker_pipe], [], 1)
                 if not worker_queue.empty():
                     worker_queue_int = worker_queue_int + worker_queue.get()
                 written = os.write(worker_pipe, worker_queue_int)
@@ -383,9 +384,11 @@ class BitbakeWorker(object):
                 self.handle_item(b"quit", self.handle_quit)
 
             for pipe in self.build_pipes:
-                self.build_pipes[pipe].read()
+                if self.build_pipes[pipe].input in ready:
+                    self.build_pipes[pipe].read()
             if len(self.build_pids):
-                self.process_waitpid()
+                while self.process_waitpid():
+                    continue
 
 
     def handle_item(self, item, func):
@@ -440,9 +443,9 @@ class BitbakeWorker(object):
         try:
             pid, status = os.waitpid(-1, os.WNOHANG)
             if pid == 0 or os.WIFSTOPPED(status):
-                return None
+                return False
         except OSError:
-            return None
+            return False
 
         workerlog_write("Exit code of %s for pid %s\n" % (status, pid))
 
@@ -464,6 +467,7 @@ class BitbakeWorker(object):
         if pid in self.cpuused:
             self.cpuavail.append(self.cpuused[pid])
             del self.cpuused[pid]
+        return True
 
     def handle_finishnow(self, _):
         if self.build_pids:
-- 
2.7.4