[oe-commits] [bitbake] branch master updated: cooker/siggen: Empty siggen cache during parsing

[git at git.openembedded.org]

This is an automated email from the git hooks/post-receive script.

rpurdie pushed a commit to branch master
in repository bitbake.

The following commit(s) were added to refs/heads/master by this push:
     new 5d98d8e  cooker/siggen: Empty siggen cache during parsing
5d98d8e is described below

commit 5d98d8e39bba42f458532b1eef3619f2321d8a2b
Author: Richard Purdie <richard.purdie at linuxfoundation.org>
AuthorDate: Mon Feb 17 16:18:25 2020 +0000

    cooker/siggen: Empty siggen cache during parsing
    
    When parsing recipes its apparent the memory usage of bitbake rises linearly
    with number of recipes parsed. It shouldn't.
    
    Using tracemalloc (thanks for the tip Joshua Lock) it was clear that the
    dependency information left behind in siggen was the culprit. Add a new
    method to allow us to drop this information. We don't need it after the recipe
    has been parsed and hashes calculated (at runtime its different but only the
    currently executing task would be in memory).
    
    This should give signficant memory usage improvements for bitbake and that
    in turn should help speed on more constrained systems, as well as when used in
    multiconfig environments.
    
    Signed-off-by: Richard Purdie <richard.purdie at linuxfoundation.org>
---
 lib/bb/cooker.py |  3 ++-
 lib/bb/siggen.py | 11 +++++++++++
 2 files changed, 13 insertions(+), 1 deletion(-)

diff --git a/lib/bb/cooker.py b/lib/bb/cooker.py
index 1124332..a05630d 100644
--- a/lib/bb/cooker.py
+++ b/lib/bb/cooker.py
@@ -1940,7 +1940,8 @@ class Parser(multiprocessing.Process):
                 except IndexError:
                     break
                 result = self.parse(*job)
-
+                # Clear the siggen cache after parsing to control memory usage, its huge
+                bb.parse.siggen.postparsing_clean_cache()
             try:
                 self.results.put(result, timeout=0.25)
             except queue.Full:
diff --git a/lib/bb/siggen.py b/lib/bb/siggen.py
index ffd8fca..c2d0c73 100644
--- a/lib/bb/siggen.py
+++ b/lib/bb/siggen.py
@@ -51,6 +51,9 @@ class SignatureGenerator(object):
     def finalise(self, fn, d, varient):
         return
 
+    def postparsing_clean_cache(self):
+        return
+
     def get_unihash(self, tid):
         return self.taskhash[tid]
 
@@ -188,6 +191,14 @@ class SignatureGeneratorBasic(SignatureGenerator):
         for task in taskdeps:
             d.setVar("BB_BASEHASH_task-%s" % task, self.basehash[fn + ":" + task])
 
+    def postparsing_clean_cache(self):
+        #
+        # After parsing we can remove some things from memory to reduce our memory footprint
+        #
+        self.gendeps = {}
+        self.lookupcache = {}
+        self.taskdeps = {}
+
     def rundep_check(self, fn, recipename, task, dep, depname, dataCache):
         # Return True if we should keep the dependency, False to drop it
         # We only manipulate the dependencies for packages not in the whitelist

-- 
To stop receiving notification emails like this one, please contact
the administrator of this repository.