[bitbake-devel] [PATCH] cooker/siggen: Empty siggen cache during parsing

[akuster808]

On 2/17/20 8:23 AM, Richard Purdie wrote:
> When parsing recipes its apparent the memory usage of bitbake rises linearly
> with number of recipes parsed. It shouldn't.
This seems back port worthy to 1.44, 1.42

- armin
>
> Using tracemalloc (thanks for the tip Joshua Lock) it was clear that the
> dependency information left behind in siggen was the culprit. Add a new
> method to allow us to drop this information. We don't need it after the recipe
> has been parsed and hashes calculated (at runtime its different but only the
> currently executing task would be in memory).
>
> This should give signficant memory usage improvements for bitbake and that
> in turn should help speed on more constrained systems, as well as when used in
> multiconfig environments.
>
> Signed-off-by: Richard Purdie <richard.purdie at linuxfoundation.org>
> ---
>  lib/bb/cooker.py |  3 ++-
>  lib/bb/siggen.py | 11 +++++++++++
>  2 files changed, 13 insertions(+), 1 deletion(-)
>
> diff --git a/lib/bb/cooker.py b/lib/bb/cooker.py
> index 1124332186..a05630d52d 100644
> --- a/lib/bb/cooker.py
> +++ b/lib/bb/cooker.py
> @@ -1940,7 +1940,8 @@ class Parser(multiprocessing.Process):
>                  except IndexError:
>                      break
>                  result = self.parse(*job)
> -
> +                # Clear the siggen cache after parsing to control memory usage, its huge
> +                bb.parse.siggen.postparsing_clean_cache()
>              try:
>                  self.results.put(result, timeout=0.25)
>              except queue.Full:
> diff --git a/lib/bb/siggen.py b/lib/bb/siggen.py
> index ffd8fcaf36..c2d0c736cf 100644
> --- a/lib/bb/siggen.py
> +++ b/lib/bb/siggen.py
> @@ -51,6 +51,9 @@ class SignatureGenerator(object):
>      def finalise(self, fn, d, varient):
>          return
>  
> +    def postparsing_clean_cache(self):
> +        return
> +
>      def get_unihash(self, tid):
>          return self.taskhash[tid]
>  
> @@ -188,6 +191,14 @@ class SignatureGeneratorBasic(SignatureGenerator):
>          for task in taskdeps:
>              d.setVar("BB_BASEHASH_task-%s" % task, self.basehash[fn + ":" + task])
>  
> +    def postparsing_clean_cache(self):
> +        #
> +        # After parsing we can remove some things from memory to reduce our memory footprint
> +        #
> +        self.gendeps = {}
> +        self.lookupcache = {}
> +        self.taskdeps = {}
> +
>      def rundep_check(self, fn, recipename, task, dep, depname, dataCache):
>          # Return True if we should keep the dependency, False to drop it
>          # We only manipulate the dependencies for packages not in the whitelist