[bitbake-devel] [PATCH] codeparser: Use hashlib for hashing, not hash()

[Armin Kuster]

From: Richard Purdie <richard.purdie at linuxfoundation.org>

"hash() is randomised by default each time you start a new instance of
recent
versions (Python3.3+) to prevent dictionary insertion DOS attacks"

which means we need to use hashlib.md5 to get consistent values for
the codeparser cache under python 3. Prior to this, the codeparser
cache was effectively useless under python3 as shown by performance
regressions.

Signed-off-by: Richard Purdie <richard.purdie at linuxfoundation.org>
(cherry picked from commit 12d43cf45ba48e3587392f15315d92a1a53482ef)

We kept running into an issue where shell scripts were not
getting generated on 32bit hosts. It seemed to be caused by this code.

Example:
.../i686/build-project/tmp/work/cortexa5hf-vfp-neon-montavista-linux-gnueabi/libpthread-stubs/0.3-r0/temp/run.sysroot_stage_all.104712:
line 121: sysroot_stage_dir: command not found
WARNING:
..../i686/build-project/tmp/work/cortexa5hf-vfp-neon-montavista-linux-gnueabi/libpthread-stubs/0.3-r0/temp/run.sysroot_stage_all.104712:1
exit 127 from
  sysroot_stage_dir $from/usr/include $to/usr/include
DEBUG: Python function do_populate_sysroot finished
ERROR: Function failed: sysroot_stage_all (log file is located at
.../i686/build-project/tmp/work/cortexa5hf-vfp-neon-montavista-linux-gnueabi/libpthread-stubs/0.3-r0/temp/log.do_populate_sysroot.104712)

Signed-off-by: Armin Kuster <akuster at mvista.com>
---
 lib/bb/codeparser.py | 7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)

diff --git a/lib/bb/codeparser.py b/lib/bb/codeparser.py
index 82a3af4..15492e4 100644
--- a/lib/bb/codeparser.py
+++ b/lib/bb/codeparser.py
@@ -3,6 +3,7 @@ import codegen
 import logging
 import os.path
 import bb.utils, bb.data
+import hashlib
 from itertools import chain
 from pysh import pyshyacc, pyshlex, sherrors
 from bb.cache import MultiProcessCache
@@ -16,6 +17,8 @@ except ImportError:
     import pickle
     logger.info('Importing cPickle failed.  Falling back to a very slow implementation.')
 
+def bbhash(s):
+    return hashlib.md5(s.encode("utf-8")).hexdigest()
 
 def check_indent(codestr):
     """If the code is indented, add a top level piece of code to 'remove' the indentation"""
@@ -248,7 +251,7 @@ class PythonParser():
         if not node or not node.strip():
             return
 
-        h = hash(str(node))
+        h = bbhash(str(node))
 
         if h in codeparsercache.pythoncache:
             self.references = set(codeparsercache.pythoncache[h].refs)
@@ -291,7 +294,7 @@ class ShellParser():
         commands it executes.
         """
 
-        h = hash(str(value))
+        h = bbhash(str(value))
 
         if h in codeparsercache.shellcache:
             self.execs = set(codeparsercache.shellcache[h].execs)
-- 
2.7.4