[oe-commits] [bitbake] branch master updated: utils/md5_file: don't iterate line-by-line

[git at git.openembedded.org]

This is an automated email from the git hooks/post-receive script.

rpurdie pushed a commit to branch master
in repository bitbake.

The following commit(s) were added to refs/heads/master by this push:
     new a0ac8d6  utils/md5_file: don't iterate line-by-line
a0ac8d6 is described below

commit a0ac8d67f1471a0c611d691b856fede67efb53f6
Author: Ross Burton <ross.burton at intel.com>
AuthorDate: Mon Aug 13 19:02:25 2018 +0100

    utils/md5_file: don't iterate line-by-line
    
    Opening a file in binary mode and iterating it seems like the simple solution
    but will still break on newlines, which for binary files isn't really useful as
    the size of the chunks could be huge or tiny.
    
    Instead, let's be a bit more clever: we'll be MD5ing lots of files, but we don't
    want to fill up memory: use mmap() to open the file and read the file in 8k
    blocks.
    
    Signed-off-by: Ross Burton <ross.burton at intel.com>
    Signed-off-by: Richard Purdie <richard.purdie at linuxfoundation.org>
---
 lib/bb/utils.py | 13 +++++++++----
 1 file changed, 9 insertions(+), 4 deletions(-)

diff --git a/lib/bb/utils.py b/lib/bb/utils.py
index 9903183..b20cdab 100644
--- a/lib/bb/utils.py
+++ b/lib/bb/utils.py
@@ -524,12 +524,17 @@ def md5_file(filename):
     """
     Return the hex string representation of the MD5 checksum of filename.
     """
-    import hashlib
-    m = hashlib.md5()
+    import hashlib, mmap
 
     with open(filename, "rb") as f:
-        for line in f:
-            m.update(line)
+        m = hashlib.md5()
+        try:
+            with mmap.mmap(f.fileno(), 0, access=mmap.ACCESS_READ) as mm:
+                for chunk in iter(lambda: mm.read(8192), b''):
+                    m.update(chunk)
+        except ValueError:
+            # You can't mmap() an empty file so silence this exception
+            pass
     return m.hexdigest()
 
 def sha256_file(filename):

-- 
To stop receiving notification emails like this one, please contact
the administrator of this repository.