[OE-core] [PATCH v2] externalsrc.bbclas: remove nostamp from do_configure

[Markus Lehtonen]

Be a bit more intelligent than mindlessly re-compiling every time.
Instead of using 'nostamp' flag for do_compile run a python function to
get a list of files to add as 'file-checksums' flag.  The intention is
to only re-run do_compile if something in the source tree content
changes.

This python function, srctree_hash_files(), works differently, depending
if the source tree is a git repository clone or not. If the source tree
is a git repository, the function runs 'git add .' with a custom git
index file in order to record all changes in the git working tree. This
custom index file is then returned as the file for the task to depend
on. The index file is changed if any changes are made in the source tree
causing the task to be re-run.

If the source tree is not a git repository, srctree_hash_files() simply
adds the whole source tree as a dependency, causing bitbake to basically
hash every file in it. Hidden files and directories in the source tree
root are ignored by the glob currently used. This has the advantage of
automatically ignoring .git directory, for example.

This method of tracking changes source tree changes to determine if
re-build is needed does not work perofectly, though. Many packages are
built under ${S} which effectively changes the source tree causing some
unwanted re-compilations.  However, if do_compile of the recipe does not
produce new/different artefacts on every run (as commonly is and should
be the case) the re-compilation loop stops. Thus, you should usually see
only one re-compilation (if any) after which the source tree is
"stabilized" and no more re-compilations happen.

During the first bitbake run preparing of the task runqueue may take
much longer if the source tree is not a git repository. The reason is
that all the files in the source tree are hashed.  Subsequent builds are
not significantly slower because (most) file hashes are found from the
cache.

[YOCTO #8853]

Signed-off-by: Markus Lehtonen <markus.lehtonen at linux.intel.com>
---
 meta/classes/externalsrc.bbclass | 25 +++++++++++++++++++++++--
 1 file changed, 23 insertions(+), 2 deletions(-)

diff --git a/meta/classes/externalsrc.bbclass b/meta/classes/externalsrc.bbclass
index b608bd0..4f25bcf 100644
--- a/meta/classes/externalsrc.bbclass
+++ b/meta/classes/externalsrc.bbclass
@@ -85,8 +85,7 @@ python () {
         d.prependVarFlag('do_compile', 'prefuncs', "externalsrc_compile_prefunc ")
         d.prependVarFlag('do_configure', 'prefuncs', "externalsrc_configure_prefunc ")
 
-        # Ensure compilation happens every time
-        d.setVarFlag('do_compile', 'nostamp', '1')
+        d.setVarFlag('do_compile', 'file-checksums', '${@srctree_hash_files(d)}')
 
         # We don't want the workdir to go away
         d.appendVar('RM_WORK_EXCLUDE', ' ' + d.getVar('PN', True))
@@ -125,3 +124,25 @@ python externalsrc_compile_prefunc() {
     # Make it obvious that this is happening, since forgetting about it could lead to much confusion
     bb.plain('NOTE: %s: compiling from external source tree %s' % (d.getVar('PN', True), d.getVar('EXTERNALSRC', True)))
 }
+
+def srctree_hash_files(d):
+    import shutil
+    import subprocess
+
+    s_dir = d.getVar('EXTERNALSRC', True)
+    git_dir = os.path.join(s_dir, '.git')
+    oe_index_file = os.path.join(git_dir, 'oe-devtool-index')
+
+    ret = " "
+    if os.path.exists(git_dir):
+        # Clone index
+        if not os.path.exists(oe_index_file):
+            shutil.copy2(os.path.join(git_dir, 'index'), oe_index_file)
+        # Update our custom index
+        env = os.environ.copy()
+        env['GIT_INDEX_FILE'] = oe_index_file
+        subprocess.check_output(['git', 'add', '.'], cwd=s_dir, env=env)
+        ret = oe_index_file + ':True'
+    else:
+        ret = d.getVar('EXTERNALSRC') + '/*:True'
+    return ret
-- 
2.6.2