[bitbake-devel] [PATCH] cooker: Improve cache handling

[Richard Purdie]

We're seeing issues where the self test, which uses tinfoil doesn't
notice the changed contents of include files. The issue is
cached_statements in the parser being reused when the files have changed.

Whilst looking at this, I realised there were some other issues:

* We need to also invalidate the mtime cache when cooker restarts
* We should pass full filenames to the file invalidation code
* We should process cached_statements as part of inotify invalidation

With these fixes, the caching is more reliable for memory resident
bitbake too. It does raise some questions about cache validation and
lifecycles and indicates bitbake does need more work in the area,
preferably with the removal of the globals. This at least highlights
and works around some of the current issues.

Signed-off-by: Richard Purdie <richard.purdie at linuxfoundation.org>

diff --git a/bitbake/lib/bb/cooker.py b/bitbake/lib/bb/cooker.py
index da8cc62..9d10943 100644
--- a/bitbake/lib/bb/cooker.py
+++ b/bitbake/lib/bb/cooker.py
@@ -144,6 +144,10 @@ class BBCooker:
         self.watcher.bbwatchedfiles = []
         self.notifier = pyinotify.Notifier(self.watcher, self.notifications)
 
+        # If being called by something like tinfoil, we need to clean cached data 
+        # which may now be invalid
+        bb.parse.__mtime_cache = {}
+        bb.parse.BBHandler.cached_statements = {}
 
         self.initConfigurationData()
 
@@ -197,13 +201,13 @@ class BBCooker:
     def config_notifications(self, event):
         if not event.pathname in self.configwatcher.bbwatchedfiles:
             return
-        if not event.path in self.inotify_modified_files:
-            self.inotify_modified_files.append(event.path)
+        if not event.pathname in self.inotify_modified_files:
+            self.inotify_modified_files.append(event.pathname)
         self.baseconfig_valid = False
 
     def notifications(self, event):
-        if not event.path in self.inotify_modified_files:
-            self.inotify_modified_files.append(event.path)
+        if not event.pathname in self.inotify_modified_files:
+            self.inotify_modified_files.append(event.pathname)
         self.parsecache_valid = False
 
     def add_filewatch(self, deps, watcher=None):
@@ -1506,6 +1510,8 @@ class BBCooker:
         # reload files for which we got notifications
         for p in self.inotify_modified_files:
             bb.parse.update_cache(p)
+            if p in bb.parse.BBHandler.cached_statements:
+                del bb.parse.BBHandler.cached_statements[p]
         self.inotify_modified_files = []
 
         if not self.baseconfig_valid: