[bitbake-devel] [PATCH 0/2] Variable tracking: Cleaned up and rebased.

[Richard Purdie]

I figured out the issue, I shouldn't have been touching current at all.
Rather than deepcopy, its also better to use our own copy operation on
the children:

--- a/bitbake/lib/bb/data_smart.py
+++ b/bitbake/lib/bb/data_smart.py
@@ -174,7 +174,14 @@ class IncludeHistory(object):
         self.filename = filename or '[TOP LEVEL]'
         self.children = []
         self.current = self
-        self.variables = {}
+        self.variables = COWDictBase.copy()
+
+    def copy(self):
+        new = IncludeHistory(self.parent, self.filename, self.datasmart)
+        for c in self.children:
+            new.children.append(c.copy())
+        new.variables = self.variables.copy()
+        return new
 
     def include(self, filename):
         newfile = IncludeHistory(self.current, filename)
@@ -615,7 +622,7 @@ class DataSmart(MutableMapping):
         # we really want this to be a DataSmart...
         data = DataSmart(seen=self._seen_overrides.copy(), special=self._special_values.copy())
         data.dict["_data"] = self.dict
-        data.history = copy.deepcopy(self.history)
+        data.history = self.history.copy()
         data.history.datasmart = data
         data._tracking = self._tracking
 
So with this applied, we get -e taking around 10s and parsing 3m26 so
about a 10% increase and the output of -e is the same before and after
that patch. I'm still not happy but that is obviously a better
proposition.

The real issue is these highly recursive and duplicate data structures
are inefficient so I think there is still much room for improvement.
Separating the variable and include history will likely help a lot.

The main point of the above is to prove how much the deepcopy hurts us
and also that there are ways we can fix it.

Cheers,

Richard