[oe] Amend nullifies cache

[Ambrose, Martin]

On Sat, Jun 19, 2010 at 19:37:00, Chris Larson wrote:

> On Sat, Jun 19, 2010 at 3:58 PM, Ambrose, Martin <martin at ti.com> wrote:
> 
> >    # Adding all amend.incs that can exist to the __depends, to ensure that
> >    # creating one of them invalidates the bitbake cache.  Note that it
> >    # requires a fix in bitbake.  Without the bitbake fix, the cache will be
> >    # completely invalidated on every bitbake execution.
> >
> > Which bitbake fix is that so that I can confirm or deny? Though I expect
> > I have it since this comment was added in an older commit (73bf2289) when the
> > file was named amend-recipes.inc. I couldn't find reference to this patch
> > in the ML maybe because of its age.
> >
> 
> See 3adb0d1c23f8b0d153859e78899a0694a27aa0af in the bitbake git repository.
> It looks like this may not have been cherry picked to 1.8, I'm not certain
> offhand.  I'd suggest trying out the 1.10 branch and seeing if you encounter
> the same problems.

Thanks for the pointer. Interesting I was coming to the same conclusion this was needed.
But reassured by the presence of a prior commit. Here I'm assuming 
git://git.openembedded.org/bitbake.git is the authoritative bitbake repo.

	$ git diff 3adb0d^ 3adb0d
	diff --git a/lib/bb/cache.py b/lib/bb/cache.py
	index 2726193..05f0d4f 100644
	--- a/lib/bb/cache.py
	+++ b/lib/bb/cache.py
	@@ -273,7 +273,7 @@ class Cache:
	             for f,old_mtime in depends:
	                 fmtime = bb.parse.cached_mtime_noerror(f)
	                 # Check if file still exists
	-                if fmtime == 0:
	+                if old_mtime != 0 and fmtime == 0:
	                     self.remove(fn)
	                     return False

I confirm this patch fixes my problem. Thank you.
Nice side benefit is that the "Handling Bitbake files" time decreased, for
handling 8500 recipes, from ~5min to ~ 30s. Sweet!

> > Looking further into the amend.bbclass I see that it doesn't add the
> > amend.inc files to the dependency list if they aren't present on the
> > filesystem. And I verified this by further debug prints. This is good
> > but leads to the question of where/when these non-existent
> > files are added to the dependency list.
> >
> 
> This is incorrect.  The files that do *not* exist are the ones explicitly
> added to __depends, if you look at the class again.

My bad. It certainly looks that way in the latest version I have (4d642fd5d):
    d.setVar("__depends", depends + [(file, 0) for file in amendfiles if not os.path.exists(file)])

I read this as if the amend file *does not* exist it *is* added to the depends list
with a file modification time of zero. But I don't understand my debug output where,
after the line above, I added:
    for f in amendfiles:
        bb.msg.debug(1, bb.msg.domain.Cache, "MJA: amendfiles contains %s" % (f))

    for f,mtime in depends:
        bb.msg.debug(1, bb.msg.domain.Cache, "MJA: depends contains %s" % (f))

Then ran 
	$ bitbake -D -b arago-feed-configs -f -c clean | grep MJA | grep -e amend | sort
	DEBUG: MJA: amendfiles contains /home/user/work/oe/arago/recipes/arago/amend.inc
	DEBUG: MJA: amendfiles contains /home/user/work/oe/arago/recipes/arago/amend.inc
	DEBUG: MJA: amendfiles contains /home/user/work/oe/arago/recipes/arago/files/amend.inc
	DEBUG: MJA: amendfiles contains /home/user/work/oe/arago/recipes/arago/files/amend.inc
	DEBUG: MJA: depends contains /home/user/work/oe/arago-oe-dev/classes/amend.bbclass
	DEBUG: MJA: depends contains /home/user/work/oe/arago-oe-dev/classes/amend.bbclass
	DEBUG: MJA: removing /home/user/work/oe/arago/recipes/arago/arago-feed-configs.bb from cache since /home/user/work/oe/arago/recipes/arago/files/amend.inc doesn't exist

There are two entries (I guess) since the amend.bbclass is invoked twice.
Here I see the amendfiles list contains the paths to the non-existent files but they 
are not added to the depends list. Probably my debug print, invocation, or grep is lacking in some way.


Regards, 
Martin