[OE-core] Looking for a way to build latest tagged releases in recipes

[chris.laplante at agilent.com]

Hi Richard,

> FWIW I do want to see the system used to do things like this. I've kind
> of envisaged you'd do it "up front" though. For example it should be
> trivial for a script to use tinfoil, iterate through and generate an
> include file of the revisions you want, using the fetcher calls.
> 
> Yes, its slightly more annoying to generate the config, then build it
> but if it stops the core becoming unmaintainable with corner cases,
> that could be the right decision.
> 
> If you do something like that it would be great to share it so others
> can see what you did and be able to build off it...

This is actually something I've been working on and thinking about a lot for the past year and a half. Our application layer has probably 30 or so recipes that use AUTOREV. For the purposes of reproducible builds, we need to generate similar output to what buildhistory-collect-srcrevs generates, e.g. 

	SRCREV_pn-my_daemon = "githash" 

I've gone back and forth and re-implemented the code to do so a half dozen times. Chronologically, these are the fundamental strategies I've attempted:

1. Simply in terms of buildhistory-collect-srcrevs. 
	Pros: dead simple
	Cons: requires that you actually build stuff first; requires buildhistory to be enabled
2. As a bbclass, injecting a task between do_fetch and do_unpack ("do_capture_srcrevs")
	Pros: can be invoked without actually building stuff, e.g. bitbake my-image --runall=do_capture_srcrevs
	Cons: adds yet another task to the recipe; a little ugly since you either need to INHERIT the class, or individually inherit it in recipes 
3. As a separate tinfoil script (as you suggest) which I run before the actual bitbake. 
	Pros: decoupled from the actual build
	Cons: since it's decoupled, it's slower - each recipe needs to be parsed
4. Hacking the srcrev cache (inside fetch2) to log all information necessary to generate the SRCREV overrides, paired with a script that reads the cache and does it
	Pros: Equally suitable for the "config-then-build" workflow ("bitbake --parse-only && generate_overrides.py >> local.conf && bitbake image") or the "build-then-config" workflow ("bitbake image && generate_overrides.py > overrides.conf")
	Cons: Invasive, required large changes to fetch2/__init__.py and some smaller changes to fetch2/git.py

In the current incarnation (#4), I also go a step further and, in cases where a SRC_URI has a parameterized branch, I generate BRANCH override lines as well. For example, if I see:

	MY_BRANCH ??= "master"
	SRC_URI = "git://git.oe.handhelds.org/git/vip.git;tag=version-1; branch=${MY_BRANCH}" 

I generate this line along with the corresponding SRCREV override:

	MY_BRANCH_pn-recipe = "master"

The algorithm for detecting the branch variable in the SRC_URI entry is as a little complicated, and not super relevant at this moment, but I want to get it off my chest :). The code first tries to find a parameterized branch by using a regex to find stuff like "branch=${VAR}". If that fails, it could be because the branch is expressed as something more complicated like this:

	BRANCH_RELEASE_NUMBER ??= "1.0"
	SRC_URI = "git://git.oe.handhelds.org/git/vip.git;tag=version-1; branch=release-${BRANCH_RELEASE_NUMBER}"

In that case, I fall back on a breadth-first search on all the variables that SRC_URI references (recursively) to find the "deepest" one that controls the branch. I wrote this in terms of d.expandWithRefs, and can detail it further if people are interested. Basically, I didn't want to enforce a particular convention on the name of the branch variable. You'd think BRANCH would cover it, but the kernel recipes also have KMETA and stuff like that. 

I realize the branch stuff as currently implemented is maybe too opinionated (and a bit dangerous) for inclusion in bitbake. But I digress. 

I'm very interested in discussing the SRCREV (and perhaps BRANCH) stuff further with anyone interested. I would love to see some strategy for doing this make its way into bitbake or poky, so I can stop reimplementing it and second guessing myself :).

Thanks,
Chris