[bitbake-devel] [RFC PATCH 0/5] Implement git shallow mirror tarball support

[Christopher Larson]

On Tue, Aug 18, 2015 at 3:44 PM, Christopher Larson <kergoth at gmail.com>
wrote:

>
> On Tue, Aug 18, 2015 at 11:51 AM, Martin Jansa <martin.jansa at gmail.com>
> wrote:
>
>> On Tue, Aug 18, 2015 at 11:35:03AM -0700, Christopher Larson wrote:
>> > On Tue, Aug 18, 2015 at 7:28 AM, Christopher Larson <kergoth at gmail.com>
>> > wrote:
>> >
>> > >
>> > > On Tue, Aug 18, 2015 at 1:45 AM, Martin Jansa <martin.jansa at gmail.com
>> >
>> > > wrote:
>> > >
>> > >> On Fri, Aug 14, 2015 at 08:57:17AM -0700, Christopher Larson wrote:
>> > >> > On Thu, Aug 13, 2015 at 4:46 PM, Christopher Larson <
>> kergoth at gmail.com>
>> > >> > wrote:
>> > >> >
>> > >> > > From: Christopher Larson <chris_larson at mentor.com>
>> > >> > >
>> > >> > > Please review the following changes for suitability for
>> inclusion. If
>> > >> you
>> > >> > > have
>> > >> > > any objections or suggestions for improvement, please respond to
>> the
>> > >> > > patches. If
>> > >> > > you agree with the changes, please provide your Acked-by.
>> > >> > >
>> > >> > > This only implements support for shallow mirror tarballs, not
>> shallow
>> > >> > > clones.
>> > >> > >
>> > >> > > The mirror tarball filename includes branch, revision, and
>> depth. To
>> > >> > > enable,
>> > >> > > use the `BB_GIT_SHALLOW` variable, or `BB_GIT_SHALLOW_<name>` for
>> > >> specific
>> > >> > > URLs. This variable can hold either a clone depth (e.g. `1` to
>> just
>> > >> get
>> > >> > > SRCREV), or a ref or commit, in which case we keep history up to
>> that
>> > >> > > commit.
>> > >> > >
>> > >> > > The shallow support will not, at this time, function correctly
>> for
>> > >> > > linux-yocto, due to its branching scheme & validation.
>> > >> > >
>> > >> > > Example:
>> > >> > >
>> > >> > >     BB_GIT_SHALLOW ?= "1"
>> > >> > >     BB_GIT_SHALLOW_pn-linux-yocto = ""
>> > >> > >     BB_GIT_SHALLOW_pn-linux-mel_mx6 = "v3.14"
>> > >> > >     BB_GIT_SHALLOW_pn-testrepo = "testbranch"
>> > >> > >
>> > >> > > This implementation will attempt to fetch a full mirror tarball
>> if it
>> > >> was
>> > >> > > unable to fetch a shallow tarball. If `BB_GIT_SHALLOW` and
>> > >> > > `BB_GENERATE_MIRROR_TARBALLS` are enabled for a given recipe, a
>> > >> shallow
>> > >> > > tarball will be emitted.
>> > >>
>> > >> It would be nice to describe what this implementation does when
>> there is
>> > >> some local git clone already (will it first try to update it with
>> "git
>> > >> fetch"
>> > >> or download shallow tarball?)
>> > >
>> > >
>> > > In that case, I believe it will update the clone and use that, unless
>> we
>> > > have a current tarball already downloaded, but I'll double check the
>> logic
>> > > and get back to you.
>> >
>> >
>> > I was correct about the behavioral inconsistency of the current
>> > implementation -- if both existed, localpath was returning the mirror
>> > tarball, needs_update was returning false regardless of the state of the
>> > clone, and then unpack would unpack the out of date clone instead of the
>> > tarball :) Not ideal, obviously -- either we prefer a shallow tarball
>> to a
>> > clone or vice versa, not somewhere in-between. I'm fixing that now.
>> >
>> > I think that since the clone is essentially a superset (or should be) of
>> > the clone tarball, it should probably be preferred when both are around,
>> > but I'm open to opinions on that.
>>
>> I agree that existing clone should be preferred over shallow tarball
>> (doing incremental update should be faster in most repositories and
>> updating local clone will be useful next time SRCREV is upgraded again).
>>
>
> The one concern I have is, what if we have a perfectly useful shallow
> tarball, and an out of date clone, and upstream is down, so the fetch to
> update it fails. Ideally, the MIRRORS (not PREMIRRORS) would then be
> fetched, and it'd realize that the shallow tarball already exists and use
> that, but I suspect unpack() would end up using the out of date clone in
> that case, since we prefer the clone. I'll have to test that case out and
> see if I need to manually check needs_update() in unpack to decide which to
> use.
>

FYI, I confirmed that this is handled, though in a slightly surprising way.
If you have an existing out of date clone, and the attempt to update it
fails, the fetcher *removes* the entire local clone and you start fresh.
Not ideal, obviously, and I have a fix for it locally, but in that case the
fallback to mirrors works painlessly, as the out of date clonedir is
removed by the time the mirror fetching and unpack happens. I'll prep
another RFC commit to change that behavior.
-- 
Christopher Larson
kergoth at gmail dot com
Founder - BitBake, OpenEmbedded, OpenZaurus
Maintainer - Tslib
Senior Software Engineer, Mentor Graphics
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openembedded.org/pipermail/bitbake-devel/attachments/20150818/dc66f1b7/attachment-0002.html>