[bitbake-devel] [master][PATCH 0/3] Implement support for shallow git mirror tarballs

[Christopher Larson]

On Sat, Apr 30, 2016 at 12:44 PM, Christopher Larson <kergoth at gmail.com>
wrote:

> From: Christopher Larson <chris_larson at mentor.com>
>
> This adds support to the git fetcher for fetching and generating mirror
> tarballs of shallow git repositories, with the specified revisions /
> depths.
>
> This implements support for shallow mirror tarballs, not shallow clones.
> Supporting shallow clones directly would be rather more problematic, as
> we'd
> need to hardcode the depth between branch HEAD and the SRCREV, and that
> depth
> would change as the branch is updated. Sadly, git's remote protocol is
> extremely limited.
>
> When BB_GIT_SHALLOW is enabled, we will always attempt to fetch a shallow
> mirror tarball. If the shallow mirror tarball cannot be fetched, it will
> try
> to fetch the full mirror tarball and use that.
>
> Multiple variables exist to exert control over what revisions and refs are
> kept at what are not.
>
> BB_GIT_SHALLOW: enable/disable shallow support as a whole, boolean
> BB_GIT_SHALLOW_DEPTH: specify commit depth for the included refs. defaults
> to 1 (the top commit alone). This can be set to 0 or the empty string.
> BB_GIT_SHALLOW_DEPTH_<name>: specify commit depth for a specific named
> url/branch
> BB_GIT_SHALLOW_REVS: specific revisions whose history should be removed,
> beyond that specified by refs+depth
> BB_GIT_SHALLOW_TRIM_REFS: by default, unused refs/branches are removed.
> set this to 0 or the empty string to keep all refs around. this is required
> for linux-yocto kernel repositories, otherwise the branch checking done by
> the kernel scripts will fail
> BB_GIT_SHALLOW_EXTRA_REFS: additional refs to keep beyond those referenced
> in SRC_URI, when TRIM_REFS is enabled
> BB_GENERATE_SHALLOW_TARBALLS: explicitly control creation of shallow
> mirror tarballs. this defaults to enabled when BB_GENERATE_MIRROR_TARBALLS
> is enabled
>
> Example usage:
>
>     BB_GIT_SHALLOW ?= "1"
>     BB_GIT_SHALLOW_DEPTH ?= "1"
>
>     # Usage in a recipe with multiple named uris or multiple named branches
>     BB_GIT_SHALLOW_DEPTH_doc = ""
>
>     # Remove the upstream history from our kernel repo, keeping our own
>     BB_GIT_SHALLOW_REVS_pn-linux-mel_mx6 = "v3.14"
>     BB_GIT_SHALLOW_DEPTH_pn-linux-mel_mx6 = ""
>
>     # Keep the branches in linux-yocto repositories
>     BB_GIT_SHALLOW_TRIM_REFS_pn-linux-yocto = "0"
>
>
> Please review the following changes for suitability for inclusion. If you
> have
> any objections or suggestions for improvement, please respond to the
> patches. If
> you agree with the changes, please provide your Acked-by.
>
> The following changes since commit
> 309f5907a3661821e041ed14645b5d165007b058:
>
>   bitbake: Switch to post release version (2016-04-29 07:41:34 +0100)
>
> are available in the git repository at:
>
>   https://github.com/kergoth/bitbake shallow-simplify
>   https://github.com/kergoth/bitbake/tree/shallow-simplify
>

This branch has been updated after rebasing post-python3-switch, so should
be ready for potential merge again. A couple things worth noting that I
failed to mention in the original post:

The shallow mirror tarballs are never unpacked into DL_DIR -- I didn't want
to create another dir structure there when there wouldn't be much benefit,
and I definitely didn't want anything shallow inside of ${DL_DIR}/git2/ to
avoid confusion and state problems, so when a shallow tarball is in play,
the download() method will return early and do nothing beyond the fetch of
the shallow tarball, and then the unpack method will extract it directly.

It's also worth noting that a current clone is preferred to either tarball,
a shallow tarball is preferred to an out of date clone, and a missing clone
will use either tarball.

I'd also like to clarify the usage of BB_GIT_SHALLOW_REVS. In a normal use
of shallow, the history removed is that beyond a certain depth of a certain
branch, so you end up with N commits left over, and everything else
removed. If BB_GIT_SHALLOW_DEPTH isn't set and BB_GIT_SHALLOW_REVS is set,
instead of only keeping a certain depth of history, we're only *removing* a
particular subset of the history. The main use case is to get more control
over what history is available to a customer when shipping sources, so e.g.
you can strip out the bits available in public repositories and just keep
the bits they can't easily re-fetch.

Richard, think you'll be able to find time to take a look at this? As long
as BB_GIT_SHALLOW isn't set, there's almost no impact at all. The only
change that isn't conditional upon this is the change in logic in bb.fetch
to simplify the tarball handling, so the risk is actually quite low.
-- 
Christopher Larson
kergoth at gmail dot com
Founder - BitBake, OpenEmbedded, OpenZaurus
Maintainer - Tslib
Senior Software Engineer, Mentor Graphics
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openembedded.org/pipermail/bitbake-devel/attachments/20160701/db2ccc34/attachment-0002.html>