Discussion: Version retention policy in oe-core

Wed Feb 23 17:05:35 UTC 2011

I think this points out something fairly fundamental to the core.  There is no
one right answer for all packages.. but there are a couple of buckets we can put
the packages into.

There is a critical set of packages, primarily toolchains, autotools, and
related libraries...  And core userspace..

For the critical set of packages, the multiple version criteria, check in
policies, default preferences, etc are often different then the rest of the
system...

for core userspace, I'd say these are relaxed compared to the critical set, but
still more controlled then "the rest of the system".

For all "oe-core" packages, the general concept of a primary package, and a
"secondary" package I think holds true.  However what determines primary vs
secondary version really needs to be classified based on the requirements of the
section.  Any guidelines we mention need to be simply that -- guidelines.

For critical packages, we may end up with 2, 3, 4, 5 versions of the toolchains
if we hit a situation where the current toolchain doesn't work on all
architectures -- but I'd be inclined to say then that the current toolchain goes
off into another layer until it works on enough architectures to limit any
disruption and damage to the rest of the system.  So this really leaves 1
primary toolchain (for all/most architectures) and possibly a secondary one that
fills in a gap of some type.

For core userspace packages, the criteria changes a bit that we want to prefer a
single version -- unless we have a specific overriding factor, such as the GPLv2
vs GPLv3 issue.  (I think that factor becomes fairly easy to document vs the
more nebulous, last, current, next versioning..)

But there will always be exceptions to this, as we deal with the exceptions, say
glib-2.28 causes issues with components outside of the core, then we need to
take the action to document the exception and why it was there -- then in the
future we need to continue to monitor if the exception is still necessary and
remove it when it no longer is.

What I would hate to happen is we make an exception for glib-2.28 for some
reason, then in 3 years we still have glib-2.28 (in the core) because an
exception to the process was granted earlier.. at some point the exceptions need
to be reconsidered and no longer valid.

Ok, so taking all of the above -- and everything below... (see inline..)

On 2/22/11 12:44 PM, Tom Rini wrote:
> On 02/22/2011 11:35 AM, Koen Kooi wrote:
>>
>> Op 22 feb 2011, om 16:33 heeft Richard Purdie het volgende geschreven:
>>
>>> On Mon, 2011-02-21 at 17:53 -0700, Tom Rini wrote:
>>>> As has been discussed in a few places, there needs to be a policy that
>>>> is followed about how long to retain old recipes within the oe-core
>>>> repository as well as what to do with older versions of things.  There
>>>> is a related question here of when to use DEFAULT_PREFERENCE = "-1".
>>>>
>>>> We also have use cases such as:
>>>> - Public distribution X uses version A (pins)
>>>> - Private distribution X uses version A
>>>> - Upstream provides provides support (new releases) and clear guidelines
>>>> on upgrading for version 4.0 (current), version 3.8 (previous and
>>>> stable) and version 3.6 (further previous, stable).  Upstream is also
>>>> working on version 4.1.x (unstable, active development).
>>>> - Upstream provides no clear policy about what's supported other than
>>>> current.
>>>>
>>>> After thinking about the brief discussion we had during todays TSC
>>>> meeting, I would like to propose the following:
>>>>
>>>> The goal of oe-core is to keep the latest stable version.  Exceptions
>>>> will been needed for the toolchain components where architecture
>>>> requirements or upstream problems may dictate that we need multiple
>>>> versions (but versions for versions sake is not required, we are working
>>>> within an SCM and retrieval of older ".z" type releases is easy).  With
>>>> a release framework of X.Y.Z, going from X.Y.Z to X.Y.$((Z + 1)) should
>>>> be a simple git mv.  When a new stable release happens (using the above
>>>> example where 3.8 and 4.0 are both stable), 4.0 should be introduced
>>>> with a DEFAULT_PREFERENCE of -1 and not moved to be the default until
>>>> validation is complete for both oe-core and meta-oe using this version
>>>> and the defined (separate item...) validation targets.
>>>
>>> I think this DEFAULT_PREFERENCE policy complicates testing a lot and I'm
>>> not sure its intuitive or useful. Just to highlight this, you're
>>> proposing someone increasing the version:
>>>
>>> a) Commit the new version with DP = "-1"
>>> b) Update some testing config to PREFER that version
>>> c) Complete the testing
>>> d) Commit a change removing the DP = "-1", remove the preference
>>> e) Remove the old version
>>>
>>> This assumes that people don't test their changes and I'd rather make
>>> the assumption that for OE core, people do test them. I know Saul queues
>>> up patch sets for testing on our autobuilder before sending pull
>>> requests and I'm expecting others doing patch aggregation to do this
>>> too. We still need to agree on testing for OE-core but I'd suggest what
>>> Yocto is doing works well as a good smoke test.
>>
>> I can give you an example from .dev where this can go wrong: glib 2.28.
>>
>> So for oe-core 2.28 would get added and a test build would get done and pass muster, but problems arise with gvfs in meta-oe, it suddenly doesn't build anymore. Using gvfs 1.7.x makes it build again, but you loose iDevice support.
>> I'm not sure what the right way is to catch these kind of problems, but for breakage prone things (glib, ffmpeg, etc) adding it with DP = -1 might be the best way. But I suspect for most things that dance isn't needed.
> 
> This indeed is the problem I had on my mind with that 5 step dance.  I 
> suppose if we had stuff being test built more widely before merge that 
> problem might have been caught and then we could... say keep 2.28.x as 
> D_P = -1 until everything is happy with it?
> 

my proposal is:

We classify the packages as critical "core" and regular "core".  Critical core
items will require additional review/signoff/etc..

We favor the current stable version... we may have to be the ones to determine
what that is for any given package.  (If the current "stable" can/should not be
used for a reason, it needs to be documented why not.)

An alternative version can and should be provided if there is a specific reason
for it.  This reason needs to be documented somewhere, so it can be determined
over time if the reason remains valid.  The primary exception that I see today
is GPLv2 vs GPLv3. (More then one alternative version should be avoided unless
there is a very very clear reason it needs to be there.)

For items that fit outside of the above, say work on the "next" version of a
component it should be done off in a layer, branch, etc where it can be done and
validated against the oe-core (primarily) and the meta-oe secondarily.

For "critical" packages, I strongly suggest development is done in a layer as it
makes it the easiest to override the core and allow the maximum people to verify
the work.  Once the component is ready, we should be able to pull it into the
oe-core and replace the existing item.

For non-critical components, I'm not sure the best way to handle experimental
(next version) software -- it may turn out this simply isn't a problem with this
class of items.  A standard pull model may be all that is needed for validation
and moving forward.

--Mark