[oe] PRINC migration questions

[Bryan Evenson]

Martin,

> -----Original Message-----
> From: Martin Jansa [mailto:martin.jansa at gmail.com]
> Sent: Friday, November 07, 2014 6:21 PM
> To: Bryan Evenson
> Cc: openembedded-devel at lists.openembedded.org
> Subject: Re: [oe] PRINC migration questions
> 
> On Fri, Nov 07, 2014 at 03:36:14PM +0000, Bryan Evenson wrote:
> > Martin,
> >
> > > -----Original Message-----
> > > From: openembedded-devel-bounces at lists.openembedded.org
> > > [mailto:openembedded-devel-bounces at lists.openembedded.org] On
> Behalf
> > > Of Martin Jansa
> > > Sent: Friday, November 07, 2014 9:36 AM
> > > To: openembedded-devel at lists.openembedded.org
> > > Subject: Re: [oe] PRINC migration questions
> > >
> > > On Fri, Nov 07, 2014 at 01:31:02PM +0000, Bryan Evenson wrote:
> > > > All,
> > > >
> > > > I am on poky/dylan and have not yet started using the PR server.
> > > > I really do
> > > want to start using the PR server and stop using PRINC.  However,
> > > even more so I really don't want to break my package feeds.  I want
> > > to make sure I do my migration correctly and I don't do something
> > > that looks like it works okay only to find out it causes problems
> > > later.  With that in mind, I am looking for advice on the proper way to
> migrate away from using PRINC.
> > > >
> > > > Let's say I have a .bbappend in my private layer with the line:
> > > >
> > > > PRINC := "${@int(PRINC) + 4}"
> > > >
> > > > The mainline layer's .bb has the line:
> > > >
> > > > PR = "${INC_PR}.0"
> > > >
> > > > And the mainline layer's .inc has the line:
> > > >
> > > > INC_PR = "r8"
> > > >
> > > > In this scenario, the resulting PR will be "r12.0".  Now let's say
> > > > I want to start
> > > using the PR server and get rid of PRINC in this recipe.  The
> > > resulting package contents are to be the same, so I want the
> > > resulting PR to be "r12.0".  From some limited testing, if I start
> > > the PR server and remove the PRINC line from my .bbappend the
> > > resulting PR is "r8.0".  So in my .bbappend should I change the PRINC line
> to:
> > > >
> > > > INC_PR = "r12"
> > > >
> > > > Or should I change it to something else?  If I change it to a
> > > > hardcoded value,
> > > then I'll have to be careful about what to do if the mainline recipe
> > > changes INC_PR.  How have other people handled this situation?
> > >
> > > PR service won't you help at all with this (it adds another .N to PR
> > > value, so it cannot fix when the level of .N goes backwards).
> > >
> > > The only really working solution is to increment INC_PR/PR values in
> > > upstream recipes atomically with PRINC drop in your layers.
> > >
> >
> > Thanks for the information.  Just a few more questions to make sure I get
> this right.  We have a local Git server which has clones of all the Yocto/oe
> layers that we are using.  So your suggestion for the example scenario would
> be to:
> > 1. Remove PRINC reference from our .bbappend 2. Change .inc in
> > mainline recipe to INC_PR = "r12"
> > 3. Push changes to our local Git repositories.
> >
> > Then if later I were to merge in updates from the official layers, there
> would be a merge conflict and I would know at merge time that I need to
> inspect INC_PR for being the correct value.  Does that sound right?
> 
> Yes that's possible, a bit of pain to maintain it, but as you say, at least you get
> the warning in form of merge conflict. Just keep in mind that you can drop
> these PR, INC_PR bumps when the upstream recipe increments PE or PV so
> you can at least drop some of them in time.
> 
> > > For PR service migration the more important part is to correctly
> > > migrate LOCALCOUNT numbers (used in SRCPVs), especially if you're
> > > also changing package architecture (e.g. t2 suffix added in daisy or
> > > selecting different DEFAULTTUNE and all your LOCALCOUNTs in PR
> > > server DB will reset to 0, because of different db key to find them.
> > > Luckily you can pre-populate them with single SQL command if you realize
> this situation before first build.
> > >
> >
> > OK, now I'm confused.  I see no reference to LOCALCOUNT anywhere in
> the Yocto Project Mega Manual.  The only reference I see to LOCALCOUNT is
> on the PR Service wiki which states that you can call "bitbake-prserv-tool
> migrate_localcount" to migrate LOCALCOUNTs to AUTOINCs (and AUTOINC is
> another variable I cannot find mentioned anywhere else).  So I don't fully
> understand how to recognize if I need to migrate LOCALCOUNT numbers or
> what the SQL command would be to perform the migration.  But it sounds
> like this is a pretty important piece to get right during migration.  Could you
> explain this piece for PR service migration in more detail, or point me to the
> proper place in the documentation that describes this part of migration?
> 
> The important question is if you're using SRCPV in some recipes you care
> about upgrade-path (e.g. linux-yocto is using it).
> 
> LOCALCOUNT was name of variable which had the N number in SRCPV (e.g.
> "gitrN+<hash>"), originally it was stored in separate sqlite database
> cache/bb_persist_data.sqlite3 and maintenance of this number was moved
> also to PR service, that's why it's not referenced in current documentation
> anymore.
> 
> AUTOINC has the same meaning, basically the fetcher will ask PR service:
> "do you have AUTOINC number for revision "deadcafe1234", recipe "abc",
> architecture "cortexa9t2-neon" and PR service will reply either with already
> stored number or max+1 for given recipe+architecture tuple and store the
> returned number.
> 
> It's important for the migration, because if you depend on N numbers in
> gitrN to be from always increasing sequence you'll get zero for all recipes if
> you don't migrate your old cache/bb_persist_data.sqlite3 to PR service's
> sqlite db.
> 
> Honestly I did that all and it was sort of working after few manual fixes, but
> I've already gave up on trusting and testing upgrade paths :/.
> 
> It's possible to use public PR service with multiple builders, but there are
> some cases where the current design wont work and other cases where it
> just isn't reliable enough to trust it - if you want reliable images on target you
> have to reflash them from time to time (or really know what you're doing
> and what you're expecting to get every time you type "opkg update && opkg
> upgrade" and most end-users don't know).
> 

Thanks for the detailed explanation.  In other words, before release take a close look at what recipes changed and what PRs updated in the resulting packages.  I think all this information would be helpful in a "PRINC Migration" section in the manual.  Given personal experience, I think a lot of people have not migrated away from using PRINC because they may not know how to without breaking package feeds.

Regards,
Bryan