[bitbake-devel] [PATCH] Fix mercurial fetching (hosttools need hd for latest revision autoinc, fix username and password usage and setting moddir needed by setup_revisions)

[Leonardo Sandoval]

On Thu, 16 Nov 2017 18:28:16 +0100
Volker Vogelhuber <v.vogelhuber at digitalendoscopy.de> wrote:

> Sorry this was my first patch sent with git send-mail. Seems like I had 
> to learn how to properly compose an email out of it.
> 

no problem.

> Regarding the patch:
> I just tried updating my existing yocto toolchain to rocko and our 
> recipes using mercurial repositories failed during parsing. We're using 
> a Mercurial repository for our internal stuff and also the AUTOREV 
> feature. So during parsing the revision in the recipes the call to 
> /usr/bin/env hg failed because it is not part of the HOSTTOOLS. As it 
> seems like bitbake is now replacing the PATH environment variable a 
> mercurial installation on the host system is not found anymore. I saw 
> that there seem to be some automatic dependency check which results in a 
> dependency to mercurial-native, but as far as I analyzed it, this comes 
> too late for the revision parsing as mercurial has to be available 
> already for that case. That's why I added hg to HOSTTOOLS_NONFATAL which 
> results in the automatic creation of a symbol link in the HOSTTOOLS_DIR 
> which solves the problem.
> The second thing is that the setup_revisions call will access ud.moddir 
> which is not yet set in the current implementation. That's why I moved 
> it to the lines below ud.basecmd where ud.moddir has been set already.
> And the third thing is that username and password were not taken into 
> account during unpack. Thats why I added it.
> 

you are introducing changes in two areas/projects (bitbake and oe-core) so you need to split this patch in at least 2. bitbake/* changes go to this mailing list and meta/* changes to openembedded-core@

I let the experts comment on your code changes


> Kind regards
>     Volker
> 
> On 16.11.2017 18:19, Leonardo Sandoval wrote:
> > 
> > subject should be shorter answering the 'what this patch does' and the description the 'why this is needed'
> > 
> > 
> > On Thu, 16 Nov 2017 15:15:41 +0100
> > Volker Vogelhuber <v.vogelhuber at digitalendoscopy.de> wrote:
> >   
> >> Signed-off-by: Volker Vogelhuber <v.vogelhuber at digitalendoscopy.de>
> >> ---
> >>   bitbake/lib/bb/fetch2/hg.py | 27 +++++++++++++++++----------
> >>   meta/conf/bitbake.conf      |  2 ++
> >>   2 files changed, 19 insertions(+), 10 deletions(-)
> >>
> >> diff --git a/bitbake/lib/bb/fetch2/hg.py b/bitbake/lib/bb/fetch2/hg.py
> >> index d0857e63f7..04620f3722 100644
> >> --- a/bitbake/lib/bb/fetch2/hg.py
> >> +++ b/bitbake/lib/bb/fetch2/hg.py
> >> @@ -66,13 +66,6 @@ class Hg(FetchMethod):
> >>           else:
> >>               ud.proto = "hg"
> >>   
> >> -        ud.setup_revisions(d)
> >> -
> >> -        if 'rev' in ud.parm:
> >> -            ud.revision = ud.parm['rev']
> >> -        elif not ud.revision:
> >> -            ud.revision = self.latest_revision(ud, d)
> >> -
> >>           # Create paths to mercurial checkouts
> >>           hgsrcname = '%s_%s_%s' % (ud.module.replace('/', '.'), \
> >>                               ud.host, ud.path.replace('/', '.'))
> >> @@ -86,6 +79,13 @@ class Hg(FetchMethod):
> >>           ud.localfile = ud.moddir
> >>           ud.basecmd = d.getVar("FETCHCMD_hg") or "/usr/bin/env hg"
> >>   
> >> +        ud.setup_revisions(d)
> >> +
> >> +        if 'rev' in ud.parm:
> >> +            ud.revision = ud.parm['rev']
> >> +        elif not ud.revision:
> >> +            ud.revision = self.latest_revision(ud, d)
> >> +
> >>           ud.write_tarballs = d.getVar("BB_GENERATE_MIRROR_TARBALLS")
> >>   
> >>       def need_update(self, ud, d):
> >> @@ -151,7 +151,7 @@ class Hg(FetchMethod):
> >>                   cmd = "%s --config auth.default.prefix=* --config auth.default.username=%s --config auth.default.password=%s --config \"auth.default.schemes=%s\" pull" % (ud.basecmd, ud.user, ud.pswd, proto)
> >>               else:
> >>                   cmd = "%s pull" % (ud.basecmd)
> >> -        elif command == "update":
> >> +        elif command == "update" or command == "up":
> >>               if ud.user and ud.pswd:
> >>                   cmd = "%s --config auth.default.prefix=* --config auth.default.username=%s --config auth.default.password=%s --config \"auth.default.schemes=%s\" update -C %s" % (ud.basecmd, ud.user, ud.pswd, proto, " ".join(options))
> >>               else:
> >> @@ -259,12 +259,19 @@ class Hg(FetchMethod):
> >>   
> >>           scmdata = ud.parm.get("scmdata", "")
> >>           if scmdata != "nokeep":
> >> +            proto = ud.parm.get('protocol', 'http')
> >>               if not os.access(os.path.join(codir, '.hg'), os.R_OK):
> >>                   logger.debug(2, "Unpack: creating new hg repository in '" + codir + "'")
> >>                   runfetchcmd("%s init %s" % (ud.basecmd, codir), d)
> >>               logger.debug(2, "Unpack: updating source in '" + codir + "'")
> >> -            runfetchcmd("%s pull %s" % (ud.basecmd, ud.moddir), d, workdir=codir)
> >> -            runfetchcmd("%s up -C %s" % (ud.basecmd, revflag), d, workdir=codir)
> >> +            if ud.user and ud.pswd:
> >> +                runfetchcmd("%s --config auth.default.prefix=* --config auth.default.username=%s --config auth.default.password=%s --config \"auth.default.schemes=%s\" pull %s" % (ud.basecmd, ud.user, ud.pswd, proto, ud.moddir), d, workdir=codir)
> >> +            else:
> >> +                runfetchcmd("%s pull %s" % (ud.basecmd, ud.moddir), d, workdir=codir)
> >> +            if ud.user and ud.pswd:
> >> +                runfetchcmd("%s --config auth.default.prefix=* --config auth.default.username=%s --config auth.default.password=%s --config \"auth.default.schemes=%s\" up -C %s" % (ud.basecmd, ud.user, ud.pswd, proto, revflag), d, workdir=codir)
> >> +            else:
> >> +                runfetchcmd("%s up -C %s" % (ud.basecmd, revflag), d, workdir=codir)
> >>           else:
> >>               logger.debug(2, "Unpack: extracting source to '" + codir + "'")
> >>               runfetchcmd("%s archive -t files %s %s" % (ud.basecmd, revflag, codir), d, workdir=ud.moddir)
> >> diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
> >> index 9696273acc..9ec527228b 100644
> >> --- a/meta/conf/bitbake.conf
> >> +++ b/meta/conf/bitbake.conf
> >> @@ -494,6 +494,8 @@ HOSTTOOLS_NONFATAL += "join nl size yes zcat"
> >>   
> >>   # Used by bzr fetcher
> >>   HOSTTOOLS_NONFATAL += "bzr"
> >> +# Used by hg fetcher
> >> +HOSTTOOLS_NONFATAL += "hg"
> >>   
> >>   CCACHE ??= ""
> >>   # ccache < 3.1.10 will create CCACHE_DIR on startup even if disabled, and
> >> -- 
> >> 2.11.0
> >>
> >> -- 
> >> _______________________________________________
> >> bitbake-devel mailing list
> >> bitbake-devel at lists.openembedded.org
> >> http://lists.openembedded.org/mailman/listinfo/bitbake-devel