[bitbake-devel] [PATCH 1/7] gitsm.py: Fix when a submodule is defined, but not initialized
Khem Raj
raj.khem at gmail.com
Wed Jan 23 03:36:27 UTC 2019
It worked.
On Tue, Jan 22, 2019 at 3:43 PM Mark Hatle <mark.hatle at windriver.com> wrote:
>
> Sorry for the delay in getting a fix back to you..
>
> See:
>
> http://git.openembedded.org/bitbake-contrib/log/?h=mgh/gitsm
>
> top two commits should resolve both this and the CLI11 issue. (I am assuming
> the third one is the same problem.)
>
> Can you please give this a try.. if it's working now, I'll send a formal pull
> request.
>
> --Mark
>
> On 1/19/19 12:44 AM, Khem Raj wrote:
> > Mark
> >
> > another regression
> >
> > http://errors.yoctoproject.org/Errors/Details/217455/
> >
> > http://cgit.openembedded.org/meta-openembedded/tree/meta-oe/recipes-core/dbus/dbus-broker_git.bb?h=master
> >
> >
> >
> > On Fri, Jan 18, 2019 at 5:27 PM Khem Raj <raj.khem at gmail.com> wrote:
> >>
> >> I cherry picked this commit on top of master and I still see failure to unpack
> >>
> >> ERROR: cli11-1.6.2+gitAUTOINC+bd4dc91184-r0 do_unpack: Fetcher
> >> failure: Fetch command export PSEUDO_DISABLED=1; export
> >> DBUS_SESSION_BUS_ADDRESS="unix:path=/run/user/1000/bus"; export
> >> SSH_AGENT_PID="1235"; export SSH_AUTH_SOCK="/tmp/ssh-agent.sock.1000";
> >> export PATH="/mnt/a/yoe/build/tmp/sysroots-uninative/x86_64-linux/usr/bin:/mnt/a/yoe/sources/openembedded-core/scripts:/mnt/a/yoe/build/tmp/work/cortexa5t2hf-neon-vfpv4-yoe-linux-musleabi/cli11/1.6.2+gitAUTOINC+bd4dc91184-r0/recipe-sysroot-native/usr/bin/arm-yoe-linux-musleabi:/mnt/a/yoe/build/tmp/work/cortexa5t2hf-neon-vfpv4-yoe-linux-musleabi/cli11/1.6.2+gitAUTOINC+bd4dc91184-r0/recipe-sysroot/usr/bin/crossscripts:/mnt/a/yoe/build/tmp/work/cortexa5t2hf-neon-vfpv4-yoe-linux-musleabi/cli11/1.6.2+gitAUTOINC+bd4dc91184-r0/recipe-sysroot-native/usr/sbin:/mnt/a/yoe/build/tmp/work/cortexa5t2hf-neon-vfpv4-yoe-linux-musleabi/cli11/1.6.2+gitAUTOINC+bd4dc91184-r0/recipe-sysroot-native/usr/bin:/mnt/a/yoe/build/tmp/work/cortexa5t2hf-neon-vfpv4-yoe-linux-musleabi/cli11/1.6.2+gitAUTOINC+bd4dc91184-r0/recipe-sysroot-native/sbin:/mnt/a/yoe/build/tmp/work/cortexa5t2hf-neon-vfpv4-yoe-linux-musleabi/cli11/1.6.2+gitAUTOINC+bd4dc91184-r0/recipe-sysroot-native/bin:/mnt/a/yoe/sources/bitbake/bin:/mnt/a/yoe/build/tmp/hosttools";
> >> export HOME="/home/kraj"; git -c core.fsyncobjectfiles=0 config
> >> core.bare false
> >> ERROR: cli11-1.6.2+gitAUTOINC+bd4dc91184-r0 do_unpack: Function
> >> failed: base_do_unpack
> >> ERROR: Logfile of failure stored in:
> >> /mnt/a/yoe/build/tmp/work/cortexa5t2hf-neon-vfpv4-yoe-linux-musleabi/cli11/1.6.2+gitAUTOINC+bd4dc91184-r0/temp/log.do_unpack.10171
> >> ERROR: Task (/mnt/a/yoe/sources/meta-openembedded/meta-oe/recipes-support/cli11/cli11_1.6.2.bb:do_unpack)
> >> failed with exit code '1'
> >>
> >> My bitbake tree is here
> >>
> >> https://github.com/YoeDistro/bitbake/commits/yoe/mut
> >>
> >> On Fri, Jan 18, 2019 at 3:06 PM Mark Hatle <mark.hatle at windriver.com> wrote:
> >>>
> >>> I think I have a fix for this. Can you try the bitbake available from:
> >>>
> >>> git://git.openembedded.org/bitbake-contrib
> >>>
> >>> branch mgh/gitsm
> >>>
> >>> The commit that resolves this issue is:
> >>>
> >>> 0a7d8bd99a5ba394b00ed35593101b4cd324a58d
> >>>
> >>> --Mark
> >>>
> >>> On 1/18/19 3:20 PM, Mark Hatle wrote:
> >>>> On 1/18/19 12:31 PM, Khem Raj wrote:
> >>>>> On Fri, Jan 18, 2019 at 10:08 AM Mark Hatle <mark.hatle at windriver.com> wrote:
> >>>>>>
> >>>>>> On 1/18/19 9:07 AM, Khem Raj wrote:
> >>>>>>> I am seeing
> >>>>>>>
> >>>>>>> http://errors.yoctoproject.org/Errors/Details/217408/
> >>>>>>> http://errors.yoctoproject.org/Errors/Details/217412/
> >>>>>>
> >>>>>> Do you know where these failing recipes are so that I can reproduce it locally?
> >>>>>>
> >>>>>> rwmem looks like it might be meta-ti?
> >>>>>> cli11 looks like something from meta-oe.
> >>>>>>
> >>>>>
> >>>>> yes thats right.
> >>>>> http://cgit.openembedded.org/meta-openembedded/tree/meta-oe/recipes-support/cli11/cli11_1.6.2.bb?h=master
> >>>>>
> >>>>> gitsm version of rwmem is not yes applied in meta-ti so you might have
> >>>>> to get it from my staging branch
> >>>>>
> >>>>> https://github.com/YoeDistro/meta-ti/commit/3418beda1869882d5f94dc405d2727f09ffe20c4
> >>>>
> >>>> The reproducer appears to be that these repositories are 'git' based, but use
> >>>> relative submodule locations from the main repository.
> >>>>
> >>>> So now the gitsm code needs to resolve relative URLs, which is something not
> >>>> previously implemented.
> >>>>
> >>>> --Mark
> >>>>
> >>>>>> --Mark
> >>>>>>
> >>>>>>> On Tue, Jan 15, 2019 at 1:33 PM Mark Hatle <mark.hatle at windriver.com> wrote:
> >>>>>>>>
> >>>>>>>> It is possible for a submodule to be defined in the .gitmodules file, but
> >>>>>>>> never initialized in the repository itself. This shows itself when searching
> >>>>>>>> for the defined module hash you will get back a empty value.
> >>>>>>>>
> >>>>>>>> Similarly we need to identify and skip defined but not initialized submodules
> >>>>>>>> during the unpack stages as well.
> >>>>>>>>
> >>>>>>>> Thanks to raphael.lisicki at siemens.com for their help is figuring out how
> >>>>>>>> to resolve this issue.
> >>>>>>>>
> >>>>>>>> Additionally a problem was found where, while unlikely, it may be possible
> >>>>>>>> for the wrong revision to have been searched using ls-tree. This has been
> >>>>>>>> resolved in the update_submodules function by keeping the correct revision
> >>>>>>>> along with the submodule path.
> >>>>>>>>
> >>>>>>>> Signed-off-by: Mark Hatle <mark.hatle at windriver.com>
> >>>>>>>> ---
> >>>>>>>> lib/bb/fetch2/gitsm.py | 32 ++++++++++++++++++++++++++++----
> >>>>>>>> 1 file changed, 28 insertions(+), 4 deletions(-)
> >>>>>>>>
> >>>>>>>> diff --git a/lib/bb/fetch2/gitsm.py b/lib/bb/fetch2/gitsm.py
> >>>>>>>> index 35729db..b7959ff 100644
> >>>>>>>> --- a/lib/bb/fetch2/gitsm.py
> >>>>>>>> +++ b/lib/bb/fetch2/gitsm.py
> >>>>>>>> @@ -64,6 +64,7 @@ class GitSM(Git):
> >>>>>>>> def update_submodules(self, ud, d):
> >>>>>>>> submodules = []
> >>>>>>>> paths = {}
> >>>>>>>> + revision = {}
> >>>>>>>> uris = {}
> >>>>>>>> local_paths = {}
> >>>>>>>>
> >>>>>>>> @@ -77,6 +78,7 @@ class GitSM(Git):
> >>>>>>>> for m, md in self.parse_gitmodules(gitmodules).items():
> >>>>>>>> submodules.append(m)
> >>>>>>>> paths[m] = md['path']
> >>>>>>>> + revision[m] = ud.revisions[name]
> >>>>>>>> uris[m] = md['url']
> >>>>>>>> if uris[m].startswith('..'):
> >>>>>>>> newud = copy.copy(ud)
> >>>>>>>> @@ -84,7 +86,17 @@ class GitSM(Git):
> >>>>>>>> uris[m] = Git._get_repo_url(self, newud)
> >>>>>>>>
> >>>>>>>> for module in submodules:
> >>>>>>>> - module_hash = runfetchcmd("%s ls-tree -z -d %s %s" % (ud.basecmd, ud.revisions[name], paths[module]), d, quiet=True, workdir=ud.clonedir)
> >>>>>>>> + try:
> >>>>>>>> + module_hash = runfetchcmd("%s ls-tree -z -d %s %s" % (ud.basecmd, revision[module], paths[module]), d, quiet=True, workdir=ud.clonedir)
> >>>>>>>> + except:
> >>>>>>>> + # If the command fails, we don't have a valid file to check. If it doesn't
> >>>>>>>> + # fail -- it still might be a failure, see next check...
> >>>>>>>> + module_hash = ""
> >>>>>>>> +
> >>>>>>>> + if not module_hash:
> >>>>>>>> + logger.debug(1, "submodule %s is defined, but is not initialized in the repository. Skipping", module)
> >>>>>>>> + continue
> >>>>>>>> +
> >>>>>>>> module_hash = module_hash.split()[2]
> >>>>>>>>
> >>>>>>>> # Build new SRC_URI
> >>>>>>>> @@ -143,7 +155,7 @@ class GitSM(Git):
> >>>>>>>> if not ud.shallow or ud.localpath != ud.fullshallow:
> >>>>>>>> self.update_submodules(ud, d)
> >>>>>>>>
> >>>>>>>> - def copy_submodules(self, submodules, ud, destdir, d):
> >>>>>>>> + def copy_submodules(self, submodules, ud, name, destdir, d):
> >>>>>>>> if ud.bareclone:
> >>>>>>>> repo_conf = destdir
> >>>>>>>> else:
> >>>>>>>> @@ -156,6 +168,18 @@ class GitSM(Git):
> >>>>>>>> srcpath = os.path.join(ud.clonedir, 'modules', md['path'])
> >>>>>>>> modpath = os.path.join(repo_conf, 'modules', md['path'])
> >>>>>>>>
> >>>>>>>> + # Check if the module is initialized
> >>>>>>>> + try:
> >>>>>>>> + module_hash = runfetchcmd("%s ls-tree -z -d %s %s" % (ud.basecmd, ud.revisions[name], md['path']), d, quiet=True, workdir=ud.clonedir)
> >>>>>>>> + except:
> >>>>>>>> + # If the command fails, we don't have a valid file to check. If it doesn't
> >>>>>>>> + # fail -- it still might be a failure, see next check...
> >>>>>>>> + module_hash = ""
> >>>>>>>> +
> >>>>>>>> + if not module_hash:
> >>>>>>>> + logger.debug(1, "submodule %s is defined, but is not initialized in the repository. Skipping", module)
> >>>>>>>> + continue
> >>>>>>>> +
> >>>>>>>> if os.path.exists(srcpath):
> >>>>>>>> if os.path.exists(os.path.join(srcpath, '.git')):
> >>>>>>>> srcpath = os.path.join(srcpath, '.git')
> >>>>>>>> @@ -188,7 +212,7 @@ class GitSM(Git):
> >>>>>>>> continue
> >>>>>>>>
> >>>>>>>> submodules = self.parse_gitmodules(gitmodules)
> >>>>>>>> - self.copy_submodules(submodules, ud, dest, d)
> >>>>>>>> + self.copy_submodules(submodules, ud, name, dest, d)
> >>>>>>>>
> >>>>>>>> def unpack(self, ud, destdir, d):
> >>>>>>>> Git.unpack(self, ud, destdir, d)
> >>>>>>>> @@ -211,7 +235,7 @@ class GitSM(Git):
> >>>>>>>> continue
> >>>>>>>>
> >>>>>>>> submodules = self.parse_gitmodules(gitmodules)
> >>>>>>>> - self.copy_submodules(submodules, ud, ud.destdir, d)
> >>>>>>>> + self.copy_submodules(submodules, ud, name, ud.destdir, d)
> >>>>>>>>
> >>>>>>>> submodules_queue = [(module, os.path.join(repo_conf, 'modules', md['path'])) for module, md in submodules.items()]
> >>>>>>>> while len(submodules_queue) != 0:
> >>>>>>>> --
> >>>>>>>> 1.8.3.1
> >>>>>>>>
> >>>>>>>> --
> >>>>>>>> _______________________________________________
> >>>>>>>> bitbake-devel mailing list
> >>>>>>>> bitbake-devel at lists.openembedded.org
> >>>>>>>> http://lists.openembedded.org/mailman/listinfo/bitbake-devel
> >>>>>>
> >>>>
> >>>
>
More information about the bitbake-devel
mailing list