[bitbake-devel] [PATCH 1/7] gitsm.py: Fix when a submodule is defined, but not initialized
Mark Hatle
mark.hatle at windriver.com
Tue Jan 22 20:43:11 UTC 2019
Sorry for the delay in getting a fix back to you..
See:
http://git.openembedded.org/bitbake-contrib/log/?h=mgh/gitsm
top two commits should resolve both this and the CLI11 issue. (I am assuming
the third one is the same problem.)
Can you please give this a try.. if it's working now, I'll send a formal pull
request.
--Mark
On 1/19/19 12:44 AM, Khem Raj wrote:
> Mark
>
> another regression
>
> http://errors.yoctoproject.org/Errors/Details/217455/
>
> http://cgit.openembedded.org/meta-openembedded/tree/meta-oe/recipes-core/dbus/dbus-broker_git.bb?h=master
>
>
>
> On Fri, Jan 18, 2019 at 5:27 PM Khem Raj <raj.khem at gmail.com> wrote:
>>
>> I cherry picked this commit on top of master and I still see failure to unpack
>>
>> ERROR: cli11-1.6.2+gitAUTOINC+bd4dc91184-r0 do_unpack: Fetcher
>> failure: Fetch command export PSEUDO_DISABLED=1; export
>> DBUS_SESSION_BUS_ADDRESS="unix:path=/run/user/1000/bus"; export
>> SSH_AGENT_PID="1235"; export SSH_AUTH_SOCK="/tmp/ssh-agent.sock.1000";
>> export PATH="/mnt/a/yoe/build/tmp/sysroots-uninative/x86_64-linux/usr/bin:/mnt/a/yoe/sources/openembedded-core/scripts:/mnt/a/yoe/build/tmp/work/cortexa5t2hf-neon-vfpv4-yoe-linux-musleabi/cli11/1.6.2+gitAUTOINC+bd4dc91184-r0/recipe-sysroot-native/usr/bin/arm-yoe-linux-musleabi:/mnt/a/yoe/build/tmp/work/cortexa5t2hf-neon-vfpv4-yoe-linux-musleabi/cli11/1.6.2+gitAUTOINC+bd4dc91184-r0/recipe-sysroot/usr/bin/crossscripts:/mnt/a/yoe/build/tmp/work/cortexa5t2hf-neon-vfpv4-yoe-linux-musleabi/cli11/1.6.2+gitAUTOINC+bd4dc91184-r0/recipe-sysroot-native/usr/sbin:/mnt/a/yoe/build/tmp/work/cortexa5t2hf-neon-vfpv4-yoe-linux-musleabi/cli11/1.6.2+gitAUTOINC+bd4dc91184-r0/recipe-sysroot-native/usr/bin:/mnt/a/yoe/build/tmp/work/cortexa5t2hf-neon-vfpv4-yoe-linux-musleabi/cli11/1.6.2+gitAUTOINC+bd4dc91184-r0/recipe-sysroot-native/sbin:/mnt/a/yoe/build/tmp/work/cortexa5t2hf-neon-vfpv4-yoe-linux-musleabi/cli11/1.6.2+gitAUTOINC+bd4dc91184-r0/recipe-sysroot-native/bin:/mnt/a/yoe/sources/bitbake/bin:/mnt/a/yoe/build/tmp/hosttools";
>> export HOME="/home/kraj"; git -c core.fsyncobjectfiles=0 config
>> core.bare false
>> ERROR: cli11-1.6.2+gitAUTOINC+bd4dc91184-r0 do_unpack: Function
>> failed: base_do_unpack
>> ERROR: Logfile of failure stored in:
>> /mnt/a/yoe/build/tmp/work/cortexa5t2hf-neon-vfpv4-yoe-linux-musleabi/cli11/1.6.2+gitAUTOINC+bd4dc91184-r0/temp/log.do_unpack.10171
>> ERROR: Task (/mnt/a/yoe/sources/meta-openembedded/meta-oe/recipes-support/cli11/cli11_1.6.2.bb:do_unpack)
>> failed with exit code '1'
>>
>> My bitbake tree is here
>>
>> https://github.com/YoeDistro/bitbake/commits/yoe/mut
>>
>> On Fri, Jan 18, 2019 at 3:06 PM Mark Hatle <mark.hatle at windriver.com> wrote:
>>>
>>> I think I have a fix for this. Can you try the bitbake available from:
>>>
>>> git://git.openembedded.org/bitbake-contrib
>>>
>>> branch mgh/gitsm
>>>
>>> The commit that resolves this issue is:
>>>
>>> 0a7d8bd99a5ba394b00ed35593101b4cd324a58d
>>>
>>> --Mark
>>>
>>> On 1/18/19 3:20 PM, Mark Hatle wrote:
>>>> On 1/18/19 12:31 PM, Khem Raj wrote:
>>>>> On Fri, Jan 18, 2019 at 10:08 AM Mark Hatle <mark.hatle at windriver.com> wrote:
>>>>>>
>>>>>> On 1/18/19 9:07 AM, Khem Raj wrote:
>>>>>>> I am seeing
>>>>>>>
>>>>>>> http://errors.yoctoproject.org/Errors/Details/217408/
>>>>>>> http://errors.yoctoproject.org/Errors/Details/217412/
>>>>>>
>>>>>> Do you know where these failing recipes are so that I can reproduce it locally?
>>>>>>
>>>>>> rwmem looks like it might be meta-ti?
>>>>>> cli11 looks like something from meta-oe.
>>>>>>
>>>>>
>>>>> yes thats right.
>>>>> http://cgit.openembedded.org/meta-openembedded/tree/meta-oe/recipes-support/cli11/cli11_1.6.2.bb?h=master
>>>>>
>>>>> gitsm version of rwmem is not yes applied in meta-ti so you might have
>>>>> to get it from my staging branch
>>>>>
>>>>> https://github.com/YoeDistro/meta-ti/commit/3418beda1869882d5f94dc405d2727f09ffe20c4
>>>>
>>>> The reproducer appears to be that these repositories are 'git' based, but use
>>>> relative submodule locations from the main repository.
>>>>
>>>> So now the gitsm code needs to resolve relative URLs, which is something not
>>>> previously implemented.
>>>>
>>>> --Mark
>>>>
>>>>>> --Mark
>>>>>>
>>>>>>> On Tue, Jan 15, 2019 at 1:33 PM Mark Hatle <mark.hatle at windriver.com> wrote:
>>>>>>>>
>>>>>>>> It is possible for a submodule to be defined in the .gitmodules file, but
>>>>>>>> never initialized in the repository itself. This shows itself when searching
>>>>>>>> for the defined module hash you will get back a empty value.
>>>>>>>>
>>>>>>>> Similarly we need to identify and skip defined but not initialized submodules
>>>>>>>> during the unpack stages as well.
>>>>>>>>
>>>>>>>> Thanks to raphael.lisicki at siemens.com for their help is figuring out how
>>>>>>>> to resolve this issue.
>>>>>>>>
>>>>>>>> Additionally a problem was found where, while unlikely, it may be possible
>>>>>>>> for the wrong revision to have been searched using ls-tree. This has been
>>>>>>>> resolved in the update_submodules function by keeping the correct revision
>>>>>>>> along with the submodule path.
>>>>>>>>
>>>>>>>> Signed-off-by: Mark Hatle <mark.hatle at windriver.com>
>>>>>>>> ---
>>>>>>>> lib/bb/fetch2/gitsm.py | 32 ++++++++++++++++++++++++++++----
>>>>>>>> 1 file changed, 28 insertions(+), 4 deletions(-)
>>>>>>>>
>>>>>>>> diff --git a/lib/bb/fetch2/gitsm.py b/lib/bb/fetch2/gitsm.py
>>>>>>>> index 35729db..b7959ff 100644
>>>>>>>> --- a/lib/bb/fetch2/gitsm.py
>>>>>>>> +++ b/lib/bb/fetch2/gitsm.py
>>>>>>>> @@ -64,6 +64,7 @@ class GitSM(Git):
>>>>>>>> def update_submodules(self, ud, d):
>>>>>>>> submodules = []
>>>>>>>> paths = {}
>>>>>>>> + revision = {}
>>>>>>>> uris = {}
>>>>>>>> local_paths = {}
>>>>>>>>
>>>>>>>> @@ -77,6 +78,7 @@ class GitSM(Git):
>>>>>>>> for m, md in self.parse_gitmodules(gitmodules).items():
>>>>>>>> submodules.append(m)
>>>>>>>> paths[m] = md['path']
>>>>>>>> + revision[m] = ud.revisions[name]
>>>>>>>> uris[m] = md['url']
>>>>>>>> if uris[m].startswith('..'):
>>>>>>>> newud = copy.copy(ud)
>>>>>>>> @@ -84,7 +86,17 @@ class GitSM(Git):
>>>>>>>> uris[m] = Git._get_repo_url(self, newud)
>>>>>>>>
>>>>>>>> for module in submodules:
>>>>>>>> - module_hash = runfetchcmd("%s ls-tree -z -d %s %s" % (ud.basecmd, ud.revisions[name], paths[module]), d, quiet=True, workdir=ud.clonedir)
>>>>>>>> + try:
>>>>>>>> + module_hash = runfetchcmd("%s ls-tree -z -d %s %s" % (ud.basecmd, revision[module], paths[module]), d, quiet=True, workdir=ud.clonedir)
>>>>>>>> + except:
>>>>>>>> + # If the command fails, we don't have a valid file to check. If it doesn't
>>>>>>>> + # fail -- it still might be a failure, see next check...
>>>>>>>> + module_hash = ""
>>>>>>>> +
>>>>>>>> + if not module_hash:
>>>>>>>> + logger.debug(1, "submodule %s is defined, but is not initialized in the repository. Skipping", module)
>>>>>>>> + continue
>>>>>>>> +
>>>>>>>> module_hash = module_hash.split()[2]
>>>>>>>>
>>>>>>>> # Build new SRC_URI
>>>>>>>> @@ -143,7 +155,7 @@ class GitSM(Git):
>>>>>>>> if not ud.shallow or ud.localpath != ud.fullshallow:
>>>>>>>> self.update_submodules(ud, d)
>>>>>>>>
>>>>>>>> - def copy_submodules(self, submodules, ud, destdir, d):
>>>>>>>> + def copy_submodules(self, submodules, ud, name, destdir, d):
>>>>>>>> if ud.bareclone:
>>>>>>>> repo_conf = destdir
>>>>>>>> else:
>>>>>>>> @@ -156,6 +168,18 @@ class GitSM(Git):
>>>>>>>> srcpath = os.path.join(ud.clonedir, 'modules', md['path'])
>>>>>>>> modpath = os.path.join(repo_conf, 'modules', md['path'])
>>>>>>>>
>>>>>>>> + # Check if the module is initialized
>>>>>>>> + try:
>>>>>>>> + module_hash = runfetchcmd("%s ls-tree -z -d %s %s" % (ud.basecmd, ud.revisions[name], md['path']), d, quiet=True, workdir=ud.clonedir)
>>>>>>>> + except:
>>>>>>>> + # If the command fails, we don't have a valid file to check. If it doesn't
>>>>>>>> + # fail -- it still might be a failure, see next check...
>>>>>>>> + module_hash = ""
>>>>>>>> +
>>>>>>>> + if not module_hash:
>>>>>>>> + logger.debug(1, "submodule %s is defined, but is not initialized in the repository. Skipping", module)
>>>>>>>> + continue
>>>>>>>> +
>>>>>>>> if os.path.exists(srcpath):
>>>>>>>> if os.path.exists(os.path.join(srcpath, '.git')):
>>>>>>>> srcpath = os.path.join(srcpath, '.git')
>>>>>>>> @@ -188,7 +212,7 @@ class GitSM(Git):
>>>>>>>> continue
>>>>>>>>
>>>>>>>> submodules = self.parse_gitmodules(gitmodules)
>>>>>>>> - self.copy_submodules(submodules, ud, dest, d)
>>>>>>>> + self.copy_submodules(submodules, ud, name, dest, d)
>>>>>>>>
>>>>>>>> def unpack(self, ud, destdir, d):
>>>>>>>> Git.unpack(self, ud, destdir, d)
>>>>>>>> @@ -211,7 +235,7 @@ class GitSM(Git):
>>>>>>>> continue
>>>>>>>>
>>>>>>>> submodules = self.parse_gitmodules(gitmodules)
>>>>>>>> - self.copy_submodules(submodules, ud, ud.destdir, d)
>>>>>>>> + self.copy_submodules(submodules, ud, name, ud.destdir, d)
>>>>>>>>
>>>>>>>> submodules_queue = [(module, os.path.join(repo_conf, 'modules', md['path'])) for module, md in submodules.items()]
>>>>>>>> while len(submodules_queue) != 0:
>>>>>>>> --
>>>>>>>> 1.8.3.1
>>>>>>>>
>>>>>>>> --
>>>>>>>> _______________________________________________
>>>>>>>> bitbake-devel mailing list
>>>>>>>> bitbake-devel at lists.openembedded.org
>>>>>>>> http://lists.openembedded.org/mailman/listinfo/bitbake-devel
>>>>>>
>>>>
>>>
More information about the bitbake-devel
mailing list