[bitbake-devel] [PATCH 1/1] bitbake: Disable pseudo in runfetchcmd()

Mon Mar 24 12:45:27 UTC 2014

On 14-03-22 11:12 +0100, Richard Purdie wrote:
> On Fri, 2014-03-21 at 09:27 +0100, Olof Johansson wrote:
> > On 14-03-20 16:14 +0100, Richard Purdie wrote:
> > > The log helps immensely since it shows its "Failure expanding variable
> > > do_patch" so its not running do_patch, it is trying to expand the
> > > variable for some reason when writing the rpm in do_package_write_rpm. 
> > > 
> > > Are you generating some kind of srpm differently from the normal
> > > usecases?
> > 
> > No we don't, as far as I know (is there something I should look
> > for?). If this isn't something we are doing, what makes it a
> > problem for us is the fetching of kernel sources from git over
> > ssh with key authentication. Not sure how common that is.
> 
> Are you using the archiver classes?

> I'm wondering if
> 
> http://git.yoctoproject.org/cgit.cgi/poky/commit/?id=2c7d20ce3849aba84e0552f0fc4bde649d96faa9
> 
> might resolve the issue.

We don't; grepping for 'archive' in our layer gives me zero hits.
And I can still reproduce the issue.

> Somehow you're referencing do_patch in the rpm write task and
> I'd like to figure out how. We should not be hitting the
> network during anything other than do_fetch...

I added an exit(1) in the git fetcher to get a traceback:

Traceback (most recent call last):
  File "/var/opt/builds/olofjn/oe/master/bitbake/bin/bitbake-worker", line 365, in <module>
    worker.serve()
  File "/var/opt/builds/olofjn/oe/master/bitbake/bin/bitbake-worker", line 267, in serve
    self.handle_item("runtask", self.handle_runtask)
  File "/var/opt/builds/olofjn/oe/master/bitbake/bin/bitbake-worker", line 283, in handle_item
    func(self.queue[(len(item) + 2):index])
  File "/var/opt/builds/olofjn/oe/master/bitbake/bin/bitbake-worker", line 317, in handle_runtask
    pid, pipein, pipeout = fork_off_task(self.cookercfg, self.data, self.workerdata, fn, task, taskname, appends, taskdepdata, quieterrors)
  File "/var/opt/builds/olofjn/oe/master/bitbake/bin/bitbake-worker", line 160, in fork_off_task
    the_data = bb.cache.Cache.loadDataFull(fn, appends, data)
  File "/var/opt/builds/olofjn/oe/master/bitbake/lib/bb/cache.py", line 398, in loadDataFull
    bb_data = cls.load_bbfile(fn, appends, cfgData)
  File "/var/opt/builds/olofjn/oe/master/bitbake/lib/bb/cache.py", line 673, in load_bbfile
    bb_data = parse.handle(bbfile, bb_data)
  File "/var/opt/builds/olofjn/oe/master/bitbake/lib/bb/parse/__init__.py", line 99, in handle
    return h['handle'](fn, data, include)
  File "/var/opt/builds/olofjn/oe/master/bitbake/lib/bb/parse/parse_py/BBHandler.py", line 169, in handle
    return ast.multi_finalize(fn, d)
  File "/var/opt/builds/olofjn/oe/master/bitbake/lib/bb/parse/ast.py", line 386, in multi_finalize
    finalize(fn, d)
  File "/var/opt/builds/olofjn/oe/master/bitbake/lib/bb/parse/ast.py", line 334, in finalize
    bb.parse.siggen.finalise(fn, d, variant)
  File "/var/opt/builds/olofjn/oe/master/bitbake/lib/bb/siggen.py", line 139, in finalise
    taskdeps = self._build_data(fn, d)
  File "/var/opt/builds/olofjn/oe/master/bitbake/lib/bb/siggen.py", line 92, in _build_data
    tasklist, gendeps, lookupcache = bb.data.generate_dependencies(d)
  File "/var/opt/builds/olofjn/oe/master/bitbake/lib/bb/data.py", line 382, in generate_dependencies
    deps[task], values[task] = build_dependencies(task, keys, shelldeps, varflagsexcl, d)
  File "/var/opt/builds/olofjn/oe/master/bitbake/lib/bb/data.py", line 329, in build_dependencies
    parsedvar = d.expandWithRefs(value, key)
  File "/var/opt/builds/olofjn/oe/master/bitbake/lib/bb/data_smart.py", line 332, in expandWithRefs
    s = __expand_python_regexp__.sub(varparse.python_sub, s)
  File "/var/opt/builds/olofjn/oe/master/bitbake/lib/bb/data_smart.py", line 129, in python_sub
    value = utils.better_eval(codeobj, DataContext(self.d))
  File "/var/opt/builds/olofjn/oe/master/bitbake/lib/bb/utils.py", line 378, in better_eval
    return eval(source, get_context(), locals)
  File "do_patch", line 1, in <module>
  File "kernel-yocto.bbclass", line 2, in get_machine_branch
  File "/var/opt/builds/olofjn/oe/master/bitbake/lib/bb/fetch2/__init__.py", line 1336, in __init__
    self.ud[url] = FetchData(url, d, localonly)
  File "/var/opt/builds/olofjn/oe/master/bitbake/lib/bb/fetch2/__init__.py", line 1038, in __init__
    self.method.urldata_init(self, d)
  File "/var/opt/builds/olofjn/oe/master/bitbake/lib/bb/fetch2/git.py", line 137, in urldata_init
    ud.revisions[name] = self.latest_revision(ud, d, name)
  File "/var/opt/builds/olofjn/oe/master/bitbake/lib/bb/fetch2/__init__.py", line 1307, in latest_revision
    revs[key] = rev = self._latest_revision(ud, d, name)
  File "/var/opt/builds/olofjn/oe/master/bitbake/lib/bb/fetch2/git.py", line 345, in _latest_revision
    output = self._lsremote(ud, d, search)
  File "/var/opt/builds/olofjn/oe/master/bitbake/lib/bb/fetch2/git.py", line 329, in _lsremote
    exit(1)
  File "/usr/lib/python2.7/site.py", line 374, in __call__
    raise SystemExit(code)
SystemExit: 1
1NOTE: Tasks Summary: Attempted 373 tasks of which 372 didn't need to be rerun and all succeeded.

I also print out the tasklist (__BBTASKS) as seen in data.py's generate_dependencies:

ERROR: tasklist = ['do_patch', 'do_populate_sysroot',
    'do_populate_sysroot_setscene', 'do_listtasks', 'do_clean',
    'do_checkuri', 'do_checkuriall', 'do_fetchall', 'do_fetch',
    'do_unpack', 'do_configure', 'do_compile', 'do_install',
    'do_build', 'do_cleansstate', 'do_cleanall', 'do_package',
    'do_package_setscene', 'do_packagedata',
    'do_packagedata_setscene', 'do_package_write_rpm_setscene',
    'do_package_write_rpm', 'do_devshell', 'do_populate_lic',
    'do_populate_lic_setscene', 'do_deploy_setscene',
    'do_bundle_initramfs', 'do_compile_kernelmodules',
    'do_savedefconfig', 'do_menuconfig', 'do_diffconfig', 'do_strip',
    'do_sizecheck', 'do_uboot_mkimage', 'do_deploy',
    'do_kernel_configme', 'do_kernel_checkout',
    'do_kernel_link_vmlinux', 'do_validate_branches',
    'do_kernel_configcheck', 'do_kernel_link_vmlinuz']

I still don't get why do_patch is expanded, but afaict it tries
to do expansion of all of the tasks in __BBTASKS, and the mere
expansion of the do_patch variable triggers the fetcher to get
things from the network.

-- 
olofjn