[bitbake-devel] [PATCH] build: Improve python execution tracebacks

[Richard Purdie]

If something fails in a exec_func_python() the current stack trace shows
incorrect filenames and linenumbers. For example:

The stack trace of python calls that resulted in this exception/failure was:
File: '/media/build1/poky/meta/recipes-sato/images/core-image-sato.bb', lineno: 200, function: <module>
     0196:                chksum = bb.utils.sha256_file(fn)
     0197:                f.write('%s\t%s\n' % (chksum, os.path.relpath(fn, baseoutpath)))
     0198:
     0199:
 *** 0200:copy_buildsystem(d)
     0201:
File: '/media/build1/poky/meta/recipes-sato/images/core-image-sato.bb', lineno: 9, function: copy_buildsystem
     0005:IMAGE_FEATURES += "splash package-management x11-base x11-sato ssh-server-dropbear hwcodecs"
     0006:
     0007:LICENSE = "MIT"
     0008:
 *** 0009:inherit core-image
     0010:
     0011:IMAGE_INSTALL += "packagegroup-core-x11-sato-games"
File: '/usr/lib/python2.7/subprocess.py', lineno: 535, function: check_call
     0531:    The arguments are the same as for the Popen constructor.  Example:
     0532:
     0533:    check_call(["ls", "-l"])
     0534:    """
 *** 0535:    retcode = call(*popenargs, **kwargs)
     0536:    if retcode:
     0537:        cmd = kwargs.get("args")
     0538:        if cmd is None:
     0539:            cmd = popenargs[0]

The problem is the use of "FILE" to obtain the current filename. Instead,
we therefore inject the function being executed into the methodpool which
allows us to correct its linenumber and filename information. We can then
clearly mark the initial piece as autogenerated and the rest of the linenumber
and filename information should be correct. Afterwards the trace starts:

The stack trace of python calls that resulted in this exception/failure was:
File: 'exec_python_func() autogenerated', lineno: 2, function: <module>
     0001:
 *** 0002:copy_buildsystem(d)
     0003:
File: '/media/build1/poky/meta/classes/populate_sdk_ext.bbclass', lineno: 66, function: copy_buildsystem
     0062:    import glob
     0063:    import oe.copy_buildsystem
     0064:    import subprocess
     0065:
 *** 0066:    subprocess.check_call("foo")
     0067:
     0068:    oe_init_env_script = d.getVar('OE_INIT_ENV_SCRIPT', True)
     0069:
     0070:    conf_bbpath = ''
File: '/usr/lib/python2.7/subprocess.py', lineno: 535, function: check_call
     0531:    The arguments are the same as for the Popen constructor.  Example:
     0532:
     0533:    check_call(["ls", "-l"])
     0534:    """
 *** 0535:    retcode = call(*popenargs, **kwargs)
     0536:    if retcode:
     0537:        cmd = kwargs.get("args")
     0538:        if cmd is None:
     0539:            cmd = popenargs[0]

We can't inject into methodpool at parsing time, since there may be
_append or other override operations against the function before its
execution.

Signed-off-by: Richard Purdie <richard.purdie at linuxfoundation.org>

diff --git a/bitbake/lib/bb/build.py b/bitbake/lib/bb/build.py
index 3039837..005ace0 100644
--- a/bitbake/lib/bb/build.py
+++ b/bitbake/lib/bb/build.py
@@ -229,17 +229,13 @@ def exec_func(func, d, dirs = None):
             exec_func_shell(func, d, runfile, cwd=adir)
 
 _functionfmt = """
-def {function}(d):
-{body}
-
 {function}(d)
 """
 logformatter = bb.msg.BBLogFormatter("%(levelname)s: %(message)s")
 def exec_func_python(func, d, runfile, cwd=None):
     """Execute a python BB 'function'"""
 
-    bbfile = d.getVar('FILE', True)
-    code = _functionfmt.format(function=func, body=d.getVar(func, False))
+    code = _functionfmt.format(function=func)
     bb.utils.mkdirhier(os.path.dirname(runfile))
     with open(runfile, 'w') as script:
         bb.data.emit_func_python(func, script, d)
@@ -254,8 +250,13 @@ def exec_func_python(func, d, runfile, cwd=None):
     bb.debug(2, "Executing python function %s" % func)
 
     try:
-        comp = utils.better_compile(code, func, bbfile)
-        utils.better_exec(comp, {"d": d}, code, bbfile)
+        text = "def %s(d):\n%s" % (func, d.getVar(func, False))
+        fn = d.getVarFlag(func, "filename", False)
+        lineno = int(d.getVarFlag(func, "lineno", False))
+        bb.methodpool.insert_method(func, text, fn, lineno - 1)
+
+        comp = utils.better_compile(code, func, "exec_python_func() autogenerated")
+        utils.better_exec(comp, {"d": d}, code, "exec_python_func() autogenerated")
     except (bb.parse.SkipRecipe, bb.build.FuncFailed):
         raise
     except: