[OE-core] Custom CLASSOVERRIDE and a "sub-sysroot" for JavaScript

[chris.laplante at agilent.com]

Hi all,

For about two years now, we have been using Emscripten together with Yocto, via an emscripten.bbclass that I wrote. In a nutshell, it works by conducting the actual build in a Docker container, whose image is where Emscripten is actually installed.  I mount all the necessary directories (e.g. ${B}, ${S}, etc.) into the container, and I've got some Python scripting that ensures that containers get cleaned up even if the build is aborted with Ctrl+C or otherwise dies. All of our Emscripten-enabled recipes happen to be using CMake, so emscripten.bbclass is CMake specific. It's basically a reimplementation of cmake.bbclass (from Poky) that happens to run in a container.

For the time being the specifics of how the Emscripten part works isn't relevant (1). I provide it as background for what I really want to ask about, which is the Yocto-specific plumbing. Here's a snippet of it:

-------
    # Copyright (C) 2020 Agilent Technologies, Inc.
    # SPDX-License-Identifier: GPL-2.0-only
    CLASSOVERRIDE = "class-emscripten"
    SPECIAL_PKGSUFFIX_class-emscripten += "-emscripten"
    BASEDEPENDS_class-emscripten = ""
    FILES_${PN} =+ "${libdir}/*.js"
    export prefix = "/usr/js"

    python emscripten_virtclass_handler() {
        pn = e.data.getVar("PN")
        if not pn.endswith("-emscripten"):
            return

        classextend = e.data.getVar('BBCLASSEXTEND') or ""
        if "emscripten" not in classextend:
            return

        def map_dependencies(varname, d, suffix = ""):
            if suffix:
                varname = varname + "_" + suffix
            deps = d.getVar(varname)
            if not deps:
                return
            deps = bb.utils.explode_deps(deps)
            newdeps = []
            for dep in deps:
                if dep == pn:
                    continue
                elif dep in ["cmake-native"]:
                    continue
                elif dep.endswith("-emscripten"):
                    newdeps.append(dep)
                elif dep.endswith("-native"):
                    bb.warn("emscripten.bbclass: Native dependency {0} will not be usable from within Emscripten Docker container".format(dep))
                    newdeps.append(dep)
            d.setVar(varname, " ".join(newdeps))

        e.data.setVar("OVERRIDES", e.data.getVar("OVERRIDES", False) + ":virtclass-emscripten")

        map_dependencies("DEPENDS", e.data)
        for pkg in [e.data.getVar("PN"), "", "${PN}"]:
            map_dependencies("RDEPENDS", e.data, pkg)
            map_dependencies("RRECOMMENDS", e.data, pkg)
            map_dependencies("RSUGGESTS", e.data, pkg)
            map_dependencies("RPROVIDES", e.data, pkg)
            map_dependencies("RREPLACES", e.data, pkg)

        provides = e.data.getVar("PROVIDES")
        nprovides = []
        for prov in provides.split():
            if prov.find(pn) != -1:
                nprovides.append(prov)
            elif not prov.endswith("-emscripten"):
                nprovides.append(prov.replace(prov, prov + "-emscripten"))
            else:
                nprovides.append(prov)
        e.data.setVar("PROVIDES", ' '.join(nprovides))
    }

    addhandler emscripten_virtclass_handler
    emscripten_virtclass_handler[eventmask] = "bb.event.RecipePreFinalise"

    do_populate_sysroot[stamp-extra-info] = ""
    do_packagedata[stamp-extra-info] = ""
-------


There are two main goals of this code. First, it sets up dependencies between -emscripten recipes. Second, it sets 'prefix' to be '/usr/js/' so that, for example, a "foo" and "foo-emscripten" recipe that both install development headers won't collide. I call this the "sub-sysroot".

I have a few concerns/questions:
[a] Is the basic premise of a custom CLASSOVERRIDE supported by Yocto? Our situation seems a bit weird, because our emscripten recipes are still sort of "target" recipes, so I feel a bit funny completely wiping CLASSOVERRIDE.
[b] At the very least, I know I did something wrong, because sometimes we get weird warnings like:

WARNING: app-emscripten-1.0+gitAUTOINC+1d1073e7c4-r0 do_compile: Manifest /home/user/yocto/build/tmp/sstate-control/manifest-x86_64_x86_64-nativesdk-library-emscripten.populate_sysroot not found in my_mach cortexa9t2hf-neon cortexa9t2hf-vfp cortexa9hf-neon cortexa9hf-vfp armv7at2hf-neon armv7ahf-neon armv7at2hf-vfp armv7ahf-vfp armv6thf-vfp armv6hf-vfp armv5tehf-vfp armv5ehf-vfp armv5thf-vfp armv5hf-vfp allarch x86_64_x86_64-nativesdk (variant '')?

... usually followed by a build failure in "app-emscripten" because "library-emscripten:do_populate_sysroot" indeed didn't run. Related to [a] or perhaps the stamp-extra-info lines?
[c] Is there a better way to do the "sub-sysroot" (/usr/js)? Would multilib cover this whole use case?  (2)

I appreciate any comments/help.
Chris


(1) I'm happy to cover it in a separate thread if people are interested. I hope to rewrite the class using Clang's WebAssembly backend, rather than call out to the Emscripten container which has always seemed like a hack. Regardless, I will make what we have public on poky-contrib soon anyway.
(2) I suppose one option is to have Emscripten flavors of all the MACHINEs we are building for, but I'd like to avoid that. We only have a handful of Emscripten recipes, so a separate MACHINE at this time would be overkill.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openembedded.org/pipermail/openembedded-core/attachments/20200124/e82fa026/attachment-0001.html>