[OE-core] [WIP][PATCH 01/12] gstreamer: Add common .inc files for meson based recipes

Mon Jan 13 20:24:11 UTC 2020

there are some plugins in meta-multimedia which are erroring out

https://errors.yoctoproject.org/Errors/Build/96752/

On Wed, Jan 8, 2020 at 3:43 PM Carlos Rafael Giani <crg7475 at mailbox.org> wrote:
>
> Sorry, I missed this message somehow. Will do that tomorrow.
>
>
> On 02.01.20 22:47, Alexander Kanavin wrote:
>
> So can you resend the patchset, without any special tags (wip/rfc etc) please? To give it a run on the autobuilder.
>
> Alex
>
> On 19 Dec 2019, at 12.57, Carlos Rafael Giani <crg7475 at mailbox.org> wrote:
>
> Well, alright, then let's merge this. You do have a point in that we are not close to a release. Plenty of time left to fix these issues afterwards.
>
> On 19.12.19 12:54, Alexander Kanavin wrote:
>
> I read them through now, and none of them are actually a big deal that need further discussion in my opinion. I’d rather get meson conversion done, and handle any issues afterwards.
>
> Alex
>
> On 19 Dec 2019, at 12.46, Carlos Rafael Giani <crg7475 at mailbox.org> wrote:
>
> First we should discuss the leftover TODOs I think. Since these may not be trivial issues, it would be better to take a look at them and then decide on a case-by-case basis whether to fix them now or later.
>
> For convenience, I copy-paste these bits from an older RFC email:
>
>
> ========
>
> Noteworthy changes:
>
> - A lot of autotools/M4 related patches are just gone. Well, obviously But it is surprising just how much fewer patches there are now.
>
> - gstreamer1.0-libav no longer builds its own FFmpeg copy. This was done in the past because there were no stable versioned FFmpeg releases. They do exist now, so in meson based builds, gst-libav just builds the actual plugin code, meaning that the build time is much shorter, most of the recipe code (and all of the patches) are gone (because they were FFmpeg specific), and the libgstlibav.so binary in /usr/lib/gstreamer-1.0/ is much smaller. This also means that FFmpeg specials do not have to be added both to the FFmpeg and to the gstreamer1.0-libav recipe anymore.
>
> - gstreamer1.0-plugins.inc was replaced by gstreamer1.0-plugins-common.inc, and gst-plugins-package.inc by gstreamer1.0-plugins-packaging.inc . These names are clearer. gst-plugins-package.inc in particular was named that way because it used to be shared between GStreamer 0.10 and 1.0 recipes. Since the former are gone by now, there's no point in keeping that name.
>
> - gstreamer1.0 ptest is now done differently. Originally, make was installed on the target and the Makefile was run by the run-ptest script. Now, gstreamer is patched to allow for installable tests that are accompanied by individual shell scripts that set up the necessary environment variables, along with .test files so that the gnome-desktop-testing test runner can be used in run-ptest to run the unit tests. This also makes it possible to inherit from ptest-gnome in the gstreamer1.0 recipe. The patch against gstreamer is necessary because by default, one runs the test by executing "meson test". I don't think installing meson along with all of the necessary meson.build scripts is a practical approach.
>
> - OpenGL packageconfigs now work quite differently in gstreamer1.0-plugins-base, since the meson options distinguish between OpenGL APIs, platforms, and windowing systems.
>
> - zlib in gstreamer1.0-plugins-base and -good is now required. I don't think this is a big deal, since it is also a dependency of GLib itself, so zlib has to be present on the rootfs anyway.
>
>
> There are some TODOs left that I want to bring up here for discussion:
>
> - The aforementioned PTest approach needs some review. Also, initially, I thought I'd have to add PTest support to several GStreamer recipes, which is why I isolated the ptest code in gstreamer1.0-ptest.inc . But it turned out that only the gstreamer1.0 recipe needs it. I am unsure if I should just merge that .inc into that recipe, since there's no reason why the tests from other recipes shouldn't be "ptest-ified" in the future. And then that .inc would be useful to have. (Tests from the plugin sets may be partially tough to run, since some of them create X11 windows etc.)
>
> - The code in gstreamer1.0-ptest.inc sets up run-ptest in a rather uncommon way, since that file has to be autogenerated.
>
> - gstreamer1.0_1.16.1.bb and gstreamer1.0-plugins-common.inc both contain a "gettext_oemeson" function to set up meson NLS options. Ideally, this functionality would be part of the gettext bbclass itself. However, my bitbake/OE knowledge does not suffice to do that, since with meson, you want to be able to configure what the enable/disable flags should be ("enabled"/"disabled", "true"/"false" etc.); that's why for example the gtk-doc bbclass defines GTKDOC_MESON_OPTION, GTKDOC_MESON_ENABLE_FLAG, GTKDOC_MESON_DISABLE_FLAG. I am not sure how to combine such variables with this kind of custom python code, and still allow for customized values in other recipes. In other words, if I first get a "NLS_MESON_DISABLE_FLAG" value with getValue() in a modified gettext_oemeson function, then I might not get the value I expect, since it will get the initial value (the one set with ?=), not necessarily the one that is redefined by a recipe.
>
> - The OpenGL packageconfigs are handled by a get_opengl_cmdline_list function in gstreamer1.0-plugins-base_1.16.1.bb . I did this so that a comma-separated list can be generated out of the packageconfigs that are (a) part of PACKAGECONFIG and (b) part of the newly added OPENGL_APIS/OPENGL_PLATFORMS/OPENGL_WINSYS lists. I do not know if there's a better approach for this. The intent is that other layers (most notably BSP layers) can add custom packageconfig names to these lists. One example would be OPENGL_WINSYS += "viv-fb" in meta-freescale. Another would be OPENGL_WINSYS += "dispmanx" in meta-raspberrypi.
>
> - gstreamer1.0-python_1.16.1.bb currently has a problem in that the gst-python code tries to load /usr/lib/libpython3.7m.so. But, in a regular installation, only /usr/lib/libpython3.7m.so.1.0 would be availale. It is possible to adjust the libpython path & name during the build. But - what would it be adjusted to? How would the recipe know that "/usr/lib/libpython3.7m.so.1.0" is the correct choice? Or perhaps would it make sense to add the /usr/lib/libpython3.7m.so symlink to the regular (as in: non-dev) Python package?
>
> ========
>
>
>
> On 18.12.19 22:14, Alexander Kanavin wrote:
>
> Thanks :) Will there be a more refined version, or should these be merged already, with followup fixing as needed?
>
> Alex
>
> On Sun, 15 Dec 2019 at 13:55, Carlos Rafael Giani <crg7475 at mailbox.org> wrote:
>>
>> * gstreamer1.0-plugins-common.inc
>>   The old gstreamer1.0-plugins.inc file, adapted for meson
>> * gstreamer1.0-plugins-packaging.inc
>>   The old gst-plugins-package.inc file, adapted for meson
>> * gstreamer1.0-ptest.inc
>>   Common code for meson based PTest support; autogenerates the run-ptest
>>   file and is designed to use the gnome-desktop-testing suite runner
>>   (which is why the .inc files inherits from ptest-gnome)
>>
>> Signed-off-by: Carlos Rafael Giani <crg7475 at mailbox.org>
>> ---
>>  .../gstreamer/gstreamer1.0-plugins-common.inc | 47 +++++++++++++
>>  .../gstreamer1.0-plugins-packaging.inc        | 70 +++++++++++++++++++
>>  .../gstreamer/gstreamer1.0-ptest.inc          | 21 ++++++
>>  3 files changed, 138 insertions(+)
>>  create mode 100644 meta/recipes-multimedia/gstreamer/gstreamer1.0-plugins-common.inc
>>  create mode 100644 meta/recipes-multimedia/gstreamer/gstreamer1.0-plugins-packaging.inc
>>  create mode 100644 meta/recipes-multimedia/gstreamer/gstreamer1.0-ptest.inc
>>
>> diff --git a/meta/recipes-multimedia/gstreamer/gstreamer1.0-plugins-common.inc b/meta/recipes-multimedia/gstreamer/gstreamer1.0-plugins-common.inc
>> new file mode 100644
>> index 0000000000..ccf55500a9
>> --- /dev/null
>> +++ b/meta/recipes-multimedia/gstreamer/gstreamer1.0-plugins-common.inc
>> @@ -0,0 +1,47 @@
>> +# This .inc file contains the common setup for the gstreamer1.0-plugins-*
>> +# plugin set recipes.
>> +
>> +# SUMMARY is set in the actual .bb recipes
>> +HOMEPAGE = "https://gstreamer.freedesktop.org/"
>> +BUGTRACKER = "https://bugzilla.gnome.org/enter_bug.cgi?product=Gstreamer"
>> +SECTION = "multimedia"
>> +
>> +DEPENDS = "gstreamer1.0 glib-2.0-native"
>> +
>> +inherit gettext meson pkgconfig upstream-version-is-even
>> +
>> +require gstreamer1.0-plugins-packaging.inc
>> +
>> +# Orc enables runtime JIT compilation of data processing routines from Orc
>> +# bytecode to SIMD instructions for various architectures (currently SSE, MMX,
>> +# MIPS, Altivec and NEON are supported).
>> +# This value is used in the PACKAGECONFIG values for each plugin set recipe.
>> +# By modifying it, Orc can be enabled/disabled in all of these recipes at once.
>> +GSTREAMER_ORC ?= "orc"
>> +
>> +PACKAGECONFIG[orc] = "-Dorc=enabled,-Dorc=disabled,orc orc-native"
>> +
>> +# TODO: put this in a gettext.bbclass patch (with variables to allow for
>> +# configuring the option name and the enabled/disabled values).
>> +def gettext_oemeson(d):
>> +    if d.getVar('USE_NLS') == 'no':
>> +        return '-Dnls=disabled'
>> +    # Remove the NLS bits if USE_NLS is no or INHIBIT_DEFAULT_DEPS is set
>> +    if d.getVar('INHIBIT_DEFAULT_DEPS') and not oe.utils.inherits(d, 'cross-canadian'):
>> +        return '-Dnls=disabled'
>> +    return '-Dnls=enabled'
>> +
>> +# Not all plugin sets contain examples, so the -Dexamples
>> +# option needs to be added conditionally.
>> +GST_PLUGIN_SET_HAS_EXAMPLES ?= "1"
>> +
>> +EXTRA_OEMESON += " \
>> +    ${@bb.utils.contains('GST_PLUGIN_SET_HAS_EXAMPLES', '1', '-Dexamples=disabled', '', d)} \
>> +    ${@gettext_oemeson(d)} \
>> +"
>> +
>> +GIR_MESON_ENABLE_FLAG = "enabled"
>> +GIR_MESON_DISABLE_FLAG = "disabled"
>> +
>> +# Dynamically generate packages for all enabled plugins
>> +PACKAGES_DYNAMIC = "^${PN}-.*"
>> diff --git a/meta/recipes-multimedia/gstreamer/gstreamer1.0-plugins-packaging.inc b/meta/recipes-multimedia/gstreamer/gstreamer1.0-plugins-packaging.inc
>> new file mode 100644
>> index 0000000000..9a7a1b6afe
>> --- /dev/null
>> +++ b/meta/recipes-multimedia/gstreamer/gstreamer1.0-plugins-packaging.inc
>> @@ -0,0 +1,70 @@
>> +# This .inc file contains functionality for automatically splitting
>> +# built plugins into individual packages for each plugin. A -meta
>> +# package is also set up that has no files of its own, but contains
>> +# the names of all plugin packages in its RDEPENDS list.
>> +#
>> +# This is mainly used by the gstreamer1.0-plugins-* plugin set recipes,
>> +# but can be used in any recipe that produces GStreamer plugins.
>> +
>> +PACKAGESPLITFUNCS_prepend = " split_gstreamer10_packages "
>> +PACKAGESPLITFUNCS_append = " set_gstreamer10_metapkg_rdepends "
>> +
>> +python split_gstreamer10_packages () {
>> +    gst_libdir = d.expand('${libdir}/gstreamer-1.0')
>> +    postinst = d.getVar('plugin_postinst')
>> +    glibdir = d.getVar('libdir')
>> +
>> +    # GStreamer libraries
>> +    do_split_packages(d, glibdir, r'^lib(.*)\.so\.*', 'lib%s', 'GStreamer 1.0 %s library', extra_depends='', allow_links=True)
>> +    # GStreamer plugin shared objects
>> +    do_split_packages(d, gst_libdir, r'libgst(.*)\.so$', d.expand('${PN}-%s'), 'GStreamer 1.0 plugin for %s', postinst=postinst, extra_depends='')
>> +    # GObject introspection files for GStreamer plugins
>> +    do_split_packages(d, glibdir+'/girepository-1.0', r'Gst(.*)-1.0\.typelib$', d.expand('${PN}-%s-typelib'), 'GStreamer 1.0 typelib file for %s', postinst=postinst, extra_depends='')
>> +    # Static GStreamer libraries for development
>> +    do_split_packages(d, gst_libdir, r'libgst(.*)\.a$', d.expand('${PN}-%s-staticdev'), 'GStreamer 1.0 plugin for %s (static development files)', extra_depends='${PN}-staticdev')
>> +}
>> +
>> +python set_gstreamer10_metapkg_rdepends () {
>> +    import os
>> +    import oe.utils
>> +
>> +    # Go through all generated packages (excluding the main package and
>> +    # the -meta package itself) and add them to the -meta package as RDEPENDS.
>> +
>> +    pn = d.getVar('PN')
>> +    metapkg =  pn + '-meta'
>> +    d.setVar('ALLOW_EMPTY_' + metapkg, "1")
>> +    d.setVar('FILES_' + metapkg, "")
>> +    blacklist = [ pn, pn + '-meta' ]
>> +    metapkg_rdepends = []
>> +    pkgdest = d.getVar('PKGDEST')
>> +    for pkg in oe.utils.packages_filter_out_system(d):
>> +        if pkg not in blacklist and pkg not in metapkg_rdepends:
>> +            # See if the package is empty by looking at the contents of its PKGDEST subdirectory.
>> +            # If this subdirectory is empty, then the package is.
>> +            # Empty packages do not get added to the meta package's RDEPENDS
>> +            pkgdir = os.path.join(pkgdest, pkg)
>> +            if os.path.exists(pkgdir):
>> +                dir_contents = os.listdir(pkgdir) or []
>> +            else:
>> +                dir_contents = []
>> +            is_empty = len(dir_contents) == 0
>> +            if not is_empty:
>> +                metapkg_rdepends.append(pkg)
>> +    d.setVar('RDEPENDS_' + metapkg, ' '.join(metapkg_rdepends))
>> +    d.setVar('DESCRIPTION_' + metapkg, pn + ' meta package')
>> +}
>> +
>> +# each plugin-dev depends on PN-dev, plugin-staticdev on PN-staticdev
>> +# so we need them even when empty (like in gst-plugins-good case)
>> +ALLOW_EMPTY_${PN} = "1"
>> +ALLOW_EMPTY_${PN}-dev = "1"
>> +ALLOW_EMPTY_${PN}-staticdev = "1"
>> +
>> +PACKAGES += "${PN}-apps ${PN}-meta ${PN}-glib"
>> +
>> +FILES_${PN} = ""
>> +FILES_${PN}-apps = "${bindir}"
>> +FILES_${PN}-glib = "${datadir}/glib-2.0"
>> +
>> +RRECOMMENDS_${PN} += "${PN}-meta"
>> diff --git a/meta/recipes-multimedia/gstreamer/gstreamer1.0-ptest.inc b/meta/recipes-multimedia/gstreamer/gstreamer1.0-ptest.inc
>> new file mode 100644
>> index 0000000000..74368c4cd8
>> --- /dev/null
>> +++ b/meta/recipes-multimedia/gstreamer/gstreamer1.0-ptest.inc
>> @@ -0,0 +1,21 @@
>> +inherit ptest-gnome
>> +
>> +TEST_FILES_PATH = "${datadir}/installed-tests/gstreamer-1.0/test-files"
>> +RUN_PTEST_FILE = "${D}${PTEST_PATH}/run-ptest"
>> +
>> +EXTRA_OEMESON += "-Dtest-files-path=${TEST_FILES_PATH}"
>> +
>> +# Using do_install_ptest_base instead of do_install_ptest, since
>> +# the default do_install_ptest_base is hardcoded to expect Makefiles.
>> +do_install_ptest_base() {
>> +    # Generate run-ptest file
>> +    echo "#!/usr/bin/env sh" > "${RUN_PTEST_FILE}"
>> +    echo "gnome-desktop-testing-runner ${GST_TEST_SUITE_NAME}" >> "${RUN_PTEST_FILE}"
>> +    chmod 0755 "${RUN_PTEST_FILE}"
>> +
>> +    # Install additional files required by tests
>> +    if [ -d "${S}/tests/files" ] ; then
>> +        install -d "${D}/${TEST_FILES_PATH}"
>> +        install -m 0644 "${S}/tests/files"/* "${D}/${TEST_FILES_PATH}"
>> +    fi
>> +}
>> --
>> 2.17.1
>>
>> --
>> _______________________________________________
>> Openembedded-core mailing list
>> Openembedded-core at lists.openembedded.org
>> http://lists.openembedded.org/mailman/listinfo/openembedded-core
>
> --
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core at lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-core