[OE-core] [PATCH 1/4] sstate: allow specifying indirect dependencies to exclude from sysroot

[André Draszik]

On Sun, 2018-06-17 at 14:13 +0100, Richard Purdie wrote:
> On Tue, 2018-05-22 at 13:25 +0100, André Draszik wrote:
> > From: André Draszik <andre.draszik at jci.com>
> > 
> > Currently, a dependency on any -native recipe will pull in
> > all dependencies of that -native recipe in the recipe
> > sysroot. This behaviour might not always be wanted, e.g.
> > when that -native recipe depends on build-tools that are
> > not relevant for the current recipe.
> > 
> > This change adds a SSTATE_EXCLUDEDEPS_SYSROOT variable,
> > which will be evaluated for such recursive dependencies to
> > be excluded. The idea is similar to
> >    http://lists.openembedded.org/pipermail/openembedded-core/2018-Jan
> > uary/146324.html
> > except that the list is not hard-coded anymore.
> > 
> > SSTATE_EXCLUDEDEPS_SYSROOT is evaluated as two regular
> > expressions of recipe and dependency to ignore, e.g. in
> > the above flex-native / bison-native use-case, one would
> > specify
> > 
> >     SSTATE_EXCLUDEDEPS_SYSROOT = ".*->(flex|bison)-native"
> > 
> > in layer.conf.
> > 
> > The existing special handling of "-initial" as well as
> > "base-passwd" and "shadow-sysroot" could also be
> > streamlined:
> > 
> >     SSTATE_EXCLUDEDEPS_SYSROOT += "\
> >         .*->.*-initial.* \
> >         .*(base-passwd|shadow-sysroot)->.* \
> >     "
> > 
> > Another anticipated user is meta-java, where certain newer
> > JDKs can only be bootstrapped (built) using older JDKs,
> > but it doesn't make much sense to copy all those older
> > JDKs and their own build tools (ant, etc.) into the
> > sysroot of recipes wanting to be built using the newer JDK
> > (only), e.g.:
> > 
> >     SSTATE_EXCLUDEDEPS_SYSROOT += "\
> >         openjdk-8-native->(ant-native|attr-native|coreutils-
> > native|icedtea7-native|libxslt-native|make-native|openssl-native|zip-
> > native|unzip-native) \
> >     "
> > 
> > Signed-off-by: André Draszik <andre.draszik at jci.com>
> > ---
> >  meta/classes/sstate.bbclass | 13 +++++++++++++
> >  1 file changed, 13 insertions(+)
> > 
> 
> I have some reservations about this series as I've previously mentioned
> but I had a try at testing it on the autobuilder:
> 
> https://autobuilder.yocto.io/builders/build-appliance/builds/1059/steps/Bu
> ildImages_1/logs/stdio

Thank you Richard.

I should maybe explain my ultimate goal here:

meta-java is completely self-hosted, which is nice. But to be able to
compile OpenJDK v8, various other Java compilers need to be compiled first.
This of course means that all those different Java compilers/VMs/runtimes
and tools needed to compile them end up in each Java recipe's sysroot.

While in theory this is fine, and all of them can be installed in parallel,
as everything resides below subdirectories in /usr/lib/jvm/... and
usr/bin/java normally is a symlink to the default chosen via update-
alternatives, this doesn't work in yocto for -native recipes.

In particular this is annoying, because 'usr/bin/java' in the recipe sysroot
is a *very* old version of java, with no possibility of control.

This has caused me a bit of headache, and if it was possible to avoid all
the recursive OpenJDK-8 build time dependencies then:
* usr/bin/java could disappear (for -native)
* tools inherited from the OpenJDK-8 build can be avoided during other
  recipe builds

In other words, as a follow-up to this series here I have a local meta-java
change to meta-java's layer.conf:

SSTATE_EXCLUDEDEPS_SYSROOT += "\
    openjdk-8-native->(ant-native|attr-native|coreutils-native|icedtea7-native|libxslt-native|make-native|openssl-native|zip-native|unzip-native) \
"

This really is a similar situation to *-initial treatment in sstate.bbclass,
but for Java / OpenJDK.


So rather than having to add knowledge about Java / OpenJDK into
sstate.bbclass, I need a way to inject that instead.


Given the above, would you be happy to consider a series where the
infrastructure is put in place, but without the flex-native and bison-native 
changes? I.e. patches 1 through 3, but not patch 4 of this series? (With
your previous review comments addressed, of course).


> Basically binutils-cross is bust with errors like:
> 
> > x86_64-poky-linux-ar: error while loading shared libraries: libfl.so.2:
> > cannot open shared object file: No such file or directory
> > make[5]: *** [libaudio_resampler_sse.la] Error 127
> 
> There are a ton of other failures, I've not waded through them but I'm
> guessing there are similar issues.

I am not sure why I didn't see these in my builds before submitting, but now
I do, too.

It looks like the cross binutils-native (or more specifically cross -ar)
dynamically links against libflex-native. So the last patch in this series
is of course responsible for that.

Comparing to my debian, ar isn't dynamically linked against libflex, I
wonder why yocto does it, or why debian doesn't.

In any case my builds succeed now by dropping the flex-native from patch 4
of this series. Alternative would be to switch to static linking for
libflex. But see above.


Cheers,
Andre'