[OE-core] [PATCH v4] libtool: fix resolve of lt_sysroot

Fri Sep 13 09:08:01 UTC 2013

On Fri, Sep 13, 2013 at 11:01 AM, Hans Beckérus <hans.beckerus at gmail.com> wrote:
> On Fri, Sep 13, 2013 at 10:52 AM, Richard Purdie
> <richard.purdie at linuxfoundation.org> wrote:
>> On Fri, 2013-09-13 at 10:06 +0200, Hans Beckérus wrote:
>>> On Fri, Sep 13, 2013 at 1:07 AM, Hans Beckerus <hans.beckerus at gmail.com> wrote:
>>> > On 2013-09-12 11:09, Hans Beckerus wrote:
>>> >>
>>> >> On 2013-09-12 8:02, Hans Beckérus wrote:
>>> >>
>>> > I now got a somewhat better picture of what is going on. I know what is
>>> > failing, and why. But currently I have no solution ready. Actually there are
>>> > some nasty traps to get caught in here :(. The problem is actually as simple
>>> > as it is obvious. For all those native packages that do work (this is a
>>> > unique problem for native packages using libtool), they all seem to share a
>>> > common thing in their recipes:
>>> >
>>> > EXTRA_OECONF += " --with-libtool-sysroot=${STAGING_DIR_NATIVE}"
>>> >
>>> > Great! This is the way to do it. But, what if someone forgets to do this?
>>> > Well the answer is; it will most likely *not* compile!
>>> > Since libtool now has been fixed to correctly pick up the sysroot from the
>>> > compiler (using --print-sysroot) if --with-libtool-sysroot is not specified
>>> > it will try to execute ${CC} --print-sysroot. Bummer! ${CC} is most likely
>>> > simply set to 'gcc' for native packages. That is, the local host compiler is
>>> > used.
>>> > The sysroot for that is of course "/". And it should be. Otherwise it will
>>> > bring in the wrong set of header files and libraries. But, there is another
>>> > problem here. We should not let libtool use "/"! Because even if we use the
>>> > local host compiler for native packages, we still use the oe-core upstream
>>> > version of libtool, and that does not like using "/" as sysroot.  If it does
>>> > everything becomes a mess. And that is exactly what seems to happen now
>>> > after the patch. Before the patch libtool rendered the SDK for libtool
>>> > enabled packages more or less useless. But, it also saved us in the native
>>> > case. Because if --with-libtool-sysroot was set, the path was used directly,
>>> > but if it was not set, lt_sysroot was also kept unset. And here is the
>>> > spooky part again. Having lt_sysroot set to nothing seems to work just as
>>> > well as setting it, provided it points to a valid location!? This magic
>>> > however did not work for the SDK which requires the sysroot to be resolved
>>> > correctly when not specified. So one conclusion could be that, for native
>>> > packages, enforcement of --with-libtool-sysroot is a possible way forward.
>>> > Would this be safe? I think so, but I might have overlooked something.  I
>>> > can also see in config.log that "configure" is fed with a lot of arguments,
>>> > even if EXTRA_OECONF is not specified. How is this handled? How can I try to
>>> > force this in for all native packages. I looked into native.bbclass but it
>>> > was not obvious to me how anything in there actually ends up in arguments to
>>> > "configure". Any hints? There are still a lot of gaps in my analysis ;) If
>>> > anyone feels like they can fill in the gaps, please do.
>>> >
>>> >
>>> Now this is getting interesting. When I said before I could reproduce
>>> the problem in a world build I did not know that it would actually
>>> succeed later. On a different build server!! The previous analysis
>>> made is mostly true. But there is something more. Why did it succeed
>>> on one of my servers but not the other one? The way to find out was to
>>> compare what lt_sysroot is resolved to in both cases. And there it
>>> was. On the machine for which it works, lt_sysroot is now picked up
>>> from $CC --print-sysroot (due to the patch), but the result is not
>>> "/", it is unset! Which is the same behavior that can be observed
>>> without the libtool patch. So this is getting even worse now. It is
>>> gcc and host dependent. If gcc has been built with a sysroot of "/" is
>>> does not work since that will be picked up by libtool, and is of
>>> course wrong in our case. But if lt_sysroot is kept unset, libtool
>>> seems to be able to resolve a working default using some other
>>> (unknown) mechanism. That is why we have not seen this problem before,
>>> even though libtool actually did the wrong thing before the patch. So
>>> this leaves me with at least this question; is it actually correct
>>> that gcc has "/" as a compiled in sysroot on some machines?
>>>
>>> I then saw two possible solutions:
>>>
>>> a) update the patch in libtool and if gcc reports a single "/", skip
>>> it and leave lt_sysroot unset.
>>> b) consistency is the key. Let all native packages force setting of a
>>> proper lt_sysroot  using --with-libtool-sysroot.
>>>
>>> This is also when I found this in autotools.bbclass
>>>
>>> def append_libtool_sysroot(d):
>>>     # Only supply libtool sysroot option for non-native packages
>>>     if not bb.data.inherits_class('native', d):
>>>         return '--with-libtool-sysroot=${STAGING_DIR_HOST}'
>>>     return ""
>>>
>>> I can somewhat understand the rationale behind this. If building for
>>> target you wish to point libtool to a proper sysroot, and leaving it
>>> unset for native seems like a good idea. But I do not think that
>>> really is the case. Even for native builds you wish to point it to the
>>> sysroot for the libtool actually being used. Not to what the compiler
>>> is pointing to, which is now the effect after applying the libtool
>>> patch. But which is actually what you want when building from an
>>> placement independent SDK toolchain for which $CC is updated with a
>>> proper --sysroot argument to gcc automatically when the environment
>>> script is sourced.
>>>
>>> So my propasal now is to update autotools.bbclass and for native
>>> packages set -with-libtool-sysroot=${STAGING_DIR_NATIVE}. This should
>>> cover all the corner cases and work in all different combinations.
>>> Again, consistency is the key. But I still need to try it. Problem now
>>> is that I need to wait until I have access to the machine for which it
>>> currently does not work to verify it properly.
>>>
>>> Please advise. Does anyone object strongly to this idea?
>>
>> Sadly its not the right thing to do. The purpose of a sysroot option is
>> to say "compile and link against things in this subdirectory, do not
>> look outside it". Our native binaries need to use the system libc and
>> headers. They are therefore expected to look outside STAGING_DIR_NATIVE
>> if they don't find what they need in there. Native is really a special
>> hybrid case.
> Here I am not with you really. Yes, the sysroot purpose is exactly what you say.
> But, the sysroot for libtool *is* different. It has nothing to do
> really with what gcc is using, it just happens to be the same on
> standard systems. The purpose of the libtool sysroot is to point to
> libtools own sysroot, and this might not actually be the same as the
> compiler sysroot. As in our case. Since for native packages, gcc and
> libtool are executed in different context, libtool *must* point to its
> proper location for which it was built and that is STAGING_DIR_NATIVE.
>
>>
>> So things should be working if "/" is specified as the sysroot, or its
>> not specified at all. I think there are deeper bugs in libtool we're
>> uncovering :(
>>
>> When I first saw the patch, it looked like the correct thing to be doing
>> but it now seems it may uncover further bugs which are going to need to
>> get fixed before we can take this patch :(.
>>
> I am testing this second patch now in a world build and so far
> everything looks good.
>
Btw, what about the first alternative? Patching "/" to nothing in
libtool. Having lt_sysroot unset seems to work ok.
I agree that patching autotools.bbclass might be wrong. It seems my
patch in libtool only cause problems if gcc return sysroot as "/", not
when it return nothing.

Thanks,
Hans

>> Cheers,
>>
>> Richard
>>
>>
>>