[OE-core] Multilib status

[Mark Hatle]

Just an FYI.  I'm working through a bunch of issues.  I found out that the RPM
dependency resolution was completely broken, due to a small bug in the
package_rpm.bbclass..  Packages were not being given the proper provide
information, so RPM was attempting best match -- and failing as noted in the
previous discussion..

Unfortunately this has opened up a small can of worms which I'm trying to deal
with.  The simplistic fix for the bug is:

--- a/meta/classes/package_rpm.bbclass
+++ b/meta/classes/package_rpm.bbclass
@@ -840,7 +840,7 @@ python do_package_rpm () {
        os.chmod(outdepends, 0755)

        # Poky / RPM Provides
-       outprovides = workdir + "/" + srcname + ".requires"
+       outprovides = workdir + "/" + srcname + ".provides"

        try:
                from __builtin__ import file

Unfortunately this has exposed a large set of problems that I didn't realize we
had....

Will do a more formal pull request as soon as I get things working again.

--Mark

On 9/16/11 10:26 AM, Mark Hatle wrote:
> On 9/16/11 10:21 AM, Xu, Dongxiao wrote:
>>> >> -----Original Message-----
>>> >> From: Mark Hatle [mailto:mark.hatle at windriver.com]
>>> >> Sent: Friday, September 16, 2011 10:51 PM
>>> >> To: Richard Purdie
>>> >> Cc: Xu, Dongxiao; openembedded-core
>>> >> Subject: Re: Multilib status
>>> >>
>>> >> On 9/16/11 4:32 AM, Richard Purdie wrote:
>>>> >>> Hi Mark/Dongxaio,
>>>> >>>
>>>> >>> We really need to get the remainder of the multilib bugs wrapped up. I
>>>> >>> think there is some confusion about the exact approach to take to fix
>>>> >>> some of them so I'm hoping to try and summarise the problems here so
>>>> >>> we can work out some resolutions.
>>> >>
>>> >> Let me explain where I am quickly.  I just spent two days working on
>>> >> reproducing the issue(s).  So far I've not reproduced the direct failures but a
>>> >> series of others.  I finally figured out part of the reason I wasn't seeing this.  I
>>> >> had a typo in my configuration:
>>> >>
>>> >> DEFAULTTUNE-virtclass-mulitlib-lib32 = "x86"
>>> >>
>>> >> This resulted in two sets of binaries normal and lib32 that were more or less
>>> >> identical in the RPM case.  We -really- need a sanity check that stupid typos
>>> >> like that don't cause problems.
>>> >>
>>> >> ---
>>> >>
>>> >> So now that I finally have a built with that done... see below.
>>> >>
>>>> >>> Problem A
>>>> >>> =========
>>>> >>>
>>>> >>> We can have multiple forms of "machine specific" packages now, one for
>>>> >>> each multilib e.g.:
>>>> >>>
>>>> >>> task-core-x11-base-1.0-r34.qemux86_64.rpm ("normal")
>>>> >>> task-core-x11-base-1.0-r34.qemux86_64.rpm ("lib64")
>>>> >>> task-core-x11-base-1.0-r34.qemux86_64.rpm ("lib32")
>>>> >>>
>>>> >>> (lets assume the first uses /lib/, the second /lib64/ and the thrid
>>>> >>> /lib32/, an artificial example I realise)
>>>> >>>
>>>> >>> I know one proposal was to map:
>>>> >>>
>>>> >>> task-core-x11-base-1.0-r34.lib32_qemux86_64.rpm to
>>>> >>> task-core-x11-base-1.0-r34.lib32_qemux86.rpm but this isn't correct
>>>> >>> since the library directories are different. In this specific case it
>>>> >>> might not matter but in general it would. What is also important is
>>>> >>> that the different versions imply different dependencies. The lib64
>>>> >>> bit version would pull in and select lib64 bit libs. Its these
>>>> >>> dependencies which are a key reason these are not "all" arch packages.
>>>> >>>
>>>> >>> The end result is therefore we likely want:
>>>> >>>
>>>> >>> task-core-x11-base-1.0-r34.qemux86_64.rpm
>>>> >>> task-core-x11-base-1.0-r34.lib64_qemux86_64.rpm
>>>> >>> task-core-x11-base-1.0-r34.lib32_qemux86_64.rpm
>>>> >>>
>>>> >>> and I believe there are some patches Dongxiao has created which do this.
>>> >>
>>> >> Yes, I'm currently testing his new MACHINE_virtclass-multilib-... patch.  So far
>>> >> it seems to be working.. assuming this is the approach we go with, we'll want to
>>> >> add some type of sanity check so that it's not possible to collide between the
>>> >> different multilib configurations.  (Alternatively we could, for each multilib,
>>> >> specify a default machine virtclass in the format you list above.
>>> >>
>>> >> An alternative I was thinking of over night would be to avoid the RPM
>>> >> remapping on the "machine" packages.. but I'm not sure if that is really a good
>>> >> idea.
>>> >> However, it would could simplify the configuration aspects.
>> > 
>> > There are two implementations towards the above result:
>> > 1) automatically setting MLPREFIX before MACHINE_ARCH.
>> > 2) user manually sets MACHINE_virtclass-multilib-lib32="lib32-qemux86-64" in local.conf.
>> > 
>> > Which one do you prefer?
> Right now I'm thinking the best approach is:
> 
> *) Allow the user to set MACHINE_virtclass-multilib-...="...."
> 
> *) If the user has NOT set it, fall back and use the setting of lib..-machine as
> in 2
> 
> This way for people who have specific requirements for external compatibility or
> just desire more control they can do it.  But for everyone else it will still
> work properly.
> 
> FYI, I'm currently working in the rootfs_rpm stuff, the system is currently
> ignoring the alternative (multilib) machine packages.  I'm hoping this will be
> fairly simple to resolve.
> 
>>> >>
>>>> >>> Problem B
>>>> >>> =========
>>>> >>>
>>>> >>> If we install task-core-x11-base-1.0-r34.qemux86_64.rpm which depends
>>>> >>> on "connman", how does the system figure out to associate this to mean
>>>> >>> we want the "x86_64" architecture packages installed?
>>>> >>>
>>>> >>> As far as I can tell there are two proposals around:
>>>> >>>
>>>> >>> 1) Set the architecture in the package provides and depends (a bit
>>>> >>> ugly)
>>>> >>>
>>>> >>> 2) Configure libzypp to understand that qemux86_64 does really map to
>>>> >>> x86_64 architecture and imply x86_64 dependencies.
>>>> >>>
>>>> >>> Solving problem A above is the first step towards resolving this
>>>> >>> either way but we don't seem to have a resolution on this second part.
>>> >>
>>> >> Agreed.
>>> >>
>>>> >>> For reference, we really do want these task packages to have an
>>>> >>> associated architecture so the user can easily build up blocks of the
>>>> >>> system using a given ABI.
>>> >>
>>> >> This is going to be somewhat of a problem I suspect, simply because that is not
>>> >> the way RPM is/was designed.  We can adjust the rootfs install time, but this
>>> >> won't address field upgrade/installs.
>>> >>
>>> >> RPM (and Red Hat Linux/Fedora systems) appear to be designed that any
>>> >> package that reasonably meets the run-time dependencies can be used.
>>> >> There is a concept of a "best" machine based on the resolver hierarchy, but ELF
>>> >> size may or may not be a factor in that decision.
>>> >>
>>> >> Doing this is quite powerful, but it also has a conscious decision set behind it.  If
>>> >> "bash" is required by a script, it really doesn't matter if it's ELF32,
>>> >> ELF64 or something else, as long as something provides bash.
>> > 
>> > If user sets MULTILIB_IMAGE_INSTALL="lib32-task-core-boot" with a qemux86-64 image, it means that he wants to install all elements required by task-core-boot by lib32 version. If the dependency is selected *randomly*, users would not have multiple libraries in his system.
> This is a special use case, yes one that I think we need to resolve, but it's
> not the typical case.
> 
> The typical case will be someone wants the 64-bit (or 32-bit) version of a
> select package for compatibility.  And they only want the dependencies for that
> package to be loaded.  Anything satisfying the dependency will do.
> 
> My recommendation right now is, lets get that working ASAP.  Once that does,
> refocus the efforts on getting the multilib "tasks" to do something reasonable.
> (i.e. select a group of packages)
> 
> If we try to solve both at once, I don't think we're going to come up with a
> reasonable solution at this time.
> 
> --Mark
> 
>> > Thanks,
>> > Dongxiao
>>> >>
>>>> >>> What I really need now is an idea of which patches you both agree on
>>>> >>> that we need to merge (I think we have some for problem A) and a
>>>> >>> resolution on problem B along with some patches so we can close this
>>>> >>> set of issues out.
>>>> >>>
>>>> >>> If there are further issues can you reply to this email either with
>>>> >>> the details or a pointer to the specific additional problems.
>>> >>
>>> >> Now that I have my stupid typo out of the way, I expect to have further
>>> >> information in a few hours as to the regularly dependency resolution failure
>>> >> Dongxiao was reporting.  (Library dependencies -are- ELF specific, so those
>>> >> have to work properly!)
>>> >>
>>> >> --Mark
>>> >>
>>>> >>> Cheers,
>>>> >>>
>>>> >>> Richard
>>>> >>>
>>>> >>>
>> > 
>