[OE-core] Multilib status

Tue Sep 20 15:01:45 UTC 2011

Hi Mark,

> -----Original Message-----
> From: Mark Hatle [mailto:mark.hatle at windriver.com]
> Sent: Saturday, September 17, 2011 2:19 AM
> To: Mark Hatle
> Cc: Xu, Dongxiao; Richard Purdie; openembedded-core
> Subject: Re: Multilib status
> 
> Just an FYI.  I'm working through a bunch of issues.  I found out that the RPM
> dependency resolution was completely broken, due to a small bug in the
> package_rpm.bbclass..  Packages were not being given the proper provide
> information, so RPM was attempting best match -- and failing as noted in the
> previous discussion..
> 
> Unfortunately this has opened up a small can of worms which I'm trying to deal
> with.  The simplistic fix for the bug is:
> 
> --- a/meta/classes/package_rpm.bbclass
> +++ b/meta/classes/package_rpm.bbclass
> @@ -840,7 +840,7 @@ python do_package_rpm () {
>         os.chmod(outdepends, 0755)
> 
>         # Poky / RPM Provides
> -       outprovides = workdir + "/" + srcname + ".requires"
> +       outprovides = workdir + "/" + srcname + ".provides"
> 
>         try:
>                 from __builtin__ import file
> 
> Unfortunately this has exposed a large set of problems that I didn't realize we
> had....
> 
> Will do a more formal pull request as soon as I get things working again.
> 
> --Mark

I just had a test after cherry-pick some of your changes in mhatle/rpm, here are some problems I found and some investigations:

1. do_rootfs failed.
The error log reports unmet dependency of pkgconfig(libpng).
This is due to the dangling link in libpng package. libpng.pc links to libpng12.pc which has been packaged into libpng12.

2. There is a wrong commit I suppose it should be reverted. When you debugging your code, please revert it.
I will discuss it with Richard tomorrow.
http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/commit/?h=mhatle/rpm&id=329d864f9bbf94ad3aae8df43d63fe10e4237e4f

3. After the above issue got fixed, rootfs succeed but we saw the final image is still in a mess. We need some x86_64 rpm to be installed, however what we got is x86 rpm package. Then I tried another approach we ever talked about that, firstly install base image, then install the multilib packages. I added a parameter "--replacepkgs" to rpm command to avoid errors like "package xxx has already installed". With this approach, the final image could boot up with multilib library installed.

Some of my testing code is located under http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/log/?h=dxu4/ml-testing

Thanks,
Dongxiao

> 
> On 9/16/11 10:26 AM, Mark Hatle wrote:
> > On 9/16/11 10:21 AM, Xu, Dongxiao wrote:
> >>> >> -----Original Message-----
> >>> >> From: Mark Hatle [mailto:mark.hatle at windriver.com]
> >>> >> Sent: Friday, September 16, 2011 10:51 PM
> >>> >> To: Richard Purdie
> >>> >> Cc: Xu, Dongxiao; openembedded-core
> >>> >> Subject: Re: Multilib status
> >>> >>
> >>> >> On 9/16/11 4:32 AM, Richard Purdie wrote:
> >>>> >>> Hi Mark/Dongxaio,
> >>>> >>>
> >>>> >>> We really need to get the remainder of the multilib bugs
> >>>> >>> wrapped up. I think there is some confusion about the exact
> >>>> >>> approach to take to fix some of them so I'm hoping to try and
> >>>> >>> summarise the problems here so we can work out some resolutions.
> >>> >>
> >>> >> Let me explain where I am quickly.  I just spent two days working
> >>> >> on reproducing the issue(s).  So far I've not reproduced the
> >>> >> direct failures but a series of others.  I finally figured out
> >>> >> part of the reason I wasn't seeing this.  I had a typo in my
> configuration:
> >>> >>
> >>> >> DEFAULTTUNE-virtclass-mulitlib-lib32 = "x86"
> >>> >>
> >>> >> This resulted in two sets of binaries normal and lib32 that were
> >>> >> more or less identical in the RPM case.  We -really- need a
> >>> >> sanity check that stupid typos like that don't cause problems.
> >>> >>
> >>> >> ---
> >>> >>
> >>> >> So now that I finally have a built with that done... see below.
> >>> >>
> >>>> >>> Problem A
> >>>> >>> =========
> >>>> >>>
> >>>> >>> We can have multiple forms of "machine specific" packages now,
> >>>> >>> one for each multilib e.g.:
> >>>> >>>
> >>>> >>> task-core-x11-base-1.0-r34.qemux86_64.rpm ("normal")
> >>>> >>> task-core-x11-base-1.0-r34.qemux86_64.rpm ("lib64")
> >>>> >>> task-core-x11-base-1.0-r34.qemux86_64.rpm ("lib32")
> >>>> >>>
> >>>> >>> (lets assume the first uses /lib/, the second /lib64/ and the
> >>>> >>> thrid /lib32/, an artificial example I realise)
> >>>> >>>
> >>>> >>> I know one proposal was to map:
> >>>> >>>
> >>>> >>> task-core-x11-base-1.0-r34.lib32_qemux86_64.rpm to
> >>>> >>> task-core-x11-base-1.0-r34.lib32_qemux86.rpm but this isn't
> >>>> >>> correct since the library directories are different. In this
> >>>> >>> specific case it might not matter but in general it would. What
> >>>> >>> is also important is that the different versions imply
> >>>> >>> different dependencies. The lib64 bit version would pull in and
> >>>> >>> select lib64 bit libs. Its these dependencies which are a key reason
> these are not "all" arch packages.
> >>>> >>>
> >>>> >>> The end result is therefore we likely want:
> >>>> >>>
> >>>> >>> task-core-x11-base-1.0-r34.qemux86_64.rpm
> >>>> >>> task-core-x11-base-1.0-r34.lib64_qemux86_64.rpm
> >>>> >>> task-core-x11-base-1.0-r34.lib32_qemux86_64.rpm
> >>>> >>>
> >>>> >>> and I believe there are some patches Dongxiao has created which do
> this.
> >>> >>
> >>> >> Yes, I'm currently testing his new MACHINE_virtclass-multilib-...
> >>> >> patch.  So far it seems to be working.. assuming this is the
> >>> >> approach we go with, we'll want to add some type of sanity check
> >>> >> so that it's not possible to collide between the different
> >>> >> multilib configurations.  (Alternatively we could, for each multilib,
> specify a default machine virtclass in the format you list above.
> >>> >>
> >>> >> An alternative I was thinking of over night would be to avoid the
> >>> >> RPM remapping on the "machine" packages.. but I'm not sure if
> >>> >> that is really a good idea.
> >>> >> However, it would could simplify the configuration aspects.
> >> >
> >> > There are two implementations towards the above result:
> >> > 1) automatically setting MLPREFIX before MACHINE_ARCH.
> >> > 2) user manually sets
> MACHINE_virtclass-multilib-lib32="lib32-qemux86-64" in local.conf.
> >> >
> >> > Which one do you prefer?
> > Right now I'm thinking the best approach is:
> >
> > *) Allow the user to set MACHINE_virtclass-multilib-...="...."
> >
> > *) If the user has NOT set it, fall back and use the setting of
> > lib..-machine as in 2
> >
> > This way for people who have specific requirements for external
> > compatibility or just desire more control they can do it.  But for
> > everyone else it will still work properly.
> >
> > FYI, I'm currently working in the rootfs_rpm stuff, the system is
> > currently ignoring the alternative (multilib) machine packages.  I'm
> > hoping this will be fairly simple to resolve.
> >
> >>> >>
> >>>> >>> Problem B
> >>>> >>> =========
> >>>> >>>
> >>>> >>> If we install task-core-x11-base-1.0-r34.qemux86_64.rpm which
> >>>> >>> depends on "connman", how does the system figure out to
> >>>> >>> associate this to mean we want the "x86_64" architecture packages
> installed?
> >>>> >>>
> >>>> >>> As far as I can tell there are two proposals around:
> >>>> >>>
> >>>> >>> 1) Set the architecture in the package provides and depends (a
> >>>> >>> bit
> >>>> >>> ugly)
> >>>> >>>
> >>>> >>> 2) Configure libzypp to understand that qemux86_64 does really
> >>>> >>> map to
> >>>> >>> x86_64 architecture and imply x86_64 dependencies.
> >>>> >>>
> >>>> >>> Solving problem A above is the first step towards resolving
> >>>> >>> this either way but we don't seem to have a resolution on this second
> part.
> >>> >>
> >>> >> Agreed.
> >>> >>
> >>>> >>> For reference, we really do want these task packages to have an
> >>>> >>> associated architecture so the user can easily build up blocks
> >>>> >>> of the system using a given ABI.
> >>> >>
> >>> >> This is going to be somewhat of a problem I suspect, simply
> >>> >> because that is not the way RPM is/was designed.  We can adjust
> >>> >> the rootfs install time, but this won't address field upgrade/installs.
> >>> >>
> >>> >> RPM (and Red Hat Linux/Fedora systems) appear to be designed that
> >>> >> any package that reasonably meets the run-time dependencies can be
> used.
> >>> >> There is a concept of a "best" machine based on the resolver
> >>> >> hierarchy, but ELF size may or may not be a factor in that decision.
> >>> >>
> >>> >> Doing this is quite powerful, but it also has a conscious
> >>> >> decision set behind it.  If "bash" is required by a script, it
> >>> >> really doesn't matter if it's ELF32,
> >>> >> ELF64 or something else, as long as something provides bash.
> >> >
> >> > If user sets MULTILIB_IMAGE_INSTALL="lib32-task-core-boot" with a
> qemux86-64 image, it means that he wants to install all elements required by
> task-core-boot by lib32 version. If the dependency is selected *randomly*, users
> would not have multiple libraries in his system.
> > This is a special use case, yes one that I think we need to resolve,
> > but it's not the typical case.
> >
> > The typical case will be someone wants the 64-bit (or 32-bit) version
> > of a select package for compatibility.  And they only want the
> > dependencies for that package to be loaded.  Anything satisfying the
> dependency will do.
> >
> > My recommendation right now is, lets get that working ASAP.  Once that
> > does, refocus the efforts on getting the multilib "tasks" to do something
> reasonable.
> > (i.e. select a group of packages)
> >
> > If we try to solve both at once, I don't think we're going to come up
> > with a reasonable solution at this time.
> >
> > --Mark
> >
> >> > Thanks,
> >> > Dongxiao
> >>> >>
> >>>> >>> What I really need now is an idea of which patches you both
> >>>> >>> agree on that we need to merge (I think we have some for
> >>>> >>> problem A) and a resolution on problem B along with some
> >>>> >>> patches so we can close this set of issues out.
> >>>> >>>
> >>>> >>> If there are further issues can you reply to this email either
> >>>> >>> with the details or a pointer to the specific additional problems.
> >>> >>
> >>> >> Now that I have my stupid typo out of the way, I expect to have
> >>> >> further information in a few hours as to the regularly dependency
> >>> >> resolution failure Dongxiao was reporting.  (Library dependencies
> >>> >> -are- ELF specific, so those have to work properly!)
> >>> >>
> >>> >> --Mark
> >>> >>
> >>>> >>> Cheers,
> >>>> >>>
> >>>> >>> Richard
> >>>> >>>
> >>>> >>>
> >> >
> >