[OE-core] Multilib status

Tue Sep 20 16:57:39 UTC 2011

> >> -----Original Message-----
> >> From: Mark Hatle [mailto:mark.hatle at windriver.com]
> >> Sent: Saturday, September 17, 2011 2:19 AM
> >> To: Mark Hatle
> >> Cc: Xu, Dongxiao; Richard Purdie; openembedded-core
> >> Subject: Re: Multilib status
> >>
> >> Just an FYI.  I'm working through a bunch of issues.  I found out
> >> that the RPM dependency resolution was completely broken, due to a
> >> small bug in the package_rpm.bbclass..  Packages were not being given
> >> the proper provide information, so RPM was attempting best match --
> >> and failing as noted in the previous discussion..
> >>
> >> Unfortunately this has opened up a small can of worms which I'm
> >> trying to deal with.  The simplistic fix for the bug is:
> >>
> >> --- a/meta/classes/package_rpm.bbclass
> >> +++ b/meta/classes/package_rpm.bbclass
> >> @@ -840,7 +840,7 @@ python do_package_rpm () {
> >>         os.chmod(outdepends, 0755)
> >>
> >>         # Poky / RPM Provides
> >> -       outprovides = workdir + "/" + srcname + ".requires"
> >> +       outprovides = workdir + "/" + srcname + ".provides"
> >>
> >>         try:
> >>                 from __builtin__ import file
> >>
> >> Unfortunately this has exposed a large set of problems that I didn't
> >> realize we had....
> >>
> >> Will do a more formal pull request as soon as I get things working again.
> >>
> >> --Mark
> >
> > I just had a test after cherry-pick some of your changes in mhatle/rpm, here
> are some problems I found and some investigations:
> >
> > 1. do_rootfs failed.
> > The error log reports unmet dependency of pkgconfig(libpng).
> > This is due to the dangling link in libpng package. libpng.pc links to libpng12.pc
> which has been packaged into libpng12.
> 
> Ahh this is a bit different then the failures I've seen.  We'll have to fix that one.
> 
> > 2. There is a wrong commit I suppose it should be reverted. When you
> debugging your code, please revert it.
> > I will discuss it with Richard tomorrow.
> > http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/commit/?h=mhatl
> > e/rpm&id=329d864f9bbf94ad3aae8df43d63fe10e4237e4f
> 
> Ahh, ya I just ran into that problem... I have a fix for that commit I just finished.
> (Do you have a newer version, if so I'll update to that and see if my change is still
> needed.)

No I haven't worked on a new version of that patch. Could you share yours?

Thanks,
Dongxiao

> 
> > 3. After the above issue got fixed, rootfs succeed but we saw the final image is
> still in a mess. We need some x86_64 rpm to be installed, however what we got
> is x86 rpm package. Then I tried another approach we ever talked about that,
> firstly install base image, then install the multilib packages. I added a parameter
> "--replacepkgs" to rpm command to avoid errors like "package xxx has already
> installed". With this approach, the final image could boot up with multilib library
> installed.
> >
> > Some of my testing code is located under
> > http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/log/?h=dxu4/ml-
> > testing
> 
> I'll run through the testing code and see if I can figure out what is different.
> 
> --Mark
> 
> > Thanks,
> > Dongxiao
> >
> >
> >>
> >> On 9/16/11 10:26 AM, Mark Hatle wrote:
> >>> On 9/16/11 10:21 AM, Xu, Dongxiao wrote:
> >>>>>>> -----Original Message-----
> >>>>>>> From: Mark Hatle [mailto:mark.hatle at windriver.com]
> >>>>>>> Sent: Friday, September 16, 2011 10:51 PM
> >>>>>>> To: Richard Purdie
> >>>>>>> Cc: Xu, Dongxiao; openembedded-core
> >>>>>>> Subject: Re: Multilib status
> >>>>>>>
> >>>>>>> On 9/16/11 4:32 AM, Richard Purdie wrote:
> >>>>>>>>> Hi Mark/Dongxaio,
> >>>>>>>>>
> >>>>>>>>> We really need to get the remainder of the multilib bugs
> >>>>>>>>> wrapped up. I think there is some confusion about the exact
> >>>>>>>>> approach to take to fix some of them so I'm hoping to try and
> >>>>>>>>> summarise the problems here so we can work out some resolutions.
> >>>>>>>
> >>>>>>> Let me explain where I am quickly.  I just spent two days
> >>>>>>> working on reproducing the issue(s).  So far I've not reproduced
> >>>>>>> the direct failures but a series of others.  I finally figured
> >>>>>>> out part of the reason I wasn't seeing this.  I had a typo in my
> >> configuration:
> >>>>>>>
> >>>>>>> DEFAULTTUNE-virtclass-mulitlib-lib32 = "x86"
> >>>>>>>
> >>>>>>> This resulted in two sets of binaries normal and lib32 that were
> >>>>>>> more or less identical in the RPM case.  We -really- need a
> >>>>>>> sanity check that stupid typos like that don't cause problems.
> >>>>>>>
> >>>>>>> ---
> >>>>>>>
> >>>>>>> So now that I finally have a built with that done... see below.
> >>>>>>>
> >>>>>>>>> Problem A
> >>>>>>>>> =========
> >>>>>>>>>
> >>>>>>>>> We can have multiple forms of "machine specific" packages now,
> >>>>>>>>> one for each multilib e.g.:
> >>>>>>>>>
> >>>>>>>>> task-core-x11-base-1.0-r34.qemux86_64.rpm ("normal")
> >>>>>>>>> task-core-x11-base-1.0-r34.qemux86_64.rpm ("lib64")
> >>>>>>>>> task-core-x11-base-1.0-r34.qemux86_64.rpm ("lib32")
> >>>>>>>>>
> >>>>>>>>> (lets assume the first uses /lib/, the second /lib64/ and the
> >>>>>>>>> thrid /lib32/, an artificial example I realise)
> >>>>>>>>>
> >>>>>>>>> I know one proposal was to map:
> >>>>>>>>>
> >>>>>>>>> task-core-x11-base-1.0-r34.lib32_qemux86_64.rpm to
> >>>>>>>>> task-core-x11-base-1.0-r34.lib32_qemux86.rpm but this isn't
> >>>>>>>>> correct since the library directories are different. In this
> >>>>>>>>> specific case it might not matter but in general it would.
> >>>>>>>>> What is also important is that the different versions imply
> >>>>>>>>> different dependencies. The lib64 bit version would pull in
> >>>>>>>>> and select lib64 bit libs. Its these dependencies which are a
> >>>>>>>>> key reason
> >> these are not "all" arch packages.
> >>>>>>>>>
> >>>>>>>>> The end result is therefore we likely want:
> >>>>>>>>>
> >>>>>>>>> task-core-x11-base-1.0-r34.qemux86_64.rpm
> >>>>>>>>> task-core-x11-base-1.0-r34.lib64_qemux86_64.rpm
> >>>>>>>>> task-core-x11-base-1.0-r34.lib32_qemux86_64.rpm
> >>>>>>>>>
> >>>>>>>>> and I believe there are some patches Dongxiao has created
> >>>>>>>>> which do
> >> this.
> >>>>>>>
> >>>>>>> Yes, I'm currently testing his new MACHINE_virtclass-multilib-...
> >>>>>>> patch.  So far it seems to be working.. assuming this is the
> >>>>>>> approach we go with, we'll want to add some type of sanity check
> >>>>>>> so that it's not possible to collide between the different
> >>>>>>> multilib configurations.  (Alternatively we could, for each
> >>>>>>> multilib,
> >> specify a default machine virtclass in the format you list above.
> >>>>>>>
> >>>>>>> An alternative I was thinking of over night would be to avoid
> >>>>>>> the RPM remapping on the "machine" packages.. but I'm not sure
> >>>>>>> if that is really a good idea.
> >>>>>>> However, it would could simplify the configuration aspects.
> >>>>>
> >>>>> There are two implementations towards the above result:
> >>>>> 1) automatically setting MLPREFIX before MACHINE_ARCH.
> >>>>> 2) user manually sets
> >> MACHINE_virtclass-multilib-lib32="lib32-qemux86-64" in local.conf.
> >>>>>
> >>>>> Which one do you prefer?
> >>> Right now I'm thinking the best approach is:
> >>>
> >>> *) Allow the user to set MACHINE_virtclass-multilib-...="...."
> >>>
> >>> *) If the user has NOT set it, fall back and use the setting of
> >>> lib..-machine as in 2
> >>>
> >>> This way for people who have specific requirements for external
> >>> compatibility or just desire more control they can do it.  But for
> >>> everyone else it will still work properly.
> >>>
> >>> FYI, I'm currently working in the rootfs_rpm stuff, the system is
> >>> currently ignoring the alternative (multilib) machine packages.  I'm
> >>> hoping this will be fairly simple to resolve.
> >>>
> >>>>>>>
> >>>>>>>>> Problem B
> >>>>>>>>> =========
> >>>>>>>>>
> >>>>>>>>> If we install task-core-x11-base-1.0-r34.qemux86_64.rpm which
> >>>>>>>>> depends on "connman", how does the system figure out to
> >>>>>>>>> associate this to mean we want the "x86_64" architecture
> >>>>>>>>> packages
> >> installed?
> >>>>>>>>>
> >>>>>>>>> As far as I can tell there are two proposals around:
> >>>>>>>>>
> >>>>>>>>> 1) Set the architecture in the package provides and depends (a
> >>>>>>>>> bit
> >>>>>>>>> ugly)
> >>>>>>>>>
> >>>>>>>>> 2) Configure libzypp to understand that qemux86_64 does really
> >>>>>>>>> map to
> >>>>>>>>> x86_64 architecture and imply x86_64 dependencies.
> >>>>>>>>>
> >>>>>>>>> Solving problem A above is the first step towards resolving
> >>>>>>>>> this either way but we don't seem to have a resolution on this
> >>>>>>>>> second
> >> part.
> >>>>>>>
> >>>>>>> Agreed.
> >>>>>>>
> >>>>>>>>> For reference, we really do want these task packages to have
> >>>>>>>>> an associated architecture so the user can easily build up
> >>>>>>>>> blocks of the system using a given ABI.
> >>>>>>>
> >>>>>>> This is going to be somewhat of a problem I suspect, simply
> >>>>>>> because that is not the way RPM is/was designed.  We can adjust
> >>>>>>> the rootfs install time, but this won't address field upgrade/installs.
> >>>>>>>
> >>>>>>> RPM (and Red Hat Linux/Fedora systems) appear to be designed
> >>>>>>> that any package that reasonably meets the run-time dependencies
> >>>>>>> can be
> >> used.
> >>>>>>> There is a concept of a "best" machine based on the resolver
> >>>>>>> hierarchy, but ELF size may or may not be a factor in that decision.
> >>>>>>>
> >>>>>>> Doing this is quite powerful, but it also has a conscious
> >>>>>>> decision set behind it.  If "bash" is required by a script, it
> >>>>>>> really doesn't matter if it's ELF32,
> >>>>>>> ELF64 or something else, as long as something provides bash.
> >>>>>
> >>>>> If user sets MULTILIB_IMAGE_INSTALL="lib32-task-core-boot" with a
> >> qemux86-64 image, it means that he wants to install all elements
> >> required by task-core-boot by lib32 version. If the dependency is
> >> selected *randomly*, users would not have multiple libraries in his system.
> >>> This is a special use case, yes one that I think we need to resolve,
> >>> but it's not the typical case.
> >>>
> >>> The typical case will be someone wants the 64-bit (or 32-bit)
> >>> version of a select package for compatibility.  And they only want
> >>> the dependencies for that package to be loaded.  Anything satisfying
> >>> the
> >> dependency will do.
> >>>
> >>> My recommendation right now is, lets get that working ASAP.  Once
> >>> that does, refocus the efforts on getting the multilib "tasks" to do
> >>> something
> >> reasonable.
> >>> (i.e. select a group of packages)
> >>>
> >>> If we try to solve both at once, I don't think we're going to come
> >>> up with a reasonable solution at this time.
> >>>
> >>> --Mark
> >>>
> >>>>> Thanks,
> >>>>> Dongxiao
> >>>>>>>
> >>>>>>>>> What I really need now is an idea of which patches you both
> >>>>>>>>> agree on that we need to merge (I think we have some for
> >>>>>>>>> problem A) and a resolution on problem B along with some
> >>>>>>>>> patches so we can close this set of issues out.
> >>>>>>>>>
> >>>>>>>>> If there are further issues can you reply to this email either
> >>>>>>>>> with the details or a pointer to the specific additional problems.
> >>>>>>>
> >>>>>>> Now that I have my stupid typo out of the way, I expect to have
> >>>>>>> further information in a few hours as to the regularly
> >>>>>>> dependency resolution failure Dongxiao was reporting.  (Library
> >>>>>>> dependencies
> >>>>>>> -are- ELF specific, so those have to work properly!)
> >>>>>>>
> >>>>>>> --Mark
> >>>>>>>
> >>>>>>>>> Cheers,
> >>>>>>>>>
> >>>>>>>>> Richard
> >>>>>>>>>
> >>>>>>>>>
> >>>>>
> >>>