[OE-core] Multilib status

Mark Hatle mark.hatle at windriver.com
Tue Sep 20 18:11:32 UTC 2011 

On 9/20/11 12:34 PM, Xu, Dongxiao wrote:
>> -----Original Message-----
>> From: Mark Hatle [mailto:mark.hatle at windriver.com]
>> Sent: Wednesday, September 21, 2011 1:06 AM
>> To: Xu, Dongxiao
>> Cc: Richard Purdie; openembedded-core
>> Subject: Re: Multilib status
>>
>> commit id aea5b9ef458099efd8f9a83428af84bbbe69eed2
>>
>> I'm really not sure what I have is right though.. it seems to be not failing..
>> ;)  But you likely know what should trigger these changes and if what I did is
>> correct or not.
> 
> What I was trying to solve with the previous commit is to handle the multiple (r)provider issue of those allarch recipes.
> 
> It seems that although your changes don't solve the above problem, they are needed to handle RPROVIDES mappings.
> 
> For the multiple provider issue, I will discuss it with Richard tomorrow.

Ya, what I sent out early was broken.. I think I have it working now:

poky-contrib.git -- commit id: d971a73d970b8f0beeded599f95a6d8ce5c32a4c

This seems to correctly specify the PROVIDES, RPROVIDES and RPROVIDES_<PN> for
the multilibs now.. I'm still not sure that this is right though, because it may
change the PROVIDES/RPROVIDES for packages that were previously cached from non
multilib builds..  but it's progress.

--Mark

> Thanks,
> Dongxiao
> 
>>
>> --Mark
>>
>> On 9/20/11 11:57 AM, Xu, Dongxiao wrote:
>>>>>> -----Original Message-----
>>>>>> From: Mark Hatle [mailto:mark.hatle at windriver.com]
>>>>>> Sent: Saturday, September 17, 2011 2:19 AM
>>>>>> To: Mark Hatle
>>>>>> Cc: Xu, Dongxiao; Richard Purdie; openembedded-core
>>>>>> Subject: Re: Multilib status
>>>>>>
>>>>>> Just an FYI.  I'm working through a bunch of issues.  I found out
>>>>>> that the RPM dependency resolution was completely broken, due to a
>>>>>> small bug in the package_rpm.bbclass..  Packages were not being
>>>>>> given the proper provide information, so RPM was attempting best
>>>>>> match -- and failing as noted in the previous discussion..
>>>>>>
>>>>>> Unfortunately this has opened up a small can of worms which I'm
>>>>>> trying to deal with.  The simplistic fix for the bug is:
>>>>>>
>>>>>> --- a/meta/classes/package_rpm.bbclass
>>>>>> +++ b/meta/classes/package_rpm.bbclass
>>>>>> @@ -840,7 +840,7 @@ python do_package_rpm () {
>>>>>>         os.chmod(outdepends, 0755)
>>>>>>
>>>>>>         # Poky / RPM Provides
>>>>>> -       outprovides = workdir + "/" + srcname + ".requires"
>>>>>> +       outprovides = workdir + "/" + srcname + ".provides"
>>>>>>
>>>>>>         try:
>>>>>>                 from __builtin__ import file
>>>>>>
>>>>>> Unfortunately this has exposed a large set of problems that I
>>>>>> didn't realize we had....
>>>>>>
>>>>>> Will do a more formal pull request as soon as I get things working again.
>>>>>>
>>>>>> --Mark
>>>>>
>>>>> I just had a test after cherry-pick some of your changes in
>>>>> mhatle/rpm, here
>>>> are some problems I found and some investigations:
>>>>>
>>>>> 1. do_rootfs failed.
>>>>> The error log reports unmet dependency of pkgconfig(libpng).
>>>>> This is due to the dangling link in libpng package. libpng.pc links
>>>>> to libpng12.pc
>>>> which has been packaged into libpng12.
>>>>
>>>> Ahh this is a bit different then the failures I've seen.  We'll have to fix that
>> one.
>>>>
>>>>> 2. There is a wrong commit I suppose it should be reverted. When you
>>>> debugging your code, please revert it.
>>>>> I will discuss it with Richard tomorrow.
>>>>> http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/commit/?h=mha
>>>>> tl e/rpm&id=329d864f9bbf94ad3aae8df43d63fe10e4237e4f
>>>>
>>>> Ahh, ya I just ran into that problem... I have a fix for that commit I just
>> finished.
>>>> (Do you have a newer version, if so I'll update to that and see if my
>>>> change is still
>>>> needed.)
>>>
>>> No I haven't worked on a new version of that patch. Could you share yours?
>>>
>>> Thanks,
>>> Dongxiao
>>>
>>>>
>>>>> 3. After the above issue got fixed, rootfs succeed but we saw the
>>>>> final image is
>>>> still in a mess. We need some x86_64 rpm to be installed, however
>>>> what we got is x86 rpm package. Then I tried another approach we ever
>>>> talked about that, firstly install base image, then install the
>>>> multilib packages. I added a parameter "--replacepkgs" to rpm command
>>>> to avoid errors like "package xxx has already installed". With this
>>>> approach, the final image could boot up with multilib library installed.
>>>>>
>>>>> Some of my testing code is located under
>>>>> http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/log/?h=dxu4/m
>>>>> l-
>>>>> testing
>>>>
>>>> I'll run through the testing code and see if I can figure out what is different.
>>>>
>>>> --Mark
>>>>
>>>>> Thanks,
>>>>> Dongxiao
>>>>>
>>>>>
>>>>>>
>>>>>> On 9/16/11 10:26 AM, Mark Hatle wrote:
>>>>>>> On 9/16/11 10:21 AM, Xu, Dongxiao wrote:
>>>>>>>>>>> -----Original Message-----
>>>>>>>>>>> From: Mark Hatle [mailto:mark.hatle at windriver.com]
>>>>>>>>>>> Sent: Friday, September 16, 2011 10:51 PM
>>>>>>>>>>> To: Richard Purdie
>>>>>>>>>>> Cc: Xu, Dongxiao; openembedded-core
>>>>>>>>>>> Subject: Re: Multilib status
>>>>>>>>>>>
>>>>>>>>>>> On 9/16/11 4:32 AM, Richard Purdie wrote:
>>>>>>>>>>>>> Hi Mark/Dongxaio,
>>>>>>>>>>>>>
>>>>>>>>>>>>> We really need to get the remainder of the multilib bugs
>>>>>>>>>>>>> wrapped up. I think there is some confusion about the exact
>>>>>>>>>>>>> approach to take to fix some of them so I'm hoping to try
>>>>>>>>>>>>> and summarise the problems here so we can work out some
>> resolutions.
>>>>>>>>>>>
>>>>>>>>>>> Let me explain where I am quickly.  I just spent two days
>>>>>>>>>>> working on reproducing the issue(s).  So far I've not
>>>>>>>>>>> reproduced the direct failures but a series of others.  I
>>>>>>>>>>> finally figured out part of the reason I wasn't seeing this.
>>>>>>>>>>> I had a typo in my
>>>>>> configuration:
>>>>>>>>>>>
>>>>>>>>>>> DEFAULTTUNE-virtclass-mulitlib-lib32 = "x86"
>>>>>>>>>>>
>>>>>>>>>>> This resulted in two sets of binaries normal and lib32 that
>>>>>>>>>>> were more or less identical in the RPM case.  We -really- need
>>>>>>>>>>> a sanity check that stupid typos like that don't cause problems.
>>>>>>>>>>>
>>>>>>>>>>> ---
>>>>>>>>>>>
>>>>>>>>>>> So now that I finally have a built with that done... see below.
>>>>>>>>>>>
>>>>>>>>>>>>> Problem A
>>>>>>>>>>>>> =========
>>>>>>>>>>>>>
>>>>>>>>>>>>> We can have multiple forms of "machine specific" packages
>>>>>>>>>>>>> now, one for each multilib e.g.:
>>>>>>>>>>>>>
>>>>>>>>>>>>> task-core-x11-base-1.0-r34.qemux86_64.rpm ("normal")
>>>>>>>>>>>>> task-core-x11-base-1.0-r34.qemux86_64.rpm ("lib64")
>>>>>>>>>>>>> task-core-x11-base-1.0-r34.qemux86_64.rpm ("lib32")
>>>>>>>>>>>>>
>>>>>>>>>>>>> (lets assume the first uses /lib/, the second /lib64/ and
>>>>>>>>>>>>> the thrid /lib32/, an artificial example I realise)
>>>>>>>>>>>>>
>>>>>>>>>>>>> I know one proposal was to map:
>>>>>>>>>>>>>
>>>>>>>>>>>>> task-core-x11-base-1.0-r34.lib32_qemux86_64.rpm to
>>>>>>>>>>>>> task-core-x11-base-1.0-r34.lib32_qemux86.rpm but this isn't
>>>>>>>>>>>>> correct since the library directories are different. In this
>>>>>>>>>>>>> specific case it might not matter but in general it would.
>>>>>>>>>>>>> What is also important is that the different versions imply
>>>>>>>>>>>>> different dependencies. The lib64 bit version would pull in
>>>>>>>>>>>>> and select lib64 bit libs. Its these dependencies which are
>>>>>>>>>>>>> a key reason
>>>>>> these are not "all" arch packages.
>>>>>>>>>>>>>
>>>>>>>>>>>>> The end result is therefore we likely want:
>>>>>>>>>>>>>
>>>>>>>>>>>>> task-core-x11-base-1.0-r34.qemux86_64.rpm
>>>>>>>>>>>>> task-core-x11-base-1.0-r34.lib64_qemux86_64.rpm
>>>>>>>>>>>>> task-core-x11-base-1.0-r34.lib32_qemux86_64.rpm
>>>>>>>>>>>>>
>>>>>>>>>>>>> and I believe there are some patches Dongxiao has created
>>>>>>>>>>>>> which do
>>>>>> this.
>>>>>>>>>>>
>>>>>>>>>>> Yes, I'm currently testing his new MACHINE_virtclass-multilib-...
>>>>>>>>>>> patch.  So far it seems to be working.. assuming this is the
>>>>>>>>>>> approach we go with, we'll want to add some type of sanity
>>>>>>>>>>> check so that it's not possible to collide between the
>>>>>>>>>>> different multilib configurations.  (Alternatively we could,
>>>>>>>>>>> for each multilib,
>>>>>> specify a default machine virtclass in the format you list above.
>>>>>>>>>>>
>>>>>>>>>>> An alternative I was thinking of over night would be to avoid
>>>>>>>>>>> the RPM remapping on the "machine" packages.. but I'm not sure
>>>>>>>>>>> if that is really a good idea.
>>>>>>>>>>> However, it would could simplify the configuration aspects.
>>>>>>>>>
>>>>>>>>> There are two implementations towards the above result:
>>>>>>>>> 1) automatically setting MLPREFIX before MACHINE_ARCH.
>>>>>>>>> 2) user manually sets
>>>>>> MACHINE_virtclass-multilib-lib32="lib32-qemux86-64" in local.conf.
>>>>>>>>>
>>>>>>>>> Which one do you prefer?
>>>>>>> Right now I'm thinking the best approach is:
>>>>>>>
>>>>>>> *) Allow the user to set MACHINE_virtclass-multilib-...="...."
>>>>>>>
>>>>>>> *) If the user has NOT set it, fall back and use the setting of
>>>>>>> lib..-machine as in 2
>>>>>>>
>>>>>>> This way for people who have specific requirements for external
>>>>>>> compatibility or just desire more control they can do it.  But for
>>>>>>> everyone else it will still work properly.
>>>>>>>
>>>>>>> FYI, I'm currently working in the rootfs_rpm stuff, the system is
>>>>>>> currently ignoring the alternative (multilib) machine packages.
>>>>>>> I'm hoping this will be fairly simple to resolve.
>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>>> Problem B
>>>>>>>>>>>>> =========
>>>>>>>>>>>>>
>>>>>>>>>>>>> If we install task-core-x11-base-1.0-r34.qemux86_64.rpm
>>>>>>>>>>>>> which depends on "connman", how does the system figure out
>>>>>>>>>>>>> to associate this to mean we want the "x86_64" architecture
>>>>>>>>>>>>> packages
>>>>>> installed?
>>>>>>>>>>>>>
>>>>>>>>>>>>> As far as I can tell there are two proposals around:
>>>>>>>>>>>>>
>>>>>>>>>>>>> 1) Set the architecture in the package provides and depends
>>>>>>>>>>>>> (a bit
>>>>>>>>>>>>> ugly)
>>>>>>>>>>>>>
>>>>>>>>>>>>> 2) Configure libzypp to understand that qemux86_64 does
>>>>>>>>>>>>> really map to
>>>>>>>>>>>>> x86_64 architecture and imply x86_64 dependencies.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Solving problem A above is the first step towards resolving
>>>>>>>>>>>>> this either way but we don't seem to have a resolution on
>>>>>>>>>>>>> this second
>>>>>> part.
>>>>>>>>>>>
>>>>>>>>>>> Agreed.
>>>>>>>>>>>
>>>>>>>>>>>>> For reference, we really do want these task packages to have
>>>>>>>>>>>>> an associated architecture so the user can easily build up
>>>>>>>>>>>>> blocks of the system using a given ABI.
>>>>>>>>>>>
>>>>>>>>>>> This is going to be somewhat of a problem I suspect, simply
>>>>>>>>>>> because that is not the way RPM is/was designed.  We can
>>>>>>>>>>> adjust the rootfs install time, but this won't address field
>> upgrade/installs.
>>>>>>>>>>>
>>>>>>>>>>> RPM (and Red Hat Linux/Fedora systems) appear to be designed
>>>>>>>>>>> that any package that reasonably meets the run-time
>>>>>>>>>>> dependencies can be
>>>>>> used.
>>>>>>>>>>> There is a concept of a "best" machine based on the resolver
>>>>>>>>>>> hierarchy, but ELF size may or may not be a factor in that decision.
>>>>>>>>>>>
>>>>>>>>>>> Doing this is quite powerful, but it also has a conscious
>>>>>>>>>>> decision set behind it.  If "bash" is required by a script, it
>>>>>>>>>>> really doesn't matter if it's ELF32,
>>>>>>>>>>> ELF64 or something else, as long as something provides bash.
>>>>>>>>>
>>>>>>>>> If user sets MULTILIB_IMAGE_INSTALL="lib32-task-core-boot" with
>>>>>>>>> a
>>>>>> qemux86-64 image, it means that he wants to install all elements
>>>>>> required by task-core-boot by lib32 version. If the dependency is
>>>>>> selected *randomly*, users would not have multiple libraries in his system.
>>>>>>> This is a special use case, yes one that I think we need to
>>>>>>> resolve, but it's not the typical case.
>>>>>>>
>>>>>>> The typical case will be someone wants the 64-bit (or 32-bit)
>>>>>>> version of a select package for compatibility.  And they only want
>>>>>>> the dependencies for that package to be loaded.  Anything
>>>>>>> satisfying the
>>>>>> dependency will do.
>>>>>>>
>>>>>>> My recommendation right now is, lets get that working ASAP.  Once
>>>>>>> that does, refocus the efforts on getting the multilib "tasks" to
>>>>>>> do something
>>>>>> reasonable.
>>>>>>> (i.e. select a group of packages)
>>>>>>>
>>>>>>> If we try to solve both at once, I don't think we're going to come
>>>>>>> up with a reasonable solution at this time.
>>>>>>>
>>>>>>> --Mark
>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>> Dongxiao
>>>>>>>>>>>
>>>>>>>>>>>>> What I really need now is an idea of which patches you both
>>>>>>>>>>>>> agree on that we need to merge (I think we have some for
>>>>>>>>>>>>> problem A) and a resolution on problem B along with some
>>>>>>>>>>>>> patches so we can close this set of issues out.
>>>>>>>>>>>>>
>>>>>>>>>>>>> If there are further issues can you reply to this email
>>>>>>>>>>>>> either with the details or a pointer to the specific additional
>> problems.
>>>>>>>>>>>
>>>>>>>>>>> Now that I have my stupid typo out of the way, I expect to
>>>>>>>>>>> have further information in a few hours as to the regularly
>>>>>>>>>>> dependency resolution failure Dongxiao was reporting.
>>>>>>>>>>> (Library dependencies
>>>>>>>>>>> -are- ELF specific, so those have to work properly!)
>>>>>>>>>>>
>>>>>>>>>>> --Mark
>>>>>>>>>>>
>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>>
>>>>>>>>>>>>> Richard
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>
>>>>>>>
>>>
>

More information about the Openembedded-core mailing list