[OE-core] About multilib packages rpm pkg name

[Robert Yang]

On 08/07/2015 04:57 AM, Mark Hatle wrote:
> On 8/5/15 9:33 PM, Robert Yang wrote:
>>
>> Hi Mark,
>>
>> Thank you very much, this makes things more clear now, I've tried
>> to change packagegroup-core-standalone-sdk-target's PACKAGE_ARCH
>> to MACHINE_ARCH or TUNE_PKGARCH, it didn't work (the rdepends lib32
>> are still not installed), I will try to use read_subpkgdata() to get
>> the required pkgs and fix the problem.
>
> During package creation we strip off the multilib prefix (since it's not
> needed).  Then during dependency calculations things often favor a 'shortest
> path' to the dependency which may not give you the 32-bit version you are expecting.
>
> What I'd like to see is a patch to smart where you can say "Use this ARCH or
> word size, unless explicitly selected".  That would help eliminate -some- of
> these issues.

Thanks, sounds good, I will try smart later, at the moment, I made a
simple patch to make the populate_sdk pull in multilib toolchains works,
there are still other issues, I will fix them and send patches.

diff --git a/meta/classes/populate_sdk_base.bbclass 
b/meta/classes/populate_sdk_base.bbclass
index a9e9bd7..c2491ea 100644
--- a/meta/classes/populate_sdk_base.bbclass
+++ b/meta/classes/populate_sdk_base.bbclass
@@ -83,8 +83,40 @@ POPULATE_SDK_POST_HOST_COMMAND_append = " 
write_host_sdk_manifest; "
  fakeroot python do_populate_sdk() {
      from oe.sdk import populate_sdk
      from oe.manifest import create_manifest, Manifest
+    import oe.packagedata

      pn = d.getVar('PN', True)
+    # Help rpm to pull in multilib requires since it may not pull in
+    # them correctly, for example,
+    # lib32-packagegroup-core-standalone-sdk-target requires
+    # lib32-libc6, rpm may pull in libc6 rather than lib32-libc6, but
+    # directly use
+    # TOOLCHAIN_TARGET_TASK_append = " lib32-libc6" works well.
+    for pkg in (d.getVar('TOOLCHAIN_TARGET_TASK', True) or "").split():
+        sub_data = oe.packagedata.read_subpkgdata(pkg, d)
+        sub_rdep = sub_data.get("RDEPENDS_" + pkg)
+        if not sub_rdep:
+            continue
+        done = sub_rdep.split()
+        next = done
+        # Find all the rdepends on the dependency chain
+        while next:
+            new = []
+            for sub_pkg in next:
+                sub_data = oe.packagedata.read_subpkgdata(sub_pkg, d)
+                sub_pkg_rdep = sub_data.get("RDEPENDS_" + sub_pkg)
+                if not sub_pkg_rdep:
+                    continue
+                for p in sub_pkg_rdep.split():
+                    if p in done:
+                        continue
+                    if not p.startswith('(') and \
+                            oe.packagedata.has_subpkgdata(p, d):
+                        # It's a new rdep
+                        done.append(p)
+                        new.append(p)
+            next = new
+        d.appendVar('TOOLCHAIN_TARGET_TASK', ' ' + " ".join(done))

// Robert

>
> The other side of the issue then is for things that do have explicit word size
> requirements (like gcc and some of the build tools) we need to fix the packages
> to have additions RDEPENDS and RPROVIDES that dictate the word size they need
> and the word size they provide.  libpam already does this, and it resolved the
> issues there.


>
> --Mark
>
>> // Robert
>>
>> On 08/05/2015 09:10 PM, Mark Hatle wrote:
>>> On 8/5/15 1:45 AM, Robert Yang wrote:
>>>>
>>>> Hello,
>>>>
>>>> For the multilib package like lib32-bash, its rpm package name
>>>> is bash-4.3.30-r0.lib32_x86.rpm, but for ipk and deb, its name is
>>>> lib32-bash.ipk and lib32-bash.deb, there is a side effect for
>>>> the naming of rpm, for example, if packagegroup-xx rdepends on bash,
>>>> the lib32-packagegroup-xx should rdepend on lib32-bash, but
>>>> the its rpm pkg knows nothing about lib32-bash, so it still rdepends
>>>> on bash, this causes 32bit toolchain libs not installed when export
>>>> multilib sdk, for example:
>>>>
>>>> Configured multilib:
>>>> MACHINE ?= "qemux86-64"
>>>> require conf/multilib.conf
>>>> MULTILIBS = "multilib:lib32"
>>>> DEFAULTTUNE_virtclass-multilib-lib32 = "x86"
>>>>
>>>> $ bitbake core-image-minimal -cpopulate_sdk
>>>>
>>>> When PACKAGE_CLASSES = "package_rpm",  the lib32-libgcc,
>>>> lib32-libgcc-dev or other lib32-xx packages are not installed,
>>>> so that the 32bit toolchain doesn't work in SDK, this is because
>>>> lib32-packagegroup-core-standalone-sdk-target can't pull in the
>>>> lib32-xx correctly.
>>>>
>>>> When PACKAGE_CLASSES = "package_ipk", it works well.
>>>>
>>>> Does anyone why we don't use the name like lib32-bash.rpm, please ?
>>>> Can we use lib32-bash, please ?
>>>
>>> Because for something like bash, there is absolutely no reason to choose a
>>> 64-bit or 32-bit version over the other.  They have equivalent functionality for
>>> the end user, and only one version can be installed at a time.
>>>
>>> This is the RPM behavior that is being copied from Red Hat and other places, and
>>> what our customers are expecting.
>>>
>>> RPM supports multiple packages, with different arch fields, being installed at
>>> the same time.  ipkg and deb have a limitation that the package name MUST be
>>> unique.  So this is the first reason why we have different style names.
>>>
>>> The idea here being that you can build the .i586 packages once, and re-use them
>>> on an i586 system and an x86-64 system.  (Note, this doesn't work in practice in
>>> the Yocto Project due to sstate issues... but that is the underlying design from
>>> a package management perspective.)
>>>
>>> Dependency processing is the second reason, see below for an explanation of the
>>> difference and how it affects us.
>>>
>>> There are different types of dependencies within the system.
>>>
>>> * Package level dependency
>>> This is of the format 'bash' or 'lib32-bash'.  The system requires a specific
>>> package.  All three package formats support this.
>>>
>>> * Virtual dependency
>>> This is similar, in that the format can look like a package, or be something
>>> like 'virtual/foo'.  Again all three package formats support this.
>>>
>>> * Library SONAME
>>> This is a correlation of the library SONAME into the dependency set, such as
>>> 'libz.so(64)'.  This is pulled directly from any ELF binaries in the system
>>> during processing and added as a direct dependency to other shared libraries.
>>> It is significantly more exact then the package or virtual dependencies above.
>>> HOWEVER, it only works in the RPM case.
>>>
>>> This is the second reason why deb and ipkg need a unique 'lib32' style name for
>>> things.  They don't have a mechanism for a package to say "I need the 64-bit
>>> version of libz.so" to run properly, while that is built into RPM itself.
>>>
>>> * Filename dependency
>>> These are dependencies that start with '/'.  This is unique to RPM as well, and
>>> allows RPM to express dependencies on '/bin/bash', vs deb/ipkg having
>>> dependencies on 'bash'.  The difference is that it puts a hard marker in place
>>> that if 'bash' doesn't provide '/bin/bash', it's not enough.
>>>
>>> This doesn't directly affect the naming, but occasionally has some indirect
>>> adjustments on things.
>>>
>>>
>>> So when building a system using RPM as the package type, the rootfs construction
>>> may be slightly different because the package manager rules are different.
>>>
>>> RPM (smart) will look at the overall dependency tree and attempt to pick the
>>> shortest path so that all of the dependencies are correct.  You can 'weight' the
>>> dependency resolver to favor one style of package over the other.  This is a
>>> place in smart where we likely need to change the behavior to make it a bit more
>>> friendly for the Yocto Project.
>>>
>>> The dependency generator is already configured to weigh a package of the same
>>> type (arch) as a bit higher then others.  The problem is that a 'noarch' (all)
>>> package is of a different type then say i586.  So when an all package says "give
>>> me bash".  There is no 'bash.all', so it looks at the list and finds 'bash' of
>>> the highest priority, IF it had not previously been selected by something else.
>>>    (How would it have previously been selected?  If the image had selected
>>> 'lib32-bash' explicitly, or the more likely case, a lib32-* package that
>>> requires bash had been selected by the image, and there were not explicit
>>> requirements for 'bash' [main lib version]).
>>>
>>>
>>> RPM itself has a setting that defines which ABI type becomes the default if you
>>> attempt to choose multiple packages that have the same executables listed.  The
>>> setting here is:
>>>
>>> # Set RPM_PREFER_ELF_ARCH to configure preferred ABI when using rpm packaging
>>> # backend to generate a rootfs, choices are:
>>> # 1: ELF32 wins
>>> # 2: ELF64 wins
>>> # 4: ELF64 N32 wins (for mips64 or mips64el only)
>>>
>>> Note, this doesn't prevent multiple packages of the same name, different arch
>>> from being installed -- what this does is selects which one wins.  When not set,
>>> I believe it's last-in wins.
>>>
>>>
>>> This setting should also help weigh the smart dep generator, but I'm not sure it
>>> does.  (Pretty easy to test though.. set the value, build a multilib rootfs, and
>>> select something (all package) that requires 'bash' and see which version is
>>> selected.)
>>>
>>> But for most things in the system this really doesn't matter at all.  For things
>>> where a 32-bit or 64-bit does matter (executable and/or libraries that are not
>>> implicitly linked elsewhere), we really do need to add dependencies into the
>>> system so that the correct set of items are selected.
>>>
>>> PAM, Perl, Python, and the on-target toolchains are things that adding a few
>>> runtime dependencies are needed.  Basically anything in the system that may have
>>> runtime loadable libraries of a specific ABI, using something like dlopen.
>>> (Remember SONAME dependencies are automatic as mentioned above.)
>>>
>>> These new runtime dependencies will not harm deb/ipkg resolution (and may help
>>> it), but they give RPM the additional information it needs to select the
>>> explicit set that is required.  For example PAM:
>>>
>>> RPROVIDES_${PN} += "${PN}-${libpam_suffix}"
>>> RPROVIDES_${PN}-runtime += "${PN}-runtime-${libpam_suffix}"
>>>
>>> RDEPENDS_${PN}-runtime = "${PN}-${libpam_suffix} \
>>>       ${MLPREFIX}pam-plugin-deny-${libpam_suffix} \
>>>       ${MLPREFIX}pam-plugin-permit-${libpam_suffix} \
>>>       ${MLPREFIX}pam-plugin-warn-${libpam_suffix} \
>>>       ${MLPREFIX}pam-plugin-unix-${libpam_suffix} \
>>>       "
>>>
>>> (other code later constructs the RPROVIDES for the plugins as well)
>>>
>>> The MLPREFIX will be filtered off when RPM constructs it's packages, as it
>>> doesn't want/need them.  However the suffix won't be filtered.
>>>
>>> libpam_suffix = "suffix${@get_multilib_bit(d)}"
>>>
>>> This way the -runtime package ensures the right other packages (with their
>>> specific PROVIDES sufffixes) are included in all cases.
>>>
>>> This is a unique case, but should be pretty easy to help fix.
>>>
>>> For the case where you want to specify 'lib32-packagegroup' and expect to ONLY
>>> get 32-bit packages... While we can't ensure it with the RPM mechanism, we can
>>> weigh things to more likely happens -if- the lib32-packagegroup is of the same
>>> .arch as the packages it's installing, instead of being 'all' packages.
>>>
>>> Hopefully this explains a bit more how all of these items fit together, and the
>>> choice made when RPM was implemented to follow more of an expected multilib
>>> workflow.  (This won't affect you if you don't use multilibs in your design!)
>>>
>>> --Mark
>>>
>>>
>
>
>