[Openembedded-architecture] Changes that switching from smart to dnf will cause

[Mark Hatle]

(I've already commented on some pieces through existing threads, but I wanted to
include a few things below as well.)

On 2/14/17 6:04 AM, Alexander Kanavin wrote:
> Hello,
> 
> I've posted the patchset for replacing the smart package manager with 
> dnf package manager to oe-core list, and wanted to write a separate 
> email here, laying out what are the changes this will cause and why 
> they're necessary or good to have. Please refer to the patchset if you 
> want to see implementation details.
> 
> 1. Smart package manager is replaced by Dnf package manager. Smart has 
> become unmaintained upstream, is not ported to Python 3.x and we need to 
> find a replacement. Dnf is the only feasible candidate.
> 
> The change in functionality is that the on-target package management 
> from remote package feeds is now done with a different tool that has a 
> different set of command line options. If you have scripts that call the 
> tool directly, or use its api (not sure if smart has an api), they need 
> to be fixed.
> 
> 2. Rpm 5.x is replaced with rpm 4.x. This is done for two major reasons:
> 
> - DNF is API incompatible with Rpm 5.x and porting it and maintaining 
> the port is non-trivial.
> 
> - rpm 5.x itself is more or less unmaintained both in the upstream and 
> in Yocto. There is no community around it, and Yocto is the sole 
> remaining user. I think this is a stronger argument for moving away from 
> it, than any technically superior features rpm 5.x may have over 4.x 
> (not sure what those would be).

Just to be clear.  I thinking moving to DNF is a good thing.  We want to get to
a more standardized (well known) interface then we previously have.

The move from RPM 5 to 4 (or whatever) is semantics to me.  The actual version
of RPM really doesn't matter.  Many of the issues that existed in the past with
rpm4 are no longer problems simply because the world has changed, not that
they've necessarily fixed the issues.

With that in mind, I'm not objecting to the changes -- but I do expect problems.

MIPS64 n32, maybe IA32 x32, maybe mixed mode ARM systems (not sure).

NSS as the default crypto library.  (beecrypt is certainly smaller..  recently
OpenSSL support has been added -- but there is some question if all of the
licenses are compatible.)

> 3. Berkeley db 6.x is removed; Berkeley db 5.x becomes default
> 
> Version 6.x of Berkeley DB has been rejected by open source community
> due to its hostile AGPLv3 license; both Fedora and Debian are sticking 
> with db 5.x - and by extension,all the open source projects are still 
> developed and tested with db 5.x
> 
> In oe-core the only thing that was requiring db 6.x was rpm 5.x, and so
> there's no reason to continue carrying db 6.x in oe-core. If someone 
> needs API features that are only available in db 6.x, it can be re-added 
> to meta-oe.
> 
> 4. Createrepo is replaced with createrepo_c
> 
> createrepo_c is the current incarnation of the tool that generates 
> remote repository metadata; it's written in C (createrepo was in 
> Python), is faster, and is maintained.
> 
> 5. architecture-independent .rpm packages are "noarch" instead of "all"
> 
> Too many places in dnf/rpm4 stack make that assumption; let's not fight 
> against it. It's only about the filenames and the architecture tag in 
> them; nothing else has changed in oe-core system, particularly 
> allarch.bbclass is untouched.

I was expecting more screaming from this one.  If nobody else has objections
with this anymore great!  When we did the original work, it was a big deal that
people wanted the OE / Debian style 'all' instead of noarch.

We do also need to be clear on the rootfs construction on the compatible
architecture configurations and make sure they are properly consulted throughout
the process by rpm, dnf, and any other pieces involved.  (RPM5 ignored the arch
field and used the contents of the package itself to determine compatibility.)

> 6. Things that need to run on target during package installation should 
> go into pkg_postinst_ontarget()
> 
> Previously, the way to achieve this was:
> 
> pkg_postinst_PACKAGENAME() {
>       if [ x"$D" = "x" ]; then
>            # Actions to carry out on the device go here
>       else
>            exit 1
>       fi
> }
> 
> The new way is simply:
> 
> pkg_postinst_ontarget_PACKAGENAME() {
>       # Actions to carry out on the device go here
> }
> 
> The old way still works, but is deprecated and will produce a warning. I 
> understand this change is orthogonal to dnf, but the current design is 
> flawed and now is a chance to fix it. The reasons are:
> 
> 1) Code is hard to read; it is not obvious that 'if D is defined then 
> exit 1' means 'defer what follows to first boot during package 
> cross-install'.
> 
> 2) Worse, this hides actual errors in the scriptlets; there is no 
> difference between scriptlet failing because it's intended to be run on 
> target and scriptlet failing because there's a bug or a regression 
> somewhere. I've already found broken scriptlets where the brokenness was 
> hidden this way (in meta-selftest/recipes-test/postinst/postinst_1.0.bb)
> 
> Plain pkg_postinst() without special tricks works exactly as before.
> 
> 7. There are various rpm-related features that have not been ported over 
> yet.
> 
> The reason is that they are all disabled by default and don't have a 
> test case - I don't want to figure out what they're supposed to do in 
> the absence of a clear way to verify them - there's already more than 
> enough changes to test and review. Specifically:
> 
> - configuring remote package feeds with PACKAGE_FEED_URIS (bug to create 
> a test case has been filed)
> - signing of remote package feeds using PACKAGE_FEED_SIGN (also has a 
> bug to create a test case)
> - signing of rpm packages using RPM_SIGN_PACKAGES (this one I will fix 
> in the next few days, as I actually did find a testcase)

Signing of packages is a really big deal for RPM4.  RPM5 autosigned packages.
This was important for validation (not authenticity).  RPM typically validates
the signature -first-, then starts loading the data.  If it's not signed, the
data is loaded and it's much easier for a corrupt package to cause security issues.

Authenticity always requires an actual signing process of course.

> - RPM_PREFER_ELF_ARCH (have no idea about what this does at all).

When a multilib install is done, you -will- end up with multiple packages trying
to install content to the same place.  RPM has a conflict resolution algorithm.
Effectively it is:

Do the two files have the same hash?
Yes - only install one
No - continue

Are the two files ELF binaries?
No - fail with a conflict
Yes - continue

Is multilib resolution enabled?
No - fail with a conflict
Yes - continue

Resolve the conflict:
Using the ELF type of the file we decide which of the (up to three) options are
installed.

If there is no preferred type, the last version in wins.  (This becomes
non-deterministic.)

If there IS a preferred type, then the preferred type in wins.  Typically you
define '64-bit' as preferred on IA32.  While you often define 32-bit as the
preferred type of MIPS and Power.

This is potentially a tri-arch switch BTW.  As there is the 'n32' and 'x32'
variants on MIPS and IA32.

This type of conflict resolution needs to work, it is likely implemented and
it's simply a switch that is flipped in the RPM macro file used for the rootfs
install and installed onto the target (if rpm is installed there.)  It may also
affect the DNF internal resolution as well.

> ... and possibly other features - there's a lot of obscure corners in 
> oe-core which I might have missed.
> 
> Regards,
> Alex
> _______________________________________________
> Openembedded-architecture mailing list
> Openembedded-architecture at lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-architecture
>