[OE-core] [PATCH RFC] Support hierarchical ipk feeds

[Richard Purdie]

On Thu, 2014-05-08 at 16:55 +0000, Paul Barker wrote:
> This is an RFC only and is not suitable for merging yet. The necessary patches
> for opkg-utils have also been posted for review but have not been merged yet. I
> want to know if there is general approval for adding this feature to oe-core. If
> there is, I'll go ahead and merge the opkg-utils patches and post a new patch
> series which includes the update to the SRCREV used for opkg-utils in oe-core.
> 
> This patch has been tested on qemuarm:
> 
>   A feed was created with IPK_HIERARCHICAL_FEED unset and opkg was pointed at
>   that feed. 'opkg update && opkg install ncurses-terminfo-base' worked
>   correctly, downloading a package file from the same directory as the package
>   feed.
>   
>   IPK_HIERARCHICAL_FEED was then set to "1" in local.conf and the feed was
>   rebuilt by running 'bitbake core-image-minimal && bitbake package-index'. On
>   qemu 'opkg update && opkg install ncurses-terminfo' worked correctly,
>   downloading the package from the subdirectory 'n/ncurses-terminfo'.
> 
>   Finally, IPK_HIERARCHICAL_FEED was unset again and the feed was rebuilt. On
>   qemu 'opkg update && opkg install ncurses' worked correctly, downloading
>   package files from the same directory as the package feed again.
> 
> Those tests prove that changes to IPK_HIERARCHICAL_FEED do not break the upgrade
> path of an existing system provided that 'bitbake pacakge-index' is re-ran after
> the change.
> 
> The proposed commit message follows...
> 
> 
> 
> package_ipk.bbclass: Support hierarchical feed
> 
> This patch allows for an optional new layout for ipk feed directories which I've
> called a 'hierarchical feed' and is based on how Debian pools package files. It
> is disabled by default and is enabled by setting IPK_HIERARCHICAL_FEED to "1".
> 
> In the traditional feed layout, package files are placed in <outdir>/<arch>/.
> This can lead to several thousand files existing in a single directory which is
> often a problem if developers want to upload a package feed to a shared web
> hosting provider. For example, in my case, listing files via FTP only shows the
> first 2000 files, breaking my scripts which attempt to upload only new and
> changed files via FTP.
> 
> In the hierarchical feed, package files are written to
> '<outdir>/<arch>/<pkg_prefix>/<pkg_subdir>', where pkg_prefix is the first
> letter of the package file name for non-lib packages or "lib" plus the 4th
> letter of the pacakge file name for lib pacakges (eg, 'l' for less, 'libc' for
> libc6).  pkg_subdir is the root of the pacakge file name, discarding the version
> and architecture parts and the common suffixes '-dbg', '-dev', '-doc',
> '-staticdev' and '-locale-*'.
> 
> This change relies on recent patches to opkg-utils which support hierarchical
> package feeds.

FWIW I'm ok with this and its something we probably want to enable by
default however:

a) I'm not sure <pkg_subdir> is that useful, we tend to only have one
version of things around in most cases and I can't imagine people having
a lot of these.

b) How are the "common suffixes" configured? I guess its just a loose
thing but I worry about the "why did my package get put into directory X
when its really part of Y?" type questions. Perhaps splitting on the
first dash in the name might be good enough? I'm thinking about
kernel-modules-xxx for example and some of our other dynamically
generated packages.

Cheers,

Richard