[OE-core] [PATCH RFC] Support hierarchical ipk feeds

[Paul Barker]

This is an RFC only and is not suitable for merging yet. The necessary patches
for opkg-utils have also been posted for review but have not been merged yet. I
want to know if there is general approval for adding this feature to oe-core. If
there is, I'll go ahead and merge the opkg-utils patches and post a new patch
series which includes the update to the SRCREV used for opkg-utils in oe-core.

This patch has been tested on qemuarm:

  A feed was created with IPK_HIERARCHICAL_FEED unset and opkg was pointed at
  that feed. 'opkg update && opkg install ncurses-terminfo-base' worked
  correctly, downloading a package file from the same directory as the package
  feed.
  
  IPK_HIERARCHICAL_FEED was then set to "1" in local.conf and the feed was
  rebuilt by running 'bitbake core-image-minimal && bitbake package-index'. On
  qemu 'opkg update && opkg install ncurses-terminfo' worked correctly,
  downloading the package from the subdirectory 'n/ncurses-terminfo'.

  Finally, IPK_HIERARCHICAL_FEED was unset again and the feed was rebuilt. On
  qemu 'opkg update && opkg install ncurses' worked correctly, downloading
  package files from the same directory as the package feed again.

Those tests prove that changes to IPK_HIERARCHICAL_FEED do not break the upgrade
path of an existing system provided that 'bitbake pacakge-index' is re-ran after
the change.

The proposed commit message follows...



package_ipk.bbclass: Support hierarchical feed

This patch allows for an optional new layout for ipk feed directories which I've
called a 'hierarchical feed' and is based on how Debian pools package files. It
is disabled by default and is enabled by setting IPK_HIERARCHICAL_FEED to "1".

In the traditional feed layout, package files are placed in <outdir>/<arch>/.
This can lead to several thousand files existing in a single directory which is
often a problem if developers want to upload a package feed to a shared web
hosting provider. For example, in my case, listing files via FTP only shows the
first 2000 files, breaking my scripts which attempt to upload only new and
changed files via FTP.

In the hierarchical feed, package files are written to
'<outdir>/<arch>/<pkg_prefix>/<pkg_subdir>', where pkg_prefix is the first
letter of the package file name for non-lib packages or "lib" plus the 4th
letter of the pacakge file name for lib pacakges (eg, 'l' for less, 'libc' for
libc6).  pkg_subdir is the root of the pacakge file name, discarding the version
and architecture parts and the common suffixes '-dbg', '-dev', '-doc',
'-staticdev' and '-locale-*'.

This change relies on recent patches to opkg-utils which support hierarchical
package feeds.

Signed-off-by: Paul Barker <paul at paulbarker.me.uk>
---
 meta/classes/package_ipk.bbclass | 21 ++++++++++++++++++++-
 1 file changed, 20 insertions(+), 1 deletion(-)

diff --git a/meta/classes/package_ipk.bbclass b/meta/classes/package_ipk.bbclass
index 2949d1d..680b9bf 100644
--- a/meta/classes/package_ipk.bbclass
+++ b/meta/classes/package_ipk.bbclass
@@ -63,7 +63,26 @@ python do_package_ipk () {
         bb.data.update_data(localdata)
         basedir = os.path.join(os.path.dirname(root))
         arch = localdata.getVar('PACKAGE_ARCH', True)
-        pkgoutdir = "%s/%s" % (outdir, arch)
+        if localdata.getVar('IPK_HIERARCHICAL_FEED') == "1":
+            # Spread packages across subdirectories so each isn't too crowded
+            if pkgname.startswith('lib'):
+                pkg_prefix = 'lib' + pkgname[3]
+            else:
+                pkg_prefix = pkgname[0]
+
+            # Keep -dbg, -dev, -doc, -staticdev and -locale-* packages together
+            if pkgname[-4:] in ('-dbg', '-dev', '-doc'):
+                pkg_subdir = pkgname[:-4]
+            elif pkgname.endswith('-staticdev'):
+                pkg_subdir = pkgname[:-10]
+            elif '-locale-' in pkgname:
+                pkg_subdir = pkgname[:pkgname.find('-locale-')]
+            else:
+                pkg_subdir = pkgname
+
+            pkgoutdir = "%s/%s/%s/%s" % (outdir, arch, pkg_prefix, pkg_subdir)
+        else:
+            pkgoutdir = "%s/%s" % (outdir, arch)
         bb.utils.mkdirhier(pkgoutdir)
         os.chdir(root)
         cleanupcontrol(root)
-- 
1.9.2