[OE-core] opkg-make-index cache failing
Martin Jansa
martin.jansa at gmail.com
Thu Mar 29 07:58:12 UTC 2012
On Tue, Nov 29, 2011 at 12:40:52PM +0100, Martin Jansa wrote:
> On Fri, Nov 25, 2011 at 08:59:06PM +0100, Koen Kooi wrote:
> >
> > Op 25 nov. 2011, om 19:15 heeft Koen Kooi het volgende geschreven:
> >
> > >
> > > Op 25 nov. 2011, om 17:53 heeft Richard Purdie het volgende geschreven:
> > >
> > >> On Fri, 2011-11-25 at 16:38 +0100, Koen Kooi wrote:
> > >>> In OE-classic the opkg-make-index cache was working pretty well,
> > >>> do_rootfs only spent a few seconds doing opkg-make-index on
> > >>> incremental builds. In the OE-core world the situation is different,
> > >>> opkg-make-index will reindex every package on each do_rootfs run, so
> > >>> after a while building images starts taking *really* long, on my
> > >>> systemd (work/ on ssd, deploy on rotating media) it now takes
> > >>> ±7minutes just do refresh the opkg indices.
> > >>>
> > >>> Since I can't blame sstate for this I suspect that pseudo gets in the
> > >>> way of the naive caching logic. Is there a simple way to turn off
> > >>> pseudo when running opkg-make-index?
> > >
> > > With a relatively small deploy:
> > >
> > > real 6m14.611s
> > > user 2m45.909s
> > > sys 0m43.674s
> > >
> > >> You could try:
> > >>
> > >> PSEUDO_DISABLED=1 opkg-make-index
> > >
> > > real 5m25.920s
> > > user 2m41.279s
> > > sys 0m41.681s
> > >
> > >> or even
> > >>
> > >> PSEUDO_UNLOAD=1 opkg-make-index
> > >
> > > real 5m5.699s
> > > user 2m37.283s
> > > sys 0m37.508s
> > >
> > > Now I need to double check to see if the patches are really having effect on the opkg caching instead of just not having pseudo overhead.
> >
> > Retesting without PSEUDO changes:
> >
> > real 5m18.412s
> > user 2m40.736s
> > sys 0m41.374s
> >
> > So the PSEUDO changes don't seem to do much :(
>
> I can confirm this change here too:
> OE om-gta02 at shr ~/shr-core $ time /usr/bin/python2.7 \
> /OE/shr-core/tmp/sysroots/x86_64-linux/usr/bin/opkg-make-index \
> -r /OE/shr-core/tmp/deploy/ipk/armv4t/Packages \
> -p /OE/shr-core/tmp/deploy/ipk/armv4t/Packages \
> -l /OE/shr-core/tmp/deploy/ipk/armv4t/Packages.filelist \
> -m /OE/shr-core/tmp/deploy/ipk/armv4t/
>
>
> real 11m47.949s
> user 4m13.848s
> sys 0m17.917s
>
> It hurts even more when someone is building ie for spitz and
> linux-kexecboot depends on initramfs image which calls another
> opkg-make-index with every image build.
>
> If I replace opkg-make-index from oe-core with old oe-classic version, I
> get a lot of messages like:
> "Lost field license, MIT-X"
>
> but the time is similar
> real 10m23.953s
> user 4m10.940s
> sys 0m15.833s
>
> And if I do the same completely in shr-unstable (based on OE-classic)
> OE om-gta02 at shr ~/shr-unstable $ time /usr/bin/python2.7 \
> /OE/shr-unstable/tmp/sysroots/x86_64-linux/usr/bin/opkg-make-index \
> -r /OE/shr-unstable/tmp/deploy/ipk/armv4t/Packages \
> -p /OE/shr-unstable/tmp/deploy/ipk/armv4t/Packages \
> -l /OE/shr-unstable/tmp/deploy/ipk/armv4t/Packages.filelist \
> -m /OE/shr-unstable/tmp/deploy/ipk/armv4t/
>
>
> real 0m6.469s
> user 0m6.056s
> sys 0m0.400s
>
> Another difference seems to be in Packages.filelist which were empty in
> OE-classic:
>
> OE om-gta02 at shr ~/shr-core $ find tmp/deploy/ipk/ -name Packages.filelist -exec wc -l {} \;
> 1173 tmp/deploy/ipk/palmpre2/Packages.filelist
> 1254 tmp/deploy/ipk/palmpre/Packages.filelist
> 107346 tmp/deploy/ipk/armv4t/Packages.filelist
> 110557 tmp/deploy/ipk/armv7a-vfp-neon/Packages.filelist
> 0 tmp/deploy/ipk/Packages.filelist
> 1365 tmp/deploy/ipk/nokia900/Packages.filelist
> 812 tmp/deploy/ipk/all/Packages.filelist
> 1643 tmp/deploy/ipk/om_gta02/Packages.filelist
> OE om-gta02 at shr ~/shr-core $ cd ../shr-unstable/
> OE om-gta02 at shr ~/shr-unstable $ find tmp/deploy/ipk/ -name Packages.filelist -exec wc -l {} \;
> 0 tmp/deploy/ipk/armv7a/Packages.filelist
> 0 tmp/deploy/ipk/om-gta01/Packages.filelist
> 0 tmp/deploy/ipk/om-gta02/Packages.filelist
> 0 tmp/deploy/ipk/palmpre2/Packages.filelist
> 0 tmp/deploy/ipk/armv6/Packages.filelist
> 0 tmp/deploy/ipk/palmpre/Packages.filelist
> 0 tmp/deploy/ipk/armv4t/Packages.filelist
> 0 tmp/deploy/ipk/armv4/Packages.filelist
> 0 tmp/deploy/ipk/armv6-novfp/Packages.filelist
> 0 tmp/deploy/ipk/Packages.filelist
> 0 tmp/deploy/ipk/htcdream/Packages.filelist
> 0 tmp/deploy/ipk/nokia900/Packages.filelist
> 0 tmp/deploy/ipk/all/Packages.filelist
> 0 tmp/deploy/ipk/armv7/Packages.filelist
>
> But that's not the cause, because even without Packages.filelist
> generation it takes about 3 min (better but it still writes info for
> every package).
>
> Weird is that there is only small diff between new and old opkg.py/opkg-make-index
> --- /OE/shr-unstable/tmp/sysroots/x86_64-linux/usr/bin/opkg.py 2011-05-27 18:35:03.000000000 +0200
> +++ /OE/shr-core/tmp/sysroots/x86_64-linux/usr/bin/opkg.py 2011-09-01 16:36:35.000000000 +0200
> @@ -145,6 +145,7 @@
> self.priority = None
> self.tags = None
> self.fn = fn
> + self.license = None
>
> if fn:
> # see if it is deb format
> @@ -319,6 +320,12 @@
> def get_section(self, section):
> return self.section
>
> + def set_license(self, license):
> + self.license = license
> +
> + def get_license(self, license):
> + return self.license
> +
> def get_file_list(self):
> if not self.fn:
> return []
> @@ -425,6 +432,7 @@
> if self.description: out = out + "Description: %s\n" % (self.description)
> if self.oe: out = out + "OE: %s\n" % (self.oe)
> if self.homepage: out = out + "HomePage: %s\n" % (self.homepage)
> + if self.license: out = out + "License: %s\n" % (self.license)
> if self.priority: out = out + "Priority: %s\n" % (self.priority)
> if self.tags: out = out + "Tags: %s\n" % (self.tags)
> out = out + "\n"
> (reverse-i-search)`uNr': diff -^Cr ~/common/conf/shr-core/ conf/
>
> --- /OE/shr-unstable/tmp/sysroots/x86_64-linux/usr/bin/opkg-make-index 2011-05-27 18:35:03.000000000 +0200
> +++ /OE/shr-core/tmp/sysroots/x86_64-linux/usr/bin/opkg-make-index 2011-11-29 12:32:39.000000000 +0100
> @@ -197,7 +197,15 @@
> names = packages.packages.keys()
> names.sort()
> for name in names:
> - for fn in packages[name].get_file_list():
> + try:
> + fnlist = packages[name].get_file_list()
> + except OSError, e:
> + sys.stderr.write("Package %s disappeared on us!\n" % (name))
> + continue
> + except IOError, e:
> + sys.stderr.write("Package %s disappeared on us!\n" % (name))
> + continue
> + for fn in fnlist:
> (h,t) = os.path.split(fn)
> if not t: continue
> if not files.has_key(t): files[t] = name+':'+fn
>
>
> Regards,
> --
> Martin 'JaMa' Jansa jabber: Martin.Jansa at gmail.com
Today I did some more tests and started using simple bash script to
reindex all Packages files just once (instead of rebuilding common arch
many times with bitbake package-index).
OPKG_MAKE_INDEX=~/shr-core/tmp-eglibc/sysroots/x86_64-linux/usr/bin/opkg-make-index
DEPLOY_IPK=~/shr-core/tmp-eglibc/deploy/ipk/
for PACKAGE_FILE in `find ${DEPLOY_IPK} -name Packages`; do
PACKAGE_DIR=`dirname ${PACKAGE_FILE}`
touch ${PACKAGE_FILE}
echo "Regenerating ${PACKAGE_FILE}"
time flock ${PACKAGE_FILE}.flock -c "${OPKG_MAKE_INDEX} -r ${PACKAGE_FILE} -p ${PACKAGE_FILE} -l ${PACKAGE_FILE}.filelist -m ${PACKAGE_DIR}/"
done
And on my machine it takes really long to finish
real 26m40.108s
user 23m6.673s
sys 0m14.464s
So I'll look into opkg-make-index changes again and to make it easier I would like
to move all oe-core local patches:
http://git.openembedded.org/openembedded-core/tree/meta/recipes-devtools/opkg-utils/opkg-utils
to yocto git repo:
http://git.yoctoproject.org/cgit/cgit.cgi/opkg-utils/
should I send pull-request for opkg-utils repo here or somewhere else? e.g. yocto ML?
Cheers,
--
Martin 'JaMa' Jansa jabber: Martin.Jansa at gmail.com
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 198 bytes
Desc: Digital signature
URL: <http://lists.openembedded.org/pipermail/openembedded-core/attachments/20120329/cd9b669b/attachment-0002.sig>
More information about the Openembedded-core
mailing list