[OE-core] opkg-make-index cache failing
Martin Jansa
martin.jansa at gmail.com
Tue Nov 29 11:40:52 UTC 2011
On Fri, Nov 25, 2011 at 08:59:06PM +0100, Koen Kooi wrote:
>
> Op 25 nov. 2011, om 19:15 heeft Koen Kooi het volgende geschreven:
>
> >
> > Op 25 nov. 2011, om 17:53 heeft Richard Purdie het volgende geschreven:
> >
> >> On Fri, 2011-11-25 at 16:38 +0100, Koen Kooi wrote:
> >>> In OE-classic the opkg-make-index cache was working pretty well,
> >>> do_rootfs only spent a few seconds doing opkg-make-index on
> >>> incremental builds. In the OE-core world the situation is different,
> >>> opkg-make-index will reindex every package on each do_rootfs run, so
> >>> after a while building images starts taking *really* long, on my
> >>> systemd (work/ on ssd, deploy on rotating media) it now takes
> >>> ±7minutes just do refresh the opkg indices.
> >>>
> >>> Since I can't blame sstate for this I suspect that pseudo gets in the
> >>> way of the naive caching logic. Is there a simple way to turn off
> >>> pseudo when running opkg-make-index?
> >
> > With a relatively small deploy:
> >
> > real 6m14.611s
> > user 2m45.909s
> > sys 0m43.674s
> >
> >> You could try:
> >>
> >> PSEUDO_DISABLED=1 opkg-make-index
> >
> > real 5m25.920s
> > user 2m41.279s
> > sys 0m41.681s
> >
> >> or even
> >>
> >> PSEUDO_UNLOAD=1 opkg-make-index
> >
> > real 5m5.699s
> > user 2m37.283s
> > sys 0m37.508s
> >
> > Now I need to double check to see if the patches are really having effect on the opkg caching instead of just not having pseudo overhead.
>
> Retesting without PSEUDO changes:
>
> real 5m18.412s
> user 2m40.736s
> sys 0m41.374s
>
> So the PSEUDO changes don't seem to do much :(
I can confirm this change here too:
OE om-gta02 at shr ~/shr-core $ time /usr/bin/python2.7 \
/OE/shr-core/tmp/sysroots/x86_64-linux/usr/bin/opkg-make-index \
-r /OE/shr-core/tmp/deploy/ipk/armv4t/Packages \
-p /OE/shr-core/tmp/deploy/ipk/armv4t/Packages \
-l /OE/shr-core/tmp/deploy/ipk/armv4t/Packages.filelist \
-m /OE/shr-core/tmp/deploy/ipk/armv4t/
real 11m47.949s
user 4m13.848s
sys 0m17.917s
It hurts even more when someone is building ie for spitz and
linux-kexecboot depends on initramfs image which calls another
opkg-make-index with every image build.
If I replace opkg-make-index from oe-core with old oe-classic version, I
get a lot of messages like:
"Lost field license, MIT-X"
but the time is similar
real 10m23.953s
user 4m10.940s
sys 0m15.833s
And if I do the same completely in shr-unstable (based on OE-classic)
OE om-gta02 at shr ~/shr-unstable $ time /usr/bin/python2.7 \
/OE/shr-unstable/tmp/sysroots/x86_64-linux/usr/bin/opkg-make-index \
-r /OE/shr-unstable/tmp/deploy/ipk/armv4t/Packages \
-p /OE/shr-unstable/tmp/deploy/ipk/armv4t/Packages \
-l /OE/shr-unstable/tmp/deploy/ipk/armv4t/Packages.filelist \
-m /OE/shr-unstable/tmp/deploy/ipk/armv4t/
real 0m6.469s
user 0m6.056s
sys 0m0.400s
Another difference seems to be in Packages.filelist which were empty in
OE-classic:
OE om-gta02 at shr ~/shr-core $ find tmp/deploy/ipk/ -name Packages.filelist -exec wc -l {} \;
1173 tmp/deploy/ipk/palmpre2/Packages.filelist
1254 tmp/deploy/ipk/palmpre/Packages.filelist
107346 tmp/deploy/ipk/armv4t/Packages.filelist
110557 tmp/deploy/ipk/armv7a-vfp-neon/Packages.filelist
0 tmp/deploy/ipk/Packages.filelist
1365 tmp/deploy/ipk/nokia900/Packages.filelist
812 tmp/deploy/ipk/all/Packages.filelist
1643 tmp/deploy/ipk/om_gta02/Packages.filelist
OE om-gta02 at shr ~/shr-core $ cd ../shr-unstable/
OE om-gta02 at shr ~/shr-unstable $ find tmp/deploy/ipk/ -name Packages.filelist -exec wc -l {} \;
0 tmp/deploy/ipk/armv7a/Packages.filelist
0 tmp/deploy/ipk/om-gta01/Packages.filelist
0 tmp/deploy/ipk/om-gta02/Packages.filelist
0 tmp/deploy/ipk/palmpre2/Packages.filelist
0 tmp/deploy/ipk/armv6/Packages.filelist
0 tmp/deploy/ipk/palmpre/Packages.filelist
0 tmp/deploy/ipk/armv4t/Packages.filelist
0 tmp/deploy/ipk/armv4/Packages.filelist
0 tmp/deploy/ipk/armv6-novfp/Packages.filelist
0 tmp/deploy/ipk/Packages.filelist
0 tmp/deploy/ipk/htcdream/Packages.filelist
0 tmp/deploy/ipk/nokia900/Packages.filelist
0 tmp/deploy/ipk/all/Packages.filelist
0 tmp/deploy/ipk/armv7/Packages.filelist
But that's not the cause, because even without Packages.filelist
generation it takes about 3 min (better but it still writes info for
every package).
Weird is that there is only small diff between new and old opkg.py/opkg-make-index
--- /OE/shr-unstable/tmp/sysroots/x86_64-linux/usr/bin/opkg.py 2011-05-27 18:35:03.000000000 +0200
+++ /OE/shr-core/tmp/sysroots/x86_64-linux/usr/bin/opkg.py 2011-09-01 16:36:35.000000000 +0200
@@ -145,6 +145,7 @@
self.priority = None
self.tags = None
self.fn = fn
+ self.license = None
if fn:
# see if it is deb format
@@ -319,6 +320,12 @@
def get_section(self, section):
return self.section
+ def set_license(self, license):
+ self.license = license
+
+ def get_license(self, license):
+ return self.license
+
def get_file_list(self):
if not self.fn:
return []
@@ -425,6 +432,7 @@
if self.description: out = out + "Description: %s\n" % (self.description)
if self.oe: out = out + "OE: %s\n" % (self.oe)
if self.homepage: out = out + "HomePage: %s\n" % (self.homepage)
+ if self.license: out = out + "License: %s\n" % (self.license)
if self.priority: out = out + "Priority: %s\n" % (self.priority)
if self.tags: out = out + "Tags: %s\n" % (self.tags)
out = out + "\n"
(reverse-i-search)`uNr': diff -^Cr ~/common/conf/shr-core/ conf/
--- /OE/shr-unstable/tmp/sysroots/x86_64-linux/usr/bin/opkg-make-index 2011-05-27 18:35:03.000000000 +0200
+++ /OE/shr-core/tmp/sysroots/x86_64-linux/usr/bin/opkg-make-index 2011-11-29 12:32:39.000000000 +0100
@@ -197,7 +197,15 @@
names = packages.packages.keys()
names.sort()
for name in names:
- for fn in packages[name].get_file_list():
+ try:
+ fnlist = packages[name].get_file_list()
+ except OSError, e:
+ sys.stderr.write("Package %s disappeared on us!\n" % (name))
+ continue
+ except IOError, e:
+ sys.stderr.write("Package %s disappeared on us!\n" % (name))
+ continue
+ for fn in fnlist:
(h,t) = os.path.split(fn)
if not t: continue
if not files.has_key(t): files[t] = name+':'+fn
Regards,
--
Martin 'JaMa' Jansa jabber: Martin.Jansa at gmail.com
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 198 bytes
Desc: Digital signature
URL: <http://lists.openembedded.org/pipermail/openembedded-core/attachments/20111129/260f7dc7/attachment-0002.sig>
More information about the Openembedded-core
mailing list