<div dir="ltr"><div class="gmail_extra"><div class="gmail_quote">On 10 November 2016 at 19:53, Patrick Ohly <span dir="ltr"><<a href="mailto:patrick.ohly@intel.com" target="_blank">patrick.ohly@intel.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><span class="gmail-">On Thu, 2016-10-13 at 13:16 -0700, Jianxun Zhang wrote:<br>
> It turns out this change between 3.12.2 and 3.13 introduces<br>
> a 2% increase of build time based on statistic data in<br>
> bz10367.<br>
<br>
</span>Let me add that this patch increased build performance in Ostro even<br>
more: apparently one big impact of the sqlite performance issue is on<br>
pseudo. Ostro depends fairly heavily on pseudo because of meta-swupd and<br>
xattrs on all files.<br>
<br>
When this patch and others recently landed in Ostro, total build times<br>
dropped from 4:46h (build #508,<br>
<a href="https://ostroproject.org/jenkins/job/build_intel-corei7-64/2763/console" rel="noreferrer" target="_blank">https://ostroproject.org/<wbr>jenkins/job/build_intel-<wbr>corei7-64/2763/console</a>)<br>
to 2:07h (build #510,<br>
<a href="https://ostroproject.org/jenkins/job/build_intel-corei7-64/2831/console" rel="noreferrer" target="_blank">https://ostroproject.org/<wbr>jenkins/job/build_intel-<wbr>corei7-64/2831/console</a>).<br>
<br>
That could also be because of other improvements, perhaps even in our CI<br>
hardware, so take those numbers with a large chunk of salt.<br>
<br>
However, in local builds with and without this patch (i.e. everything<br>
else the same) I also see big differences for pseudo-heavy operations:<br>
<br>
$ buildstats-diff --diff-attr walltime --min-val 60 with-patch/ without-patch/<br>
Ignoring tasks less than 01:00.0 (60.0s)<br>
Ignoring differences less than 00:02.0 (2.0s)<br>
<br>
PKG TASK ABSDIFF RELDIFF WALLTIME1 -> WALLTIME2<br>
...<br>
bundle-ostro-image-swupd-qa-<wbr>bundle-b do_rootfs 78.1s +115.4% 67.7s -> 145.8s<br>
bundle-ostro-image-swupd-qa-<wbr>bundle-a do_rootfs 80.3s +116.8% 68.8s -> 149.1s<br>
bundle-ostro-image-swupd-qa-<wbr>bundle-a do_image 106.8s +291.9% 36.6s -> 143.3s<br>
bundle-ostro-image-swupd-qa-<wbr>bundle-b do_image 107.9s +298.2% 36.2s -> 144.1s<br>
bundle-ostro-image-swupd-mega do_image 244.4s +74.2% 329.2s -> 573.6s<br>
bundle-ostro-image-swupd-<wbr>world-dev do_rootfs 246.7s +207.2% 119.1s -> 365.8s<br>
bundle-ostro-image-swupd-<wbr>world-dev do_image 269.2s +83.5% 322.6s -> 591.7s<br>
bundle-ostro-image-swupd-mega do_rootfs 272.6s +246.1% 110.8s -> 383.3s<br>
ostro-image-swupd do_rootfs 676.1s +808.1% 83.7s -> 759.8s<br>
bundle-ostro-image-swupd-<wbr>world-dev do_copy_bundle_contents 1339.5s +2957.6% 45.3s -> 1384.8s<br>
bundle-ostro-image-swupd-qa-<wbr>bundle-b do_copy_bundle_contents 1475.0s +3147.8% 46.9s -> 1521.9s<br>
bundle-ostro-image-swupd-qa-<wbr>bundle-a do_copy_bundle_contents 1503.9s +3283.0% 45.8s -> 1549.8s<br>
<br>
Cumulative walltime:<br>
6070.9s +326.9% 30:57.3 (1857.3s) -> 2:12:08.2 (7928.2s)<br></blockquote><div> </div><div>yikes.</div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
So it really seems that the sqlite change is a very relevant<br>
improvement.<br>
<br>
That leads me to a bigger question: has upstream been notified about<br>
this?<br>
<br>
Our observation may also be relevant to other sqlite users. Besides, not<br>
getting this fixed upstream means that we'll have to do the same tricky<br>
revert for the next upstream version update.<span class="gmail-HOEnZb"><font color="#888888"><br></font></span></blockquote><div><br></div><div>Completely true. It may also be worthwhile to profile what pseudo is really doing. The feeling I had was that there could be something going wrong there as recipes that install lots of files not only take very long but also create _very_ large database file: The increases do not seem even close to linear as one might expect. Seebs gave some advice about this in the bug (<a href="https://bugzilla.yoctoproject.org/show_bug.cgi?id=10367#c11">https://bugzilla.yoctoproject.org/show_bug.cgi?id=10367#c11</a>). </div><div><br></div><div>Jussi<br></div><div><br></div><div><br></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><span class="gmail-HOEnZb"><font color="#888888">
--<br>
Best Regards, Patrick Ohly<br>
<br>
The content of this message is my personal opinion only and although<br>
I am an employee of Intel, the statements I make here in no way<br>
represent Intel's position on the issue, nor am I authorized to speak<br>
on behalf of Intel on this matter.<br>
</font></span><div class="gmail-HOEnZb"><div class="gmail-h5"><br>
<br>
<br>
--<br>
______________________________<wbr>_________________<br>
Openembedded-core mailing list<br>
<a href="mailto:Openembedded-core@lists.openembedded.org">Openembedded-core@lists.<wbr>openembedded.org</a><br>
<a href="http://lists.openembedded.org/mailman/listinfo/openembedded-core" rel="noreferrer" target="_blank">http://lists.openembedded.org/<wbr>mailman/listinfo/openembedded-<wbr>core</a><br>
</div></div></blockquote></div><br></div></div>