<div dir="ltr">I've created<div><a href="https://bugzilla.yoctoproject.org/show_bug.cgi?id=12434">https://bugzilla.yoctoproject.org/show_bug.cgi?id=12434</a><br></div><div>to track this issue.</div><div><br></div><div>I get this QA error now 100% reproducible when the "right" do_package sstate archive is being used, but still haven't found the root cause (aka why/how this bad sstate archive was created in first place).</div></div><div class="gmail_extra"><br><div class="gmail_quote">On Mon, Oct 16, 2017 at 5:50 PM, Martin Jansa <span dir="ltr"><<a href="mailto:martin.jansa@gmail.com" target="_blank">martin.jansa@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div>Yes, package-output.lock is very often the cause for "^inode mismatch":</div><div><br></div><div>martin@jama ~/pseudo-qa $ cat log.world.qemu* | grep '^inode mismatch' | wc -l</div><div>1485</div><div>martin@jama ~/pseudo-qa $ cat log.world.qemu* | grep '^inode mismatch' | grep package-output.lock | wc -l</div><div>1168</div><div><br></div><div>log.do_install also causes "^inode mismatch" in many cases:</div><div><br></div><div><div>martin@jama ~/pseudo-qa $ cat log.world.qemu* | grep '^inode mismatch' | grep -v package-output.lock | wc -l</div><div>317</div><div>martin@jama ~/pseudo-qa $ cat log.world.qemu* | grep '^inode mismatch' | grep -v package-output.lock | grep temp/log.do_install | wc -l</div><div>309</div></div><div><br></div><div>That leaves only couple other files:</div><div><div>martin@jama ~/pseudo-qa $ cat log.world.qemu* | grep '^inode mismatch' | grep -v package-output.lock | grep -v temp/log.do_install</div><div>inode mismatch: '/home/jenkins/oe/world/shr-<wbr>core/tmp-glibc/work/armv5te-<wbr>oe-linux-gnueabi/qtbase/5.9.2+<wbr>gitAUTOINC+73573fce29-r0/<wbr>image/usr/include/qt5/<wbr>QtNetwork/qt_temp.rhjRGj' ino 691255 in db, 691256 in request.</div><div>inode mismatch: '/home/jenkins/.python-<wbr>history' ino 1066494 in db, 1066493 in request.</div><div>inode mismatch: '/home/jenkins/.python-<wbr>history' ino 1066494 in db, 1066493 in request.</div><div>inode mismatch: '/home/jenkins/.python-<wbr>history' ino 1066729 in db, 1062665 in request.</div><div>inode mismatch: '/home/jenkins/.python-<wbr>history' ino 1066729 in db, 1062665 in request.</div><div>inode mismatch: '/home/jenkins/oe/world/shr-<wbr>core/tmp-glibc/work/armv5te-<wbr>oe-linux-gnueabi/qt3d/5.9.2+<wbr>gitAUTOINC+9d8c9ada16-r0/<wbr>image/usr/include/qt5/<wbr>Qt3DInput/qt_temp.rhjRGj' ino 520536 in db, 520539 in request.</div><div>inode mismatch: '/home/jenkins/oe/world/shr-<wbr>core/tmp-glibc/work/armv5te-<wbr>oe-linux-gnueabi/libqofono/0.<wbr>87+gitrAUTOINC+54435de3be-r0/<wbr>image/usr/lib/libqofono-qt5/<wbr>tests/qt_temp.rhjRGj' ino 876399 in db, 876404 in request.</div><div>inode mismatch: '/home/jenkins/oe/world/shr-<wbr>core/tmp-glibc/work/armv5te-<wbr>oe-linux-gnueabi/libqofono/0.<wbr>87+gitrAUTOINC+54435de3be-r0/<wbr>image/usr/lib/libqofono-qt5/<wbr>tests/qt_temp.rhjRGj' ino 876404 in db, 876408 in request.</div></div><div><br></div><div>Similarly with "creat for", but there is more wide selection of files triggering this one:</div><div><div>martin@jama ~/pseudo-qa $ cat log.world.qemu* | grep -v "grep -e" | grep -v 'inode mismatch' | grep 'creat for.* replaces' | wc -l</div><div>1771</div></div><div>martin@jama ~/pseudo-qa $ cat log.world.qemu* | grep -v "grep -e" | grep -v 'inode mismatch' | grep 'creat for' | grep package-output.lock | wc -l<br></div><div><div>198</div></div><div><br></div><div>for "creat for" the log.do_install appears only twice, but various temp/run.* files are quite common cause:</div><div><div>martin@jama ~/pseudo-qa $ cat log.world.qemu* | grep -v "grep -e" | grep -v 'inode mismatch' | grep 'creat for.* replaces' | grep temp/run | wc -l</div><div>294</div></div><div><br></div><div>pm-qa is triggering this on many files, e.g. /home/jenkins/oe/world/<wbr>shr-core/tmp-glibc/work/core2-<wbr>64-oe-linux/pm-qa/0.5.2-r0/<wbr>image/usr/bin/cpuhotplug_07.sh</div><div><div>martin@jama ~/pseudo-qa $ cat log.world.qemu* | grep -v "grep -e" | grep -v 'inode mismatch' | grep 'creat for.* replaces' | grep temp/run -v | grep pm-qa | wc -l</div><div>229</div></div><div><br></div><div><div>martin@jama ~/pseudo-qa $ cat log.world.qemu* | grep -v "grep -e" | grep -v 'inode mismatch' | grep 'creat for.* replaces' | grep temp/run -v | grep pm-qa | grep cpuhotplug_05.sh</div><div>creat for '/home/jenkins/oe/world/shr-<wbr>core/tmp-glibc/work/armv5te-<wbr>oe-linux-gnueabi/pm-qa/0.5.2-<wbr>r0/image/usr/bin/cpuhotplug_<wbr>05.sh' replaces existing 902737 ['NAMELESS FILE'].</div><div>creat for '/home/jenkins/oe/world/shr-<wbr>core/tmp-glibc/work/armv5te-<wbr>oe-linux-gnueabi/pm-qa/0.5.2-<wbr>r0/image/usr/bin/cpuhotplug_<wbr>05.sh' replaces existing 1264757 ['NAMELESS FILE'].</div><div>creat for '/home/jenkins/oe/world/shr-<wbr>core/tmp-glibc/work/i586-oe-<wbr>linux/pm-qa/0.5.2-r0/image/<wbr>usr/bin/cpuhotplug_05.sh' replaces existing 976636 ['NAMELESS FILE'].</div><div>creat for '/home/jenkins/oe/world/shr-<wbr>core/tmp-glibc/work/core2-64-<wbr>oe-linux/pm-qa/0.5.2-r0/image/<wbr>usr/bin/cpuhotplug_05.sh' replaces existing 576004 ['NAMELESS FILE'].</div><div>creat for '/home/jenkins/oe/world/shr-<wbr>core/tmp-glibc/work/core2-64-<wbr>oe-linux/pm-qa/0.5.2-r0/image/<wbr>usr/bin/cpuhotplug_05.sh' replaces existing 986226 ['NAMELESS FILE'].</div></div><div><br></div><div>then stamps files</div><div><div>martin@jama ~/pseudo-qa $ cat log.world.qemu* | grep -v "grep -e" | grep -v 'inode mismatch' | grep 'creat for.* replaces' | grep temp/run -v | grep -v pm-qa | grep -v package-output.lock | grep glibc/stamps | wc -l</div><div>130</div></div><div><br></div><div>so it looks pretty rancom, but most of these errors are replacing 'NAMELESS FILE'</div><div><div>martin@jama ~/pseudo-qa $ cat log.world.qemu* | grep -v "grep -e" | grep -v 'inode mismatch' | grep 'creat for.* replaces' | grep 'NAMELESS FILE' | wc -l</div><div>1632</div></div><div><br></div><div>which leaves only 139 other files:</div><div><div>martin@jama ~/pseudo-qa $ cat log.world.qemu* | grep -v "grep -e" | grep -v 'inode mismatch' | grep 'creat for.* replaces' | grep 'replaces.*NAMELESS FILE' -v | wc -l</div><div>139</div></div><div><br></div><div>from which couple of them are package-output.lock again:</div><div><div>martin@jama ~/pseudo-qa $ cat log.world.qemu* | grep -v "grep -e" | grep -v 'inode mismatch' | grep 'creat for.* replaces' | grep 'replaces.*NAMELESS FILE' -v | grep replaces.*package-output.lock | wc -l</div><div>31</div></div><div><br></div><div>and then random selection of builds, most of them from glibc-locale:</div><div><div>martin@jama ~/pseudo-qa $ cat log.world.qemu* | grep -v "grep -e" | grep -v 'inode mismatch' | grep 'creat for.* replaces' | grep 'replaces.*NAMELESS FILE' -v | grep replaces.*package-output.lock -v | grep glibc-locale | wc -l<br></div><div>34</div><div>martin@jama ~/pseudo-qa $ cat log.world.qemu* | grep -v "grep -e" | grep -v 'inode mismatch' | grep 'creat for.* replaces' | grep 'replaces.*NAMELESS FILE' -v | grep replaces.*package-output.lock -v | grep llvm | wc -l</div><div>3</div></div><div><div>martin@jama ~/pseudo-qa $ cat log.world.qemu* | grep -v "grep -e" | grep -v 'inode mismatch' | grep 'creat for.* replaces' | grep 'replaces.*NAMELESS FILE' -v | grep replaces.*package-output.lock -v | grep linux-yocto | wc -l</div><div>7</div></div><div><br></div><div>"^creat ignored" is again mostly 195/202 package-output.lock:</div><div><br></div><div><div>martin@jama ~/pseudo-qa $ cat log.world.qemu* | grep -v "grep -e" | grep -v 'inode mismatch' | grep -v 'creat for.* replaces' | grep -v indicates | grep -v '^Setup complete' | grep -v '^db cleanup' | grep -v '^pid ' | grep -v '^ \[' | grep -v '^memory-to-file' | grep -v '^debug_logfile: ' | grep '^creat ignored ' | wc -l</div><div>202</div><div>martin@jama ~/pseudo-qa $ cat log.world.qemu* | grep -v "grep -e" | grep -v 'inode mismatch' | grep -v 'creat for.* replaces' | grep -v indicates | grep -v '^Setup complete' | grep -v '^db cleanup' | grep -v '^pid ' | grep -v '^ \[' | grep -v '^memory-to-file' | grep -v '^debug_logfile: ' | grep '^creat ignored ' | grep package-output.lock | wc -l</div><div>195</div><div>martin@jama ~/pseudo-qa $ cat log.world.qemu* | grep -v "grep -e" | grep -v 'inode mismatch' | grep -v 'creat for.* replaces' | grep -v indicates | grep -v '^Setup complete' | grep -v '^db cleanup' | grep -v '^pid ' | grep -v '^ \[' | grep -v '^memory-to-file' | grep -v '^debug_logfile: ' | grep '^creat ignored ' | grep package-output.lock -v </div><div>creat ignored for existing file '/home/jenkins/oe/world/shr-<wbr>core/tmp-glibc/work/armv5te-<wbr>oe-linux-gnueabi/qtbase/5.9.2+<wbr>gitAUTOINC+73573fce29-r0/<wbr>image/usr/include/qt5/<wbr>QtNetwork/qt_temp.rhjRGj'.</div><div>creat ignored for existing file '/home/jenkins/oe/world/shr-<wbr>core/tmp-glibc/work/armv5te-<wbr>oe-linux-gnueabi/qt3d/5.9.2+<wbr>gitAUTOINC+9d8c9ada16-r0/<wbr>image/usr/include/qt5/<wbr>Qt3DInput/qt_temp.rhjRGj'.</div><div>creat ignored for existing file '/home/jenkins/oe/world/shr-<wbr>core/tmp-glibc/work/armv5te-<wbr>oe-linux-gnueabi/libqofono/0.<wbr>87+gitrAUTOINC+54435de3be-r0/<wbr>image/usr/lib/libqofono-qt5/<wbr>tests/qt_temp.rhjRGj'.</div><div>creat ignored for existing file '/home/jenkins/oe/world/shr-<wbr>core/tmp-glibc/work/armv5te-<wbr>oe-linux-gnueabi/libqofono/0.<wbr>87+gitrAUTOINC+54435de3be-r0/<wbr>image/usr/lib/libqofono-qt5/<wbr>tests/qt_temp.rhjRGj'.</div><div>creat ignored for existing file '/home/jenkins/oe/world/shr-<wbr>core/tmp-glibc/sstate-control/<wbr>index-armv5te.lock'.</div><div>creat ignored for existing file '/home/jenkins/oe/world/shr-<wbr>core/tmp-glibc/sstate-control/<wbr>index-armv5te.lock'.</div><div>creat ignored for existing file '/home/jenkins/oe/world/shr-<wbr>core/tmp-glibc/work/i586-oe-<wbr>linux/usermode/1.111-r0/build/<wbr>po/.intltool-merge-cache'.</div></div><div><br></div><div>^dir err: also happens a lot in the log files where other issues appear as well (because the QA check isn't triggered by this one):</div><div><div>martin@jama ~/pseudo-qa $ cat log.world.qemu* | grep -v "grep -e" | grep -v 'inode mismatch' | grep -v 'creat for.* replaces' | grep -v indicates | grep -v '^Setup complete' | grep -v '^db cleanup' | grep -v '^pid ' | grep -v '^ \[' | grep -v '^memory-to-file' | grep -v '^debug_logfile: ' | grep '^creat ignored ' -v | grep '^dir err ' | wc -l</div><div>744</div></div><div><br></div><div>couple ^symlink err:</div><div><div>martin@jama ~/pseudo-qa $ cat log.world.qemu* | grep -v "grep -e" | grep -v 'inode mismatch' | grep -v 'creat for.* replaces' | grep -v indicates | grep -v '^Setup complete' | grep -v '^db cleanup' | grep -v '^pid ' | grep -v '^ \[' | grep -v '^memory-to-file' | grep -v '^debug_logfile: ' | grep '^creat ignored ' -v | grep '^dir err ' -v | grep '^symlink err' | wc -l</div><div>223</div></div><div><br></div><div><div>martin@jama ~/pseudo-qa $ cat log.world.qemu* | grep -v "grep -e" | grep -v 'inode mismatch' | grep -v 'creat for.* replaces' | grep -v indicates | grep -v '^Setup complete' | grep -v '^db cleanup' | grep -v '^pid ' | grep -v '^ \[' | grep -v '^memory-to-file' | grep -v '^debug_logfile: ' | grep '^creat ignored ' -v | grep '^dir err ' -v | grep '^symlink err' -v | grep '^symlink mismatch: ' | wc -l</div><div>309</div></div><div><br></div><div>I fail to see useful patterns in this, maybe fixing the package-output.lock might help to filter the noise a bit.</div><div><div class="h5"><div><br></div><div class="gmail_extra"><div class="gmail_quote">On Mon, Oct 16, 2017 at 5:10 PM, Burton, Ross <span dir="ltr"><<a href="mailto:ross.burton@intel.com" target="_blank">ross.burton@intel.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><span class="m_4890801203757336322gmail-">On 16 October 2017 at 16:01, Martin Jansa <span dir="ltr"><<a href="mailto:martin.jansa@gmail.com" target="_blank">martin.jansa@gmail.com</a>></span> wrote:<br></span><div class="gmail_extra"><div class="gmail_quote"><span class="m_4890801203757336322gmail-"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">The world builds with this check applied show that *a lot* of components report one of these errors and often many of errors, the most complete logs from last couple days:</div></blockquote><div><br></div></span><div>A lot of the errors in my logs were due to package-output.lock being accessed inside/outside pseudo context. Can you grep all of your logs and collate some common patterns?</div><span class="m_4890801203757336322gmail-HOEnZb"><font color="#888888"><div><br></div><div>Ross</div></font></span></div></div></div>
</blockquote></div><br></div></div></div></div>
</blockquote></div><br></div>