[OE-core] [RFC][PATCHv2] insane.bbclass: Add do_qa_pseudo function to check for common errors listed in pseudo.log

[Martin Jansa]

Yes, package-output.lock is very often the cause for "^inode mismatch":

martin at jama ~/pseudo-qa $ cat log.world.qemu* | grep '^inode mismatch' | wc
-l
1485
martin at jama ~/pseudo-qa $ cat log.world.qemu* | grep '^inode mismatch' |
grep package-output.lock | wc -l
1168

log.do_install also causes "^inode mismatch" in many cases:

martin at jama ~/pseudo-qa $ cat log.world.qemu* | grep '^inode mismatch' |
grep -v package-output.lock | wc -l
317
martin at jama ~/pseudo-qa $ cat log.world.qemu* | grep '^inode mismatch' |
grep -v package-output.lock | grep temp/log.do_install | wc -l
309

That leaves only couple other files:
martin at jama ~/pseudo-qa $ cat log.world.qemu* | grep '^inode mismatch' |
grep -v package-output.lock | grep -v temp/log.do_install
inode mismatch:
'/home/jenkins/oe/world/shr-core/tmp-glibc/work/armv5te-oe-linux-gnueabi/qtbase/5.9.2+gitAUTOINC+73573fce29-r0/image/usr/include/qt5/QtNetwork/qt_temp.rhjRGj'
ino 691255 in db, 691256 in request.
inode mismatch: '/home/jenkins/.python-history' ino 1066494 in db, 1066493
in request.
inode mismatch: '/home/jenkins/.python-history' ino 1066494 in db, 1066493
in request.
inode mismatch: '/home/jenkins/.python-history' ino 1066729 in db, 1062665
in request.
inode mismatch: '/home/jenkins/.python-history' ino 1066729 in db, 1062665
in request.
inode mismatch:
'/home/jenkins/oe/world/shr-core/tmp-glibc/work/armv5te-oe-linux-gnueabi/qt3d/5.9.2+gitAUTOINC+9d8c9ada16-r0/image/usr/include/qt5/Qt3DInput/qt_temp.rhjRGj'
ino 520536 in db, 520539 in request.
inode mismatch:
'/home/jenkins/oe/world/shr-core/tmp-glibc/work/armv5te-oe-linux-gnueabi/libqofono/0.87+gitrAUTOINC+54435de3be-r0/image/usr/lib/libqofono-qt5/tests/qt_temp.rhjRGj'
ino 876399 in db, 876404 in request.
inode mismatch:
'/home/jenkins/oe/world/shr-core/tmp-glibc/work/armv5te-oe-linux-gnueabi/libqofono/0.87+gitrAUTOINC+54435de3be-r0/image/usr/lib/libqofono-qt5/tests/qt_temp.rhjRGj'
ino 876404 in db, 876408 in request.

Similarly with "creat for", but there is more wide selection of files
triggering this one:
martin at jama ~/pseudo-qa $ cat log.world.qemu* | grep -v "grep -e" | grep -v
'inode mismatch' | grep 'creat for.* replaces'  | wc -l
1771
martin at jama ~/pseudo-qa $ cat log.world.qemu* | grep -v "grep -e" | grep -v
'inode mismatch' | grep 'creat for'  | grep package-output.lock | wc -l
198

for "creat for" the log.do_install appears only twice, but various
temp/run.* files are quite common cause:
martin at jama ~/pseudo-qa $ cat log.world.qemu* | grep -v "grep -e" | grep -v
'inode mismatch' | grep 'creat for.* replaces'  | grep temp/run | wc -l
294

pm-qa is triggering this on many files,
e.g. /home/jenkins/oe/world/shr-core/tmp-glibc/work/core2-64-oe-linux/pm-qa/0.5.2-r0/image/usr/bin/cpuhotplug_07.sh
martin at jama ~/pseudo-qa $ cat log.world.qemu* | grep -v "grep -e" | grep -v
'inode mismatch' | grep 'creat for.* replaces'  | grep temp/run -v  | grep
pm-qa | wc -l
229

martin at jama ~/pseudo-qa $ cat log.world.qemu* | grep -v "grep -e" | grep -v
'inode mismatch' | grep 'creat for.* replaces'  | grep temp/run -v  | grep
pm-qa | grep cpuhotplug_05.sh
creat for
'/home/jenkins/oe/world/shr-core/tmp-glibc/work/armv5te-oe-linux-gnueabi/pm-qa/0.5.2-r0/image/usr/bin/cpuhotplug_05.sh'
replaces existing 902737 ['NAMELESS FILE'].
creat for
'/home/jenkins/oe/world/shr-core/tmp-glibc/work/armv5te-oe-linux-gnueabi/pm-qa/0.5.2-r0/image/usr/bin/cpuhotplug_05.sh'
replaces existing 1264757 ['NAMELESS FILE'].
creat for
'/home/jenkins/oe/world/shr-core/tmp-glibc/work/i586-oe-linux/pm-qa/0.5.2-r0/image/usr/bin/cpuhotplug_05.sh'
replaces existing 976636 ['NAMELESS FILE'].
creat for
'/home/jenkins/oe/world/shr-core/tmp-glibc/work/core2-64-oe-linux/pm-qa/0.5.2-r0/image/usr/bin/cpuhotplug_05.sh'
replaces existing 576004 ['NAMELESS FILE'].
creat for
'/home/jenkins/oe/world/shr-core/tmp-glibc/work/core2-64-oe-linux/pm-qa/0.5.2-r0/image/usr/bin/cpuhotplug_05.sh'
replaces existing 986226 ['NAMELESS FILE'].

then stamps files
martin at jama ~/pseudo-qa $ cat log.world.qemu* | grep -v "grep -e" | grep -v
'inode mismatch' | grep 'creat for.* replaces'  | grep temp/run -v  | grep
-v pm-qa | grep -v package-output.lock | grep glibc/stamps | wc -l
130

so it looks pretty rancom, but most of these errors are replacing 'NAMELESS
FILE'
martin at jama ~/pseudo-qa $ cat log.world.qemu* | grep -v "grep -e" | grep -v
'inode mismatch' | grep 'creat for.* replaces' | grep 'NAMELESS FILE' | wc
-l
1632

which leaves only 139 other files:
martin at jama ~/pseudo-qa $ cat log.world.qemu* | grep -v "grep -e" | grep -v
'inode mismatch' | grep 'creat for.* replaces' | grep 'replaces.*NAMELESS
FILE' -v | wc -l
139

from which couple of them are package-output.lock again:
martin at jama ~/pseudo-qa $ cat log.world.qemu* | grep -v "grep -e" | grep -v
'inode mismatch' | grep 'creat for.* replaces' | grep 'replaces.*NAMELESS
FILE' -v | grep replaces.*package-output.lock | wc -l
31

and then random selection of builds, most of them from glibc-locale:
martin at jama ~/pseudo-qa $ cat log.world.qemu* | grep -v "grep -e" | grep -v
'inode mismatch' | grep 'creat for.* replaces' | grep 'replaces.*NAMELESS
FILE' -v | grep replaces.*package-output.lock -v | grep glibc-locale | wc -l
34
martin at jama ~/pseudo-qa $ cat log.world.qemu* | grep -v "grep -e" | grep -v
'inode mismatch' | grep 'creat for.* replaces' | grep 'replaces.*NAMELESS
FILE' -v | grep replaces.*package-output.lock -v | grep llvm | wc -l
3
martin at jama ~/pseudo-qa $ cat log.world.qemu* | grep -v "grep -e" | grep -v
'inode mismatch' | grep 'creat for.* replaces' | grep 'replaces.*NAMELESS
FILE' -v | grep replaces.*package-output.lock -v | grep linux-yocto | wc -l
7

"^creat ignored" is again mostly 195/202 package-output.lock:

martin at jama ~/pseudo-qa $ cat log.world.qemu* | grep -v "grep -e" | grep -v
'inode mismatch' | grep -v 'creat for.* replaces' | grep -v indicates |
grep -v '^Setup complete' | grep -v '^db cleanup' | grep -v '^pid ' | grep
-v '^ \[' | grep -v '^memory-to-file' | grep -v '^debug_logfile: ' | grep
'^creat ignored ' | wc -l
202
martin at jama ~/pseudo-qa $ cat log.world.qemu* | grep -v "grep -e" | grep -v
'inode mismatch' | grep -v 'creat for.* replaces' | grep -v indicates |
grep -v '^Setup complete' | grep -v '^db cleanup' | grep -v '^pid ' | grep
-v '^ \[' | grep -v '^memory-to-file' | grep -v '^debug_logfile: ' | grep
'^creat ignored ' | grep package-output.lock | wc -l
195
martin at jama ~/pseudo-qa $ cat log.world.qemu* | grep -v "grep -e" | grep -v
'inode mismatch' | grep -v 'creat for.* replaces' | grep -v indicates |
grep -v '^Setup complete' | grep -v '^db cleanup' | grep -v '^pid ' | grep
-v '^ \[' | grep -v '^memory-to-file' | grep -v '^debug_logfile: ' | grep
'^creat ignored ' | grep package-output.lock -v
creat ignored for existing file
'/home/jenkins/oe/world/shr-core/tmp-glibc/work/armv5te-oe-linux-gnueabi/qtbase/5.9.2+gitAUTOINC+73573fce29-r0/image/usr/include/qt5/QtNetwork/qt_temp.rhjRGj'.
creat ignored for existing file
'/home/jenkins/oe/world/shr-core/tmp-glibc/work/armv5te-oe-linux-gnueabi/qt3d/5.9.2+gitAUTOINC+9d8c9ada16-r0/image/usr/include/qt5/Qt3DInput/qt_temp.rhjRGj'.
creat ignored for existing file
'/home/jenkins/oe/world/shr-core/tmp-glibc/work/armv5te-oe-linux-gnueabi/libqofono/0.87+gitrAUTOINC+54435de3be-r0/image/usr/lib/libqofono-qt5/tests/qt_temp.rhjRGj'.
creat ignored for existing file
'/home/jenkins/oe/world/shr-core/tmp-glibc/work/armv5te-oe-linux-gnueabi/libqofono/0.87+gitrAUTOINC+54435de3be-r0/image/usr/lib/libqofono-qt5/tests/qt_temp.rhjRGj'.
creat ignored for existing file
'/home/jenkins/oe/world/shr-core/tmp-glibc/sstate-control/index-armv5te.lock'.
creat ignored for existing file
'/home/jenkins/oe/world/shr-core/tmp-glibc/sstate-control/index-armv5te.lock'.
creat ignored for existing file
'/home/jenkins/oe/world/shr-core/tmp-glibc/work/i586-oe-linux/usermode/1.111-r0/build/po/.intltool-merge-cache'.

^dir err: also happens a lot in the log files where other issues appear as
well (because the QA check isn't triggered by this one):
martin at jama ~/pseudo-qa $ cat log.world.qemu* | grep -v "grep -e" | grep -v
'inode mismatch' | grep -v 'creat for.* replaces' | grep -v indicates |
grep -v '^Setup complete' | grep -v '^db cleanup' | grep -v '^pid ' | grep
-v '^ \[' | grep -v '^memory-to-file' | grep -v '^debug_logfile: ' | grep
'^creat ignored ' -v | grep '^dir err ' | wc -l
744

couple ^symlink err:
martin at jama ~/pseudo-qa $ cat log.world.qemu* | grep -v "grep -e" | grep -v
'inode mismatch' | grep -v 'creat for.* replaces' | grep -v indicates |
grep -v '^Setup complete' | grep -v '^db cleanup' | grep -v '^pid ' | grep
-v '^ \[' | grep -v '^memory-to-file' | grep -v '^debug_logfile: ' | grep
'^creat ignored ' -v | grep '^dir err ' -v | grep '^symlink err' | wc -l
223

martin at jama ~/pseudo-qa $ cat log.world.qemu* | grep -v "grep -e" | grep -v
'inode mismatch' | grep -v 'creat for.* replaces' | grep -v indicates |
grep -v '^Setup complete' | grep -v '^db cleanup' | grep -v '^pid ' | grep
-v '^ \[' | grep -v '^memory-to-file' | grep -v '^debug_logfile: ' | grep
'^creat ignored ' -v | grep '^dir err ' -v | grep '^symlink err' -v | grep
'^symlink mismatch: ' | wc -l
309

I fail to see useful patterns in this, maybe fixing the package-output.lock
might help to filter the noise a bit.

On Mon, Oct 16, 2017 at 5:10 PM, Burton, Ross <ross.burton at intel.com> wrote:

> On 16 October 2017 at 16:01, Martin Jansa <martin.jansa at gmail.com> wrote:
>
>> The world builds with this check applied show that *a lot* of components
>> report one of these errors and often many of errors, the most complete logs
>> from last couple days:
>>
>
> A lot of the errors in my logs were due to package-output.lock being
> accessed inside/outside pseudo context.  Can you grep all of your logs and
> collate some common patterns?
>
> Ross
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openembedded.org/pipermail/openembedded-core/attachments/20171016/2c2734fb/attachment-0002.html>