[OE-core] [RFC][PATCHv2] insane.bbclass: Add do_qa_pseudo function to check for common errors listed in pseudo.log

[Martin Jansa]

I've created
https://bugzilla.yoctoproject.org/show_bug.cgi?id=12434
to track this issue.

I get this QA error now 100% reproducible when the "right" do_package
sstate archive is being used, but still haven't found the root cause (aka
why/how this bad sstate archive was created in first place).

On Mon, Oct 16, 2017 at 5:50 PM, Martin Jansa <martin.jansa at gmail.com>
wrote:

> Yes, package-output.lock is very often the cause for "^inode mismatch":
>
> martin at jama ~/pseudo-qa $ cat log.world.qemu* | grep '^inode mismatch' |
> wc -l
> 1485
> martin at jama ~/pseudo-qa $ cat log.world.qemu* | grep '^inode mismatch' |
> grep package-output.lock | wc -l
> 1168
>
> log.do_install also causes "^inode mismatch" in many cases:
>
> martin at jama ~/pseudo-qa $ cat log.world.qemu* | grep '^inode mismatch' |
> grep -v package-output.lock | wc -l
> 317
> martin at jama ~/pseudo-qa $ cat log.world.qemu* | grep '^inode mismatch' |
> grep -v package-output.lock | grep temp/log.do_install | wc -l
> 309
>
> That leaves only couple other files:
> martin at jama ~/pseudo-qa $ cat log.world.qemu* | grep '^inode mismatch' |
> grep -v package-output.lock | grep -v temp/log.do_install
> inode mismatch: '/home/jenkins/oe/world/shr-core/tmp-glibc/work/armv5te-
> oe-linux-gnueabi/qtbase/5.9.2+gitAUTOINC+73573fce29-r0/
> image/usr/include/qt5/QtNetwork/qt_temp.rhjRGj' ino 691255 in db, 691256
> in request.
> inode mismatch: '/home/jenkins/.python-history' ino 1066494 in db,
> 1066493 in request.
> inode mismatch: '/home/jenkins/.python-history' ino 1066494 in db,
> 1066493 in request.
> inode mismatch: '/home/jenkins/.python-history' ino 1066729 in db,
> 1062665 in request.
> inode mismatch: '/home/jenkins/.python-history' ino 1066729 in db,
> 1062665 in request.
> inode mismatch: '/home/jenkins/oe/world/shr-core/tmp-glibc/work/armv5te-
> oe-linux-gnueabi/qt3d/5.9.2+gitAUTOINC+9d8c9ada16-r0/
> image/usr/include/qt5/Qt3DInput/qt_temp.rhjRGj' ino 520536 in db, 520539
> in request.
> inode mismatch: '/home/jenkins/oe/world/shr-core/tmp-glibc/work/armv5te-
> oe-linux-gnueabi/libqofono/0.87+gitrAUTOINC+54435de3be-r0/
> image/usr/lib/libqofono-qt5/tests/qt_temp.rhjRGj' ino 876399 in db,
> 876404 in request.
> inode mismatch: '/home/jenkins/oe/world/shr-core/tmp-glibc/work/armv5te-
> oe-linux-gnueabi/libqofono/0.87+gitrAUTOINC+54435de3be-r0/
> image/usr/lib/libqofono-qt5/tests/qt_temp.rhjRGj' ino 876404 in db,
> 876408 in request.
>
> Similarly with "creat for", but there is more wide selection of files
> triggering this one:
> martin at jama ~/pseudo-qa $ cat log.world.qemu* | grep -v "grep -e" | grep
> -v 'inode mismatch' | grep 'creat for.* replaces'  | wc -l
> 1771
> martin at jama ~/pseudo-qa $ cat log.world.qemu* | grep -v "grep -e" | grep
> -v 'inode mismatch' | grep 'creat for'  | grep package-output.lock | wc -l
> 198
>
> for "creat for" the log.do_install appears only twice, but various
> temp/run.* files are quite common cause:
> martin at jama ~/pseudo-qa $ cat log.world.qemu* | grep -v "grep -e" | grep
> -v 'inode mismatch' | grep 'creat for.* replaces'  | grep temp/run | wc -l
> 294
>
> pm-qa is triggering this on many files, e.g. /home/jenkins/oe/world/
> shr-core/tmp-glibc/work/core2-64-oe-linux/pm-qa/0.5.2-r0/
> image/usr/bin/cpuhotplug_07.sh
> martin at jama ~/pseudo-qa $ cat log.world.qemu* | grep -v "grep -e" | grep
> -v 'inode mismatch' | grep 'creat for.* replaces'  | grep temp/run -v  |
> grep pm-qa | wc -l
> 229
>
> martin at jama ~/pseudo-qa $ cat log.world.qemu* | grep -v "grep -e" | grep
> -v 'inode mismatch' | grep 'creat for.* replaces'  | grep temp/run -v  |
> grep pm-qa | grep cpuhotplug_05.sh
> creat for '/home/jenkins/oe/world/shr-core/tmp-glibc/work/armv5te-
> oe-linux-gnueabi/pm-qa/0.5.2-r0/image/usr/bin/cpuhotplug_05.sh' replaces
> existing 902737 ['NAMELESS FILE'].
> creat for '/home/jenkins/oe/world/shr-core/tmp-glibc/work/armv5te-
> oe-linux-gnueabi/pm-qa/0.5.2-r0/image/usr/bin/cpuhotplug_05.sh' replaces
> existing 1264757 ['NAMELESS FILE'].
> creat for '/home/jenkins/oe/world/shr-core/tmp-glibc/work/i586-oe-
> linux/pm-qa/0.5.2-r0/image/usr/bin/cpuhotplug_05.sh' replaces existing
> 976636 ['NAMELESS FILE'].
> creat for '/home/jenkins/oe/world/shr-core/tmp-glibc/work/core2-64-
> oe-linux/pm-qa/0.5.2-r0/image/usr/bin/cpuhotplug_05.sh' replaces existing
> 576004 ['NAMELESS FILE'].
> creat for '/home/jenkins/oe/world/shr-core/tmp-glibc/work/core2-64-
> oe-linux/pm-qa/0.5.2-r0/image/usr/bin/cpuhotplug_05.sh' replaces existing
> 986226 ['NAMELESS FILE'].
>
> then stamps files
> martin at jama ~/pseudo-qa $ cat log.world.qemu* | grep -v "grep -e" | grep
> -v 'inode mismatch' | grep 'creat for.* replaces'  | grep temp/run -v  |
> grep -v pm-qa | grep -v package-output.lock | grep glibc/stamps | wc -l
> 130
>
> so it looks pretty rancom, but most of these errors are replacing
> 'NAMELESS FILE'
> martin at jama ~/pseudo-qa $ cat log.world.qemu* | grep -v "grep -e" | grep
> -v 'inode mismatch' | grep 'creat for.* replaces' | grep 'NAMELESS FILE' |
> wc -l
> 1632
>
> which leaves only 139 other files:
> martin at jama ~/pseudo-qa $ cat log.world.qemu* | grep -v "grep -e" | grep
> -v 'inode mismatch' | grep 'creat for.* replaces' | grep
> 'replaces.*NAMELESS FILE' -v | wc -l
> 139
>
> from which couple of them are package-output.lock again:
> martin at jama ~/pseudo-qa $ cat log.world.qemu* | grep -v "grep -e" | grep
> -v 'inode mismatch' | grep 'creat for.* replaces' | grep
> 'replaces.*NAMELESS FILE' -v | grep replaces.*package-output.lock | wc -l
> 31
>
> and then random selection of builds, most of them from glibc-locale:
> martin at jama ~/pseudo-qa $ cat log.world.qemu* | grep -v "grep -e" | grep
> -v 'inode mismatch' | grep 'creat for.* replaces' | grep
> 'replaces.*NAMELESS FILE' -v | grep replaces.*package-output.lock -v | grep
> glibc-locale | wc -l
> 34
> martin at jama ~/pseudo-qa $ cat log.world.qemu* | grep -v "grep -e" | grep
> -v 'inode mismatch' | grep 'creat for.* replaces' | grep
> 'replaces.*NAMELESS FILE' -v | grep replaces.*package-output.lock -v | grep
> llvm | wc -l
> 3
> martin at jama ~/pseudo-qa $ cat log.world.qemu* | grep -v "grep -e" | grep
> -v 'inode mismatch' | grep 'creat for.* replaces' | grep
> 'replaces.*NAMELESS FILE' -v | grep replaces.*package-output.lock -v | grep
> linux-yocto | wc -l
> 7
>
> "^creat ignored" is again mostly 195/202 package-output.lock:
>
> martin at jama ~/pseudo-qa $ cat log.world.qemu* | grep -v "grep -e" | grep
> -v 'inode mismatch' | grep -v 'creat for.* replaces' | grep -v indicates |
> grep -v '^Setup complete' | grep -v '^db cleanup' | grep -v '^pid ' | grep
> -v '^ \[' | grep -v '^memory-to-file' | grep -v '^debug_logfile: ' | grep
> '^creat ignored ' | wc -l
> 202
> martin at jama ~/pseudo-qa $ cat log.world.qemu* | grep -v "grep -e" | grep
> -v 'inode mismatch' | grep -v 'creat for.* replaces' | grep -v indicates |
> grep -v '^Setup complete' | grep -v '^db cleanup' | grep -v '^pid ' | grep
> -v '^ \[' | grep -v '^memory-to-file' | grep -v '^debug_logfile: ' | grep
> '^creat ignored ' | grep package-output.lock | wc -l
> 195
> martin at jama ~/pseudo-qa $ cat log.world.qemu* | grep -v "grep -e" | grep
> -v 'inode mismatch' | grep -v 'creat for.* replaces' | grep -v indicates |
> grep -v '^Setup complete' | grep -v '^db cleanup' | grep -v '^pid ' | grep
> -v '^ \[' | grep -v '^memory-to-file' | grep -v '^debug_logfile: ' | grep
> '^creat ignored ' | grep package-output.lock -v
> creat ignored for existing file '/home/jenkins/oe/world/shr-
> core/tmp-glibc/work/armv5te-oe-linux-gnueabi/qtbase/5.9.2+
> gitAUTOINC+73573fce29-r0/image/usr/include/qt5/QtNetwork/qt_temp.rhjRGj'.
> creat ignored for existing file '/home/jenkins/oe/world/shr-
> core/tmp-glibc/work/armv5te-oe-linux-gnueabi/qt3d/5.9.2+
> gitAUTOINC+9d8c9ada16-r0/image/usr/include/qt5/Qt3DInput/qt_temp.rhjRGj'.
> creat ignored for existing file '/home/jenkins/oe/world/shr-
> core/tmp-glibc/work/armv5te-oe-linux-gnueabi/libqofono/0.
> 87+gitrAUTOINC+54435de3be-r0/image/usr/lib/libqofono-qt5/
> tests/qt_temp.rhjRGj'.
> creat ignored for existing file '/home/jenkins/oe/world/shr-
> core/tmp-glibc/work/armv5te-oe-linux-gnueabi/libqofono/0.
> 87+gitrAUTOINC+54435de3be-r0/image/usr/lib/libqofono-qt5/
> tests/qt_temp.rhjRGj'.
> creat ignored for existing file '/home/jenkins/oe/world/shr-
> core/tmp-glibc/sstate-control/index-armv5te.lock'.
> creat ignored for existing file '/home/jenkins/oe/world/shr-
> core/tmp-glibc/sstate-control/index-armv5te.lock'.
> creat ignored for existing file '/home/jenkins/oe/world/shr-
> core/tmp-glibc/work/i586-oe-linux/usermode/1.111-r0/build/
> po/.intltool-merge-cache'.
>
> ^dir err: also happens a lot in the log files where other issues appear as
> well (because the QA check isn't triggered by this one):
> martin at jama ~/pseudo-qa $ cat log.world.qemu* | grep -v "grep -e" | grep
> -v 'inode mismatch' | grep -v 'creat for.* replaces' | grep -v indicates |
> grep -v '^Setup complete' | grep -v '^db cleanup' | grep -v '^pid ' | grep
> -v '^ \[' | grep -v '^memory-to-file' | grep -v '^debug_logfile: ' | grep
> '^creat ignored ' -v | grep '^dir err ' | wc -l
> 744
>
> couple ^symlink err:
> martin at jama ~/pseudo-qa $ cat log.world.qemu* | grep -v "grep -e" | grep
> -v 'inode mismatch' | grep -v 'creat for.* replaces' | grep -v indicates |
> grep -v '^Setup complete' | grep -v '^db cleanup' | grep -v '^pid ' | grep
> -v '^ \[' | grep -v '^memory-to-file' | grep -v '^debug_logfile: ' | grep
> '^creat ignored ' -v | grep '^dir err ' -v | grep '^symlink err' | wc -l
> 223
>
> martin at jama ~/pseudo-qa $ cat log.world.qemu* | grep -v "grep -e" | grep
> -v 'inode mismatch' | grep -v 'creat for.* replaces' | grep -v indicates |
> grep -v '^Setup complete' | grep -v '^db cleanup' | grep -v '^pid ' | grep
> -v '^ \[' | grep -v '^memory-to-file' | grep -v '^debug_logfile: ' | grep
> '^creat ignored ' -v | grep '^dir err ' -v | grep '^symlink err' -v | grep
> '^symlink mismatch: ' | wc -l
> 309
>
> I fail to see useful patterns in this, maybe fixing the
> package-output.lock might help to filter the noise a bit.
>
> On Mon, Oct 16, 2017 at 5:10 PM, Burton, Ross <ross.burton at intel.com>
> wrote:
>
>> On 16 October 2017 at 16:01, Martin Jansa <martin.jansa at gmail.com> wrote:
>>
>>> The world builds with this check applied show that *a lot* of components
>>> report one of these errors and often many of errors, the most complete logs
>>> from last couple days:
>>>
>>
>> A lot of the errors in my logs were due to package-output.lock being
>> accessed inside/outside pseudo context.  Can you grep all of your logs and
>> collate some common patterns?
>>
>> Ross
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openembedded.org/pipermail/openembedded-core/attachments/20171219/70098c72/attachment-0002.html>