[OE-core] [RFC][PATCHv2] insane.bbclass: Add do_qa_pseudo function to check for common errors listed in pseudo.log
Martin Jansa
martin.jansa at gmail.com
Mon Oct 16 15:01:39 UTC 2017
The world builds with this check applied show that *a lot* of components
report one of these errors and often many of errors, the most complete logs
from last couple days:
366 in:
http://logs.nslu2-linux.org/buildlogs/oe/world/rocko/log.report.20171012_161223.log
http://logs.nslu2-linux.org/buildlogs/oe/world/rocko/log.world.qemuarm.20171010_225937.log/qa.log
http://logs.nslu2-linux.org/buildlogs/oe/world/rocko/log.world.qemux86.20171011_084917.log/qa.log
http://logs.nslu2-linux.org/buildlogs/oe/world/rocko/log.world.qemux86-64.20171010_012728.log/qa.log
376 in:
http://logs.nslu2-linux.org/buildlogs/oe/world/rocko/log.report.20171001_085046.log
http://logs.nslu2-linux.org/buildlogs/oe/world/rocko/log.world.qemuarm.20170926_004402.log/qa.log
http://logs.nslu2-linux.org/buildlogs/oe/world/rocko/log.world.qemux86.20170926_004401.log/qa.log
http://logs.nslu2-linux.org/buildlogs/oe/world/rocko/log.world.qemux86-64.20170927_070922.log/qa.log
The number of errors differs in various builds, e.g.:
tcl-8.6.7: This
/work/armv5te-oe-linux-gnueabi/tcl/8.6.7-r0/pseudo/pseudo.log indicates 7
errors [qa_pseudo]
tcl-8.6.7: This
/work/armv5te-oe-linux-gnueabi/tcl/8.6.7-r0/pseudo/pseudo.log indicates 5
errors [qa_pseudo]
tcl-8.6.7: This /work/i586-oe-linux/tcl/8.6.7-r0/pseudo/pseudo.log
indicates 7 errors [qa_pseudo]
tcl-8.6.7: This /work/i586-oe-linux/tcl/8.6.7-r0/pseudo/pseudo.log
indicates 8 errors [qa_pseudo]
tcl-8.6.7: This /work/core2-64-oe-linux/tcl/8.6.7-r0/pseudo/pseudo.log
indicates 6 errors [qa_pseudo]
tcl-8.6.7: This /work/core2-64-oe-linux/tcl/8.6.7-r0/pseudo/pseudo.log
indicates 6 errors [qa_pseudo]
glibc-locale-2.26: This
/work/armv5e-oe-linux-gnueabi/glibc-locale/2.26-r0/pseudo/pseudo.log
indicates 17 errors [qa_pseudo]
glibc-locale-2.26: This
/work/armv5e-oe-linux-gnueabi/glibc-locale/2.26-r0/pseudo/pseudo.log
indicates 5 errors [qa_pseudo]
glibc-locale-2.26: This
/work/i586-oe-linux/glibc-locale/2.26-r0/pseudo/pseudo.log indicates 12
errors [qa_pseudo]
glibc-locale-2.26: This
/work/i586-oe-linux/glibc-locale/2.26-r0/pseudo/pseudo.log indicates 10
errors [qa_pseudo]
glibc-locale-2.26: This
/work/core2-64-oe-linux/glibc-locale/2.26-r0/pseudo/pseudo.log indicates 11
errors [qa_pseudo]
glibc-locale-2.26: This
/work/core2-64-oe-linux/glibc-locale/2.26-r0/pseudo/pseudo.log indicates 16
errors [qa_pseudo]
If I filter out the architecture specific path to pseudo.log and the number
of errors it still gives 368 different builds with errors:
acl-2.2.52 alsa-oss-1.0.17 alsa-plugins-1.1.4 apache2-2.4.27 apmd-3.2.2-15
aspell-0.60.6.1 attr-2.4.47 aufs-util-4.4+gitAUTOINC+89afb1806c
autofs-5.1.2 avahi-ui-0.6.32 base-passwd-3.5.29 binutils-2.29
blktrace-1.1.0+gitAUTOINC+43fc870ce0 bluez5-5.46 bmap-tools-3.4
bonnie++-1.03e boost-1.64.0 bootchart2-0.14.8 busybox-1.24.1
cantarell-fonts-0.0.24 can-utils-0.0+gitrAUTOINC+4c8fb05cb4 canutils-4.0.6
cdparanoia-10.2 chrony-2.4 ckermit-302 connman-1.34 conntrack-tools-1.4.4
console-tools-0.3.2 coreutils-8.27 corosync-2.4.2 cpufrequtils-008
cryptodev-module-1.9 ctdb-2.5.6 cups-2.2.4 curlpp-0.8.1
cwautomacros-20110201 cxxtest-4.3 daemontools-0.76 dbus-test-1.10.20
ddrescue-1.19 dfu-util-0.9 dhcp-4.3.6 diffutils-3.6 dmidecode-3.1
docbook-xml-dtd4-4.5 dovecot-2.2.29 ebtables-2.0.10-4 efivar-0.31
eglinfo-fb-1.0.0 espeak-1.48.04 evtest-1.33+AUTOINC+ab140a2dab expat-2.2.3
f2fs-tools-1.8.0 firefox-l10n-ff-45.9.0esr firefox-l10n-si-45.9.0esr
flac-1.3.2 flatbuffers-1.7.1 fontforge-20170731 frame-2.5.0
freeradius-3.0.15 fribidi-0.19.7 fuse-2.9.7
gateone-1.2+gitAUTOINC+f7a9be46cb gawk-4.1.4 gd-2.2.4 gdb-8.0 gegl-0.3.18
geoip-1.6.9 gettext-0.19.8.1 giflib-5.1.4 git-2.13.3 glade-3.20.0
glib-2.0-2.52.3 glibc-2.26 glibc-locale-2.26 glib-networking-2.50.0
gnupg-2.2.0 gnuplot-5.0.5 go-1.9 gobject-introspection-1.52.1
gptfdisk-1.0.3 gstreamer-0.10.36 gtkmm-2.24.5 hiredis-0.13.1 hplip-3.12.6
icon-slicer-0.3 icu-59.1 inetutils-1.9.4 init-ifupdown-1.0 iotop-0.6
iperf3-3.2 ipsec-tools-0.8.2 iptables-1.6.1 iputils-s20151218
iscsitarget-1.4.20.3+svn502 jack-1.9.10+gitAUTOINC+2d1d323505 kbd-2.0.4
kconfig-frontends-4.11.0.1 kexec-tools-2.0.14 konkretcmpi-0.9.2 krb5-1.15.1
lcdproc-0.5.8+gitAUTOINC+f5156e2e41 less-487 libatasmart-0.19
libatomic-ops-7.6.0+gitAUTOINC+73c60c5ef1 libbsd-0.8.6 libcanberra-0.30
libconnman-qt5-1.1.10+gitAUTOINC+4d6def3053
libcrypt-openssl-random-perl-0.11 libdrm-2.4.83 libencode-perl-2.83
libevdev-1.5.7 libfastjson-0.99.4 libgnome-2.32.1 libiec61883-1.2.0
libinput-1.8.1 libldb-1.1.29 liblogging-1.0.5 libmms-0.6.4 libotr-4.1.1
libpciaccess-0.13.5 libqofono-0.87+gitrAUTOINC+54435de3be librelp-1.2.14
librsvg-2.40.18 libsombok3-2.4.0 libtext-wrapi18n-perl-0.06
libtool-cross-2.4.6 libusb-compat-0.1.5 libusbg-0.1.0 libuser-0.62
libvncserver-0.9.11 libwmf-0.2.8.4 libwnck3-3.20.1 libxcursor-1.1.14
libxml2-2.9.4 libxml-perl-0.08 libxslt-1.1.29 libxxf86vm-1.1.4
linux-atm-2.5.2 linux-yocto-4.12.12+gitAUTOINC+eda4d18ce4_16de014967
linux-yocto-4.12.12+gitAUTOINC+eda4d18ce4_67b62d8d7b lirc-0.9.4d llvm-5.0
lmbench-3.0-a9 lockfile-progs-0.1.17 log4cpp-1.1.2 logfsprogs-git
logrotate-3.12.3 lowpan-tools-0.3.1+gitAUTOINC+38f42dbfce lprng-3.8.C
lttng-tools-2.9.5 lvm2-2.02.171
maliit-framework-qt5-0.99.0+gitAUTOINC+60b1b10de1 mariadb-5.5.57
matchbox-keyboard-0.1.1 mc-4.8.19 mce-inject-git md5deep-4.4
memcached-1.5.0 menulibre-2.1.3 mesa-demos-8.3.0 meson-0.40.1
minicoredumper-2.0.0 mongodb-3.4.6+gitAUTOINC+c55eb86ef4 mosquitto-1.4.14
mozilla-devscripts-0.30 mpich-3.2 multipath-tools-0.7.1 nana-2.5
ndisc6-1.0.4+gitAUTOINC+4c794b5512 netcat-0.7.1 net-snmp-5.7.3 nettle-3.3
nginx-1.12.1 nss-3.31.1 ntp-4.2.8p10 onboard-1.4.1 onig-5.9.6
openconnect-7.08 openhpi-3.6.1 openipmi-2.0.24 open-isns-0.97
openlmi-tools-0.10.5 opensaf-5.17.07 opensc-0.16.0 openssh-7.5p1
openssl-1.0.2l opkg-utils-0.3.5 opus-tools-0.1.8 os-release-1.0
ossp-uuid-1.6.2 packagegroup-core-ssh-openssh-1.0
packagegroup-core-x11-xserver-1.0 pango-1.40.6 pciutils-3.5.5 perf-1.0
perl-5.24.1 php-7.1.9 phpmyadmin-4.7.3 pixz-1.0.6 pkcs11-helper-1.11
pm-qa-0.5.2 pm-utils-1.4.1 poppler-data-0.4.7 portaudio-v19-v190600
postfix-3.2.2 postgresql-9.4.13 powertop-2.8 procps-3.3.12 proftpd-1.3.6
pure-ftpd-1.0.42 puzzles-0.0+gitAUTOINC+8dfe5cec31 pyrtm-0.4.2
python-2.7.13 python3-3.5.3 python3-babel-2.5.1 python3-chardet-3.0.4
python3-coverage-4.4.1 python3-cryptography-2.0.3 python3-cython-0.27.1
python3-django-1.11.5 python3-docutils-0.14 python3-isort-4.2.15
python3-mako-1.0.7 python3-ndg-httpsclient-0.4.3 python3-nose-1.3.7
python3-ntplib-0.3.3 python3-pandas-0.20.3 python3-pip-9.0.1
python3-psutil-5.3.1 python3-pykickstart-2.35 python3-pylint-1.7.2
python3-pylint-1.7.4 python3-pyparted-3.10.7+gitAUTOINC+69c4a478e4
python3-serpent-1.23 python3-simplejson-3.11.1 python3-xlrd-1.1.0
python-asn1crypto-0.23.0 python-cheetah-2.4.4 python-cpuset-1.5.7
python-cson-git python-cython-0.26.1 python-django-1.11.5
python-epydoc-3.0.1 python-gevent-1.2.2 python-imaging-1.1.7
python-jsonpatch-1.16 python-jsonpointer-1.12 python-ndg-httpsclient-0.4.3
python-netaddr-0.7.19 python-nose-1.3.7 python-numeric-24.2
python-numpy-1.13.1 python-pandas-0.20.3 python-pbr-3.1.1
python-pyflakes-1.2.3 python-pylint-1.7.2 python-pylint-1.7.4
python-pyro4-4.62 python-pyserial-3.4 python-pytest-3.2.2
python-scons-2.5.1 python-selectors34-1.2 python-setuptools-36.2.7
python-snakefood-1.4 python-snimpy-0.8.11 python-twisted-17.9.0
python-urllib3-1.22 python-vobject-0.9.5 python-xlrd-1.1.0
python-yappi-0.98 pywbem-0.8.0 qemu-2.10.0 qt3d-5.9.2+gitAUTOINC+9d8c9ada16
qt5-creator-3.5.1 qt5-plugin-generic-vboxtouch-1.1.3+gitrAUTOINC+d613ad1cc6
qtbase-5.9.2+gitAUTOINC+73573fce29 qtcharts-5.9.2+gitAUTOINC+1f47b1a7ae
qtconnectivity-5.9.2+gitAUTOINC+94cc2b914a
qtdatavis3d-5.9.2+gitAUTOINC+3f8ae713df
qtdeclarative-5.9.2+gitAUTOINC+7c45b035b9
qtenginio-5.9.2+gitAUTOINC+0555cf73c8 qtgamepad-5.9.2+gitAUTOINC+1fc40deed6
qtgraphicaleffects-5.9.2+gitAUTOINC+b6ff12df54
qtimageformats-5.9.2+gitAUTOINC+144a3535db
qtlocation-5.9.2+gitAUTOINC+c832af7897_9ecbe3642f
qtmultimedia-5.9.2+gitAUTOINC+343e281f0e
qtnetworkauth-5.9.2+gitAUTOINC+3d93f64365
qtquick1-5.9.2+gitAUTOINC+9bf0edd9bd
qtquickcontrols2-5.9.2+gitAUTOINC+63f2f55462
qtremoteobjects-5.9.2+gitAUTOINC+6e0341bc87
qtscript-5.9.2+gitAUTOINC+86ee61734e qtscxml-5.9.2+gitAUTOINC+a1dff1a11a
qtsensors-5.9.2+gitAUTOINC+1673dbed00
qtserialbus-5.9.2+gitAUTOINC+1d37c357c8
qtserialport-5.9.2+gitAUTOINC+93b539140c qtsmarthome-1.0
qtsvg-5.9.2+gitAUTOINC+0fcd016066 qtsystems-5.9.2+gitAUTOINC+f364358a81
qttools-5.9.2+gitAUTOINC+fdc5749b56 qtwayland-5.9.2+gitAUTOINC+7f70da6a64
qtwebchannel-5.9.2+gitAUTOINC+b5229df6a0
qtwebengine-5.9.2+gitAUTOINC+99f84ffd2c_21508b5b54
qtwebkit-5.9.2+gitAUTOINC+97c4a80a12
qtwebsockets-5.9.2+gitAUTOINC+fb71c823a7
qtwebview-5.9.2+gitAUTOINC+e4016a67bb
qtx11extras-5.9.2+gitAUTOINC+f65d50eb06
qtxmlpatterns-5.9.2+gitAUTOINC+fcfa824402 quagga-1.2.1 rabbitmq-c-0.7.0
radvd-2.17 readline-7.0 ristretto-0.8.2 rsync-3.1.2 rsyslog-8.29.0
ruby-2.4.1 samba-4.6.7 satyr-0.23 sblim-sfcb-1.4.9 sblim-sfcc-2.2.8
scsirastools-1.6.6 sed-4.2.2 sg3-utils-1.42 shadow-4.2.1 slang-2.3.1a
snort-2.9.7.5 speexdsp-1.2rc3 sqlite3-3.20.0 squid-3.5.27 sthttpd-2.27.1
strace-4.18 systemd-234 systemtap-3.1 tcl-8.6.7 telepathy-python-0.15.19
tftp-hpa-5.2 tipcutils-2.2.0+gitAUTOINC+7ab2211b87 tomoyo-tools-2.5.0
tslib-1.1 ttf-gentium-1.02 ubi-utils-klibc-1.5.2 udisks-1.0.5 ufw-0.33
upm-1.3.0-gitAUTOINC+106b6c7062 usbredir-0.7.1+gitAUTOINC+39aa3c69f6
usermode-1.111 vblade-22 vlc-2.2.2 vte-0.48.3 weston-init-1.0
xcb-util-wm-0.4.1 xcursor-transparent-theme-0.1.1+gitAUTOINC+23c8af5ba4
xdebug-2.5.1 xdelta3-3.0.11 xf86-input-evdev-2.10.5 xf86-input-mouse-1.9.2
xfce4-closebutton-plugin-0.1.0+gitrAUTOINC+02b74f13ad
xfce4-hotcorner-plugin-0.0.2 xfce4-mixer-4.11.0 xfce4-power-manager-1.6.0
xfce4-verve-plugin-1.1.0 xfwm4-themes-4.10.0 xserver-xorg-1.19.3
xstdcmap-1.0.3 xwd-1.0.6 xz-5.2.3 ypbind-mt-2.4
zlog-1.2.12+gitAUTOINC+13904dab28
Any other insight what might be causing these?
On Tue, Sep 19, 2017 at 5:18 PM, Martin Jansa <martin.jansa at gmail.com>
wrote:
> * we often see QA warnings like:
> glibc-locale-2.26: glibc-locale: /glibc-binary-localedata-en-
> gb/usr/lib/locale/en_GB/LC_MEASUREMENT is owned by uid 3004, which is the
> same as the user running bitbake. This may be due to host contamination
> [host-user-contaminated]
> glibc-locale-2.26: glibc-locale: /glibc-binary-localedata-nn-
> no.iso-8859-1/usr/lib/locale/nn_NO.ISO-8859-1/LC_MEASUREMENT is owned by
> uid 3004, which is the same as the user running bitbake. This may be due to
> host contamination [host-user-contaminated]
> but we don't know the root cause of it.
> * don't trigger error from '^inode mismatch'
> * show parts of pseudo.log in console
> many builds are executed in tmpfs, so I cannot check the pseudo.log later
> * lower from error to warn as I hate all my builds reporting failure
> just because of this pseudo debug
> * the only theory we currently have is that it's a bug in pseudo when
> inode is being reused for different files, which is supported by
> pseudo.log entries:
>
> Good build:
> pseudo$ grep -v "^path mismatch" pseudo.log
> debug_logfile: fd 2
> pid 7975 [parent 7974], doing new pid setup and server start
> Setup complete, sending SIGUSR1 to pid 7974.
> db cleanup for server shutdown, 17:33:58.787
> memory-to-file backup complete, 17:33:58.787.
> db cleanup finished, 17:33:58.787
>
> Build with QA host-user-contaminated issue:
> ERROR: foo-1.0.0-r0 do_package_qa: QA Issue: foo: file-with-wrong-UID is
> owned
> by uid 2001, which is the same as the user running bitbake. This may be
> due to
> host contamination [host-user-contaminated]
>
> pseudo$ grep "file-with-wrong-UID" pseudo.log
> inode mismatch: 'file-with-wrong-UID' ino 13242270 in db, 13242271 in
> request.
> creat ignored for existing file 'file-with-wrong-UID'.
> inode mismatch: 'file-with-wrong-UID' ino 13242270 in db, 13242271 in
> request.
> inode mismatch: 'file-with-wrong-UID' ino 13242270 in db, 13242271 in
> request.
> inode mismatch: 'file-with-wrong-UID' ino 13242270 in db, 13242271 in
> request.
> inode mismatch: 'file-with-wrong-UID' ino 13242270 in db, 13242271 in
> request.
> inode mismatch: 'file-with-wrong-UID' ino 13242270 in db, 13242271 in
> request.
> inode mismatch: 'file-with-wrong-UID' ino 13242270 in db, 13242271 in
> request.
> path mismatch [1 link]: ino 13242270 db 'file-with-wrong-UID' req
> 'some-other-unrelated-file'.
> creat for 'some-other-unrelated-file' replaces existing 13242270
> ['file-with-wrong-UID'].
> db cleanup for server shutdown, 02:16:23.685
> memory-to-file backup complete, 02:16:23.685.
> db cleanup finished, 02:16:23.685
>
> And some-other-unrelated-file is really some different file, not just
> hardlink
> to the same file from some different directory (like between WORKDIR and
> sysroot
> other "path mismatch" entries show).
>
> Signed-off-by: Martin Jansa <Martin.Jansa at gmail.com>
> ---
> meta/classes/insane.bbclass | 43 ++++++++++++++++++++++++++++++
> +++++++++++++
> 1 file changed, 43 insertions(+)
>
> diff --git a/meta/classes/insane.bbclass b/meta/classes/insane.bbclass
> index 0a3b528ddb..76cd92709b 100644
> --- a/meta/classes/insane.bbclass
> +++ b/meta/classes/insane.bbclass
> @@ -1238,6 +1238,44 @@ python do_qa_unpack() {
> bb.warn('%s: the directory %s (%s) pointed to by the S variable
> doesn\'t exist - please set S within the recipe to point to where the
> source has been unpacked to' % (d.getVar('PN'), d.getVar('S', False),
> s_dir))
> }
>
> +python do_qa_pseudo() {
> + ############################################################
> ###############
> + # Check pseudo.log for unexpected errors
> + #
> + # Typical pseudo.log contains many "^path mismatch" lines for all the
> hardlinked files
> + # e.g. in some smaller component I see 231/237 lines to be "^path
> mismatch" other 6
> + # lines are setup and cleanup lines like this:
> + # debug_logfile: fd 2
> + # pid 7975 [parent 7974], doing new pid setup and server start
> + # Setup complete, sending SIGUSR1 to pid 7974.
> + # db cleanup for server shutdown, 17:33:58.787
> + # memory-to-file backup complete, 17:33:58.787.
> + # db cleanup finished, 17:33:58.787
> + #
> + # but if there is one of:
> + # "^inode mismatch"
> + # "^creat ignored for existing file"
> + # "^creat for.*replaces existing"
> + # then there might be some bigger issue which sometimes results in
> host-user-contaminated QA warnings
> + ############################################################
> ###############
> +
> + import subprocess
> +
> + pseudodir = d.getVar('PSEUDO_LOCALSTATEDIR')
> + bb.note("Checking pseudo.log for common errors")
> + pseudolog = os.path.join(pseudodir, "pseudo.log")
> + statement = "grep" \
> + " -e '^creat ignored for existing file'" \
> + " -e '^creat for.*replaces existing'" \
> + " %s" % pseudolog
> + if subprocess.call("%s -q" % statement, shell=True) == 0:
> + statement2 = "grep -v '^path mismatch' %s" % pseudolog
> + statement2_count = "grep -v '^path mismatch' -c %s" % pseudolog
> + output = subprocess.check_output(statement2,
> shell=True).decode("utf-8")
> + output_count = subprocess.check_output(statement2_count,
> shell=True).decode("utf-8").replace('\n', '')
> + bb.warn("This %s indicates %s errors, see %s or %s [qa_pseudo]:
> %s" % (pseudolog, output_count, statement, statement2, output))
> +}
> +
> # The Staging Func, to check all staging
> #addtask qa_staging after do_populate_sysroot before do_build
> do_populate_sysroot[postfuncs] += "do_qa_staging "
> @@ -1250,6 +1288,11 @@ do_configure[postfuncs] += "do_qa_configure "
> # Check does S exist.
> do_unpack[postfuncs] += "do_qa_unpack"
>
> +# Check pseudo.log for unexpected errors
> +# For some reason do_build postfunction isn't executed
> +# do_build[postfuncs] += "do_qa_pseudo"
> +addtask do_qa_pseudo after do_populate_sysroot do_packagedata do_package
> before do_build
> +
> python () {
> import re
>
> --
> 2.14.1
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openembedded.org/pipermail/openembedded-core/attachments/20171016/3cd486a2/attachment-0002.html>
More information about the Openembedded-core
mailing list