[OE-core] [PATCH 0/1] Uprev to pseudo 1.2

Wed Nov 2 22:47:04 UTC 2011

On Wed, Nov 02, 2011 at 05:31:59PM -0500, Mark Hatle wrote:
> On 11/2/11 5:11 PM, Martin Jansa wrote:
> > On Wed, Nov 02, 2011 at 04:39:42PM -0500, Mark Hatle wrote:
> >> On 11/2/11 4:22 PM, Martin Jansa wrote:
> >>> On Wed, Nov 02, 2011 at 04:11:57PM -0500, Mark Hatle wrote:
> >>>> Uprev to pseudo 1.2.
> >>>>
> >>>> This was heavily tested by building oe-core with numerous image 
> >>>> configurations.  Each configuration generated the same image before and 
> >>>> after the change from the current to new version of pseudo.
> >>>>
> >>>> The primary purpose of this change is to enable the PSEUDO_UNLOAD 
> >>>> functionality that may be used for performance enhancements in future 
> >>>> versions of oe-core.
> >>>>
> >>>> The following changes since commit 37579d7d74d127c90c1e078d05c5bf4ba0b3f755:
> >>>>
> >>>>   meta: glib-2.0: don't apply qsort_r test removable patch for native version (2011-11-02 09:08:21 +0000)
> >>>>
> >>>> are available in the git repository at:
> >>>>   git://git.pokylinux.org/poky-contrib mhatle/pseudo
> >>>>   http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=mhatle/pseudo
> >>>
> >>> Is there branch with pseudo-1.2 for oe-core?
> >>
> >> ^^^^^^^^^^^^^^^^^^^^^^^^
> >>
> >> git://git.pokylinux.org/poky-contrib mhatle/pseudo
> >>
> >> Grab that -- or simply grab the top commit and add it to your oe-core checkout.
> > 
> > Yes, but git cherry-pick would be faster and adding poky-contrib as
> > another git remote is not very efficient as it doesn't share any git
> > objects with oe-core repo so it checkouts "whole poky" again :/.
> > 
> > So I'll use 2nd easiest option "pw-am.sh 14227".
> > 
> >>> I would like to try it because we have strange errors when libstc++
> >>> isn't found when building with pseudo, but same binaries are working
> >>> fine without pseudo (seems related to /etc/ld.so.cache being ignored
> >>> sometimes from pseudo env).
> >>>
> >>> And while I was trying to debug it I've noticed that files.db and
> >>> pseudo.log is quite big.
> >>> -rw-r--r-- 1 bitbake bitbake 240M Aug 15 13:49 files.db
> >>> -rw-r--r-- 1 bitbake bitbake  91K Aug 15 13:50 logs.db
> >>> -rw-r--r-- 1 bitbake bitbake 305M Aug 15 13:49 pseudo.log
> >>
> >> The files.db is based on the number of files accessed during the time pseudo is
> >> running.
> > 
> > Ah ok, this was in sysroot which was created "Aug  9" so it gets pretty
> > big fast, lets hope that sqlite3 can handle it efficiently, or is there
> > some way to "cleanup" stale entries in files table?
> 
> When you clean the build it will restart the pseudo DB.
> 
> > ie I'm using eglibc-2.14, but files table is full of eglibc-2.13 entries
> > and _none_ from eglibc-2.14, is it supposed to be like this or is it
> > sign of some pseudo problems?
> > 
> > sqlite> select count(*) from files where path like '%eglibc-2.13%';
> > 20221
> > sqlite> select count(*) from files where path like '%eglibc-2.14%';
> > 0
> > sqlite> select count(*) from files;
> > 1085380
> 
> The pseudo DB is specific to the tmp work directory.  If the work directory
> hasn't changed (i.e. you aren't updating versions and pr) then the pseudo DB
> will continue to grow to accommodate the new files.
> 
> There are flags within the DB if an erase operation occurs, but entries are only
> tagged as erased -- but not removed.  This is due to speed issues.  pseudo is
> significantly slower on remove and replace operations if the DB entries are removed.
> 
> >> logs.db and pseudo.log are based on the debug settings of pseudo.  If you don't
> >> have debugging enabled (you would have to do this manually) and the logs are
> >> that big, something is going seriously wrong with the path/inode translation...
> >>  It's fairly normal to have a set of logs that are in the 10-20k range for
> >> complex packages.. (These are documenting cases where pseudo has to guess about
> >> inode to path mapping..)
> > 
> > This was generated without debug enabled.. but maybe because of rm_work,
> > see below...
> > 
> >>> with pseudo.log full of errors like this
> >>> pseudo: path mismatch [12 links]: ino 39721500 db
> >>> '/OE/shr-core/tmp/work/armv4t-oe-linux-gnueabi/vim-7.2.446-r10.3/image/usr/share/vim/vim72/tutor'
> >>> req
> >>> '/OE/shr-core/tmp/work/om_gta02-oe-linux-gnueabi/task-shr-feed-1.0-r96'.
> >>
> >> Are you building on a NFS filesystem?  NFS (and a few other filesystems) have a
> >> habit of remapping inodes out from under the filesystem.  pseudo uses the inodes
> >> as a validation that the file hasn't changed.  If these inodes are changing,
> >> pseudo will do it's best to keep things in sync, but there is a chance it won't
> >> always succeed.
> > 
> > No I'm building on local ext3 filesystem, but I'm also using rm_work, so 
> > /OE/shr-core/tmp/work/armv4t-oe-linux-gnueabi/vim-7.2.446-r10.3/image/usr/share/vim/vim72/tutor
> > is probably long gone and inode 39721500 wasn't marked as unused in
> > pseudo world or do I read it wrong?
> 
> If you use rm_work, then the pseudo DB located within the work directory will
> also be removed.  So I'm not sure why you would be having such a large DB.

But I was talking about pseudo db here:
tmp/sysroots/x86_64-linux/var/pseudo/

now with pseudo-1.2 I still have quite a few warnings about inodes in
pseudo.log now in work directory, ie python build:
OE @ ~/shr-core/tmp/work/armv4t-oe-linux-gnueabi/python-2.7.2-r0.1 $ 
-rw-r--r--  1 bitbake bitbake 5.2M Nov  2 23:44 files.db
-rw-r--r--  1 bitbake bitbake 4.0K Nov  2 23:44 logs.db
-rw-r--r--  1 bitbake bitbake  46K Nov  2 23:44 pseudo.log

pseudo: dir err : 1240102 ['/OE/shr-core/tmp/work/armv4t-oe-linux-gnueabi/python-2.7.2-r0.1/deploy-ipks/armv4t/IPKG_BUILD.25064'] (db 'no path') db mode 0100644, header mode 040755 (unlinking db)
pseudo: creat for '/OE/shr-core/tmp/work/armv4t-oe-linux-gnueabi/python-2.7.2-r0.1/deploy-ipks/armv4t/stvITxha' replaces existing 1240107 ['no path'].
wxr-xr-x  1 bitbake bitbake    0 Nov  2 23:44 pseudo.socket
pseudo: path mismatch [2 links]: ino 1240673 db '/OE/shr-core/tmp/work/armv4t-oe-linux-gnueabi/python-2.7.2-r0.1/sstate-build-package/package/usr/bin/python2.7' req '/OE/shr-core/tmp/work/armv4t-oe-linux-gnueabi/python-2.7.2-r0.1/sstate-build-package/package/usr/bin/python'.

-- 
Martin 'JaMa' Jansa     jabber: Martin.Jansa at gmail.com
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 198 bytes
Desc: Digital signature
URL: <http://lists.openembedded.org/pipermail/openembedded-core/attachments/20111102/25e373b4/attachment-0002.sig>