[Openembedded-architecture] Bitbake server rework

Tue Aug 22 13:59:07 UTC 2017

Hello,

I just got a bit confusing error when conf/bblayers.conf was referencing
deleted layer.

I know it's hard to provide nice error message for every possible corner
case what might go wrong, but if you need some examples here is one:

OE qemux86@ ~/build/oe-core $ bitbake -c cleansstate grub-efi; bitbake -c
install grub-efi
NOTE: Starting bitbake server...
ERROR: Unable to start bitbake server
ERROR: Last 10 lines of server log
/OE/build/oe-core/bitbake-cookerdaemon.log:
Running command ['setConfig', 'cmd', 'install']
Running command ['updateConfig', {'abort': True, 'tryaltconfigs': False,
'force': False, 'invalidate_stamp': None, 'verbose': False, 'debug': 0,
'dry_run': False, 'dump_signatures': [], 'debug_domains': [],
'extra_assume_provided': [], 'profile': False, 'prefile': [], 'postfile':
[], 'tracking': False, 'server_timeout': None}, {'LC_ALL': 'en_US.UTF-8',
'MACHINE': 'qemux86', 'USER': 'bitbake', 'PWD': '/OE/build/oe-core',
'HOME': '/OE', 'BB_ENV_EXTRAWHITE': 'MACHINE DISTRO http_proxy ftp_proxy
https_proxy all_proxy ALL_PROXY no_proxy SSH_AGENT_PID SSH_AUTH_SOCK
BB_SRCREV_POLICY SDKMACHINE BB_NUMBER_THREADS GIT_PROXY_COMMAND
PSEUDO_DISABLED PSEUDO_BUILD', 'SHELL': '/bin/bash', 'TERM': 'screen',
'LOGNAME': 'bitbake', 'PATH':
'/OE/build/oe-core/openembedded-core/scripts:/OE/build/oe-core/bitbake/bin:/OE/bin:/usr/x86_64-pc-linux-gnu/gcc-bin/7.1.0:/usr/local/bin:/usr/bin:/bin:/opt/bin',
'LD_LIBRARY_PATH': '', 'LS_COLORS':
'rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=01;05;37;41:mi=01;05;37;41:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arc=01;31:*.arj=01;31:*.taz=01;31:*.lha=01;31:*.lz4=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.tzo=01;31:*.t7z=01;31:*.zip=01;31:*.z=01;31:*.Z=01;31:*.dz=01;31:*.gz=01;31:*.lrz=01;31:*.lz=01;31:*.lzo=01;31:*.xz=01;31:*.zst=01;31:*.tzst=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.alz=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.cab=01;31:*.wim=01;31:*.swm=01;31:*.dwm=01;31:*.esd=01;31:*.jpg=01;35:*.jpeg=01;35:*.mjpg=01;35:*.mjpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.ogv=01;35:*.ogx=01;35:*.cfg=00;32:*.conf=00;32:*.diff=00;32:*.doc=00;32:*.ini=00;32:*.log=00;32:*.patch=00;32:*.pdf=00;32:*.ps=00;32:*.tex=00;32:*.txt=00;32:*.aac=00;36:*.au=00;36:*.flac=00;36:*.m4a=00;36:*.mid=00;36:*.midi=00;36:*.mka=00;36:*.mp3=00;36:*.mpc=00;36:*.ogg=00;36:*.ra=00;36:*.wav=00;36:*.oga=00;36:*.opus=00;36:*.spx=00;36:*.xspf=00;36:',
'STY': '3598.x', 'LANG': 'C', 'LESS': '-MSwiQ', 'DISPLAY': ':0.0',
'OPENGL_PROFILE': 'xorg-x11', 'OLDPWD': '/OE', 'CONFIG_PROTECT_MASK':
'/etc/sandbox.d /etc/gentoo-release /etc/terminfo /etc/ca-certificates.conf
/etc/revdep-rebuild', 'EDITOR': 'vim', 'GCC_SPECS': '', 'PAGER':
'/usr/bin/less', 'WINDOW': '0', 'BUILDDIR': '/OE/build/oe-core', 'SHLVL':
'2', 'MANPATH':
'/usr/share/gcc-data/x86_64-pc-linux-gnu/7.1.0/man:/usr/share/binutils-data/x86_64-pc-linux-gnu/2.28.1/man:/usr/local/share/man:/usr/share/man',
'XAUTHORITY': '/root/.xauthEJHUgh', 'PS1': '\\[\\033[01;32m\\]OE
$MACHINE@$DISTRO\\[\\033[01;34m\\]
\\w \\$\\[\\033[00m\\] ', 'INFOPATH':
'/usr/share/gcc-data/x86_64-pc-linux-gnu/7.1.0/info:/usr/share/binutils-data/x86_64-pc-linux-gnu/2.28.1/info:/usr/share/info:/usr/share/info/emacs-25',
'TERMCAP': 'SC|screen|VT 100/ANSI X3.64 virtual
terminal:\\\n\t:DO=\\E[%dB:LE=\\E[%dD:RI=\\E[%dC:UP=\\E[%dA:bs:bt=\\E[Z:\\\n\t:cd=\\E[J:ce=\\E[K:cl=\\E[H\\E[J:cm=\\E[%i%d;%dH:ct=\\E[3g:\\\n\t:do=^J:nd=\\E[C:pt:rc=\\E8:rs=\\Ec:sc=\\E7:st=\\EH:up=\\EM:\\\n\t:le=^H:bl=^G:cr=^M:it#8:ho=\\E[H:nw=\\EE:ta=^I:is=\\E)0:\\\n\t:li#36:co#100:am:xn:xv:LP:sr=\\EM:al=\\E[L:AL=\\E[%dL:\\\n\t:cs=\\E[%i%d;%dr:dl=\\E[M:DL=\\E[%dM:dc=\\E[P:DC=\\E[%dP:\\\n\t:im=\\E[4h:ei=\\E[4l:mi:IC=\\E[%d@:ks=\\E[?1h\\E=:\\\n\t:ke=\\E[?1l\\E>:vi=\\E[?25l:ve=\\E[34h\\E[?25h:vs=\\E[34l:\\\n\t:ti=\\E[?1049h:te=\\E[?1049l:us=\\E[4m:ue=\\E[24m:so=\\E[3m:\\\n\t:se=\\E[23m:mb=\\E[5m:md=\\E[1m:mh=\\E[2m:mr=\\E[7m:\\\n\t:me=\\E[m:ms:\\\n\t:Co#8:pa#64:AF=\\E[3%dm:AB=\\E[4%dm:op=\\E[39;49m:AX:\\\n\t:vb=\\Eg:G0:as=\\E(0:ae=\\E(B:\\\n\t:ac=\\140\\140aaffggjjkkllmmnnooppqqrrssttuuvvwwxxyyzz{{||}}~~..--++,,hhII00:\\\n\t:Km=\\E[M:k0=\\E[10~:k1=\\EOP:k2=\\EOQ:k3=\\EOR:k4=\\EOS:\\\n\t:k5=\\E[15~:k6=\\E[17~:k7=\\E[18~:k8=\\E[19~:k9=\\E[20~:\\\n\t:k;=\\E[21~:F1=\\E[23~:F2=\\E[24~:kb=\x7f:kh=\\E[1~:@1=\\E[1~:\\\n\t:kH=\\E[4~:@7=\\E[4~:kN=\\E[6~:kP=\\E[5~:kI=\\E[2~:kD=\\E[3~:\\\n\t:ku=\\EOA:kd=\\EOB:kr=\\EOC:kl=\\EOD:km:',
'LESSOPEN': '|lesspipe %s', '_': '/OE/build/oe-core/bitbake/bin/bitbake'},
['/OE/build/oe-core/bitbake/bin/bitbake', '-c', 'install',
'grub-efi-native']]
Running command ['buildTargets', ['grub-efi-native'], 'install']
Connecting Client
Disconnecting Client
No timeout, exiting.
Exiting
Starting bitbake server pid 30625
NOTE: Starting bitbake server...
ERROR: Layer directory '/OE/build/oe-core/meta-openembedded/meta-ruby' does
not exist! Please check BBLAYERS in /OE/build/oe-core/conf/bblayers.conf

NOTE: Starting bitbake server...
ERROR: Unable to start bitbake server
ERROR: Last 10 lines of server log
/OE/build/oe-core/bitbake-cookerdaemon.log:
Connecting Client
Disconnecting Client
No timeout, exiting.
Exiting
Starting bitbake server pid 30625
NOTE: Starting bitbake server...
ERROR: Layer directory '/OE/build/oe-core/meta-openembedded/meta-ruby' does
not exist! Please check BBLAYERS in /OE/build/oe-core/conf/bblayers.conf
Starting bitbake server pid 30636
NOTE: Starting bitbake server...
ERROR: Layer directory '/OE/build/oe-core/meta-openembedded/meta-ruby' does
not exist! Please check BBLAYERS in /OE/build/oe-core/conf/bblayers.conf

On Fri, Jul 21, 2017 at 3:05 PM, Richard Purdie <
richard.purdie at linuxfoundation.org> wrote:

> On Fri, 2017-07-21 at 14:51 +0200, Martin Jansa wrote:
> > On Fri, Jul 21, 2017 at 08:55:10AM +0100, Richard Purdie wrote:
> > >
> > > I just wanted to give people a heads up that a pretty significant
> > > change in bitbake's internal behaviour has merged.
> > >
> > > In short, the server backend to bitbake has been reworked and
> > > bitbake
> > > now always forks off a "cooker" or server process (if needed) and
> > > then
> > > connects to it using a unix domain socket.
> > >
> > > The xmlrpc and process server backends are replaced with one
> > > unified
> > > interface where there is always a process server which can
> > > optionally
> > > provide an xmlrpc interface for remote connections.
> > >
> > > This means you get "memres" behaviour simply by setting
> > > BB_SERVER_TIMEOUT to some positive value and then the cooker will
> > > remain around for that length of time before unloading from memory.
> > >
> > > Right now, the default is 0 giving the same behaviour as before,
> > > with
> > > the change that the cooker is always an independent process.
> > >
> > > BB_SERVER_TIMEOUT being non-zero is known to break some oe-
> > > selftests
> > > right now, that is some we can start to address now we have the
> > > code
> > > baseline merged. It does seem to work well in normal use although
> > > the
> > > internal server resets for new connections may need optimisation
> > > too.
> > >
> > > I do expect we may have some teething issues with this as its a
> > > major
> > > change to the way we've done things. I (and other bitbake
> > > developers)
> > > do believe this is the best way forward for the project though and
> > > now
> > > is the best time to merge this and get people using it. Not least,
> > > it
> > > replaces a load of horrible code nobody really understood with
> > > something simpler and more understandable and capable. I am
> > > delaying M2
> > > a little to ensure we get this in now and have time to fix issues
> > > before 2.4.
> >
> > Are there any changes needed for PRserv (or it's configuration)?
>
> No, no changes are needed. We are seeing that server shutdown can be
> slower than new clients starting in some cases though and if that
> happens, you see the retry message. It should be harmless if a little
> annoying on the console.
>
> I've been trying to pin down the circumstances that happens under, it
> could be PRServ is a factor (buildhistory may be too).
>
> Cheers,
>
> Richard
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openembedded.org/pipermail/openembedded-architecture/attachments/20170822/489ffa3c/attachment-0002.html>