[bitbake-devel] [PATCH 0/1] process.py, build.py: Fix log truncation problems with flush()

[Jason Wessel]

Separately Richard had asked me if this proposed patch to flush things
out to disk immediately had a negative performance impact.  It is
actually the opposite effect due to a very subtle problem that was
fixed.

My litmus test is to wipe the .o files, drop kernel caches, force
compile the kernel which will generate 100's of very small single line
logs under the size of the buffer limit.

Without the log flushing patch compiling a kernel:
   real    3m33.906s
   user    0m53.523s
   sys     0m24.518s

With the log flushing patch compiling a kernel:
   real    3m28.102s
   user    0m53.251s
   sys     0m24.982s

Why is it faster you ask?

It has to do with the alternating blocking reads on stdin and stderr.
First the existing code blocks on stdout until the buffer fills up in
the python logger.  Once the buffer is full, it will switch to
blocking on stderr, and if nothing ever comes it may wait all the way
until the task completes and then tries to chunk the buffer out to disk
at the end which is quite wasteful.  

Sometimes the existing behavior also leads to having any bitbake
injected error messages into the log file appear out of chronological
order with respect to how ever much data was in the log because bitbake
will emit the error to stderr and stderr could get flushed before
stdout.

Cheers,
Jason.