[bitbake-devel] [PATCH 1/1] process.py, build.py: Fix log truncation problems with flush()
Richard Purdie
richard.purdie at linuxfoundation.org
Thu Jun 21 16:12:39 UTC 2012
On Thu, 2012-06-21 at 11:04 -0500, Jason Wessel wrote:
> On 06/21/2012 10:55 AM, Richard Purdie wrote:
> > On Thu, 2012-06-14 at 09:58 -0500, Jason Wessel wrote:
> >> There are two problems with the _logged_communicate that are both
> >> caused as a result of buffering I/O, instead of flushing it out to the
> >> log files as it arrives.
> >>
> >> 1) log truncation when python dumps
> >> I have seen the task logs missing data that was not flushed when
> >> bitbake crashes.
> >>
> >> 2) While a bitbake task is running it is impossible to see what is
> >> going on if it is only writing a small incremental log that is
> >> smaller than the buffer, or you get only a partial log, up until
> >> the task exists. It is worse in the case that stderr and stdout
> >> are separate file handles, because previous code blocks on the read
> >> of stdout and then stderr, serially.
> >>
> >> The right approach is simply to use select() to determine if there is
> >> data available and then to flush the log buffers as they arrive. This
> >> still makes use of buffering in the cases where there is more than 1
> >> byte of data, but the buffers are much more dynamic.
> >>
> >> Signed-off-by: Jason Wessel <jason.wessel at windriver.com>
> >> ---
> >> lib/bb/build.py | 3 ++-
> >> lib/bb/process.py | 29 +++++++++++++++++++++++++++--
> >> 2 files changed, 29 insertions(+), 3 deletions(-)
> > For what its worth I'm seeing a small but consistent increase in real,
> > sys and user times with this patch which is why I'm pausing to look at
> > it a little further :(
>
>
> In my experience, it varied quite a bit. If need be, we simply allow
> allow a config variable like BB_FORCE_LOG_FLUSH = "1", and get rid of
> the flushes by default.
>
> At the distro level, I'll turn this on, but still allow users to
> override it, because timely logging of builds is critical, and loss of
> data due to a python crash is completely unacceptable, it just makes
> build failures even harder to diagnose.
Python has some better tools to handle things like this, such as
something like:
try:
<normal case>
finally:
x.flush()
so it would be interesting to see if that helps.
Cheers,
Richard
More information about the bitbake-devel
mailing list