[OE-core] [RFC PATCH 0/6] Task progress reporting

[Paul Eggleton]

I've often thought it would be nice if we could print progress information
for individual tasks so that you could get some idea of whether the task
is doing anything and how much longer it might take, and so I've been working
on a set of patches to implement just that in the background for a while now
and this is the result.  Unfortunately this isn't necessarily practical to do
for all tasks, but things like do_fetch are running processes that do output
progress information so it's not too hard to tap into that and present it in
the form of a progress bar next to the task, which is what this patchset
does. I was also able to enable progress for do_compile in recipes that use
cmake (since that outputs % complete as it runs) as well as do_rootfs where
it has a number of internal steps that can be counted and weighted.

Note that this patchset is on top of Poky because there are changes required
in both BitBake and OE-Core. However, for those that prefer BitBake + OE-core
separately I have pushed a paule/progress-bb branch to poky-contrib (on top
of bitbake master) and paule/progress-oecore to openembedded-core-contrib on
top of openembedded-core master containing the same changes - ensure you have
both if you go down this route.

I have contemplated and even prototyped going a step further and using the
count of lines in the output to track progress for *all* long-running tasks,
which does work reasonably well - the caveat is you actually need the
expected line count ahead of time for each task, so you have to be able to
get that from somewhere. I  haven't worked out a practical means of
delivering that, suggestions would be very much welcome - my prototype stuff
is on poky-contrib paule/progress-wip and relies on someone profiling a
build containing all the tasks beforehand and that information somehow being
made available to everyone else.

Anyway, please let me know what you think.


Please review the following changes for suitability for inclusion. If you have
any objections or suggestions for improvement, please respond to the patches. If
you agree with the changes, please provide your Acked-by.

The following changes since commit 7ca60ec8bff7656b4e52f5a4d238913e851da089:

  test-empty-image: Fix LIC_FILES_CHKSUM typo (2016-05-06 10:48:06 +0100)

are available in the git repository at:

  git://git.yoctoproject.org/poky-contrib paule/progress
  http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=paule/progress

Paul Eggleton (6):
  knotty: import latest python-progressbar
  lib: implement basic task progress support
  fetch2: implement progress support
  bitbake.conf: whitelist progress varflag
  classes/cmake: enable progress for do_compile
  classes/image: implement progress support for do_rootfs

 bitbake/lib/bb/build.py                           | 143 ++++++++
 bitbake/lib/bb/fetch2/__init__.py                 |   4 +-
 bitbake/lib/bb/fetch2/git.py                      |  47 ++-
 bitbake/lib/bb/fetch2/wget.py                     |  26 +-
 bitbake/lib/bb/ui/knotty.py                       |  76 ++++-
 bitbake/lib/bb/ui/uihelper.py                     |   7 +-
 bitbake/lib/progressbar.py                        | 384 ---------------------
 bitbake/lib/progressbar/LICENSE.txt               |  52 +++
 bitbake/lib/progressbar/__init__.py               |  49 +++
 bitbake/lib/progressbar/compat.py                 |  44 +++
 bitbake/lib/progressbar/progressbar.py            | 315 +++++++++++++++++
 bitbake/lib/progressbar/widgets.py                | 391 ++++++++++++++++++++++
 meta/classes/cmake.bbclass                        |   1 +
 meta/classes/image.bbclass                        |   9 +-
 meta/conf/bitbake.conf                            |   3 +-
 meta/lib/oe/rootfs.py                             | 101 +++++-
 meta/recipes-devtools/cmake/cmake-native_3.4.3.bb |   2 +
 17 files changed, 1238 insertions(+), 416 deletions(-)
 delete mode 100644 bitbake/lib/progressbar.py
 create mode 100644 bitbake/lib/progressbar/LICENSE.txt
 create mode 100644 bitbake/lib/progressbar/__init__.py
 create mode 100644 bitbake/lib/progressbar/compat.py
 create mode 100644 bitbake/lib/progressbar/progressbar.py
 create mode 100644 bitbake/lib/progressbar/widgets.py

-- 
2.5.5