[OE-core] [PATCH v2 00/10] Show progress during build and eSDK install

[Paul Eggleton]

Show progress during build and extensible SDK installation.

I've often thought it would be nice if we could print progress information
for individual tasks so that you could get some idea of whether the task
is doing anything and how much longer it might take, and so I've been working
on a set of patches to implement just that in the background for a while now
and this is the result.  Unfortunately this isn't necessarily practical to do
for all tasks, but things like do_fetch are running processes that do output
progress information so it's not too hard to tap into that and present it in
the form of a progress bar next to the task, which is what this patchset
does. I was also able to enable progress for do_compile in recipes that use
cmake (since that outputs % complete as it runs) as well as do_rootfs where
it has a number of internal steps that can be counted and weighted.

I have contemplated and even prototyped going a step further and using the
count of lines in the output to track progress for *all* long-running tasks,
which does work reasonably well - the caveat is you actually need the
expected line count ahead of time for each task, so you have to be able to
get that from somewhere. I  haven't worked out a practical means of
delivering that, suggestions would be very much welcome - my prototype stuff
is on poky-contrib paule/progress-wip and relies on someone profiling a
build containing all the tasks beforehand and that information somehow being
made available to everyone else.

Also included in this patchset is support for showing progress in a couple
of other stages during the build - when checking for the availability of
sstate artifacts, when initialising the tasks (formerly "Preparing RunQueue")
and overall progress of running setscene and normal tasks.

There are also a few semi-related changes to improve progress reporting and
speed of extensible SDK installation.

NOTE: this patchset requires the corresponding set of changes just sent
to the bitbake-devel list in order to work.

Changes since v1:
* Recalculate stage weightings for do_rootfs, ensure we call finish() and
  add a comment mentioning how to recalculate the weightings if needed in
  future
* Fix extensible SDK install printing out errors twice on failure
* Print out list of unexpected tasks in an order closer to actual
  execution order than previously
* Add a patch fixing output of lines if SDK testing fails (not directly
  related, but I noticed this issue when fixing the SDK logging)


The following changes since commit 9431698a88430f0fa892d9b270c0849c4d3d2486:

  build-appliance-image: Update to master head revision (2016-06-21 14:02:36 +0100)

are available in the git repository at:

  git://git.openembedded.org/openembedded-core-contrib paule/startup-oe
  http://cgit.openembedded.org/cgit.cgi/openembedded-core-contrib/log/?h=paule/startup-oe

Paul Eggleton (10):
  bitbake.conf: whitelist progress varflag
  classes/cmake: enable progress for do_compile
  classes/image: implement progress support for do_rootfs
  classes/sstate: show progress during sstate object availability check
  classes/sstate: add a mode to error if sstate package unavailable
  classes/testsdk: print output correctly on failure
  classes/populate_sdk_ext: eliminate double execution on install
  lib/oe/sstatesig: print locked sigs file message only when explicitly
    called
  classes/populate_sdk_ext: show progress when preparing build system
  toolchain-shar-extract.sh: allow TERM through into SDK installer env

 meta/classes/cmake.bbclass                        |   1 +
 meta/classes/image.bbclass                        |  19 +++-
 meta/classes/populate_sdk_ext.bbclass             |   7 +-
 meta/classes/sstate.bbclass                       |  29 +++++-
 meta/classes/testsdk.bbclass                      |  14 ++-
 meta/conf/bitbake.conf                            |   6 +-
 meta/files/ext-sdk-prepare.py                     | 117 ++++++++--------------
 meta/files/toolchain-shar-extract.sh              |   1 +
 meta/lib/oe/rootfs.py                             | 101 ++++++++++++++++---
 meta/lib/oe/sstatesig.py                          |  10 +-
 meta/recipes-devtools/cmake/cmake-native_3.5.2.bb |   2 +
 scripts/oe-buildenv-internal                      |   2 +-
 12 files changed, 204 insertions(+), 105 deletions(-)

-- 
2.5.5