[bitbake-devel] request for comments - dynamic task selection

[Richard Purdie]

On Thu, 2014-02-20 at 19:17 +0000, Damian, Alexandru wrote:
> I have a RFC patch that attempts to improve build performance by
> dynamically selecting the next-to-run task based on  currently running
> tasks. The general idea is if we overload the network with too many
> fetch tasks, it's better to start doing unpack tasks while the disks
> idle instead of starting new fetch tasks. The concept can be generally
> applied, e.g. it's better to schedule package tasks in parallel with
> compile tasks instead of running all the compile tasks first and then
> all package tasks.
>
> This patch just looks at currently running tasks and selects the next
> available task that hasn't a similar-type task already running.
>
> I'm seeing a roughly 2% build time reduction when building the
> virtual:native components, with just 26 tasks reordered out of 244
> tasks executed. I am attaching a toaster.sqlite file that captures the
> test - build no.1 is done with the origin/master source, and build no.
> 2 is done with this patch applied.

The trouble with scheduler changes is that they're hard to evaluate as
the benefits may appear on one machine but cause issues on another.

Firstly, I'd like to see this as a new scheduler, not changing the
existing one. I'd then like Stefan to try this against our standard
performance tests, see what it does to our various benchmarks there.

I'd also note that do_fetch is not the best task for optimisation since
most people have relatively up to date source directories locally at any
point after their first build.

Cheers,

Richard