[oe] RFC: How to improve packaged-staging and setscene

[Richard Purdie]

There are still some things bothering me about packaged-staging. As
anyone who's looked into what it does behind the scenes will agree, its
hard to follow what it does and the paths are insane.

Why? The simple reason is that its trying to capture the output from all
the tasks in one step. The tricks it plays with the stamp files are also
evil.

My proposal is to do something a bit more radical, to capture the output
on a per task basis. The structure would therefore be the following:

# Set the name of the setscene function for this task
do_foo[setscene] = "do_foo_setscene"
# The name of functions to call after do_doo completes successfully
do_foo[postfuncs] = "do_foo_packager"
do_foo_setscene () {
	# Call a shared function stating the task checksum and seeing 
	# whether a prebuilt binary is available
	# If yes, decompress to WORKDIR, install it using 
        # do_foo_installer() (optional step useful for say staging), exit 0
	# If no, exit 1
}
do_foo () {
	# Some task which generates output to ${FOO_OUT}

	# As the last step install it
	do_foo_installer()
}
do_foo_packager () {
	# Call a shared function to package up the tasks output
}


The workflow would then be that runqueue builds a list of tasks to
execute as normal. Before it does the usual processing, it inverts the
list starting at the targets and looks for tasks with setscene
functions. If it finds them, it runs them. If they succeed it marks that
part of the task list as complete (including all dependencies of that
task).

In real terms this means that for say an image target foo-image with a
fully populated cache, if would search until it found the setscene
functions for the do_package_write_rpm tasks. These would be found and
run, which would make the .rpm files available. Also, binaries needed
for image creation would be found, their do_populate_staging setscene
tasks would be run and the binaries would be come available. The image
would then build.

If you then enabled debian packaging with a cache that wasn't full and
ran the image again, the setscene tasks for each package's do_package
function would run. The resulting directory would be packaged by debain
packaging and then an image would be build from it.


This all sounds ideal but it may not be quite this simple. Its hard to
tell which recursive dependencies to follow and which are met by the
results of the setscene function. In the above example we need the
write_rpm function to run for all dependent packages. I suspect the
existing code dependency code will result in too many tasks being run
(all package_write_rpm functions even for build utilities which we
wouldn't need packaged for the image?). Regardless, I suspect the best
way forward is to experiment.

The good side of this is that the bitbake implementation is relatively
simple and backwards compatible.

Questions/further thoughts/objections?

Cheers,

Richard