[OE-core] [PATCH 1/1] sstate.bbclass: Improve sstate_installpkg performance
Richard Purdie
richard.purdie at linuxfoundation.org
Fri May 11 00:17:02 UTC 2012
On Thu, 2012-05-10 at 18:13 -0500, Mark Hatle wrote:
> In a pathological case, lots of files to process, the sstate_installpkg
> performance was very poor. It interated over each file and ran 3
> individual sed commands per file. Changing this to keep iterating
> but running only a single command took about 1/3 time time.
>
> However, when looking at the corresponding sstate_hardcode_path
> function, it was clear we could optimize this further.
>
> Using the same encoding logic to specify only the minimumal sed
> operation necessary, and using xargs to avoid the os.system call the
> install step was able to be performed in 13% of the original time.
>
> Example timing numbers for perl:
>
> 3m7s original code
> 1m20s single sed, but interating
> 0m26s using xargs and limited sed
>
> Signed-off-by: Mark Hatle <mark.hatle at windriver.com>
> ---
> meta/classes/sstate.bbclass | 28 +++++++++++++++++++++-------
> 1 files changed, 21 insertions(+), 7 deletions(-)
>
> diff --git a/meta/classes/sstate.bbclass b/meta/classes/sstate.bbclass
> index a8c98e5..ad7d121 100644
> --- a/meta/classes/sstate.bbclass
> +++ b/meta/classes/sstate.bbclass
> @@ -174,18 +174,29 @@ def sstate_installpkg(ss, d):
> bb.build.exec_func('sstate_unpack_package', d)
>
> # Fixup hardcoded paths
> + #
> + # Note: The logic below must match the reverse logic in
> + # sstate_hardcode_path(d)
> +
> fixmefn = sstateinst + "fixmepath"
> if os.path.isfile(fixmefn):
> staging = d.getVar('STAGING_DIR', True)
> staging_target = d.getVar('STAGING_DIR_TARGET', True)
> staging_host = d.getVar('STAGING_DIR_HOST', True)
> - fixmefd = open(fixmefn, "r")
> - fixmefiles = fixmefd.readlines()
> - fixmefd.close()
> - for file in fixmefiles:
> - os.system("sed -i -e s:FIXMESTAGINGDIRTARGET:%s:g %s" % (staging_target, sstateinst + file))
> - os.system("sed -i -e s:FIXMESTAGINGDIRHOST:%s:g %s" % (staging_host, sstateinst + file))
> - os.system("sed -i -e s:FIXMESTAGINGDIR:%s:g %s" % (staging, sstateinst + file))
> +
> + if bb.data.inherits_class('native', d) or bb.data.inherits_class('nativesdk', d) or bb.data.inherits_class('crosssdk', d) or bb.data.inherits_class('cross-canadian', d):
> + sstate_sed_cmd = "sed -i -e 's:FIXMESTAGINGDIR:%s:g'" % (staging)
> + elif bb.data.inherits_class('cross', d):
> + sstate_sed_cmd = "sed -i -e 's:FIXMESTAGINGDIRTARGET:%s:g; s:FIXMESTAGINGDIR:%s:g'" % (staging_target, staging)
> + else:
> + sstate_sed_cmd = "sed -i -e 's:FIXMESTAGINGDIRHOST:%s:g'" % (staging_host)
> +
> + # Add sstateinst to each filename in fixmepath, use xargs to efficiently call sed
> + sstate_hardcode_cmd = "sed -e 's:^:%s:g' %s | xargs %s" % (sstateinst, fixmefn, sstate_sed_cmd)
> +
> + print "Replacing fixme paths in sstate package: %s" % (sstate_hardcode_cmd)
> + os.system(sstate_hardcode_cmd)
> +
> # Need to remove this or we'd copy it into the target directory and may
> # conflict with another writer
> os.remove(fixmefn)
> @@ -300,6 +311,9 @@ python sstate_cleanall() {
> def sstate_hardcode_path(d):
> # Need to remove hardcoded paths and fix these when we install the
> # staging packages.
> + #
> + # Note: the logic in this function needs to match the reverse logic
> + # in sstate_installpkg(ss, d)
>
> staging = d.getVar('STAGING_DIR', True)
> staging_target = d.getVar('STAGING_DIR_TARGET', True)
I was thinking this looked familiar, it was the other half of the
problem we originally solved:
http://git.yoctoproject.org/cgit.cgi/poky/commit/meta/classes/sstate.bbclass?id=db94ad4cf32d8ce3e97a8287d1c89a58d008a142
:)
Cheers,
Richard
More information about the Openembedded-core
mailing list