[OE-core] package: Add cachedpath optimisation
Richard Purdie
richard.purdie at linuxfoundation.org
Mon Mar 18 16:53:35 UTC 2013
On Thu, 2013-03-14 at 20:02 -0700, Chris Larson wrote:
>
> On Thu, Mar 14, 2013 at 6:57 PM, Richard Purdie
> <richard.purdie at linuxfoundation.org> wrote:
> Currently, various standard library operations like os.walk(),
> os.path.isdir() and os.path.islink() each call stat or lstat
> which
> involves a syscall into the kernel. There is no caching since
> they could
> conceivably have changed on disk. The result is that for
> something like
> the do_package task of the kernel we're spending over two
> minutes making
> 868,000 individual stat calls for 23,000 files. This is
> suboptimal.
>
> This patch adds lib/oe/cachedpath.py which are a set of
> replacement
> functions for these operations which use cached stat data
> rather than
> hitting the kernel each time. It gives a nice performance
> improvement
> halving the build time of the kernel do_package.
>
> Signed-off-by: Richard Purdie
> <richard.purdie at linuxfoundation.org>
>
> Have you considered having the initcache() return a cache object,
> either with the rest as methods on it, or pass that in? I think it'd
> be substantially cleaner than messing with globals, particularly given
> with globals we tend to have issues with cache lifetime and
> invalidation — if the cache was returned, it would go away when the
> object gets collected.
Agreed. Due to the way package.bbclass is structured, we currently need
a global there but that is no reason we shouldn't have a sensible
class/object structure for the new code. I've sent out a v2 which
hopefully does better in that regard.
Cheers,
Richard
>
More information about the Openembedded-core
mailing list