[OE-core] [master][PATCH] gen-lockedsig-cache: Replace glob lookup with hash to filename lookup

[Konrad Scherer]

On 10/2/19 6:17 AM, Richard Purdie wrote:
> On Fri, 2019-09-27 at 14:56 -0400, Konrad Scherer wrote:
>> From: Konrad Scherer <Konrad.Scherer at windriver.com>
>>
>> Using the glob function to map signatures to sstate files is very
>> slow
>> when the sstate is large and accessed over nfs. The lookup now only
>> loads the necessary prefixes and doesn't use glob as all.
>>
>> Unfortunately I don't have access to the systems where the
>> performance
>> isse was noticed and on my test system the glob is fast enough that
>> the performance numbers aren't useful. I could verify that file list
>> returned by the new code is the same.
>>
>> [YOCTO #13539]
>>
>> Signed-off-by: Konrad Scherer <Konrad.Scherer at windriver.com>
>> ---
>>   meta/lib/oe/copy_buildsystem.py |  3 ++-
>>   scripts/gen-lockedsig-cache     | 44 +++++++++++++++++++++++++++++
>> ----
>>   2 files changed, 41 insertions(+), 6 deletions(-)
> 
> Thanks for this, its hugely appreciated!
> 
> I aimed to do some profile measurements to show the difference in speed
> but when testing it failed with:
> 
> Filtering out gdb-cross-x86_64:do_patch
> Filtering out gdb-cross-x86_64:do_prepare_recipe_sysroot
> Filtering out gdb-cross-x86_64:do_unpack
> Gathering file list
> Traceback (most recent call last):
>    File "/home/pokybuild/yocto-worker/qa-extras/build/scripts/gen-lockedsig-cache", line 77, in <module>
>      sstate_content_cache[prefix] = build_sha_cache(prefix)
>    File "/home/pokybuild/yocto-worker/qa-extras/build/scripts/gen-lockedsig-cache", line 39, in build_sha_cache
>      map_sha_to_files(sstate_dir, prefix, sha_map)
>    File "/home/pokybuild/yocto-worker/qa-extras/build/scripts/gen-lockedsig-cache", line 29, in map_sha_to_files
>      sha = extract_sha(f)
>    File "/home/pokybuild/yocto-worker/qa-extras/build/scripts/gen-lockedsig-cache", line 21, in extract_sha
>      return filename.split(':')[7].split('_')[0]
> IndexError: list index out of range
> 
> and then when I fixed that by ignoring files which don't match the pattern:
> 
> Gathering file list
> Traceback (most recent call last):
>    File "/home/pokybuild/yocto-worker/qa-extras/build/scripts/gen-lockedsig-cache", line 80, in <module>
>      sstate_content_cache[prefix] = build_sha_cache(prefix)
>    File "/home/pokybuild/yocto-worker/qa-extras/build/scripts/gen-lockedsig-cache", line 45, in build_sha_cache
>      map_sha_to_files(native_sstate_dir, prefix, sha_map)
>    File "/home/pokybuild/yocto-worker/qa-extras/build/scripts/gen-lockedsig-cache", line 27, in map_sha_to_files
>      sstate_files = os.listdir(sstate_prefix_path)
> FileNotFoundError: [Errno 2] No such file or directory: '/srv/autobuilder/autobuilder.yoctoproject.org/pub/sstateuniversal/b6/'
> 
> 
> I therefore added:
> 
> diff --git a/scripts/gen-lockedsig-cache b/scripts/gen-lockedsig-cache
> index ae5e09d89f..48cb67112f 100755
> --- a/scripts/gen-lockedsig-cache
> +++ b/scripts/gen-lockedsig-cache
> @@ -26,10 +26,13 @@ def map_sha_to_files(dir_, prefix, sha_map):
>       sstate_prefix_path = dir_ + '/' + prefix + '/'
>       sstate_files = os.listdir(sstate_prefix_path)
>       for f in sstate_files:
> -        sha = extract_sha(f)
> -        if sha not in sha_map:
> -            sha_map[sha] = []
> -        sha_map[sha].append(sstate_prefix_path + f)
> +        try:
> +            sha = extract_sha(f)
> +            if sha not in sha_map:
> +                sha_map[sha] = []
> +            sha_map[sha].append(sstate_prefix_path + f)
> +        except IndexError:
> +            continue
>   
>   # given a prefix build a map of hash to list of files
>   def build_sha_cache(prefix):
> @@ -38,7 +41,7 @@ def build_sha_cache(prefix):
>       sstate_dir = sys.argv[2]
>       map_sha_to_files(sstate_dir, prefix, sha_map)
>   
> -    native_sstate_dir = sys.argv[2] + sys.argv[4]
> +    native_sstate_dir = sys.argv[2] + '/' + sys.argv[4]
>       map_sha_to_files(native_sstate_dir, prefix, sha_map)
>   
>       return sha_map
> 
> 
> My benchmark before was seeing it spend over 30 minutes in bitbake
> core-image-minimal:do_populate_sdk_ext on an otherwise idle autobuilder
> cluster/NAS (35 minutes from a clean tmpdir).
> 
> With the patch applied and my above tweak, I saw:
> 
> real	6m58.120s
> 
> and I'd note this was with a full build running on the other workers so
> the NAS was under load. I could try and get an exact time for the above
> but didn't really see the point in spending another 30 minutes on it.
> 
> This is the time for the whole SDK image, not just the time this script
> takes but its enough for me to say its a vast improvement! :)

Great! It was frustrating to develop the code without a good reproducer.

> Konrad: Mind if I squash in the above tweaks?

Not at all. Sorry that you had to spend time debugging my code. That sha extraction code should have been more robust.

> Also, the new code is a bit too chatty and leads to a lot of log
> output, we'll need to tweak that too.

Please do.

-- 
Konrad Scherer, MTS, Linux Products Group, Wind River