[OE-core] [master][PATCH] gen-lockedsig-cache: Replace glob lookup with hash to filename lookup

[richard.purdie at linuxfoundation.org]

On Wed, 2019-10-02 at 10:56 -0400, Konrad Scherer wrote:
> On 10/2/19 6:17 AM, Richard Purdie wrote:
> > My benchmark before was seeing it spend over 30 minutes in bitbake
> > core-image-minimal:do_populate_sdk_ext on an otherwise idle
> > autobuilder
> > cluster/NAS (35 minutes from a clean tmpdir).
> > 
> > With the patch applied and my above tweak, I saw:
> > 
> > real	6m58.120s
> > 
> > and I'd note this was with a full build running on the other
> > workers so
> > the NAS was under load. I could try and get an exact time for the
> > above
> > but didn't really see the point in spending another 30 minutes on
> > it.
> > 
> > This is the time for the whole SDK image, not just the time this
> > script
> > takes but its enough for me to say its a vast improvement! :)
> 
> Great! It was frustrating to develop the code without a good
> reproducer.

Unfortunately I think we may be celebrating prematurely. On an
otherwise unloaded system this looked ok however in real world builds
the sdk creation is still excruciatingly slow.

We don't even need to use the script to understand why:

pokybuild at debian9-ty-2:~$ time ls /srv/autobuilder/autobuilder.yoctoproject.org/pub/sstate/c6

real	9m3.140s
user	0m4.432s
sys	0m5.164s

So we'd expect the script to take 2*9*255 minutes :(

(214054 files in there incidentally, I got 1m49 the second try)

I've cc'd our sysadmin.

> > Konrad: Mind if I squash in the above tweaks?
> 
> Not at all. Sorry that you had to spend time debugging my code. That
> sha extraction code should have been more robust.
> 
> > Also, the new code is a bit too chatty and leads to a lot of log
> > output, we'll need to tweak that too.
> 
> Please do.

I've queued such a patch in master-next and am testing, its an
improvement but I'm worried its not resolving the problem.

Options are to prune the sstate cache, or to teach the code to generate
the full filenames (would mean refactoring)...

We could give you access to the autobuilder to help reproduce the
problem but I think its clear where the delays are...

Cheers,

Richard