[oe] Issues with .list files

[Phil Blundell]

On Thu, 2010-02-04 at 19:24 -0800, Michael Morrell wrote:
> If I understand the algorithm in package_do_shlibs correctly, each package creates a .list file which is a list of all the shared libraries provided by that package.  It also generates a list of all the libraries that it needs and looks for those in .list files created by other packages so it can create a runtime dependency against that package.
> 
> The way it looks for .list files is to simply scan all *.list files in the directory.  This has two problems that I can see.
> 
> First, it is inefficient.  It should only have to look through .list files creates by packages created by bb files which were DEPENDed on by this bb file.

That's an interesting point.  I think this probably is true, although it
would lead to some issues with packages that currently have incomplete
DEPENDS (or which rely on the historically-transitive behaviour of
DEPENDS for correct operation); there are probably quite a lot of these.
If you're going to make this change then I think you would also need to
change the current "couldn't find shlib provider for..." warning into a
fatal error.

Do you have any feel for how much time would actually be saved by
eliminating the unnecessary scan of other .list files?

> Second, when a bitbake run is done with lots of parallelism, it is possible that it tries to open a nonexistent .list file (it was there when it did the os.listdir call, but that package later removed it before it recreates it), causing a crash here.  We are seeing such crashes reasonably often.

This one is definitely just a bug (and, I think, unrelated to the first
issue).  Any time you create or modify a file in a live part of staging,
it needs to be done atomically and it sounds like package.bbclass is
currently failing to do that.  Feel free to make a patch, or I can look
at this instead if you want.

p.