[bitbake-devel] [PATCH 2/2] parse: don't add attempted files to dependencies

[Richard Purdie]

On Thu, 2018-01-18 at 16:57 +0800, Robert Yang wrote:
> The attempts are the files that it tries to search but don't exist,
> it searches the file in BBPATH until find it, so the attempts might
> be very long when there are many layers, which causes bb_cache.dat
> very big (I have 54 layers, the bb_cache.dat can be 119M, and even
> much bigger depends on the order of BBLAYERS).
> 
> Here is the testing data of 54 layers before and after the patch.
> 
>                     Before          After
> Parsing time        33s             9s
> Loading time        30s             5s
> Cache size          119M            20M
> 
> The time and size can be more or less depends on the order of
> BBLAYERS before the patch.
> 
> I checked the code, but didn't find why we need the attempts as
> dependencies, the one that I can think of is the file doesn't exist
> when parsing, and added to the attempts location after parsing, but
> the parseBaseConfiguration() can detect and handle this well (and
> will cause reparse in such a case), so I think that we can safely
> remove them from dependencies since they have side effects.

If you have a memory resident bitbake you need to know which files to
watch for with inotify. If a file is created somewhere which would have
been parsed had it existed when we originally parsed, we need to
invalidate the cache and reparse. This needs to happen for the metadata
itself, not just the base configuration.

I therefore strongly suspect we do need these unfortunately. There may
be some ways we can condense the dependency information though to
reduce the cache load times.

Cheers,

Richard