[OE-core] [PATCH] cmake.bbclass: make cleaning of build directory on configure optional

[Douglas Royds]

On 18/01/19 12:59 PM, Randy MacLeod wrote:

> On 1/16/19 7:38 PM, Douglas Royds wrote:
>> On 17/01/19 12:01 PM, Richard Purdie wrote:
>>
>>> On Thu, 2019-01-17 at 11:51 +1300, Douglas Royds wrote:
>>>> From: Sam Nobs <samuel.nobs at taitradio.com>
>>>>
>>>> The do_configure task is sensitive to changes in recipes and
>>>> configuration files.
>>>> This can be time consuming because cmake.bbclass deletes the ${B}
>>>> directory
>>>> at the beginning of the do_configure task. CMake figures
>>>> out what to do when it's run again, so unless your cmake files are
>>>> buggy, there shouldn't
>>>> be any need to erase the build directory and start afresh.
>>>>
>>>> This change allows you to turn this behaviour off, either globally or
>>>> on a per-recipe basis, e.g.
>>>>
>>>> OECMAKE_CLEAN_BUILDDIR_ON_CONFIGURE_pn-yourrecipe = ""
>>>>
>>>> Signed-off-by: Sam Nobs <samuel.nobs at taitradio.com>
>>>> ---
>>>>   meta/classes/cmake.bbclass | 23 ++++++++++++++++++-----
>>>>   1 file changed, 18 insertions(+), 5 deletions(-)
>>> This may give a speed boost for some cases but it does come at a
>>> potential risk of determinism issues.
>>>
>>> I appreciate you've made it configurable but that actually further
>>> concerns me as we now have two code paths which potentially both have
>>> to be tested.
>>>
>>> Is this change really worth it?
>>>
>>> Cheers,
>>>
>>> Richard
>>
>>
>> For us, the benefit is beyond question - on our proprietary C++ 
>> component, the compile from scratch after a configure is punishing, 
>> 10s of minutes per MACHINE.
>>
>> We are counting on binary reproducibility to flush out any 
>> determinism issues with the CMake code, that is, we will be comparing 
>> our incremental builds with our clean builds. Note also that we are 
>> deleting all the generated CMake artefacts, but leaving the object 
>> files behind. CMake will force a complete rebuild should there be any 
>> change in the CFLAGS and friends.
>
> Douglas,
>
> What workflow are you trying to optimize?
>
> If you're strictly changing existing source files
> then you shouldn't need to run configure again so I'm confused.
>
> Also, have you tried using ccache as per:
>    d014c8c11f cmake.bbclass: Make it work with ccache
> and if so why doesn't that work for you?
>
> Thanks,
> ../Randy


We are not only changing existing source files, we routinely add or 
rename files or entire directories. This is a current product, very much 
under active development. You're absolutely right though, we should 
rarely need to reconfigure, indeed a compile (normal behaviour under 
externalsrc) will cause CMake to notice (under the bonnet) that it needs 
to rebuild_cache, which is all we need.

The problem is that reconfiguration happens quite frequently for other 
reasons, notably changes in any library that our proprietary components 
depend on (some of them also proprietary), or non-whitelisted changes in 
OE variables or functions.

We use icecc, which gives similar performance to ccache, it's quite 
impressive. I cheerfully concede that we could use ccache to reduce the 
load on our icecc network, and for a further marginal improvement. There 
is however a great deal of linking to be done, and regrettably our 
non-trivial code base might have less than ideal build dependencies 
within it (within the cmake code), leading to less than perfect 
parallelism in compilation and linking ... Unnecessary rebuilding from 
scratch imposes an unwelcome burden on our developers, and quite some 
muttering in the ranks. I wish to feel safe making my cup of coffee.

None of this is a problem if we don't delete object files at 
configure-time. I see no benefit from doing so. Our proposed patch 
deletes only CMake artefacts, which should be adequate in all cases.

Hope this clarifies things,
Douglas.