[OE-core] [PATCH v2 1/3] ccache.bbclass: enable max size setup for ccache dir

[Andre McCurdy]

On Fri, Jan 20, 2017 at 8:47 AM, Yannick GICQUEL
<yannick.gicquel at iot.bzh> wrote:
> 2017-01-20 17:23 GMT+01:00 Yannick GICQUEL <yannick.gicquel at iot.bzh>:
>> 2017-01-19 20:02 GMT+01:00 Andre McCurdy <armccurdy at gmail.com>:
>>> On Thu, Jan 19, 2017 at 7:01 AM, Yannick Gicquel
>>> <yannick.gicquel at iot.bzh> wrote:
>>>> ccache directories are limited to 1G by default.
>>>
>>> 1G has been the default for a long time, so maybe it's time to propose
>>> an upstream change to increase, e.g. to 10G, giving the size of the
>>> webkitgtk build as justification?
>>
>> You are right, 1GB was the default cache size limit in old versions
>> and it's up to 5GB since release 3.2 [1].
>>
>>> Since we build ccache-native within oe-core, any such patch could be
>>> applied to our version of ccache right now (ie we don't need to wait
>>> for it to be accepted and merged upstream).
>>
>> Current ccache.bbclass implementation is using the host ccache, so the
>> default limit can be different depending on the version deployed on
>> host distro. The point is that even if we use ccache-native, their
>> still will be a default size limit set.
>> Even if 5GB (or 10GB) limit should fit the webkitgtk cache size
>> requirements, some other recipes can require more space to gain full
>> benefit of the ccache (e.g. chromium from meta-browser) and for those
>> it can be useful to tune the size.
>>
>> Does it make sense?

Yes, it does. If 5GB isn't enough, then enabling compression of files
in ccache ( https://ccache.samba.org/manual.html#_cache_compression )
may be a better solution than removing the 5GB limit, but I do see the
argument that since we don't place arbitrary limits on downloads or
sstate directory sizes, why limit ccache?

However, calling "ccache -M XXX" after every recipe's do_configure()
still doesn't seem ideal. Firstly because the command will run many
more times than necessary but also because, for users who over-ride
CCACHE_DIR to try to share ccache files between builds, it will
silently and permanently over-write any existing "max_size" option
which they may have manually configured.

As an alternative, does simply exporting CCACHE_MAXSIZE=0 from
ccache.bbclass work?

(In general, I'm all in favour of improving ccache support in oe-core
so it's good to see these changes being proposed and discussed).

>> Yannick
>>
>> [1]: https://ccache.samba.org/releasenotes.html#_ccache_3_2
>>
>>>
>>>> This patch enables the configuration of their limits, and as default
>>>> location is TMPDIR, it proposes a size limit to "0" (unlimited).
>>>>
>>>> The setup can be overloaded in local.conf by setting
>>>> CCACHE_MAX_SIZE to a custom value if needed.
>>>>
>>>> Signed-off-by: Yannick Gicquel <yannick.gicquel at iot.bzh>
>>>> ---
>>>>  meta/classes/ccache.bbclass | 8 ++++++++
>>>>  1 file changed, 8 insertions(+)
>>>>
>>>> diff --git a/meta/classes/ccache.bbclass b/meta/classes/ccache.bbclass
>>>> index 93fcaca..b6643a1 100644
>>>> --- a/meta/classes/ccache.bbclass
>>>> +++ b/meta/classes/ccache.bbclass
>>>> @@ -1,6 +1,14 @@
>>>>  CCACHE = "${@bb.utils.which(d.getVar('PATH'), 'ccache') and 'ccache '}"
>>>>  export CCACHE_DIR ?= "${TMPDIR}/ccache/${MULTIMACH_HOST_SYS}/${PN}"
>>>>  CCACHE_DISABLE[unexport] = "1"
>>>> +CCACHE_MAX_SIZE ?= "0"
>>>>
>>>>  do_configure[dirs] =+ "${CCACHE_DIR}"
>>>>  do_kernel_configme[dirs] =+ "${CCACHE_DIR}"
>>>> +
>>>> +ccache_init() {
>>>> +    if [ -n "${CCACHE}" ]; then
>>>> +        ${CCACHE} -M ${CCACHE_MAX_SIZE}
>>>> +    fi
>>>> +}
>>>> +do_configure[postfuncs] += "ccache_init"
>>>> --
>>>> 1.9.1
>>>>