[OE-core] [PATCH v2 1/3] ccache.bbclass: enable max size setup for ccache dir

[Yannick GICQUEL]

2017-01-24 0:20 GMT+01:00 Andre McCurdy <armccurdy at gmail.com>:
> On Fri, Jan 20, 2017 at 8:47 AM, Yannick GICQUEL
> <yannick.gicquel at iot.bzh> wrote:
>> 2017-01-20 17:23 GMT+01:00 Yannick GICQUEL <yannick.gicquel at iot.bzh>:
>>> 2017-01-19 20:02 GMT+01:00 Andre McCurdy <armccurdy at gmail.com>:
>>>> On Thu, Jan 19, 2017 at 7:01 AM, Yannick Gicquel
>>>> <yannick.gicquel at iot.bzh> wrote:
>>>>> ccache directories are limited to 1G by default.
>>>>
>>>> 1G has been the default for a long time, so maybe it's time to propose
>>>> an upstream change to increase, e.g. to 10G, giving the size of the
>>>> webkitgtk build as justification?
>>>
>>> You are right, 1GB was the default cache size limit in old versions
>>> and it's up to 5GB since release 3.2 [1].
>>>
>>>> Since we build ccache-native within oe-core, any such patch could be
>>>> applied to our version of ccache right now (ie we don't need to wait
>>>> for it to be accepted and merged upstream).
>>>
>>> Current ccache.bbclass implementation is using the host ccache, so the
>>> default limit can be different depending on the version deployed on
>>> host distro. The point is that even if we use ccache-native, their
>>> still will be a default size limit set.
>>> Even if 5GB (or 10GB) limit should fit the webkitgtk cache size
>>> requirements, some other recipes can require more space to gain full
>>> benefit of the ccache (e.g. chromium from meta-browser) and for those
>>> it can be useful to tune the size.
>>>
>>> Does it make sense?
>
> Yes, it does. If 5GB isn't enough, then enabling compression of files
> in ccache ( https://ccache.samba.org/manual.html#_cache_compression )
> may be a better solution than removing the 5GB limit, but I do see the
> argument that since we don't place arbitrary limits on downloads or
> sstate directory sizes, why limit ccache?
>
> However, calling "ccache -M XXX" after every recipe's do_configure()
> still doesn't seem ideal. Firstly because the command will run many
> more times than necessary but also because, for users who over-ride

This add a call to ccache binary once per recipe build. In term of
execution time, IMHO it looks quite reasonable compared to the
drawback of a rebuild with limited cache size, but...

> CCACHE_DIR to try to share ccache files between builds, it will
> silently and permanently over-write any existing "max_size" option
> which they may have manually configured.

... I agree : the default behavior proposed can silently change some
user's custom setup, which may not be a good thing. (Same remark can
be done on the activation of compression, which may be kept to the
user's preferences)

>
> As an alternative, does simply exporting CCACHE_MAXSIZE=0 from
> ccache.bbclass work?

With recent ccache release, yes it does (v3.3.3 at least). But
depending on the host distro, it may not.
I had some experiment about using "ccache-native" binary, and I also
agree it can provides more flexibility : most (all?) of the
configuration variable can be tuned by exporting relevant environment
variable.

Updating ccache.bbclass to use ccache-native is not an easy task :
basically we could add a DEPENDS on "ccache-native" to all recipe
which inherit this class, but it can bring some dependency loops on
ccache recipe itself, which I do not know how to fix in a generic way.

About this patch serie, I propose to split it and keep this "max_size"
tune feature on a separate scope.

>
> (In general, I'm all in favour of improving ccache support in oe-core
> so it's good to see these changes being proposed and discussed).

You are welcome, thanks for your feedbacks.

>
>>> Yannick
>>>
>>> [1]: https://ccache.samba.org/releasenotes.html#_ccache_3_2
>>>
>>>>
>>>>> This patch enables the configuration of their limits, and as default
>>>>> location is TMPDIR, it proposes a size limit to "0" (unlimited).
>>>>>
>>>>> The setup can be overloaded in local.conf by setting
>>>>> CCACHE_MAX_SIZE to a custom value if needed.
>>>>>
>>>>> Signed-off-by: Yannick Gicquel <yannick.gicquel at iot.bzh>
>>>>> ---
>>>>>  meta/classes/ccache.bbclass | 8 ++++++++
>>>>>  1 file changed, 8 insertions(+)
>>>>>
>>>>> diff --git a/meta/classes/ccache.bbclass b/meta/classes/ccache.bbclass
>>>>> index 93fcaca..b6643a1 100644
>>>>> --- a/meta/classes/ccache.bbclass
>>>>> +++ b/meta/classes/ccache.bbclass
>>>>> @@ -1,6 +1,14 @@
>>>>>  CCACHE = "${@bb.utils.which(d.getVar('PATH'), 'ccache') and 'ccache '}"
>>>>>  export CCACHE_DIR ?= "${TMPDIR}/ccache/${MULTIMACH_HOST_SYS}/${PN}"
>>>>>  CCACHE_DISABLE[unexport] = "1"
>>>>> +CCACHE_MAX_SIZE ?= "0"
>>>>>
>>>>>  do_configure[dirs] =+ "${CCACHE_DIR}"
>>>>>  do_kernel_configme[dirs] =+ "${CCACHE_DIR}"
>>>>> +
>>>>> +ccache_init() {
>>>>> +    if [ -n "${CCACHE}" ]; then
>>>>> +        ${CCACHE} -M ${CCACHE_MAX_SIZE}
>>>>> +    fi
>>>>> +}
>>>>> +do_configure[postfuncs] += "ccache_init"
>>>>> --
>>>>> 1.9.1
>>>>>