[OE-core] Over-pruning the sstate cache

[Mike Crowe]

On Wednesday 13 April 2016 at 11:11:11 -0300, Otavio Salvador wrote:
> On Wed, Apr 13, 2016 at 11:01 AM, Mike Crowe <mac at mcrowe.com> wrote:
> > On Wednesday 13 April 2016 at 10:47:13 -0300, Otavio Salvador wrote:
> >> Couldn't this to be done, similar to the fetchall task?
> >
> > That's the sort of thing I was thinking of with my "all_populate_sysroot"
> > task in my original question.
> >
> > But, I've a working script using Richard's method now. I'll share it once
> > I've tested it a bit more.
> 
> Having it as a task allows it to avoid races and like, when running
> inside of the BitBake. It also can reuse the metadata database
> information and ought to be faster. Once you share the script I will
> see if I can rework it to a task and see how it looks.

I'm not really sure how much my (not very pretty) script will have in
common with a task-based solution.

I agree that integrating it into Bitbake would be better though. At the
moment generating the locked-sigs.inc file is by far the slowest part of
the operation though.

Since you've requested it, here's the not-really-tested-properly script.

Mike.
-------------- next part --------------
#!/usr/bin/python2
#
# touch-sstate
#
#  Update atime on sstate-cache files for locked-sigs.inc file.
#
# Mike Crowe <mac at mcrowe.com>
#
# If nothing needs recompiling when building an image then bitbake
# won't access the sstate files that are required to populate the
# sysroot. This means that they might be expired as being unused. To
# avoid the problem this script processes the "locked-sigs.inc" file
# generated by running "bitbake XXX --dump-signatures=none".

import glob
import os
import re
import sys
import time

# These variables may need tweaking when the build environment changes
native_lsb_string = "Debian-8"
target_vendor = ""
target_os = "oe-linux-gnueabi"
host_os = "linux"
sstate_version = "3"
native_arches = [ 'x86-64' ]

sstate_cache_dir = "sstate-cache/"

# The start of a variable declaration that tells us the arch
variable_re = re.compile('^SIGGEN_LOCKEDSIGS_t-([^ ]*) = \"\\\\$')

# A single hash for a task
sig_re = re.compile('^ *([^:]+):([^:]+):([^:]+) \\\\$')

# Ignore the trailing quote, the final line and blank lines
ignore_re = re.compile('^( *\")|(SIGGEN_LOCKEDSIGS_TYPES_)|$')

arch1 = "noarch"
arch2 = "noarch"

def TouchMatches(pattern, deadline):
    matches = glob.glob(pattern)
    for match in matches:
        # It's likely that we won't need to touch the file so it's
        # cheaper to do the stat to check first than to always open
        # the file for reading. The inode might be in cache but the
        # file contents not.
        if os.stat(match).st_atime < deadline:
            print match

            # We can't use os.utime for this since we may not own the
            # file.
            f = open(match, 'r')

            # We actually have to read something from the file in
            # order to update atime.
            f.read(1)
            f.close()
        
# In order to avoid unnecessarily thrashing the disk we only want to
# modify the atime of the file if the atime is more than a day ago.
# This matches the behaviour of the "relatime" mount option. We allow
# a bit of leeway in case we take a long time to run and the expiry
# script wants to expire stuff that's only more than a day old though.
deadline = time.time() - 23 * 3600

line_number = 0

for line in open("locked-sigs.inc", "r"):
    line_number += 1
    line = line.rstrip()
    variable_match = variable_re.match(line)
    if variable_match:
        arch = variable_match.group(1)
        arch_underscore = arch.replace("-", "_")
        if arch in native_arches:
            dir = sstate_cache_dir + native_lsb_string + "/"
            arch1 = arch_underscore + "-" + host_os
            arch2 = arch_underscore
        else:
            dir = sstate_cache_dir
            arch1 = arch_underscore + target_vendor + "-" + target_os
            arch2 = arch_underscore
    else:
        sig_match = sig_re.match(line)
        if sig_match:
            pn = sig_match.group(1)
            task = sig_match.group(2)[3:]
            sig = sig_match.group(3)

            arch_file = dir + sig[0:2] + "/sstate:" + pn + ":" + arch1 + ":*:*:" + arch2 + ":" + sstate_version + ":" + sig + "_" + task + ".tgz"
            noarch_file = dir + sig[0:2] + "/sstate:" + pn + "::*:*::" + sstate_version + ":" + sig + "_" + task + ".tgz"

            TouchMatches(arch_file, deadline)
            TouchMatches(noarch_file, deadline)
        elif ignore_re.match(line):
            pass
        else:
            sys.stderr.write('touch-sstate: Unparsed line %d\n' % line_number)
            sys.exit(1)