[oe] [oe-commits] Richard Purdie : sstate.bbclass: Detect overwriting files in sstate controlled directories

[Andreas Müller]

On Fri, Aug 31, 2012 at 2:38 AM,  <git at git.openembedded.org> wrote:
> Module: openembedded-core.git
> Branch: master
> Commit: 1f1b6f93d6b7aa8c9bd9bb5b1826997812e36932
> URL:    http://git.openembedded.org/?p=openembedded-core.git&a=commit;h=1f1b6f93d6b7aa8c9bd9bb5b1826997812e36932
>
> Author: Richard Purdie <richard.purdie at linuxfoundation.org>
> Date:   Sat Aug 25 15:48:14 2012 +0000
>
> sstate.bbclass: Detect overwriting files in sstate controlled directories
>
> We have a potential problem where two sstate packages try and touch the same
> file. This adds code which will print a warning whenever this happens.
>
> The implementation does but by maintaining a master file list and comparing
> file accesses against this. There are a number of places we have duplicate
> accesses which are harmless, mostly in the deploy directory so these
> are whitelisted.
>
> For now the code prints warnings, this could be strengthened in future to
> become error messages. Whilst working on this code on and off over the past
> few months various issues were uncovered, some serious.
>
> [YOCTO #238]
>
> Signed-off-by: Richard Purdie <richard.purdie at linuxfoundation.org>
>
> ---
>
>  meta/classes/sstate.bbclass |   45 ++++++++++++++++++++++++++++++++++++++++++-
>  1 files changed, 44 insertions(+), 1 deletions(-)
>
> diff --git a/meta/classes/sstate.bbclass b/meta/classes/sstate.bbclass
> index 6762e32..02594a7 100644
> --- a/meta/classes/sstate.bbclass
> +++ b/meta/classes/sstate.bbclass
> @@ -3,6 +3,7 @@ SSTATE_VERSION = "2"
>  SSTATE_MANIFESTS ?= "${TMPDIR}/sstate-control"
>  SSTATE_MANFILEBASE = "${SSTATE_MANIFESTS}/manifest-${SSTATE_MANMACH}-"
>  SSTATE_MANFILEPREFIX = "${SSTATE_MANFILEBASE}${PN}"
> +SSTATE_MASTERMANIFEST = "${SSTATE_MANIFESTS}/master.list"
>
>  def generate_sstatefn(spec, hash, d):
>      if not hash:
> @@ -17,6 +18,7 @@ SSTATE_EXTRAPATH   = ""
>  SSTATE_EXTRAPATHWILDCARD = ""
>  SSTATE_PATHSPEC   = "${SSTATE_DIR}/${SSTATE_EXTRAPATHWILDCARD}*/${SSTATE_PKGSPEC}"
>
> +SSTATE_DUPWHITELIST = "${DEPLOY_DIR_IMAGE}/ ${DEPLOY_DIR}/licenses/ ${DEPLOY_DIR_IPK}/all/ ${DEPLOY_DIR_RPM}/all ${DEPLOY_DIR_DEB}/all/"
>
>  SSTATE_SCAN_FILES ?= "*.la *-config *_config"
>  SSTATE_SCAN_CMD ?= 'find ${SSTATE_BUILDDIR} \( -name "${@"\" -o -name \"".join(d.getVar("SSTATE_SCAN_FILES", True).split())}" \) -type f'
> @@ -125,7 +127,6 @@ def sstate_install(ss, d):
>          locks.append(bb.utils.lockfile(lock))
>
>      for state in ss['dirs']:
> -        oe.path.copytree(state[1], state[2])
>          for walkroot, dirs, files in os.walk(state[1]):
>              bb.debug(2, "Staging files from %s to %s" % (state[1], state[2]))
>              for file in files:
> @@ -140,9 +141,35 @@ def sstate_install(ss, d):
>                  if not dstdir.endswith("/"):
>                      dstdir = dstdir + "/"
>                  shareddirs.append(dstdir)
> +
> +    # Check the file list for conflicts against the master manifest
> +    mastermanifest = d.getVar("SSTATE_MASTERMANIFEST", True)
> +    whitelist = d.getVar("SSTATE_DUPWHITELIST", True)
> +    lock = bb.utils.lockfile(mastermanifest + ".lock")
> +    fileslist = [line.strip() for line in open(mastermanifest)]
> +    bb.utils.unlockfile(lock)
> +    match = []
> +    for f in sharedfiles:
> +        if f in fileslist:
> +            realmatch = True
> +            for w in whitelist:
> +                if f.startswith(w):
> +                    realmatch = False
> +                    break
> +            if realmatch:
> +                match.append(f)
> +    if match:
> +        bb.warn("The recipe is trying to install files into a shared area when those files already exist. Those files are:\n   %s" % "\n   ".join(match))
> +
> +    # Write out the manifest and add to the task's manifest file
> +    lock = bb.utils.lockfile(mastermanifest + ".lock")
> +    mf = open(mastermanifest, "a")
>      f = open(manifest, "w")
>      for file in sharedfiles:
> +        mf.write(file + "\n")
>          f.write(file + "\n")
> +    bb.utils.unlockfile(lock)
> +
>      # We want to ensure that directories appear at the end of the manifest
>      # so that when we test to see if they should be deleted any contents
>      # added by the task will have been removed first.
> @@ -152,6 +179,10 @@ def sstate_install(ss, d):
>          f.write(di + "\n")
>      f.close()
>
> +    # Run the actual file install
> +    for state in ss['dirs']:
> +        oe.path.copytree(state[1], state[2])
> +
>      for postinst in (d.getVar('SSTATEPOSTINSTFUNCS', True) or '').split():
>          bb.build.exec_func(postinst, d)
>
> @@ -268,6 +299,18 @@ def sstate_clean_manifest(manifest, d):
>          except OSError:
>              pass
>
> +    # Remove the entries from the master manifest
> +    mastermanifest = d.getVar("SSTATE_MASTERMANIFEST", True)
> +    lock = bb.utils.lockfile(mastermanifest + ".lock")
> +    mf = open(mastermanifest + ".new", "w")
> +    for line in open(mastermanifest, "r"):
> +        if not line or line in entries:
> +            continue
> +        mf.write(line)
> +    mf.close()
> +    os.rename(mastermanifest + ".new", mastermanifest)
> +    bb.utils.unlockfile(lock)
> +
>      oe.path.remove(manifest)
>
>  def sstate_clean(ss, d):
>
>
I think it started with this one: I often do play around with
different kernel versions/sources to bisect when a bug was born. Doing
so I now get wallpapers with

WARNING: The recipe is trying to install files into a shared area when
those files already exist. Those files are:
   /home/Superandy/tmp/oe-core-eglibc/sysroots/overo/usr/src/kernel/Module.symvers
   /home/Superandy/tmp/oe-core-eglibc/sysroots/overo/usr/src/kernel/CREDITS
   /home/Superandy/tmp/oe-core-eglibc/sysroots/overo/usr/src/kernel/kernel-abiversion
   /home/Superandy/tmp/oe-core-eglibc/sysroots/overo/usr/src/kernel/Makefile
   /home/Superandy/tmp/oe-core-eglibc/sysroots/overo/usr/src/kernel/kernel-image-name
   /home/Superandy/tmp/oe-core-eglibc/sysroots/overo/usr/src/kernel/uImage
   /home/Superandy/tmp/oe-core-eglibc/sysroots/overo/usr/src/kernel/README
   /home/Superandy/tmp/oe-core-eglibc/sysroots/overo/usr/src/kernel/COPYING
   /home/Superandy/tmp/oe-core-eglibc/sysroots/overo/usr/src/kernel/REPORTING-BUGS
   /home/Superandy/tmp/oe-core-eglibc/sysroots/overo/usr/src/kernel/System.map-3.5.0
...
(log.do_populate_sysroot is 2.8 MB)

Would it cause huge effort to add a variable which is in scope of
recipe avoiding this warning because it scrolls out all other
interesting messages?

Andreas