[OE-core] [PATCH 1/4] sstate: allow specifying indirect dependencies to exclude from sysroot

[André Draszik]

On Tue, 2018-05-22 at 13:36 +0100, Richard Purdie wrote:
> On Tue, 2018-05-22 at 13:25 +0100, André Draszik wrote:
> > diff --git a/meta/classes/sstate.bbclass
> > b/meta/classes/sstate.bbclass
> > index 1a95f8f2b9..e5b86ad705 100644
> > --- a/meta/classes/sstate.bbclass
> > +++ b/meta/classes/sstate.bbclass
> > @@ -897,6 +897,7 @@ def setscene_depvalid(task, taskdependees,
> > notneeded, d, log=None):
> >      # task is included in taskdependees too
> >      # Return - False - We need this dependency
> >      #        - True - We can skip this dependency
> > +    import re
> >  
> >      def logit(msg, log):
> >          if log is not None:
> > @@ -957,6 +958,18 @@ def setscene_depvalid(task, taskdependees,
> > notneeded, d, log=None):
> >              # Nothing need depend on libc-initial/gcc-cross-initial
> >              if "-initial" in taskdependees[task][0]:
> >                  continue
> > +            # Allow excluding certain recursive dependencies. If a
> > recipe needs it should add a
> > +            # specific dependency itself, rather than relying on one
> > of its dependees to pull
> > +            # them in.
> > +            # See also http://lists.openembedded.org/pipermail/opene
> > mbedded-core/2018-January/146324.html
> > +            not_needed = False
> > +            for excl in (d.getVar('SSTATE_EXCLUDEDEPS_SYSROOT') or
> > "").split():
> > +                if re.match(excl.split('->', 1)[0],
> > taskdependees[dep][0]):
> > +                    if re.match(excl.split('->', 1)[1],
> > taskdependees[task][0]):
> > +                        not_needed = True
> > +                        break
> > +            if not_needed:
> > +                continue
> >              # For meta-extsdk-toolchain we want all sysroot
> > dependencies
> >              if taskdependees[dep][0] == 'meta-extsdk-toolchain':
> >                  return False
> 
> Have you looked at the performance impact of this change? It looks like
> it will be compiling the regexp each time inside a tight loop which
> gets called once per task dependency which will show up significantly
> on the performance chart.

I guess I'll have to do some more tests or so, but the change doesn't seem
to affect performance noticeably using the following regex
grep -wE '(match|setscene_depvalid)' profile.log.processed:

before my change:
   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
   472862    0.096    0.000    0.096    0.000 {method 'match' of '_sre.SRE_Pattern' objects}
   150956    0.040    0.000    0.106    0.000 /usr/lib/python3.6/re.py:169(match)
      112    0.004    0.000    0.044    0.000 .../poky/meta/classes/sstate.bbclass:895(setscene_depvalid)
     4602    0.001    0.000    0.001    0.000 /usr/lib/python3.6/sre_parse.py:248(match)

after my change:
   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
   473139    0.095    0.000    0.095    0.000 {method 'match' of '_sre.SRE_Pattern' objects}
   151114    0.042    0.000    0.108    0.000 /usr/lib/python3.6/re.py:169(match)
      112    0.003    0.000    0.040    0.000 .../poky/meta/classes/sstate.bbclass:895(setscene_depvalid)
     4602    0.001    0.000    0.001    0.000 /usr/lib/python3.6/sre_parse.py:248(match)

(it reports executing 1082 + 3981 tasks)

Maybe I'm looking at the wrong lines...


Cheers,
Andre'