[bitbake-devel] [PATCH 2/2] runqueue.py: Initial implementation of per task process limits

[Mark Hatle]

On 5/14/18 9:49 AM, akuster wrote:
> 
> 
> On 05/14/2018 07:21 AM, Mark Hatle wrote:
>> On high core machines, in do_fetch, it is possible to DDoS your own machine.
> If this is the case, isn't a CVE warranted?  Also, which version does
> this issue affect?

No, cause you are doing it to yourself..

Run a do_fetch, and ask 192 external machines all to send you data at the same
time..  depending on ISP, rate limiting, etc.. it's very possible for you to
(temporarily) DDoS yourself... no different then if you fork bomb your own
machine, or use curl recursively.. :)

--Mark

> Thanks for fixing this.
> Armin
>> A method to limit any arbitrary task type to a certain number of simultaneous
>> threads is needed.  (Similar to how BB_NUMBER_THREADS works in the general
>> case.)  The format of this new limitation is:
>>
>>     do_fetch[number_threads] = "2"
>>
>> This should be set globally.  If it is set in individual recipes it could
>> result in unpredictable behavior.
>>
>> Note: a value for number_threads > BB_NUMBER_THREADS will have no effect.
>>
>> Signed-off-by: Mark Hatle <mark.hatle at windriver.com>
>> ---
>>  lib/bb/runqueue.py | 21 +++++++++++++++++++++
>>  1 file changed, 21 insertions(+)
>>
>> diff --git a/lib/bb/runqueue.py b/lib/bb/runqueue.py
>> index a937a0b..2d9e18d 100644
>> --- a/lib/bb/runqueue.py
>> +++ b/lib/bb/runqueue.py
>> @@ -134,6 +134,7 @@ class RunQueueScheduler(object):
>>          self.prio_map = [self.rqdata.runtaskentries.keys()]
>>  
>>          self.buildable = []
>> +        self.skip_maxthread = {}
>>          self.stamps = {}
>>          for tid in self.rqdata.runtaskentries:
>>              (mc, fn, taskname, taskfn) = split_tid_mcfn(tid)
>> @@ -150,8 +151,25 @@ class RunQueueScheduler(object):
>>          self.buildable = [x for x in self.buildable if x not in self.rq.runq_running]
>>          if not self.buildable:
>>              return None
>> +
>> +        # Filter out tasks that have a max number of threads that have been exceeded
>> +        skip_buildable = {}
>> +        for running in self.rq.runq_running.difference(self.rq.runq_complete):
>> +            rtaskname = taskname_from_tid(running)
>> +            if rtaskname not in self.skip_maxthread:
>> +                self.skip_maxthread[rtaskname] = self.rq.cfgData.getVarFlag(rtaskname, "number_threads")
>> +            if not self.skip_maxthread[rtaskname]:
>> +                continue
>> +            if rtaskname in skip_buildable:
>> +                skip_buildable[rtaskname] += 1
>> +            else:
>> +                skip_buildable[rtaskname] = 1
>> +
>>          if len(self.buildable) == 1:
>>              tid = self.buildable[0]
>> +            taskname = taskname_from_tid(tid)
>> +            if taskname in skip_buildable and skip_buildable[taskname] >= int(self.skip_maxthread[taskname]):
>> +                return None
>>              stamp = self.stamps[tid]
>>              if stamp not in self.rq.build_stamps.values():
>>                  return tid
>> @@ -164,6 +182,9 @@ class RunQueueScheduler(object):
>>          best = None
>>          bestprio = None
>>          for tid in self.buildable:
>> +            taskname = taskname_from_tid(tid)
>> +            if taskname in skip_buildable and skip_buildable[taskname] >= int(self.skip_maxthread[taskname]):
>> +                continue
>>              prio = self.rev_prio_map[tid]
>>              if bestprio is None or bestprio > prio:
>>                  stamp = self.stamps[tid]
>