[OE-core] [PATCH] local.conf.sample.extended: Document how to limit parallel fetch tasks

[Andre McCurdy]

On Wed, Feb 19, 2020 at 3:16 PM Richard Purdie
<richard.purdie at linuxfoundation.org> wrote:
>
> On Wed, 2020-02-19 at 14:05 -0800, Andre McCurdy wrote:
> > On Wed, Feb 19, 2020 at 1:43 PM Richard Purdie
> > <richard.purdie at linuxfoundation.org> wrote:
> > > [YOCTO #13235]
> > >
> > > Signed-off-by: Richard Purdie <richard.purdie at linuxfoundation.org>
> > > ---
> > >  meta/conf/local.conf.sample.extended | 8 ++++++++
> > >  1 file changed, 8 insertions(+)
> > >
> > > diff --git a/meta/conf/local.conf.sample.extended
> > > b/meta/conf/local.conf.sample.extended
> > > index fcf4a9b7c1a..443b4e83f9a 100644
> > > --- a/meta/conf/local.conf.sample.extended
> > > +++ b/meta/conf/local.conf.sample.extended
> > > @@ -23,6 +23,14 @@
> > >  #
> > >  # For a quad-core machine, BB_NUMBER_THREADS = "4", PARALLEL_MAKE
> > > = "-j 4" would
> > >  # be appropriate for example.
> > > +#
> > > +# Some users are behind firewalls or use servers where the number
> > > of parallel connections
> > > +# is limited. In such cases you can limit the number of fetch
> > > tasks which run in parallel by
> > > +# setting the option below, in this case limiting to a maximum of
> > > 4 fetch tasks in parallel:
> > > +#
> > > +#do_fetch[number_threads] = "4"
> >
> > How does this interact with BB_NUMBER_THREADS? If, for example, both
> > are set to 4 does it mean there could be 4 fetch threads and 4
> > compile threads running in parallel?
>
> BB_NUMBER_THREADS is an upper limit which includes fetch tasks.
>
> > A related issue when the number of parallel connections is limited is
> > parsing a large number of AUTOREV recipes. From what I remember the
> > workaround for that was to set BB_NUMBER_PARSE_THREADS. Maybe that
> > should be documented in local.conf.sample.extended too? Perhaps even
> > better would be to define a single new variable which limits the
> > number of parallel connections in both cases?
>
> I don't know how you'd implement a total number of connections limit
> from a code perspective :/.

A single variable such as BB_MAX_CONNECTION_THREADS which, if defined,
would set the limit for both do_fetch[number_threads] and
BB_NUMBER_PARSE_THREADS ?

> You'd probably need something like a make job server where you obtained
> tokens before making connections and I can see this becoming "fun"
> since you'd have to markup every code section where a network
> connection was made.
>
> We have a conflict between code complexity, maintainability,
> performance and functionality and I'm not sure the use case here
> warrants the complexity...

Of course. I just wanted to share it since it is a real world problem
(I've seen a development team stumped for a few days before finding
BB_NUMBER_PARSE_THREADS, which isn't nearly as well documented as
BB_NUMBER_THREADS and PARALLEL_MAKE). A comment for
BB_NUMBER_PARSE_THREADS in local.conf.sample.extended might have sped
up that debugging process.