<div dir="auto"><div><br><div class="gmail_extra"><br><div class="gmail_quote">On Feb 17, 2018 18:25, "Khem Raj" <<a href="mailto:raj.khem@gmail.com">raj.khem@gmail.com</a>> wrote:<br type="attribution"><blockquote class="quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div class="quoted-text"><br>
<br>
On 2/16/18 12:14 PM, Martin Jansa wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
+ v = max(limit, v)<br>
<br>
this seems to be doing the opposite of what the limit was used for before:<br>
<br>
- v = min(64, int(v))<br>
<br>
</blockquote>
<br></div>
moreover, I think the limit should be based on ncpu() instead of hard coding it.<br></blockquote></div></div></div><div dir="auto"><br></div><div dir="auto">I don't know about that. Doesn't PARALLEL_MAKE already default to something based on ncpu()? My understanding of the limit was that some recipes (boost? waf?) have a limit on the amount of parallism they can reasonably stand, and you don't want those recipes to break if someone sets <span style="font-family:sans-serif">PARALLEL_MAKE="-j 1000".</span></div><div dir="auto"><span style="font-family:sans-serif"><br></span></div><div dir="auto">As long as that is true I think this is correct. Whether those recipes still need to limit their parallism is another discussion.</div><div dir="auto"><br></div><div dir="auto"><div class="gmail_extra"><div class="gmail_quote"><blockquote class="quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div class="quoted-text">
On Mon, Feb 12, 2018 at 8:39 PM, Joshua Watt <<a href="mailto:jpewhacker@gmail.com" target="_blank">jpewhacker@gmail.com</a> <mailto:<a href="mailto:jpewhacker@gmail.com" target="_blank">jpewhacker@gmail.com</a>>> wrote:<br>
<br>
The code to extract the integer number of parallel build threads and<br>
construct a new argument from them has started to be copied in multiple<br>
locations, so create two new helper utilities to aid recipes.<br>
<br>
The first helper (parallel_make()) extracts the integer number of<br>
parallel build threads from PARALLEL_MAKE. The second<br>
(parallel_make_argument()) does the same and then puts the result back<br>
into a format string, optionally clamping it to some maximum value.<br>
<br>
Additionally, rework the oe-core recipes that were manually doing this<br>
to use the new helper utilities.<br>
<br>
Signed-off-by: Joshua Watt <<a href="mailto:JPEWhacker@gmail.com" target="_blank">JPEWhacker@gmail.com</a><br></div>
<mailto:<a href="mailto:JPEWhacker@gmail.com" target="_blank">JPEWhacker@gmail.com</a>>><div class="quoted-text"><br>
---<br>
meta/classes/go.bbclass | 19 +---------------<br>
meta/classes/waf.bbclass | 24 +-------------------<br>
meta/lib/oe/utils.py | 43<br>
++++++++++++++++++++++++++++++<wbr>++++++<br></div>
meta/recipes-core/ovmf/<a href="http://ovmf_git.bb" rel="noreferrer" target="_blank">ovmf_<wbr>git.bb</a> <<a href="http://ovmf_git.bb" rel="noreferrer" target="_blank">http://ovmf_git.bb</a>> | 2 +-<div class="elided-text"><br>
meta/recipes-support/boost/b<wbr>oost.inc | 24 +-------------------<br>
5 files changed, 47 insertions(+), 65 deletions(-)<br>
<br>
diff --git a/meta/classes/go.bbclass b/meta/classes/go.bbclass<br>
index 09b01a84c37..7ecd8c92541 100644<br>
--- a/meta/classes/go.bbclass<br>
+++ b/meta/classes/go.bbclass<br>
@@ -1,23 +1,6 @@<br>
inherit goarch ptest<br>
<br>
-def get_go_parallel_make(d):<br>
- pm = (d.getVar('PARALLEL_MAKE') or '').split()<br>
- # look for '-j' and throw other options (e.g. '-l') away<br>
- # because they might have a different meaning in golang<br>
- while pm:<br>
- opt = pm.pop(0)<br>
- if opt == '-j':<br>
- v = pm.pop(0)<br>
- elif opt.startswith('-j'):<br>
- v = opt[2:].strip()<br>
- else:<br>
- continue<br>
-<br>
- return '-p %d' % int(v)<br>
-<br>
- return ""<br>
-<br>
-GO_PARALLEL_BUILD ?= "${@get_go_parallel_make(d)}"<br>
+GO_PARALLEL_BUILD ?= "${@oe.utils.parallel_make_arg<wbr>ument(d, '-p %d')}"<br>
<br>
GOROOT_class-native = "${STAGING_LIBDIR_NATIVE}/go"<br>
GOROOT_class-nativesdk = "${STAGING_DIR_TARGET}${libdir<wbr>}/go"<br>
diff --git a/meta/classes/waf.bbclass b/meta/classes/waf.bbclass<br>
index bdbdc56767c..f9a851d0759 100644<br>
--- a/meta/classes/waf.bbclass<br>
+++ b/meta/classes/waf.bbclass<br>
@@ -3,28 +3,6 @@ DISABLE_STATIC = ""<br>
<br>
EXTRA_OECONF_append = " ${PACKAGECONFIG_CONFARGS}"<br>
<br>
-def get_waf_parallel_make(d):<br>
- pm = d.getVar('PARALLEL_MAKE')<br>
- if pm:<br>
- # look for '-j' and throw other options (e.g. '-l') away<br>
- # because they might have different meaning in bjam<br>
- pm = pm.split()<br>
- while pm:<br>
- v = None<br>
- opt = pm.pop(0)<br>
- if opt == '-j':<br>
- v = pm.pop(0)<br>
- elif opt.startswith('-j'):<br>
- v = opt[2:].strip()<br>
- else:<br>
- v = None<br>
-<br>
- if v:<br>
- v = min(64, int(v))<br>
- return '-j' + str(v)<br>
-<br>
- return ""<br>
-<br>
python waf_preconfigure() {<br>
import subprocess<br>
from distutils.version import StrictVersion<br>
@@ -47,7 +25,7 @@ waf_do_configure() {<br>
<br>
do_compile[progress] = "outof:^\[\s*(\d+)/\s*(\d+)\]\<wbr>s+"<br>
waf_do_compile() {<br>
- ${S}/waf build ${@get_waf_parallel_make(d)}<br>
+ ${S}/waf build ${@oe.utils.parallel_make_argu<wbr>ment(d, '-j%d',<br>
limit=64)}<br>
}<br>
<br>
waf_do_install() {<br>
diff --git a/meta/lib/oe/utils.py b/meta/lib/oe/utils.py<br>
index 7a79d752b69..ec91927233a 100644<br>
--- a/meta/lib/oe/utils.py<br>
+++ b/meta/lib/oe/utils.py<br>
@@ -156,6 +156,49 @@ def any_distro_features(d, features,<br>
truevalue="1", falsevalue=""):<br>
"""<br>
return bb.utils.contains_any("DISTRO_<wbr>FEATURES", features,<br>
truevalue, falsevalue, d)<br>
<br>
+def parallel_make(d):<br>
+ """<br>
+ Return the integer value for the number of parallel threads to<br>
use when<br>
+ building, scraped out of PARALLEL_MAKE. If no parallelization<br>
option is<br>
+ found, returns None<br>
+<br>
+ e.g. if PARALLEL_MAKE = "-j 10", this will return 10 as an integer.<br>
+ """<br>
+ pm = (d.getVar('PARALLEL_MAKE') or '').split()<br>
+ # look for '-j' and throw other options (e.g. '-l') away<br>
+ while pm:<br>
+ opt = pm.pop(0)<br>
+ if opt == '-j':<br>
+ v = pm.pop(0)<br>
+ elif opt.startswith('-j'):<br>
+ v = opt[2:].strip()<br>
+ else:<br>
+ continue<br>
+<br>
+ return int(v)<br>
+<br>
+ return None<br>
+<br>
+def parallel_make_argument(d, fmt, limit=None):<br>
+ """<br>
+ Helper utility to construct a parallel make argument from the<br>
number of<br>
+ parallel threads specified in PARALLEL_MAKE.<br>
+<br>
+ Returns the input format string `fmt` where a single '%d' will<br>
be expanded<br>
+ with the number of parallel threads to use. If `limit` is<br>
specified, the<br>
+ number of parallel threads will be no larger than it. If no<br>
parallelization<br>
+ option is found in PARALLEL_MAKE, returns an empty string<br>
+<br>
+ e.g. if PARALLEL_MAKE = "-j 10", parallel_make_argument(d, "-n<br>
%d") will return<br>
+ "-n 10"<br>
+ """<br>
+ v = parallel_make(d)<br>
+ if v:<br>
+ if limit:<br>
+ v = max(limit, v)<br>
+ return fmt % v<br>
+ return ''<br>
+<br>
def packages_filter_out_system(d):<br>
"""<br>
Return a list of packages from PACKAGES with the "system"<br>
packages such as<br></div>
diff --git a/meta/recipes-core/ovmf/<a href="http://ovmf_git.bb" rel="noreferrer" target="_blank">ovmf_<wbr>git.bb</a> <<a href="http://ovmf_git.bb" rel="noreferrer" target="_blank">http://ovmf_git.bb</a>><br>
b/meta/recipes-core/ovmf/<a href="http://ovmf_git.bb" rel="noreferrer" target="_blank">ovmf_<wbr>git.bb</a> <<a href="http://ovmf_git.bb" rel="noreferrer" target="_blank">http://ovmf_git.bb</a>><br>
index fa0d66291d1..8750b3c528d 100644<br>
--- a/meta/recipes-core/ovmf/<a href="http://ovmf_git.bb" rel="noreferrer" target="_blank">ovmf_<wbr>git.bb</a> <<a href="http://ovmf_git.bb" rel="noreferrer" target="_blank">http://ovmf_git.bb</a>><br>
+++ b/meta/recipes-core/ovmf/<a href="http://ovmf_git.bb" rel="noreferrer" target="_blank">ovmf_<wbr>git.bb</a> <<a href="http://ovmf_git.bb" rel="noreferrer" target="_blank">http://ovmf_git.bb</a>><div class="elided-text"><br>
@@ -151,7 +151,7 @@ do_compile_class-native() {<br>
<br>
do_compile_class-target() {<br>
export LFLAGS="${LDFLAGS}"<br>
- PARALLEL_JOBS="${@ '${PARALLEL_MAKE}'.replace('-j<wbr>', '-n ')}"<br>
+ PARALLEL_JOBS="${@oe.utils.par<wbr>allel_make_argument(d, '-n %d')}"<br>
OVMF_ARCH="X64"<br>
if [ "${TARGET_ARCH}" != "x86_64" ] ; then<br>
OVMF_ARCH="IA32"<br>
diff --git a/meta/recipes-support/boost/b<wbr>oost.inc<br>
b/meta/recipes-support/boost/b<wbr>oost.inc<br>
index 41fc90fb211..0461ec6fcf1 100644<br>
--- a/meta/recipes-support/boost/b<wbr>oost.inc<br>
+++ b/meta/recipes-support/boost/b<wbr>oost.inc<br>
@@ -135,29 +135,7 @@ BJAM_TOOLS = "--ignore-site-config \<br>
<br>
# use PARALLEL_MAKE to speed up the build, but limit it by -j 64,<br>
greater parallelism causes bjam to segfault or to ignore -j<br>
# <a href="https://svn.boost.org/trac/boost/ticket/7634" rel="noreferrer" target="_blank">https://svn.boost.org/trac/boo<wbr>st/ticket/7634</a><br>
<<a href="https://svn.boost.org/trac/boost/ticket/7634" rel="noreferrer" target="_blank">https://svn.boost.org/trac/bo<wbr>ost/ticket/7634</a>><br>
-def get_boost_parallel_make(d):<br>
- pm = d.getVar('PARALLEL_MAKE')<br>
- if pm:<br>
- # look for '-j' and throw other options (e.g. '-l') away<br>
- # because they might have different meaning in bjam<br>
- pm = pm.split()<br>
- while pm:<br>
- v = None<br>
- opt = pm.pop(0)<br>
- if opt == '-j':<br>
- v = pm.pop(0)<br>
- elif opt.startswith('-j'):<br>
- v = opt[2:].strip()<br>
- else:<br>
- v = None<br>
-<br>
- if v:<br>
- v = min(64, int(v))<br>
- return '-j' + str(v)<br>
-<br>
- return ""<br>
-<br>
-BOOST_PARALLEL_MAKE = "${@get_boost_parallel_make(d)<wbr>}"<br>
+BOOST_PARALLEL_MAKE = "${@oe.utils.parallel_make_arg<wbr>ument(d,<br>
'-j%d', limit=64)}"<br>
BJAM_OPTS = '${BOOST_PARALLEL_MAKE} -d+2 -q \<br>
${BJAM_TOOLS} \<br>
-sBOOST_BUILD_USER_CONFIG=${WO<wbr>RKDIR}/user-config.jam \<br>
--<br>
2.14.3<br>
<br>
--<br>
______________________________<wbr>_________________<br>
Openembedded-core mailing list<br>
<a href="mailto:Openembedded-core@lists.openembedded.org" target="_blank">Openembedded-core@lists.openem<wbr>bedded.org</a><br></div>
<mailto:<a href="mailto:Openembedded-core@lists.openembedded.org" target="_blank">Openembedded-core@list<wbr>s.openembedded.org</a>><br>
<a href="http://lists.openembedded.org/mailman/listinfo/openembedded-core" rel="noreferrer" target="_blank">http://lists.openembedded.org/<wbr>mailman/listinfo/openembedded-<wbr>core</a><br>
<<a href="http://lists.openembedded.org/mailman/listinfo/openembedded-core" rel="noreferrer" target="_blank">http://lists.openembedded.org<wbr>/mailman/listinfo/openembedded<wbr>-core</a>><br>
<br>
<br>
<br>
</blockquote><div class="elided-text">
-- <br>
______________________________<wbr>_________________<br>
Openembedded-core mailing list<br>
<a href="mailto:Openembedded-core@lists.openembedded.org" target="_blank">Openembedded-core@lists.openem<wbr>bedded.org</a><br>
<a href="http://lists.openembedded.org/mailman/listinfo/openembedded-core" rel="noreferrer" target="_blank">http://lists.openembedded.org/<wbr>mailman/listinfo/openembedded-<wbr>core</a><br>
</div></blockquote></div><br></div></div></div>