[OE-core] [PATCH] image_types.bbclass: Set 70% RAM memory usage limit for xz

[Khem Raj]

On Fri, Feb 8, 2019 at 7:27 PM Andre McCurdy <armccurdy at gmail.com> wrote:
>
> On Fri, Feb 8, 2019 at 5:29 PM Khem Raj <raj.khem at gmail.com> wrote:
> >
> > when building with opkg backend and huge packages e.g. chromium/llvm all
> > going in parallel, memory pressure causes xz to catapult with
> >
> > do_package_write_ipk: Failed to create package, opkg-build failed with: xz: (stdin): Cannot allocate memory
> >
> > since there are many tasks going on in parallel, xz adds to memory pressure
> > and it wants it all, put an upper limit for memory xz can use
> >
> > We add a variable XZ_MAXRAM with 70% of RAM limit and can be customized
> > if builders have more memory one can set it like
> >
> > XZMAX_RAM = "-M 0"
>
> This needs a little more explanation. How does a change to command
> used to create .xz images help when running out of memory in
> do_package_write_ipk? Doesn't the image get created a lot later?
>

its not related to image alone. xz is used in various places e.g.
opkgs switched to
using xz as default compression format recently, so every ipk creation
task involves
xz, and you might be in multiple of ipk creation tasks based on task parallelism

hope that clarifies. If not feel free to ask more

> > Signed-off-by: Khem Raj <raj.khem at gmail.com>
> > Cc: Alejandro del Castillo <alejandro.delcastillo at ni.com>
> > ---
> >  meta/classes/image_types.bbclass | 3 ++-
> >  1 file changed, 2 insertions(+), 1 deletion(-)
> >
> > diff --git a/meta/classes/image_types.bbclass b/meta/classes/image_types.bbclass
> > index 70bd315306..9c8ea0643c 100644
> > --- a/meta/classes/image_types.bbclass
> > +++ b/meta/classes/image_types.bbclass
> > @@ -57,6 +57,7 @@ def imagetypes_getdepends(d):
> >  XZ_COMPRESSION_LEVEL ?= "-3"
> >  XZ_INTEGRITY_CHECK ?= "crc32"
> >  XZ_THREADS ?= "-T 0"
> > +XZ_MAXRAM ?= "-M 70%"
> >
> >  ZIP_COMPRESSION_LEVEL ?= "-9"
> >
> > @@ -284,7 +285,7 @@ CONVERSIONTYPES = "gz bz2 lzma xz lz4 lzo zip sum md5sum sha1sum sha224sum sha25
> >  CONVERSION_CMD_lzma = "lzma -k -f -7 ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}"
> >  CONVERSION_CMD_gz = "pigz -f -9 -n -c ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type} > ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.gz"
> >  CONVERSION_CMD_bz2 = "pbzip2 -f -k ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}"
> > -CONVERSION_CMD_xz = "xz -f -k -c ${XZ_COMPRESSION_LEVEL} ${XZ_THREADS} --check=${XZ_INTEGRITY_CHECK} ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type} > ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.xz"
> > +CONVERSION_CMD_xz = "xz -f -k -c ${XZ_COMPRESSION_LEVEL} ${XZ_THREADS} ${XZ_MAXRAM} --check=${XZ_INTEGRITY_CHECK} ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type} > ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.xz"
> >  CONVERSION_CMD_lz4 = "lz4 -9 -z -l ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type} ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.lz4"
> >  CONVERSION_CMD_lzo = "lzop -9 ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}"
> >  CONVERSION_CMD_zip = "zip ${ZIP_COMPRESSION_LEVEL} ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.zip ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}"
> > --
> > 2.20.1
> >
> > --
> > _______________________________________________
> > Openembedded-core mailing list
> > Openembedded-core at lists.openembedded.org
> > http://lists.openembedded.org/mailman/listinfo/openembedded-core