[OE-core] [PATCH] image_types.bbclass: Set 70% RAM memory usage limit for xz

[Khem Raj]

On Fri, Feb 8, 2019 at 7:27 PM Andre McCurdy <armccurdy at gmail.com> wrote:
>
> On Fri, Feb 8, 2019 at 5:29 PM Khem Raj <raj.khem at gmail.com> wrote:
> >
> > when building with opkg backend and huge packages e.g. chromium/llvm all
> > going in parallel, memory pressure causes xz to catapult with
> >
> > do_package_write_ipk: Failed to create package, opkg-build failed with: xz: (stdin): Cannot allocate memory
> >
> > since there are many tasks going on in parallel, xz adds to memory pressure
> > and it wants it all, put an upper limit for memory xz can use
> >
> > We add a variable XZ_MAXRAM with 70% of RAM limit and can be customized
> > if builders have more memory one can set it like
> >
> > XZMAX_RAM = "-M 0"
>
> This needs a little more explanation. How does a change to command
> used to create .xz images help when running out of memory in
> do_package_write_ipk? Doesn't the image get created a lot later?

I think I have sent v1 patch which was not the one I intended to send.
will send v2 soon which should be complete.

>
> > Signed-off-by: Khem Raj <raj.khem at gmail.com>
> > Cc: Alejandro del Castillo <alejandro.delcastillo at ni.com>
> > ---
> >  meta/classes/image_types.bbclass | 3 ++-
> >  1 file changed, 2 insertions(+), 1 deletion(-)
> >
> > diff --git a/meta/classes/image_types.bbclass b/meta/classes/image_types.bbclass
> > index 70bd315306..9c8ea0643c 100644
> > --- a/meta/classes/image_types.bbclass
> > +++ b/meta/classes/image_types.bbclass
> > @@ -57,6 +57,7 @@ def imagetypes_getdepends(d):
> >  XZ_COMPRESSION_LEVEL ?= "-3"
> >  XZ_INTEGRITY_CHECK ?= "crc32"
> >  XZ_THREADS ?= "-T 0"
> > +XZ_MAXRAM ?= "-M 70%"
> >
> >  ZIP_COMPRESSION_LEVEL ?= "-9"
> >
> > @@ -284,7 +285,7 @@ CONVERSIONTYPES = "gz bz2 lzma xz lz4 lzo zip sum md5sum sha1sum sha224sum sha25
> >  CONVERSION_CMD_lzma = "lzma -k -f -7 ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}"
> >  CONVERSION_CMD_gz = "pigz -f -9 -n -c ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type} > ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.gz"
> >  CONVERSION_CMD_bz2 = "pbzip2 -f -k ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}"
> > -CONVERSION_CMD_xz = "xz -f -k -c ${XZ_COMPRESSION_LEVEL} ${XZ_THREADS} --check=${XZ_INTEGRITY_CHECK} ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type} > ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.xz"
> > +CONVERSION_CMD_xz = "xz -f -k -c ${XZ_COMPRESSION_LEVEL} ${XZ_THREADS} ${XZ_MAXRAM} --check=${XZ_INTEGRITY_CHECK} ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type} > ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.xz"
> >  CONVERSION_CMD_lz4 = "lz4 -9 -z -l ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type} ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.lz4"
> >  CONVERSION_CMD_lzo = "lzop -9 ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}"
> >  CONVERSION_CMD_zip = "zip ${ZIP_COMPRESSION_LEVEL} ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.zip ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}"
> > --
> > 2.20.1
> >
> > --
> > _______________________________________________
> > Openembedded-core mailing list
> > Openembedded-core at lists.openembedded.org
> > http://lists.openembedded.org/mailman/listinfo/openembedded-core