[OE-core] [PATCH 0/3] Fix xz compression command and optimize compression time

[Andrea Adami]

On Fri, Jul 13, 2012 at 8:45 AM, Koen Kooi <koen at dominion.thruhere.net> wrote:
>
> Op 12 jul. 2012, om 21:22 heeft Saul Wold het volgende geschreven:
>
>> On 07/12/2012 11:58 AM, Koen Kooi wrote:
>>> Any volunteers to test this on a system with >4 real cores?
>>>
>>
>> Koen,
>>
>> Does OE-Core or Poky have an image setup for using .xz by default?
>
> No, and as you can see from Andrei's patches, it would have never worked :)
>
> regards,
>
> Koen
>

This leads to a similar remark for lzma:

The xz/lzma have been added with commit  38334ac and at that time both
where using XZ_COMPRESSION_LEVEL.

The code has been refactored afterwards and what happened is that lzma
now defaults to compression 7 (the default) while xz is way too high
(default is 6).

There is a well hidden pitfall with that, being that any image,
including the initramfs, will use that compression factor.

As you can notice in that (old) man xz [2]  the memory requirements
are very high and what did happen to me was that the cpio.lzma could
not be decompressed thus kernel could not boot (we override now the
value setting "-e2" in our BSP).

Adding more threads will multiply that figures.

My 2 cents

Andrea

[1] http://cgit.openembedded.org/openembedded-core/log/meta/classes/image_types.bbclass
[2] http://www.linuxcertif.com/man/1/xz

>>
>> I will run it on a machine I have, just want to make sure I am doing the same tests that Andrei is running.
>>
>> I was about to ask what msm asked about the Memory info.
>>
>>
>> Sau!
>>
>>> Op 12 jul. 2012, om 20:13 heeft Andrei Gherzan het volgende geschreven:
>>>
>>>> Koen suggested to add -T to xz commands. We have this option implemented in our current version .1alpha.
>>>>
>>>> -T threads, --threads=threads
>>>>              Specify the number of worker threads to use.  Setting threads to
>>>>              a special value 0 makes xz use as many threads as there are  CPU
>>>>              cores  on  the system.  The actual number of threads can be less
>>>>              than threads if the input file is not big enough  for  threading
>>>>              with  the  given  settings or if using more threads would exceed
>>>>              the memory usage limit.
>>>>
>>>> -------------------
>>>>
>>>> Memory: 7.8 GiB
>>>> Processor:: Intel® Core™ i5-2540M CPU @ 2.60GHz × 4
>>>>
>>>> File to compress: 1.9G
>>>>
>>>> xz -f -k -e -9 --check=crc32 (current configuration)
>>>> Memory: 673Mb
>>>> real    6m37.170s
>>>> user    6m35.389s
>>>> sys     0m0.884s
>>>> Compressed file size: 3.4Mb
>>>>
>>>> xz -f -k -e -9 -T 4 --check=crc32
>>>> Memory: Cannot allocate memory
>>>>
>>>> xz -f -k -e -8 -T 4 --check=crc32
>>>> Memory: 1.8G
>>>> real    3m24.462s
>>>> user    12m8.502s
>>>> sys     0m2.180s
>>>> Compressed file size: 3.4Mb
>>>>
>>>> xz -f -k -T 4 --check=crc32 (-e defaults to -6)
>>>> Memory: 471Mb
>>>> real    1m9.265s
>>>> user    4m8.972s
>>>> sys     0m0.944s
>>>> Compressed file size: 3.4Mb
>>>>
>>>> So my conclusion would be to use the default -e -6 with -T 0.
>>>>
>>>> The following changes since commit 90ad663909c0c8a405b22a510c9f957007d02669:
>>>>
>>>>  upstream_tracking: update boost (2012-07-09 17:21:38 +0100)
>>>>
>>>> are available in the git repository at:
>>>>
>>>>  git://git.yoctoproject.org/poky-contrib ag/xz
>>>>  http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=ag/xz
>>>>
>>>> Andrei Gherzan (3):
>>>>  image_types.bbclass: Fix COMPRESS_CMD for xz to redirect compressed
>>>>    data to file
>>>>  image_types.bbclass: Add XZ variable to set number of threads to be
>>>>    used while compressing
>>>>  image_types.bbclass: Default XZ_COMPRESSION_LEVEL to -e -6
>>>>
>>>> meta/classes/image_types.bbclass |    5 +++--
>>>> 1 file changed, 3 insertions(+), 2 deletions(-)
>>>>
>>>> --
>>>> 1.7.9.5
>>>>
>>>
>>>
>>> _______________________________________________
>>> Openembedded-core mailing list
>>> Openembedded-core at lists.openembedded.org
>>> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core
>>>
>>>
>>
>
>
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core at lists.openembedded.org
> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core