[OE-core] [PATCH] image: Convert vmdk/vdi/qcow2 to strict CONVERSION_CMD types
Jonathan Liu
net147 at gmail.com
Fri Jul 28 11:35:37 UTC 2017
Hi Tom,
On 28 July 2017 at 21:31, Tom Rini <trini at konsulko.com> wrote:
> On Fri, Jul 28, 2017 at 09:09:55PM +1000, Jonathan Liu wrote:
>> Hi Tom,
>>
>> On 26 July 2017 at 05:58, Tom Rini <trini at konsulko.com> wrote:
>> > The vmdk/vdi/qcow2 IMAGE_FSTYPEs predate wic. As such, they provide
>> > some similar underlying functionality in order to produce a "disk" image
>> > that in turn can be converted into different formats that various
>> > hypervisor types work with. They do not however provide the ability for
>> > other disk image types to be converted into these same output types.
>> > Furthermore, they are less flexible than what wic does provide. This
>> > drops the old style vmdk/vdi/qcow2 types and re-introduces them under
>> > the CONVERSION_CMD framework. The equivalent of vmdk is now wic.vmdk
>> > and so forth for the other types.
>> >
>> > Signed-off-by: Tom Rini <trini at konsulko.com>
>> > ---
>> > This depends on my previous series to correct chaining compression
>> > support. I had attempted to keep the vmdk/vdi/qcow2 IMAGE_FSTYPES for
>> > compatibility sake using IMAGE_TYPEDEP_vmdk = "wic" and introducing an
>> > oe_qemuimg function to run to do the conversion. This was working, but
>> > I could not get it to have the symlinks created automatically. At this
>> > timeI think it makes most sense to not hide that the underlying disk
>> > content has changed at least slightly from the old vmdk type to the new
>> > type and that we can simply handle this change in documentation. As
>> > such, if there's agreement about dropping the types I'll include some
>> > documentation changes in the next version.
>> >
>> > meta/classes/image-vm.bbclass | 171 ---------------------
>> > meta/classes/image.bbclass | 3 -
>> > meta/classes/image_types.bbclass | 12 +-
>> > .../images/build-appliance-image_15.0.0.bb | 6 +-
>> > 4 files changed, 10 insertions(+), 182 deletions(-)
>> > delete mode 100644 meta/classes/image-vm.bbclass
>> >
>> > diff --git a/meta/classes/image-vm.bbclass b/meta/classes/image-vm.bbclass
>> > deleted file mode 100644
>> > index b52df9f..0000000
>> > --- a/meta/classes/image-vm.bbclass
>> > +++ /dev/null
>> > @@ -1,171 +0,0 @@
>> > -# image-vm.bbclass
>> > -# (loosly based off image-live.bbclass Copyright (C) 2004, Advanced Micro Devices, Inc.)
>> > -#
>> > -# Create an image which can be placed directly onto a harddisk using dd and then
>> > -# booted.
>> > -#
>> > -# This uses syslinux. extlinux would have been nice but required the ext2/3
>> > -# partition to be mounted. grub requires to run itself as part of the install
>> > -# process.
>> > -#
>> > -# The end result is a 512 boot sector populated with an MBR and partition table
>> > -# followed by an msdos fat16 partition containing syslinux and a linux kernel
>> > -# completed by the ext2/3 rootfs.
>> > -#
>> > -# We have to push the msdos parition table size > 16MB so fat 16 is used as parted
>> > -# won't touch fat12 partitions.
>> > -
>> > -inherit live-vm-common
>> > -
>> > -do_bootdirectdisk[depends] += "dosfstools-native:do_populate_sysroot \
>> > - virtual/kernel:do_deploy \
>> > - syslinux:do_populate_sysroot \
>> > - syslinux-native:do_populate_sysroot \
>> > - parted-native:do_populate_sysroot \
>> > - mtools-native:do_populate_sysroot \
>> > - ${PN}:do_image_${VM_ROOTFS_TYPE} \
>> > - "
>> > -
>> > -IMAGE_TYPEDEP_vmdk = "${VM_ROOTFS_TYPE}"
>> > -IMAGE_TYPEDEP_vdi = "${VM_ROOTFS_TYPE}"
>> > -IMAGE_TYPEDEP_qcow2 = "${VM_ROOTFS_TYPE}"
>> > -IMAGE_TYPEDEP_hdddirect = "${VM_ROOTFS_TYPE}"
>> > -IMAGE_TYPES_MASKED += "vmdk vdi qcow2 hdddirect"
>> > -
>> > -VM_ROOTFS_TYPE ?= "ext4"
>> > -ROOTFS ?= "${IMGDEPLOYDIR}/${IMAGE_LINK_NAME}.${VM_ROOTFS_TYPE}"
>> > -
>> > -# Used by bootloader
>> > -LABELS_VM ?= "boot"
>> > -ROOT_VM ?= "root=/dev/sda2"
>> > -# Using an initramfs is optional. Enable it by setting INITRD_IMAGE_VM.
>> > -INITRD_IMAGE_VM ?= ""
>> > -INITRD_VM ?= "${@'${IMGDEPLOYDIR}/${INITRD_IMAGE_VM}-${MACHINE}.cpio.gz' if '${INITRD_IMAGE_VM}' else ''}"
>> > -do_bootdirectdisk[depends] += "${@'${INITRD_IMAGE_VM}:do_image_complete' if '${INITRD_IMAGE_VM}' else ''}"
>> > -
>> > -BOOTDD_VOLUME_ID ?= "boot"
>> > -BOOTDD_EXTRA_SPACE ?= "16384"
>> > -
>> > -DISK_SIGNATURE ?= "${DISK_SIGNATURE_GENERATED}"
>> > -DISK_SIGNATURE[vardepsexclude] = "DISK_SIGNATURE_GENERATED"
>> > -
>> > -build_boot_dd() {
>> > - HDDDIR="${S}/hdd/boot"
>> > - HDDIMG="${S}/hdd.image"
>> > - IMAGE=${IMGDEPLOYDIR}/${IMAGE_NAME}.hdddirect
>> > -
>> > - populate_kernel $HDDDIR
>> > -
>> > - if [ "${PCBIOS}" = "1" ]; then
>> > - syslinux_hddimg_populate $HDDDIR
>> > - fi
>> > - if [ "${EFI}" = "1" ]; then
>> > - efi_hddimg_populate $HDDDIR
>> > - fi
>> > -
>> > - BLOCKS=`du -bks $HDDDIR | cut -f 1`
>> > - BLOCKS=`expr $BLOCKS + ${BOOTDD_EXTRA_SPACE}`
>> > -
>> > - # Remove it since mkdosfs would fail when it exists
>> > - rm -f $HDDIMG
>> > - mkdosfs -n ${BOOTDD_VOLUME_ID} ${MKDOSFS_EXTRAOPTS} -C $HDDIMG $BLOCKS
>> > - mcopy -i $HDDIMG -s $HDDDIR/* ::/
>> > -
>> > - if [ "${PCBIOS}" = "1" ]; then
>> > - syslinux_hdddirect_install $HDDIMG
>> > - fi
>> > - chmod 644 $HDDIMG
>> > -
>> > - ROOTFSBLOCKS=`du -Lbks ${ROOTFS} | cut -f 1`
>> > - TOTALSIZE=`expr $BLOCKS + $ROOTFSBLOCKS`
>> > - END1=`expr $BLOCKS \* 1024`
>> > - END2=`expr $END1 + 512`
>> > - END3=`expr \( $ROOTFSBLOCKS \* 1024 \) + $END1`
>> > -
>> > - echo $ROOTFSBLOCKS $TOTALSIZE $END1 $END2 $END3
>> > - rm -rf $IMAGE
>> > - dd if=/dev/zero of=$IMAGE bs=1024 seek=$TOTALSIZE count=1
>> > -
>> > - parted $IMAGE mklabel msdos
>> > - parted $IMAGE mkpart primary fat16 0 ${END1}B
>> > - parted $IMAGE unit B mkpart primary ext2 ${END2}B ${END3}B
>> > - parted $IMAGE set 1 boot on
>> > -
>> > - parted $IMAGE print
>> > -
>> > - awk "BEGIN { printf \"$(echo ${DISK_SIGNATURE} | sed 's/\(..\)\(..\)\(..\)\(..\)/\\x\4\\x\3\\x\2\\x\1/')\" }" | \
>> > - dd of=$IMAGE bs=1 seek=440 conv=notrunc
>> > -
>> > - OFFSET=`expr $END2 / 512`
>> > - if [ "${PCBIOS}" = "1" ]; then
>> > - dd if=${STAGING_DATADIR}/syslinux/mbr.bin of=$IMAGE conv=notrunc
>> > - fi
>> > -
>> > - dd if=$HDDIMG of=$IMAGE conv=notrunc seek=1 bs=512
>> > - dd if=${ROOTFS} of=$IMAGE conv=notrunc seek=$OFFSET bs=512
>> > -
>> > - cd ${IMGDEPLOYDIR}
>> > -
>> > - ln -sf ${IMAGE_NAME}.hdddirect ${IMGDEPLOYDIR}/${IMAGE_LINK_NAME}.hdddirect
>> > -}
>> > -
>> > -python do_bootdirectdisk() {
>> > - validate_disk_signature(d)
>> > - set_live_vm_vars(d, 'VM')
>> > - if d.getVar("PCBIOS") == "1":
>> > - bb.build.exec_func('build_syslinux_cfg', d)
>> > - if d.getVar("EFI") == "1":
>> > - bb.build.exec_func('build_efi_cfg', d)
>> > - bb.build.exec_func('build_boot_dd', d)
>> > -}
>> > -
>> > -def generate_disk_signature():
>> > - import uuid
>> > -
>> > - signature = str(uuid.uuid4())[:8]
>> > -
>> > - if signature != '00000000':
>> > - return signature
>> > - else:
>> > - return 'ffffffff'
>> > -
>> > -def validate_disk_signature(d):
>> > - import re
>> > -
>> > - disk_signature = d.getVar("DISK_SIGNATURE")
>> > -
>> > - if not re.match(r'^[0-9a-fA-F]{8}$', disk_signature):
>> > - bb.fatal("DISK_SIGNATURE '%s' must be an 8 digit hex string" % disk_signature)
>> > -
>> > -DISK_SIGNATURE_GENERATED := "${@generate_disk_signature()}"
>> > -
>> > -run_qemu_img (){
>> > - type="$1"
>> > - qemu-img convert -O $type ${IMGDEPLOYDIR}/${IMAGE_LINK_NAME}.hdddirect ${IMGDEPLOYDIR}/${IMAGE_NAME}.$type
>> > -
>> > - ln -sf ${IMAGE_NAME}.$type ${IMGDEPLOYDIR}/${IMAGE_LINK_NAME}.$type
>> > -}
>> > -create_vmdk_image () {
>> > - run_qemu_img vmdk
>> > -}
>> > -
>> > -create_vdi_image () {
>> > - run_qemu_img vdi
>> > -}
>> > -
>> > -create_qcow2_image () {
>> > - run_qemu_img qcow2
>> > -}
>> > -
>> > -python do_vmimg() {
>> > - if 'vmdk' in d.getVar('IMAGE_FSTYPES'):
>> > - bb.build.exec_func('create_vmdk_image', d)
>> > - if 'vdi' in d.getVar('IMAGE_FSTYPES'):
>> > - bb.build.exec_func('create_vdi_image', d)
>> > - if 'qcow2' in d.getVar('IMAGE_FSTYPES'):
>> > - bb.build.exec_func('create_qcow2_image', d)
>> > -}
>> > -
>> > -addtask bootdirectdisk before do_vmimg
>> > -addtask vmimg after do_bootdirectdisk before do_image_complete
>> > -do_vmimg[depends] += "qemu-native:do_populate_sysroot"
>> > diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
>> > index dee17ae..40bd614 100644
>> > --- a/meta/classes/image.bbclass
>> > +++ b/meta/classes/image.bbclass
>> > @@ -138,9 +138,6 @@ def build_live(d):
>> > IMAGE_TYPE_live = "${@build_live(d)}"
>> > inherit ${IMAGE_TYPE_live}
>> >
>> > -IMAGE_TYPE_vm = '${@bb.utils.contains_any("IMAGE_FSTYPES", ["vmdk", "vdi", "qcow2", "hdddirect"], "image-vm", "", d)}'
>> > -inherit ${IMAGE_TYPE_vm}
>> > -
>> > IMAGE_TYPE_container = '${@bb.utils.contains("IMAGE_FSTYPES", "container", "image-container", "", d)}'
>> > inherit ${IMAGE_TYPE_container}
>> >
>> > diff --git a/meta/classes/image_types.bbclass b/meta/classes/image_types.bbclass
>> > index 4fdcde2..138d7a5 100644
>> > --- a/meta/classes/image_types.bbclass
>> > +++ b/meta/classes/image_types.bbclass
>> > @@ -254,10 +254,6 @@ IMAGE_TYPES = " \
>> > ubi ubifs multiubi \
>> > tar tar.gz tar.bz2 tar.xz tar.lz4 \
>> > cpio cpio.gz cpio.xz cpio.lzma cpio.lz4 \
>> > - vmdk \
>> > - vdi \
>> > - qcow2 \
>> > - hdddirect \
>> > elf \
>> > wic wic.gz wic.bz2 wic.lzma \
>> > container \
>> > @@ -269,7 +265,7 @@ IMAGE_TYPES = " \
>> > # CONVERSION_CMD/DEPENDS.
>> > COMPRESSIONTYPES ?= ""
>> >
>> > -CONVERSIONTYPES = "gz bz2 lzma xz lz4 lzo zip sum md5sum sha1sum sha224sum sha256sum sha384sum sha512sum bmap u-boot ${COMPRESSIONTYPES}"
>> > +CONVERSIONTYPES = "gz bz2 lzma xz lz4 lzo zip sum md5sum sha1sum sha224sum sha256sum sha384sum sha512sum bmap u-boot vmdk vdi qcow2 ${COMPRESSIONTYPES}"
>> > CONVERSION_CMD_lzma = "lzma -k -f -7 ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}"
>> > CONVERSION_CMD_gz = "gzip -f -9 -c ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type} > ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.gz"
>> > CONVERSION_CMD_bz2 = "pbzip2 -f -k ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}"
>> > @@ -287,6 +283,9 @@ CONVERSION_CMD_sha384sum = "sha384sum ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}
>> > CONVERSION_CMD_sha512sum = "sha512sum ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type} > ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.sha512sum"
>> > CONVERSION_CMD_bmap = "bmaptool create ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type} -o ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.bmap"
>> > CONVERSION_CMD_u-boot = "mkimage -A ${UBOOT_ARCH} -O linux -T ramdisk -C none -n ${IMAGE_NAME} -d ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type} ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.u-boot"
>> > +CONVERSION_CMD_vmdk = "qemu-img convert -O vmdk ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type} ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.vmdk"
>> > +CONVERSION_CMD_vdi = "qemu-img convert -O vdi ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type} ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.vdi"
>> > +CONVERSION_CMD_qcow2 = "qemu-img convert -O qcow2 ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type} ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.qcow2"
>> > CONVERSION_DEPENDS_lzma = "xz-native"
>> > CONVERSION_DEPENDS_gz = "pigz-native"
>> > CONVERSION_DEPENDS_bz2 = "pbzip2-native"
>> > @@ -297,6 +296,9 @@ CONVERSION_DEPENDS_zip = "zip-native"
>> > CONVERSION_DEPENDS_sum = "mtd-utils-native"
>> > CONVERSION_DEPENDS_bmap = "bmap-tools-native"
>> > CONVERSION_DEPENDS_u-boot = "u-boot-mkimage-native"
>> > +CONVERSION_DEPENDS_vmdk = "qemu-native"
>> > +CONVERSION_DEPENDS_vdi = "qemu-native"
>> > +CONVERSION_DEPENDS_qcow2 = "qemu-native"
>> >
>> > RUNNABLE_IMAGE_TYPES ?= "ext2 ext3 ext4"
>> > RUNNABLE_MACHINE_PATTERNS ?= "qemu"
>> > diff --git a/meta/recipes-core/images/build-appliance-image_15.0.0.bb b/meta/recipes-core/images/build-appliance-image_15.0.0.bb
>> > index f145b5e..927a931 100644
>> > --- a/meta/recipes-core/images/build-appliance-image_15.0.0.bb
>> > +++ b/meta/recipes-core/images/build-appliance-image_15.0.0.bb
>> > @@ -19,7 +19,7 @@ IMAGE_ROOTFS_EXTRA_SPACE = "41943040"
>> > APPEND += "rootfstype=ext4 quiet"
>> >
>> > DEPENDS = "zip-native python3-pip-native"
>> > -IMAGE_FSTYPES = "vmdk"
>> > +IMAGE_FSTYPES = "wic.vmdk"
>> >
>> > inherit core-image module-base setuptools3
>> >
>> > @@ -120,7 +120,7 @@ create_bundle_files () {
>> > cd ${WORKDIR}
>> > mkdir -p Yocto_Build_Appliance
>> > cp *.vmx* Yocto_Build_Appliance
>> > - ln -sf ${IMGDEPLOYDIR}/${IMAGE_NAME}.vmdk Yocto_Build_Appliance/Yocto_Build_Appliance.vmdk
>> > + ln -sf ${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.wic.vmdk Yocto_Build_Appliance/Yocto_Build_Appliance.vmdk
>> > zip -r ${IMGDEPLOYDIR}/Yocto_Build_Appliance-${DATETIME}.zip Yocto_Build_Appliance
>> > ln -sf Yocto_Build_Appliance-${DATETIME}.zip ${IMGDEPLOYDIR}/Yocto_Build_Appliance.zip
>> > }
>> > @@ -130,4 +130,4 @@ python do_bundle_files() {
>> > bb.build.exec_func('create_bundle_files', d)
>> > }
>> >
>> > -addtask bundle_files after do_vmimg before do_image_complete
>> > +addtask bundle_files after do_image_wic before do_image_complete
>>
>> I know some people rely on DISK_SIGNATURE. Does DISK_SIGNATURE still
>> work after this change?
>
> It does not, because the code is gone. Can you point me to users of it?
> We could move it to the "common" file if there's users still. FWIW, it
> may be better to leverage the UUID support in wic.
>
> --
> Tom
Here is an example of a user having an issue with DISK_SIGNATURE:
http://lists.openembedded.org/pipermail/openembedded-core/2017-May/136940.html
I fixed the issue in:
http://git.yoctoproject.org/cgit.cgi/poky/commit/?id=d2390c0646eb6c66ea2bbce69cea9d2c412bdd93
Does wic support 32-bit MBR disk signature (byte offset 440 in MBR
partition table)?
Regards,
Jonathan
More information about the Openembedded-core
mailing list