[oe] [meta-ota][PATCH] meta-ota: add support for binary-delta images in a new layer

[Bartosz Golaszewski]

From: Bartosz Golaszewski <bgolaszewski at baylibre.com>

Over-The-Air updates are a crucial part of IoT systems based on linux.
There are several OTA update frameworks available and many offer some
sort of support in yocto (e.g. meta-mender, meta-rauc). There are certain
operations that are common to all of them such as: generating binary
delta patches, system recovery, creating provisioning images etc.

This patch proposes to add a new layer in meta-openembedded dedicated to
OTA. As the first functionality it adds a bbclass for generating binary
delta images using two popular algorithms - vcdiff and rsync.

Such images can then be easily packaged in update artifacts for different
OTA frameworks.

Signed-off-by: Bartosz Golaszewski <bgolaszewski at baylibre.com>
---
Hi,

I've noticed that when launching CE products and implementing OTA support
I find myself implementing the same functionalities over again. I recently
started abstracting them and would like to upstream them in meta-openembedded
so that at least a part of that becomes standardized.

This patch proposes to add a new layer to meta-openembedded and I'll gladly
maintain it. If accepted I intend on extending it with more functionalities
such as creating provisioning (factory) images for updatable systems, recovery
system images etc.

 meta-ota/README                       | 27 +++++++++++
 meta-ota/classes/binary-delta.bbclass | 64 +++++++++++++++++++++++++++
 meta-ota/conf/layer.conf              | 17 +++++++
 3 files changed, 108 insertions(+)
 create mode 100644 meta-ota/README
 create mode 100644 meta-ota/classes/binary-delta.bbclass
 create mode 100644 meta-ota/conf/layer.conf

diff --git a/meta-ota/README b/meta-ota/README
new file mode 100644
index 000000000..07a1debdd
--- /dev/null
+++ b/meta-ota/README
@@ -0,0 +1,27 @@
+meta-ota
+========
+
+This layer provides support for creating Over-The-Air update images.
+
+Dependencies
+------------
+
+This layer depends on:
+
+URI: git://github.com/openembedded/oe-core.git
+subdirectory: meta
+branch: master
+revision: HEAD
+
+URI: git://github.com/openembedded/meta-oe.git
+subdirectory: meta-oe
+branch: master
+revision: HEAD
+
+Maintenance
+-----------
+
+Send patches / pull requests to openembedded-devel at lists.openembedded.org
+with '[meta-ota]' in the subject.
+
+Layer maintainer: Bartosz Golaszewski <bgolaszewski at baylibre.com>
diff --git a/meta-ota/classes/binary-delta.bbclass b/meta-ota/classes/binary-delta.bbclass
new file mode 100644
index 000000000..24fb20b1b
--- /dev/null
+++ b/meta-ota/classes/binary-delta.bbclass
@@ -0,0 +1,64 @@
+# SPDX-License-Identifier: MIT
+#
+# Copyright (C) 2020 BayLibre SAS
+# Author: Bartosz Golaszewski <bgolaszewski at baylibre.com>
+#
+# This class provides image conversions which - given a baseline partition
+# image - can generate a binary delta patch. Such patch can be applied on top
+# of a deployed baseline partition to generate an updated image.
+#
+# For now the supported algorithms are rsync and vcdiff (supplied by rdiff and
+# xdelta3 respectively). We also provide several compression methods.
+#
+# We only support binary delta on ext[2-4] and btrfs images as it doesn't make
+# much sense on high-entropy, compressed filesystems.
+
+BINARY_DELTA_BASELINE ?= ""
+
+verify_baseline() {
+    if [ -z ${BINARY_DELTA_BASELINE} ] ; then
+        bbfatal "variable BINARY_DELTA_BASELINE cannot be empty."
+    fi
+
+    if ! [ -f ${BINARY_DELTA_BASELINE} ] ; then
+        bbfatal "${BINARY_DELTA_BASELINE} doesn't exist"
+    fi
+}
+
+gen_rdiff() {
+    local IMAGE=$1
+    local SIGNATURE=$1.sig
+    local PATCH=$1.rdiff
+
+    verify_baseline
+
+    rdiff signature ${BINARY_DELTA_BASELINE} $SIGNATURE
+    rdiff delta $SIGNATURE $IMAGE $PATCH
+    rm $SIGNATURE
+}
+
+gen_vcdiff() {
+    local IMAGE=$1
+    local PATCH=$1.vcdiff
+
+    verify_baseline
+
+    xdelta3 -e -s ${BINARY_DELTA_BASELINE} $IMAGE $PATCH
+}
+
+IMAGE_TYPES += " \
+    ext2.rdiff ext3.rdiff ext4.rdiff btrfs.rdiff \
+    ext2.rdiff.gz ext3.rdiff.gz ext4.rdiff.gz btrfs.rdiff.gz \
+    ext2.rdiff.bz2 ext3.rdiff.bz2 ext4.rdiff.bz2 btrfs.rdiff.bz2 \
+    ext2.rdiff.xz ext3.rdiff.xz ext4.rdiff.xz btrfs.rdiff.xz \
+    ext2.vcdiff ext3.vcdiff ext4.vcdiff brtfs.vcdiff \
+    ext2.vcdiff.gz ext3.vcdiff.gz ext4.vcdiff.gz brtfs.vcdiff.gz \
+    ext2.vcdiff.bz2 ext3.vcdiff.bz2 ext4.vcdiff.bz2 brtfs.vcdiff.bz2 \
+    ext2.vcdiff.xz ext3.vcdiff.xz ext4.vcdiff.xz brtfs.vcdiff.xz \
+"
+
+CONVERSIONTYPES += "rdiff vcdiff"
+CONVERSION_CMD_rdiff = "gen_rdiff ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}"
+CONVERSION_CMD_vcdiff = "gen_vcdiff ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}"
+CONVERSION_DEPENDS_rdiff = "librsync-native"
+CONVERSION_DEPENDS_vcdiff = "xdelta3-native"
diff --git a/meta-ota/conf/layer.conf b/meta-ota/conf/layer.conf
new file mode 100644
index 000000000..05c6b0f96
--- /dev/null
+++ b/meta-ota/conf/layer.conf
@@ -0,0 +1,17 @@
+# Layer configuration for meta-ota layer
+# Copyright (C) 2020 BayLibre SAS
+
+# We have a conf and classes directory, add to BBPATH
+BBPATH .= ":${LAYERDIR}"
+
+BBFILE_COLLECTIONS += "ota"
+BBFILE_PATTERN_ota := "^${LAYERDIR}/"
+BBFILE_PRIORITY_ota = "6"
+
+# This should only be incremented on significant changes that will
+# cause compatibility issues with other layers
+LAYERVERSION_ota = "1"
+
+LAYERDEPENDS_ota = "core openembedded-layer"
+
+LAYERSERIES_COMPAT_ota = "zeus"
-- 
2.19.1