[OE-core] make-mod-scripts question

Mon Sep 9 20:25:05 UTC 2019

hi,

things are getting worse , my initial problem was only the visible
piece of the iceberg ;)

On Thu, Sep 5, 2019 at 5:42 PM Bruce Ashfield <bruce.ashfield at gmail.com> wrote:
>
> On Thu, Sep 5, 2019 at 10:43 AM Nicolas Dechesne
> <nicolas.dechesne at linaro.org> wrote:
> >
> > hi,
> >
> > I was looking into a build issue today, related to signing an external
> > kernel module. My build failed when calling the kernel utility
> > sign-file, the error was that libcrypto.so was not found.
> >
> > Looking a bit more into that, I realized that make-mod-scripts will
> > built some kernel utilities, including scripts/sign-file which link
> > against libcrypto, however we 'copy' sign-file into
> > STAGING_KERNEL_BUILDDIR.
>
> Indeed, the build and redirected output dir will get the binaries, but
> not the libraries. Either static linking of a smarter install of the
> binaries would be a good idea.
>
> >
> > Later on, when my own recipe uses sign-file from
> > STAGING_KERNEL_BUILDDIR, it still requires the libcrypto.so file which
> > is in the make-mod-scripts per recipe sysroot. So basically my recipe
> > needs a binary from STAGING_KERNEL_BUILDDIR and a shared library from
> > another recipe sysroot. That looks wrong to me.
> >
> > My problem is fixed by adding a dependency on make-mod-scripts, it
> > still feels wrong to reply on another recipe sysroot , doesn't it go
> > against the whole idea of per recipe sysroot?
>
> Agreed.
>
> I'm wondering make-mod-scripts should both create a package and
> install into the staged kernel build dir.
>
> That way if you had a dependency on make-mod-scripts, you'd use both
> the binary and the library from the recipe sysroot, but we'd also get
> that shared copy for the needs in the shared kernel build directory.
> Obviously the shared kernel build artifacts are a special case, and
> I'm not even sure if what I'm thinking is even possible .. or is worse
> than the symptom you are describing.
>
> Just thinking out loud. Hopefully someone with more knowledge on the
> packaging/internal/sstate internals will see this and have more
> details.

you are correct, and i hope we can fix that. However what I have now
realized is that building external modules with modules signing
enabled seems to be broken.

The kernel generates the keys pair and certificate when it's built and
the public key is embedded in the kernel image. The keys are put in
STAGING_KERNEL_DIR, where module_do_compile() expects it. And it works
just fine, until you delete TMPDIR, and build again. For example if
you have a change in your module recipe, and build again. Because it's
an external module, it has:
do_configure[depends] += "make-mod-scripts:do_compile"

and make-mod-scripts has the following depends:
do_configure[depends] += "virtual/kernel:do_shared_workdir
openssl-native:do_populate_sysroot"
do_compile[depends] += "virtual/kernel:do_compile_kernelmodules"

So the kernel will effectively be rebuilt , and a new pair of keys is
generated and placed in STAGING_KERNEL_DIR. Then the external module
is recompiled and signed with the *new* key. However since the kernel
recipe didn't change the kernel do_deploy() is not run, and artifacts
are used from the sstate-cache. So we eventually end up with
vmlinuz/Image built with the *old* key and the new kernel module
signed with the *new* key.

Here is a log of what happens in my build env that demonstrates this
sequence of task:

NOTE: recipe linux-msm-4.9-r5: task do_deploy_setscene: Started
NOTE: recipe linux-msm-4.9-r5: task do_populate_lic_setscene: Started
NOTE: recipe linux-msm-4.9-r5: task do_populate_lic_setscene: Succeeded
NOTE: recipe linux-msm-4.9-r5: task do_package_write_ipk_setscene: Started
NOTE: recipe linux-msm-4.9-r5: task do_package_write_ipk_setscene: Succeeded
NOTE: recipe linux-msm-4.9-r5: task do_deploy_setscene: Succeeded
NOTE: recipe linux-msm-4.9-r5: task do_package_qa_setscene: Started
NOTE: recipe linux-msm-4.9-r5: task do_package_qa_setscene: Succeeded
NOTE: recipe linux-msm-4.9-r5: task do_packagedata_setscene: Started
NOTE: recipe linux-msm-4.9-r5: task do_populate_sysroot_setscene: Started
NOTE: recipe linux-msm-4.9-r5: task do_populate_sysroot_setscene: Succeeded
NOTE: recipe linux-msm-4.9-r5: task do_packagedata_setscene: Succeeded

NOTE: recipe linux-msm-4.9-r5: task do_fetch: Started

                [3/9744]
NOTE: recipe linux-msm-4.9-r5: task do_fetch: Succeeded
NOTE: recipe linux-msm-4.9-r5: task do_prepare_recipe_sysroot: Started
NOTE: recipe linux-msm-4.9-r5: task do_unpack: Started
NOTE: recipe qcacld-hl-git-r0: task do_fetch: Started
NOTE: recipe qcacld-hl-git-r0: task do_fetch: Succeeded
NOTE: recipe linux-msm-4.9-r5: task do_prepare_recipe_sysroot: Succeeded
NOTE: recipe linux-msm-4.9-r5: task do_unpack: Succeeded
NOTE: recipe linux-msm-4.9-r5: task do_patch: Started
NOTE: recipe linux-msm-4.9-r5: task do_patch: Succeeded
NOTE: recipe linux-msm-4.9-r5: task do_configure: Started
NOTE: recipe linux-msm-4.9-r5: task do_configure: Succeeded
NOTE: recipe linux-msm-4.9-r5: task do_compile: Started
NOTE: recipe linux-msm-4.9-r5: task do_compile: Succeeded
NOTE: recipe linux-msm-4.9-r5: task do_shared_workdir: Started
NOTE: recipe linux-msm-4.9-r5: task do_shared_workdir: Succeeded
NOTE: recipe linux-msm-4.9-r5: task do_compile_kernelmodules: Started
NOTE: recipe linux-msm-4.9-r5: task do_compile_kernelmodules: Succeeded
NOTE: recipe qti-console-image-1.0-r0: task do_prepare_recipe_sysroot: Started
NOTE: recipe qti-console-image-1.0-r0: task do_prepare_recipe_sysroot: Succeeded
NOTE: recipe qti-console-image-1.0-r0: task do_make_bootimg: Started
NOTE: recipe qti-console-image-1.0-r0: task do_make_bootimg: Succeeded
NOTE: recipe qcacld-hl-git-r0: task do_prepare_recipe_sysroot: Started
NOTE: recipe qcacld-hl-git-r0: task do_unpack: Started
NOTE: recipe qcacld-hl-git-r0: task do_unpack: Succeeded
NOTE: recipe qcacld-hl-git-r0: task do_patch: Started
NOTE: recipe qcacld-hl-git-r0: task do_prepare_recipe_sysroot: Succeeded
NOTE: recipe qcacld-hl-git-r0: task do_patch: Succeeded
NOTE: recipe qcacld-hl-git-r0: task do_populate_lic: Started
NOTE: recipe qcacld-hl-git-r0: task do_configure: Started
NOTE: recipe qcacld-hl-git-r0: task do_configure: Succeeded
NOTE: recipe qcacld-hl-git-r0: task do_populate_lic: Succeeded
NOTE: recipe qcacld-hl-git-r0: task do_compile: Started
NOTE: recipe qcacld-hl-git-r0: task do_compile: Succeeded
NOTE: recipe qcacld-hl-git-r0: task do_install: Started
NOTE: recipe qcacld-hl-git-r0: task do_install: Succeeded
NOTE: recipe qcacld-hl-git-r0: task do_populate_sysroot: Started
NOTE: recipe qcacld-hl-git-r0: task do_package: Started
NOTE: recipe qcacld-hl-git-r0: task do_populate_sysroot: Succeeded
NOTE: recipe qcacld-hl-git-r0: task do_package: Succeeded
NOTE: recipe qcacld-hl-git-r0: task do_packagedata: Started
NOTE: recipe qcacld-hl-git-r0: task do_packagedata: Succeeded
NOTE: recipe qcacld-hl-git-r0: task do_package_write_ipk: Started
NOTE: recipe qcacld-hl-git-r0: task do_package_qa: Started
NOTE: recipe qcacld-hl-git-r0: task do_package_qa: Succeeded
NOTE: recipe qcacld-hl-git-r0: task do_package_write_ipk: Succeeded
NOTE: recipe qcacld-hl-git-r0: task do_rm_work: Started
NOTE: recipe qcacld-hl-git-r0: task do_rm_work: Succeeded
NOTE: recipe qti-console-image-1.0-r0: task do_rootfs: Started
NOTE: recipe qti-console-image-1.0-r0: task do_rootfs: Succeeded
NOTE: recipe qti-console-image-1.0-r0: task do_makesystem: Started
NOTE: recipe qti-console-image-1.0-r0: task do_image_qa: Started
NOTE: recipe qti-console-image-1.0-r0: task do_makesystem: Succeeded
NOTE: recipe qti-console-image-1.0-r0: task do_image_qa: Succeeded
NOTE: recipe qti-console-image-1.0-r0: task do_image: Started
NOTE: recipe qti-console-image-1.0-r0: task do_image: Succeeded
NOTE: recipe qti-console-image-1.0-r0: task do_image_ext4: Started
NOTE: recipe qti-console-image-1.0-r0: task do_image_ext4: Succeeded
NOTE: recipe qti-console-image-1.0-r0: task do_image_complete: Started
NOTE: recipe qti-console-image-1.0-r0: task do_image_complete: Succeeded
NOTE: recipe qti-console-image-1.0-r0: task do_populate_lic_deploy: Started
NOTE: recipe qti-console-image-1.0-r0: task do_populate_lic_deploy: Succeeded
NOTE: recipe qti-console-image-1.0-r0: task do_rm_work: Started
NOTE: recipe qti-console-image-1.0-r0: task do_rm_work: Succeeded

where linux-msm is the kernel recipe (inherit kernel), qcacld-hl a
kernel module (which inherit module), and qti-console-image a simple
console image.

If this reasoning is correct that means that kernel signing + external
module is broken. Note that I am working out of Thud for now, i am
sending this email for now to get some feedback, and will try to
reproduce without all our custom layers and with master.

Should we deploy the keys/certificates in the kernel recipe instead of
adding them in STAGING_KERNEL_DIR? Would that be enough?

>
> Bruce
>
> >
> > Here is the sign-file in the kernel shared workdir:
> >
> > $ find tmp-glibc -name sign-file
> > tmp-glibc/work-shared/open-q-212a-homehub/kernel-build-artifacts/scripts/sign-file
> >
> > And here we can see the dependency against make-mod-scripts sysroot:
> > $ ldd tmp-glibc/work-shared/open-q-212a-homehub/kernel-build-artifacts/scripts/sign-file
> >         linux-vdso.so.1 =>  (0x00007fff8299d000)
> >         libcrypto.so.1.1 =>
> > /local/mnt2/workspace2/ndec/le.um.3.4.1/build-wlan/tmp-glibc/work/open_q_212a_homehub-oe-linux-gnueabi/make-mod-scripts/1.0-r0/recipe-sysroot-native/usr/lib/libcrypto.so.1.1
> > (0x00007fcb2247d000)
> >         libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fcb220b3000)
> >         libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007fcb21eaf000)
> >         libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0
> > (0x00007fcb21c92000)
> >         /lib64/ld-linux-x86-64.so.2 (0x00007fcb2293c000)
> >
> > or like that:
> > $ strings tmp-glibc/work-shared/open-q-212a-homehub/kernel-build-artifacts/scripts/sign-file
> > | grep tmp-glib
> > /local/mnt2/workspace2/ndec/le.um.3.4.1/build-wlan/tmp-glibc/work/open_q_212a_homehub-oe-linux-gnueabi/make-mod-scripts/1.0-r0/recipe-sysroot-native/usr/lib:/local/mnt2/workspace2/ndec/le.um.3.4.1/build-wlan/tmp-glibc/work/open_q_212a_homehub-oe-linux-gnueabi/make-mod-scripts/1.0-r0/recipe-sysroot-native/lib
> >
> > cheers
> > nico
>
>
>
> --
> - Thou shalt not follow the NULL pointer, for chaos and madness await
> thee at its end
> - "Use the force Harry" - Gandalf, Star Trek II