[OE-core] [PATCH v3 5/5] crosstap: replace script with new python based implementation

[Victor Kamensky]

On Mon, 14 May 2018, Burton, Ross wrote:

> On 5 April 2018 at 19:25, Victor Kamensky <kamensky at cisco.com> wrote:
>> New crosstap python implementation is total replacement for
>> crosstap shell script, that has superseding capabilities.
>> New script support cross compiling of SystemTap scripts
>> for user-land, by using supplied image rootfs. Whereas old
>> script could only deal with scripts against kernel. New script
>> has more complex logic and additional capabilities.
>
> ...
>
>> +        benv_kernel = BitbakeEnv("virtual/kernel")
>> +        (self.staging_bindir_toolchain,
>> +         self.target_prefix,
>> +         self.target_arch,
>> +         self.target_kernel_builddir
>> +        ) = benv_kernel.get_vars(
>> +             ("STAGING_BINDIR_TOOLCHAIN",
>> +              "TARGET_PREFIX",
>> +              "TRANSLATED_TARGET_ARCH",
>> +              "B"
>> +            ))
>
>> +        if not os.path.isdir(self.target_kernel_builddir):
>> +            print("ERROR: Cannot find '" + self.target_kernel_builddir +
>> +                  "' directory. Was 'kernel/virtual' built?")
>> +            ret = False
>
> You can't assume that B exists and is useful, as if virtual/kernel was
> installed from sstate then all the installed pieces will be present
> but $B will be empty.

Ross, thank you for pointing to this. I will look at it.

Please note use of B of kernel/virtual by crosstap preceeded my
commit:

c6da6b64 (Tom Zanussi 2012-08-31 00:06:57 -0500 109) 
TARGET_KERNEL_BUILDDIR=$(echo "$BITBAKE_VARS" | grep ^B= \
c6da6b64 (Tom Zanussi 2012-08-31 00:06:57 -0500 110)   | cut -d '=' -f2 | 
cut -d '"' -f2)

I just followed the same logic.

> For example, I re-ran oe-selftest on the autobuilder, the kernel was
> extracted from sstate as the previous run built it, and the test
> failed:

Systemtap translator generates kernel module source and it needs
cross compiled environment to build it against proper kernel sources
build artifacts. I.e cross compilation of kernel module should
be supported when kernel is extracted from sstate.

So my thought was to figure out how to cross compile kernel module
against kernel/virtual sstate, I can take a look how recipe that
'inherit module' works in such case. So I've tried the following in
tree pulled today (cryptodev-module is recipe example that
does 'inherit module'):

Variant1:

bitbake cryptodev-module
bitbake -c clean linux-yocto
bitbake -c cleansstate cryptodev-module
bitbake cryptodev-module

Variant2:

bitbake cryptodev-module
mv tmp-glibc tmp-glibc.bak
bitbake linux-yocto
bitbake cryptodev-module

In both cases I see that full kernel rebuild (pull source, compile,
etc) is retriggered by building cryptodev-module, even if
linux-yocto initially is pulled from sstate. In Variant1
cryptodev-module build actually failed because some host utilities
from kernel tree (like scripts/basic/fixdep) were not rebuilt.

I don't think it is proper for crosstap utility to retrigger
full kernel/virtual build. So unless kernel module build against
kernel/virtual sstate without full rebuild of kernel/virtual is
solved I am not sure what I can do in crosstap script.

Currently I see that work-shared/$MACHINE/kernel-source and
work-shared/$MACHINE/kernel-build-artifacts are created and it
seems their purpose is to facilitate external kernel modules
build, and maybe need to full kernel rebuild will be fixed soon.

Do I miss something? Any advise how to address this problem
is appreciated.

Thanks,
Victor

> 2018-05-13 22:36:08,905 - oe-selftest - INFO - FAIL:
> test_crosstap_can_use_systemtap_on_qemu (crosstap.CrossTapTest)
> 2018-05-13 22:36:08,906 - oe-selftest - INFO -
> ----------------------------------------------------------------------
> 2018-05-13 22:36:08,906 - oe-selftest - INFO - Traceback (most recent
> call last):
>  File "/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/meta/lib/oeqa/selftest/cases/crosstap.py",
> line 25, in test_crosstap_can_use_systemtap_on_qemu
>    self.assertIn('hello world', result.output, 'Crosstap failed.')
> AssertionError: 'hello world' not found in 'ERROR: Cannot find
> \'/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/build/tmp/work/qemux86_64-poky-linux/linux-yocto/4.14.30+gitAUTOINC+ea9330894e_74f6cd2b69-r0/linux-qemux86_64-standard-build\'
> directory. Was \'kernel/virtual\' built?
>
> Ross
>