[Openembedded-architecture] Sharing sstate cache and detecting build host paths in -native recipes

[Phil Wise]

On 29.03.2017 12:13, Richard Purdie wrote:
> On Wed, 2017-03-29 at 11:06 +0200, Phil Wise wrote:
>> Recently we've started using SSTATE_MIRRORS to share intermediate
>> build output between machines, and have run into a few issues where
>> -native recipes end up baking in hard-coded paths to their build
>> hosts. One example would be [1], but many Linux executables hard-code 
>> paths to resources, and I would consider it to be 'normal' Linux
>> behavior.
>>
>> So far we've fixed this in a whack-a-mole fashion by either changing
>> code to pass in paths at run time, or by making the recipe explicit
>> about it's dependency:
>>
>> SYSROOT_DIR = "${STAGING_DIR_TARGET}"
>> SYSROOT_DIR_class-native = "${STAGING_DIR_NATIVE}"
>> do_configure[vardeps] += "SYSROOT_DIR"
>>
>>
>> What is the best way to solve this permanently?
> 
> We've been doing this for a while on the yocto project autobuilders and
> so far in practice, we've not found there to be huge amounts of issues.
> We've been approaching it with fixing issues if/as/when we see them.
> The recently change to use RSS in master basically makes it a hard
> requirement for things to relocate and if they don't errors are much
> more obvious so we do have a commitment to making this work.

OK, perhaps we'll go down the 'test it and see what breaks' route.  I'll
have to make sure that the build directories change between the 'create
sstate cache' and 'test sstate cache' steps, but this isn't too hard to
manage.  We also need to make sure that these operations (which involve
talking to remote servers) are exercised when testing the build from
sstate cache.

> We do have a few tricks to fix things. If you grep oe-core for
> "create_wrapper", you'll see recipes like git, perl, rpm and so on use
> this trick to set environment variables in a wrapper script which then
> allows them to operate correctly. We have patches sources in some cases
> to allow ensure we have the right environment variables available.

I hadn't seen create_wrapper before, thank you for the pointer.  We
actually control the invocations of the tool that is getting built, but
maybe it is better style to use the wrapper script.

> The trick mentioned in #9883 is an extension to this, one we already
> use in nativesdk for some binaries where we move symbols into their own
> section and rewrite this at installation time. This is more invasive
> than the wrapper scripts and likely not upstreamable though so we only
> use it for key problems like gcc/glibc right now.
> 
> We also pass parameters into some of the tools we use to ensure they do
> use the right paths.
> 
> Hopefully you're only running into a small number of problems and once
> fixed, you should be good.

I think we've hit 2 or 3 issues so far, but they have been found rather
late in the testing process because our automated tests didn't check the
operation of the SSTATE cache.

Best Regards,

Phil
-- 
Phil Wise, ATS Advanced Telematic Systems GmbH
Kantstrasse 162, 10623 Berlin
Managing Directors: Dirk Pöschl, Armin G. Schmidt
Register Court: HRB 151501 B, Amtsgericht Charlottenburg