[Openembedded-architecture] Host tools available from PATH

[Patrick Ohly]

On Thu, 2017-03-09 at 11:39 +0000, Richard Purdie wrote:
> Firstly, why is this important? OE has a reputation for not being 100%
> deterministic. RSS goes a long way to address that however this current
> issue is probably going to mean we can't shake that reputation unless
> we solve it too. I believe if we merge the blacklist approach, whilst
> on a technical level there isn't an issue, our reputation will suffer
> badly. I'm therefore fast coming to the conclusion that we need to
> implement some kind of whitelist ASAP. I believe strongly enough in
> this that I may hold 2.3 M3 (and hence the feature freeze) until we
> sort this.

I agree that resolving this is critical. Personally, I'd prefer the
whitelisting approach that you outlined here because I expect it to be
more complete.

> My patch is rather creative/evil. By adding it in layer.conf, it works
> before any real parsing happens and therefore gets injected very early
> into the process. That is good and the issues of the currently bad user
> feedback can be fixed. The downside to it is that its not user
> extensible. If you have a missing utility that you need for local
> changes (e.g. tasksel in my case

tasksel or taskset? I lacked taskset myself.

>  or Martin's use of icecc), you're out
> of luck in being able to add them.
> 
> I see a few possible options:
> 
> a) My current approach and then later we give users the option of 
>    extending the list (and the directory) at ConfigParsed time
> b) Parse with the system PATH and switch to a restricted set at   
>    ConfigParsed time.
> c) Just use the restricted set in bitbake worker (execution time).
> d) Some other combination
> 
> I think I'm leaning towards a). That would allow the later second stage
> processing to handle user or class additions to the toolset. In fact
> thinking about this more, I may be able to do better as there are
> probably only a limited set of things needed for the ConfigParsed
> stage, maybe even just uname.

Anonymous python functions run before ConfigParsed, right? They might
depend on additional tools, although I can't come up with a specific
example.

Overall approach a) sounds reasonable.

> I'm sending this to ask whether people feel this is worth getting right
> for 2.3 and whether we should hold M3 and the feature freeze to get
> something implemented?

My vote would be for doing it in 2.3.

>  This would assume we then are prepared to take
> the pain of actually populating these lists (I've made a start for OE-
> Core at least).

Regarding that: what is "flex" needed for? My build host hasn't had it
and I never ran into problems.

If a tool is missing, there should be a good error message. Right now, I
only got:
        
          File "/work/poky/bitbake/lib/bb/utils.py", line 1538, in setup_native_bindir(dest='/fast/build/poky/x86/tmp/hosttools', toolsvar='HOSTTOOLS', d=<bb.data_smart.DataSmart object at 0x7fcb53071fd0>):
                     if not os.path.exists(desttool):
                         srctool = which(path, tool)
            >            os.symlink(srctool, desttool)
        FileNotFoundError: [Errno 2] No such file or directory: '' -> '/fast/build/poky/x86/tmp/hosttools/flex'
        
Some tool is needed for "bitbake -c devshell".

systemd-systemctl-native.bb needs taskset:

        ERROR: Traceback (most recent call last):
          File "/work/poky/bitbake/bin/bitbake-worker", line 274, in
        child
            subprocess.check_output("taskset -pc %s %s" % (setcpu,
        os.getpid()), shell=True)
          File "/usr/lib/python3.4/subprocess.py", line 620, in
        check_output
            raise CalledProcessError(retcode, process.args,
        output=output)
        subprocess.CalledProcessError: Command 'taskset -pc 1 26875'
        returned non-zero exit status 127
        
        ERROR: Task
        (/work/poky/meta/recipes-core/systemd/systemd-systemctl-native.bb:do_configure) failed with exit code '1'

With taskset added to your list, I get:

    ....
        WARNING: Setting to cpu 10
        WARNING: Setting to cpu 10
        WARNING: Setting to cpu 8
        ...

> Also to be clear, if we do want to get this into 2.3, it is going to
> mean moving fairly fast. I think we can get this right quite quickly
> based on my experiments so far and I know others are also running them.
> That means if anyone objects to this (and moving quickly) I need to
> know about it now.

I had wondered how this would affect ASSUME_PROVIDED. Clearly if
ASSUME_PROVIDED is for a certain native tool, that tool now needs to be
added to the list. Bonus points if setting ASSUME_PROVIDED automatically
extends the list, at least as far as known components are concerned.

But what about ASSUME_PROVIDED += "libsdl-native"? That still works
because the build host's pkg config paths explicitly gets made available
in qemu-native's run.do_configure via qemu.inc:

        do_configure_prepend_class-native() {
                # Append build host pkg-config paths for native target since the host may provide sdl
                BHOST_PKGCONFIG_PATH=$(PATH=/usr/bin:/bin pkg-config --variable pc_path pkg-config || echo "")
                if [ ! -z "$BHOST_PKGCONFIG_PATH" ]; then
                        export PKG_CONFIG_PATH=$PKG_CONFIG_PATH:$BHOST_PKGCONFIG_PATH
                fi

So that loophole exists only for qemu-native, right? In that case, other
native recipes avoid potential contamination with the host's build
files, which probably is good enough.

-- 
Best Regards, Patrick Ohly

The content of this message is my personal opinion only and although
I am an employee of Intel, the statements I make here in no way
represent Intel's position on the issue, nor am I authorized to speak
on behalf of Intel on this matter.