[Openembedded-architecture] Host tools available from PATH

[Martin Jansa]

On Thu, Mar 09, 2017 at 11:39:14AM +0000, Richard Purdie wrote:
> We have a determinism problem with recipe specific sysroots (RSS)
> highlighted in that tools from the host in PATH can mean recipes work
> on some systems and not others. The host having pkg-config installed
> and the recipe missing an "inherit pkgconfig" is one such example.
> 
> There are two basic ways we can improve the situation, one is the
> approach Peter sent patches for which is to have "poisoned" tools in
> PATH so the host ones don't get used. This does mean adding such a tool
> for each one we want to intercept.
> 
> The other approach is to create a "known good" list of tools and put
> this as a replacement for PATH. I've created such a patch as a proof of
> concept to see what challenges such an approach might bring (included
> at the end of this mail).
> 
> Firstly, why is this important? OE has a reputation for not being 100%
> deterministic. RSS goes a long way to address that however this current
> issue is probably going to mean we can't shake that reputation unless
> we solve it too. I believe if we merge the blacklist approach, whilst
> on a technical level there isn't an issue, our reputation will suffer
> badly. I'm therefore fast coming to the conclusion that we need to
> implement some kind of whitelist ASAP. I believe strongly enough in
> this that I may hold 2.3 M3 (and hence the feature freeze) until we
> sort this.
> 
> My patch is rather creative/evil. By adding it in layer.conf, it works
> before any real parsing happens and therefore gets injected very early
> into the process. That is good and the issues of the currently bad user
> feedback can be fixed. The downside to it is that its not user
> extensible. If you have a missing utility that you need for local
> changes (e.g. tasksel in my case or Martin's use of icecc), you're out
> of luck in being able to add them.
> 
> I see a few possible options:
> 
> a) My current approach and then later we give users the option of 
>    extending the list (and the directory) at ConfigParsed time
> b) Parse with the system PATH and switch to a restricted set at   
>    ConfigParsed time.
> c) Just use the restricted set in bitbake worker (execution time).
> d) Some other combination
> 
> I think I'm leaning towards a). That would allow the later second stage
> processing to handle user or class additions to the toolset. In fact
> thinking about this more, I may be able to do better as there are
> probably only a limited set of things needed for the ConfigParsed
> stage, maybe even just uname.

I'm leaning towards c), because of the "then later we give users the
option of extending the list" in option a).

I can imagine many host tools being used in some other layers and yes
it's dangerous and yes people should provide native recipes for sake for
determinism, but without easy way to extend (or even completely disable)
this without modifying oe-core repo itself it might hurt test-coverate
close to the release.

You probably have better data, but from my experience most
undeterministic behavior was actually from execution time (like tasks
failing only on some hosts) and very rarely from parse time.

So in my POV c) would prevent most of the issues and a) can be
implemented as improvement later (together with extending) in 2.4

Just my 2c
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 195 bytes
Desc: Digital signature
URL: <http://lists.openembedded.org/pipermail/openembedded-architecture/attachments/20170309/5f805dc7/attachment-0002.sig>