[OE-core] python recipes with extensions fail to find/link with proper libraries

[Francesco Giancane]

Hi Alejandro,

Thank you for your response.
I know that openblas recipe is a pain, given that its Makefile is not
correctly written with respect to cross-compilation. However, the library
has been built locally on the target (sigh!) and then it is being deployed
with a recipe which unpacks the built binary and copy them into the sys
root.

So, from the openblas side, everything is OK.

What is actually wrong is the configuration step for python3-numpy and
python3-scipy: in the setup.py they are instructed to search for some
libraries (namely, openblas amongst them) but they fail to search for them
given that the path they are searching for is the sysroot-native, rather
than sysroot. (you can see it clearly from the logs attached in the first
message).

Installing the packages using pip3 install scipy numpy on the target
results on a correct execution (and libraries are properly found and
linked).

Thank you,
Francesco Giancane




Il giorno ven 4 mag 2018 alle ore 03:01 Alejandro Enedino Hernandez
Samaniego <alejandr at xilinx.com> ha scritto:

> Hello Francesco,
>
>
>
> It would be helpful if you could provide more information about this, e.g.
> access to your layer or your recipes, to make sure they are actually being
> built correctly, also I would start by manually linking it against the
> built library to see if the error is just not having them on the correct
> path, but I seem to remember a lot of issues when trying to cross compile
> openblas, specially for an arm MACHINE.
>
>
>
> Alejandro
>
>
>
> *From:* openembedded-core-bounces at lists.openembedded.org [mailto:
> openembedded-core-bounces at lists.openembedded.org] *On Behalf Of *Francesco
> Giancane
> *Sent:* Wednesday, May 2, 2018 2:33 PM
> *To:* OE-core <openembedded-core at lists.openembedded.org>
> *Subject:* [OE-core] python recipes with extensions fail to find/link
> with proper libraries
>
>
>
> Hi Everyone,
>
> I found a problem working with python(3) and C extensions.
>
>
>
> The use-case is the recipe for python(3)-numpy, which can be linked
> against a BLAS/LAPACK library to use accelerated matricial and vectorial
> operations.
>
>
>
> As it is currently shipped, the recipe fails to find any proper library
> and then does not use extensions.
>
>
>
> I wrote a custom recipe for openblas which is correctly built & deploy
> using standard commands.
>
> openblas is also added as a DEPENDS in a bbappend file for numpy.
>
>
>
> However, the do_compile phase cannot find the aforementioned library.
>
>
>
> Log attached:
>
>
>
> DEBUG: Executing shell function do_compile
>
> blas_opt_info:
>
> blas_mkl_info:
>
> customize UnixCCompiler
>
>   libraries mkl_rt not found in
> ['/home/fgp/poky/rpi3_64/tmp/work/cortexa7hf-neon-vfpv4-poky-linux-gnueabi/python3-numpy/1.14.2-r0/recipe-sysroot-native/usr/lib']
>
>   NOT AVAILABLE
>
>
>
> blis_info:
>
> customize UnixCCompiler
>
>   libraries blis not found in
> ['/home/fgp/poky/rpi3_64/tmp/work/cortexa7hf-neon-vfpv4-poky-linux-gnueabi/python3-numpy/1.14.2-r0/recipe-sysroot-native/usr/lib']
>
>   NOT AVAILABLE
>
>
>
> openblas_info:
>
> customize UnixCCompiler
>
> customize UnixCCompiler
>
>   libraries openblas not found in
> ['/home/fgp/poky/rpi3_64/tmp/work/cortexa7hf-neon-vfpv4-poky-linux-gnueabi/python3-numpy/1.14.2-r0/recipe-sysroot-native/usr/lib']
>
>   NOT AVAILABLE
>
>
>
> atlas_3_10_blas_threads_info:
>
> Setting PTATLAS=ATLAS
>
> customize UnixCCompiler
>
>   libraries tatlas not found in
> ['/home/fgp/poky/rpi3_64/tmp/work/cortexa7hf-neon-vfpv4-poky-linux-gnueabi/python3-numpy/1.14.2-r0/recipe-sysroot-native/usr/lib']
>
>   NOT AVAILABLE
>
>
>
> atlas_3_10_blas_info:
>
> customize UnixCCompiler
>
>   libraries satlas not found in
> ['/home/fgp/poky/rpi3_64/tmp/work/cortexa7hf-neon-vfpv4-poky-linux-gnueabi/python3-numpy/1.14.2-r0/recipe-sysroot-native/usr/lib']
>
>   NOT AVAILABLE
>
>
>
> atlas_blas_threads_info:
>
> Setting PTATLAS=ATLAS
>
> customize UnixCCompiler
>
>   libraries ptf77blas,ptcblas,atlas not found in
> ['/home/fgp/poky/rpi3_64/tmp/work/cortexa7hf-neon-vfpv4-poky-linux-gnueabi/python3-numpy/1.14.2-r0/recipe-sysroot-native/usr/lib']
>
>   NOT AVAILABLE
>
>
>
> atlas_blas_info:
>
> customize UnixCCompiler
>
>   libraries f77blas,cblas,atlas not found in
> ['/home/fgp/poky/rpi3_64/tmp/work/cortexa7hf-neon-vfpv4-poky-linux-gnueabi/python3-numpy/1.14.2-r0/recipe-sysroot-native/usr/lib']
>
>   NOT AVAILABLE
>
>
>
> blas_info:
>
> customize UnixCCompiler
>
>   libraries blas not found in
> ['/home/fgp/poky/rpi3_64/tmp/work/cortexa7hf-neon-vfpv4-poky-linux-gnueabi/python3-numpy/1.14.2-r0/recipe-sysroot-native/usr/lib']
>
>   NOT AVAILABLE
>
>
>
> blas_src_info:
>
>   NOT AVAILABLE
>
>
>
>   NOT AVAILABLE
>
>
>
> ...And it is correct for it to fail as it is searching for openblas in the
> wrong path (sysroot-native vs sysroot).
>
>
>
> I would assume that this is a setuptools.bbclass issue, does anyone know
> what to do? Is it trivial to fix?
>
>
>
> I noticed that all the shipped python extensions were built as if we were
> building for x86, even though the MACHINE is an arm target.
>
>
>
> Thank you for your attention,
>
>
> Francesco Giancane
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openembedded.org/pipermail/openembedded-core/attachments/20180504/ac0bec81/attachment-0002.html>