[OE-core] [PATCH v2] python3{,-native}: update to 3.7.0

Wed Sep 12 21:02:41 UTC 2018

Hello Jens,

On 9/12/2018 4:53 AM, Jens Rehsack wrote:
>
>
>> Am 11.09.2018 um 20:56 schrieb Alejandro Enedino Hernandez Samaniego 
>> <alejandro.enedino.hernandez-samaniego at xilinx.com 
>> <mailto:alejandro.enedino.hernandez-samaniego at xilinx.com>>:
>>
>> Hey Jens,
>>
> Hey Alejandro,
>
>> On 09/10/2018 11:58 PM, Jens Rehsack wrote:
>>>
>>>
>>>> Am 10.09.2018 um 23:33 schrieb Alejandro Enedino Hernandez 
>>>> Samaniego <alejandro.enedino.hernandez-samaniego at xilinx.com 
>>>> <mailto:alejandro.enedino.hernandez-samaniego at xilinx.com>>:
>>>>
>>>> Hey Jens,
>>>>
>>>>
>>>> As I explained before, when you create a manifest for python 
>>>> (target), it uses the native build as base (it literally runs the 
>>>> native python that was just built), it is assumed its the same 
>>>> version as target and contains all the modules provided by 
>>>> upstream, otherwise the missing modules cannot be checked for 
>>>> dependencies, and the manifest becomes incoherent, so its not an 
>>>> option to have an incomplete python native build.
>>>>
>>> In that case, uuid for target never gets deployed, but it is. And I 
>>> didn't see any packaging issues for `python3` nor for 
>>> `nativesdk-python3`
>>
>>
>> I don't see what that has to do with anything, fixing the native 
>> build should not affect what gets deployed on target, thats exactly 
>> why we have a manifest, so they user can decide what to install and 
>> what not to install.
>
> The manifest isn't used for python3-native.
> You try to argue whether there is an error to be fixed - and I don't 
> agree.
> Each fix requires effort - and that's why some errors will never 
> become fixed. When the impact is reasonable or high enough, fixes are 
> more likely.
>
> So: before you try to force me into "that's all ugly und must be 
> seriously beautified just because ..." I will not do anything.
> When you give me a sane reason, I try to understand and make a decision.

Ok first of all, I am not forcing you to do anything, and I've never 
said its all ugly, so please don't reply aggressively since I am just 
trying to help you get your patch merged, and it needs to work correctly 
for that to happen, this is a community and if you don't want to do 
something just don't do it, we're all trying to make this work in the 
best possible way, whether we get paid for it or do it for fun.

Second of all, I know the manifest is not used for python3-native, I 
never said python3-native used the manifest, but I am trying to explain 
to you why the fix is necessary, and I am even telling you what the fix 
is, so you don't have to waste your time and effort on that, here it 
goes once again:

There are python3-native and python3 packages.

The python3 package uses a manifest file, to go through the installed 
files by the whole python installation from upstream, and separates it 
in several packages, providing more granularity and giving users the 
possibility of installing a trimmed version of python, only with what 
they require on their image.

I think we're good until here.

The problem here comes because we don't control upstream python and it 
changes with every release, so new files show up, for example what I 
mentioned before sha3.*.so.

Since that file is not on the manifest, during packaging, it will end up 
on python3-misc, which contains basically everything that doesn't belong 
anywhere else, although clearly this file in specific should belong in 
python3-crypt.

What this means for the user, is that he/she will install python3-crypt 
and expect it to work at runtime, but this wont happen, and when sha3 
gets imported (or something that depends on it to run), it will fail,  
this will cause a bug on our system, which we would need to fix (yes, 
fixing bugs takes effort and time).

At this point an usual fix would be to just install python3-misc as 
well, because it contains the required library, but python3-misc also 
contains other unnecessary stuff that the user doesn't need, but anyways 
an RDEPENDS to python3-misc would somewhat solve it. The correct fix 
however, is to add the sha3.*.so file to the manifest, on the 
python3-crypt package, that way, it will get installed and the user 
would be able to use it correctly. This would solve one problem, but its 
still not enough, because, I can see (from the log), that there are some 
dependencies from other packages to sha3, which means that yes, it will 
work correctly if we decide to install python3-crypt, but if we just 
install one of the other packages that need it, it won't, again, it will 
fail to import it and its the same story, its introducing a bug into the 
system.

OK, up to this point I've explained how this will cause bugs in our 
system, now, what does this fix that I mentioned has to do with 
everything?.

There is a special task for the python3 package (not native), that 
creates a manifest for us, this task tries to import every module from 
the manifest and get its dependencies, for example, if the 
python3-numbers package needed the sha3 library (this is just an example 
I really don't know if numbers needs this), this will be picked up, and 
the python3-numbers package will RDEPEND on python3-crypt, hence solving 
both the bugs I mentioned before, even before they exist.

Now why is a fix to python3-native important for python3 target?

Because of the way the previously mentioned script works, it calls 
python3-native (it has to, because it can't use python from host for 
example, because its not guaranteed its the same version of python3 
target, and we need it to contain exactly the same files) to import all 
modules in an automated manner and get their dependencies, that's how we 
know if there are new files and/or rdependencies between python3 packages.

So what will happen if the python3-native build isn't complete?, 
whatever file its missing, we wont be able to catch dependencies to it, 
causing them to end up in python3-misc, causing the issue #1, and also, 
in any case, there wont be an RDEPENDS created to python3-misc (which 
would solve issue #1), sometimes files do belong in python3-misc, but we 
need to know that so we can add an RDEPENDS to it, but again if we just 
add an RDEPENDS to python3-misc we'll be causing issue/bug #2, which we 
won't catch, until someone actually tries to run something on a running 
image and gets an import failure on their application.

I'm not even asking you to fix the manifest, I am simply asking you to 
provide a functional native package (in this case, functional means with 
all modules built), so it doesn't break the current functionality that 
helps us catch these bugs. I hope its clear now that I am not asking you 
to just change something because I want "beautiful" code everywhere and 
I never was, I am asking you to do it because your patch will introduce 
bugs into the system, and I am trying to help you fix them.

Alejandro

>
>> I know you didn't see any packaging issues, but that doesn't mean 
>> they don't exist, just from the log I showed you, I can tell you that 
>> the python3-crypt package is not created correctly, for example, if 
>> you do:
>>
>> IMAGE_INSTALL_append = " python3-crypt"
>>
>> Boot the image, run python3 and you try to import sha3, it will fail, 
>> because the sha3*.so library won't be on the filesystem.
>>
>>
>> And thats because the sha3.*.so library was just introduced in this 
>> upgrade, and our manifest isn't aware it exists, hence it'll end up 
>> on python3-misc and you have just created an unnecessary dependency 
>> from python3-crypt to python3-misc (and worse, a dependency which 
>> were not even aware of, until we test it manually), which beats the 
>> whole purpose of the manifest.
>>
>> The do_package function is not gonna fail just because, so you won't 
>> see errors, but the files will be packaged incorrectly, causing 
>> runtime errors as a consequence, the create_manifest task tries to 
>> solve these runtime issues before they happen.
>
> That's something completely different. I'm not a python guy and I 
> don't get from the ChangeLog what I have to change in the manifest.
> I can and will do such stuff when doing the perl update. For python, I 
> ask for review because of things I don't know.
>
> You can give me a reasonable list of such changes and I update the 
> manifest accordingly when I rebase the patch after Ross PGO patches 
> are committed.
>
>> Cheers,
>>
>> Alejandro
>>
>>
>>>> Yes you probably need a patch to look at the correct directories 
>>>> for the h files, as well as a dependency to make the h files 
>>>> available on recipe-sysroot-native.
>>>>
>>> Please check the submission.
>>
>> I did, its not checking inside recipe-sysroot-native.
>
> I know. You could check the submission anyway and find the right patch 
> and then argue differently.
>
> Cheers
> -- 
> Jens Rehsack - rehsack at gmail.com <mailto:rehsack at gmail.com>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openembedded.org/pipermail/openembedded-core/attachments/20180912/4b7b4797/attachment-0002.html>