[OE-core] [PATCH v2] python3{,-native}: update to 3.7.0

Wed Sep 19 06:11:59 UTC 2018

On 9/17/2018 12:00 PM, Jens Rehsack wrote:
> Hi Alejandro,
>
> Am Mi., 12. Sep. 2018 um 23:03 Uhr schrieb Alejandro Hernandez
> <alejandro.enedino.hernandez-samaniego at xilinx.com>:
>> Hello Jens,
>>
>>
>> On 9/12/2018 4:53 AM, Jens Rehsack wrote:
>>
>>
>>
>> Am 11.09.2018 um 20:56 schrieb Alejandro Enedino Hernandez Samaniego <alejandro.enedino.hernandez-samaniego at xilinx.com>:
>>
>> Hey Jens,
>>
>> Hey Alejandro,
>>
>> On 09/10/2018 11:58 PM, Jens Rehsack wrote:
>>
>>
>>
>> Am 10.09.2018 um 23:33 schrieb Alejandro Enedino Hernandez Samaniego <alejandro.enedino.hernandez-samaniego at xilinx.com>:
>>
>> Hey Jens,
>>
>>
>> As I explained before, when you create a manifest for python (target), it uses the native build as base (it literally runs the native python that was just built), it is assumed its the same version as target and contains all the modules provided by upstream, otherwise the missing modules cannot be checked for dependencies, and the manifest becomes incoherent, so its not an option to have an incomplete python native build.
>>
>> In that case, uuid for target never gets deployed, but it is. And I didn't see any packaging issues for `python3` nor for `nativesdk-python3`
>>
>>
>>
>> I don't see what that has to do with anything, fixing the native build should not affect what gets deployed on target, thats exactly why we have a manifest, so they user can decide what to install and what not to install.
>>
>>
>> The manifest isn't used for python3-native.
>> You try to argue whether there is an error to be fixed - and I don't agree.
>> Each fix requires effort - and that's why some errors will never become fixed. When the impact is reasonable or high enough, fixes are more likely.
>>
>> So: before you try to force me into "that's all ugly und must be seriously beautified just because ..." I will not do anything.
>> When you give me a sane reason, I try to understand and make a decision.
>>
>>
>>
>> Ok first of all, I am not forcing you to do anything, and I've never said its all ugly, so please don't reply aggressively since I am just trying to help you get your patch merged, and it needs to work correctly for that to happen, this is a community and if you don't want to do something just don't do it, we're all trying to make this work in the best possible way, whether we get paid for it or do it for fun.
> Sounded a bit different, thanks for clarification.
>
>> Second of all, I know the manifest is not used for python3-native, I never said python3-native used the manifest, but I am trying to explain to you why the fix is necessary, and I am even telling you what the fix is, so you don't have to waste your time and effort on that, here it goes once again:
>>
>>
>> There are python3-native and python3 packages.
>>
>> The python3 package uses a manifest file, to go through the installed files by the whole python installation from upstream, and separates it in several packages, providing more granularity and giving users the possibility of installing a trimmed version of python, only with what they require on their image.
>>
>> I think we're good until here.
>>
>> The problem here comes because we don't control upstream python and it changes with every release, so new files show up, for example what I mentioned before sha3.*.so.
>>
>> Since that file is not on the manifest, during packaging, it will end up on python3-misc, which contains basically everything that doesn't belong anywhere else, although clearly this file in specific should belong in python3-crypt.
>>
>> What this means for the user, is that he/she will install python3-crypt and expect it to work at runtime, but this wont happen, and when sha3 gets imported (or something that depends on it to run), it will fail,  this will cause a bug on our system, which we would need to fix (yes, fixing bugs takes effort and time).
>>
>> At this point an usual fix would be to just install python3-misc as well, because it contains the required library, but python3-misc also contains other unnecessary stuff that the user doesn't need, but anyways an RDEPENDS to python3-misc would somewhat solve it. The correct fix however, is to add the sha3.*.so file to the manifest, on the python3-crypt package, that way, it will get installed and the user would be able to use it correctly. This would solve one problem, but its still not enough, because, I can see (from the log), that there are some dependencies from other packages to sha3, which means that yes, it will work correctly if we decide to install python3-crypt, but if we just install one of the other packages that need it, it won't,  again, it will fail to import it and its the same story, its introducing a bug into the system.
> Everything clear until here.
>
>> OK, up to this point I've explained how this will cause bugs in our system, now, what does this fix that I mentioned has to do with everything?.
>>
>> There is a special task for the python3 package (not native), that creates a manifest for us, this task tries to import every module from the manifest and get its dependencies, for example, if the python3-numbers package needed the sha3 library (this is just an example I really don't know if numbers needs this), this will be picked up, and the python3-numbers package will RDEPEND on python3-crypt, hence solving both the bugs I mentioned before, even before they exist.
> I get lost at this point and probably getting that resolved - which
> task is it, how is the python3-manifest.json involved or better: how
> can this become automatically updated?
>
>> Now why is a fix to python3-native important for python3 target?
>>
>> Because of the way the previously mentioned script works, it calls python3-native (it has to, because it can't use python from host for example, because its not guaranteed its the same version of python3 target, and we need it to contain exactly the same files) to import all modules in an automated manner and get their dependencies, that's how we know if there are new files and/or rdependencies between python3 packages.
>>
>> So what will happen if the python3-native build isn't complete?, whatever file its missing, we wont be able to catch dependencies to it, causing them to end up in python3-misc, causing the issue #1, and also, in any case, there wont be an RDEPENDS created to python3-misc (which would solve issue #1), sometimes files do belong in python3-misc, but we need to know that so we can add an RDEPENDS to it, but again if we just add an RDEPENDS to python3-misc we'll be causing issue/bug #2, which we won't catch, until someone actually tries to run something on a running image and gets an import failure on their application.
>>
>>
>> I'm not even asking you to fix the manifest, I am simply asking you to provide a functional native package (in this case, functional means with all modules built), so it doesn't break the current functionality that helps us catch these bugs. I hope its clear now that I am not asking you to just change something because I want "beautiful" code everywhere and I never was, I am asking you to do it because your patch will introduce bugs into the system, and I am trying to help you fix them.
> That's all an implication of one big unknown - that
> python3-manifest.json has to be upgraded by invoking a special task.
> Once I understand what's to be done, I'll add an appropriate note for
> future updaters.

I mentioned this task even on the first reply to the patch v1, it is on 
the recipe itself, its called do_create_manifest and IIRC it does indeed 
have a note that says that it needs to be called with every new release 
of Python and that it should be called only with a fully working 
python3-native build.

Alejandro

>
>> Alejandro
>>
>>
>>
>> I know you didn't see any packaging issues, but that doesn't mean they don't exist, just from the log I showed you, I can tell you that the python3-crypt package is not created correctly, for example, if you do:
>>
>> IMAGE_INSTALL_append = " python3-crypt"
>>
>> Boot the image, run python3 and you try to import sha3, it will fail, because the sha3*.so library won't be on the filesystem.
>>
>>
>> And thats because the sha3.*.so library was just introduced in this upgrade, and our manifest isn't aware it exists, hence it'll end up on python3-misc and you have just created an unnecessary dependency from python3-crypt to python3-misc (and worse, a dependency which were not even aware of, until we test it manually), which beats the whole purpose of the manifest.
>>
>> The do_package function is not gonna fail just because, so you won't see errors, but the files will be packaged incorrectly, causing runtime errors as a consequence, the create_manifest task tries to solve these runtime issues before they happen.
>>
>>
>> That's something completely different. I'm not a python guy and I don't get from the ChangeLog what I have to change in the manifest.
>> I can and will do such stuff when doing the perl update. For python, I ask for review because of things I don't know.
>>
>> You can give me a reasonable list of such changes and I update the manifest accordingly when I rebase the patch after Ross PGO patches are committed.
>>
>> Cheers,
>>
>> Alejandro
>>
>>
>> Yes you probably need a patch to look at the correct directories for the h files, as well as a dependency to make the h files available on recipe-sysroot-native.
>>
>> Please check the submission.
>>
>>
>> I did, its not checking inside recipe-sysroot-native.
>>
>>
>> I know. You could check the submission anyway and find the right patch and then argue differently.
>>
>> Cheers
>> --
>> Jens Rehsack - rehsack at gmail.com
>>
>> --
>> _______________________________________________
>> Openembedded-core mailing list
>> Openembedded-core at lists.openembedded.org
>> http://lists.openembedded.org/mailman/listinfo/openembedded-core