[OE-core] Missing venv module in Python 3

Fri Oct 19 13:13:08 UTC 2018

Hello Alejandro,

Thank you for your help.

Unfortunately, the pyvenv package is not in the manifest, I cannot therefore add the path where you are suggesting.

It is not immediately clear to me where the package is created, however, files are added to the pyvenv package (using FILES_${PN}-pyvenv += ) in the python3_3.5.6.bb recipe. This is why I added the ${libdir}/${PYTHON_MAJMIN}/venv file in a bbapend to the python3_3.5.6.bb recipe.

I am able to successfully run the create_manifest3.py script but the generated manifest content has not changed. I did not expect it to change because pyvenv was not there in the first place.

On a different note; I managed to include the venv module by including the python3-misc package to the build. I can create a virtual environment using python -m venv. The issue with that is that python3-misc includes other modules I do not need and thus needlessly increases size. Another issue is that I have to include python3-runpy (to use the -m option when creating the virtual environment).

I also realised that python3-pip is a depencency to venv.

In conclusion, can you think of how dependencies (venv and pip) can be added to the python3-pyvenv package? This will allow reduction of the image size by removing python3-misc and python3-runpy.

Hugues
________________________________
From: Alejandro Enedino Hernandez Samaniego <aehs29 at gmail.com>
Sent: 19 October 2018 13:37:14
To: Hugues Kamba
Cc: Burton, Ross; Jonathan Haigh; OE-core; Jeremy Johnson
Subject: Re: [OE-core] Missing venv module in Python 3

Hello Huges,

The path needs to be added to the manifest itself (the json file) under the files section for the pyvenv package not in a bbappend.

And to run the create manifest task you would do:

$ bitbake python3 -c create_manifest

Once I get to my computer I will be able to provide more help.

Alejandro

On Fri, Oct 19, 2018, 10:08 AM Hugues Kamba <Hugues.Kamba at arm.com<mailto:Hugues.Kamba at arm.com>> wrote:

Hi Alejandro,

Thank you for your suggestion.

The following has been added to a python3_%.bbapend:

# Add missing venv module dependency.
FILES_${PN}-pyvenv += "${libdir}/${PYTHON_MAJMIN}/venv"

However, it is not quite obvious how to successfully run the create_manifest3.py script.
Attempts were made to run it with "python3 create_manifest3.py python3.5" but it complains about an undefined variable (nativelibfolder).
I also tried to run it using "bitbake pyhton -c create_manifest3.py" as specified in the docstring but execution fails because of missing tasks (no do_..).

Could you or others on the mailing list, provide some guidance on how to go about running (arguments, etc..) the create_manifest3.py script.

[cid:bbaed11e-d9b2-456f-bd15-200669ad237e]

Hugues Kamba | Senior Software Engineer
hugues.kamba at arm.com<mailto:sandi.shannon at arm.com>
CPC1, Capital Park, Cambridge Road, Fulbourn, CB21 5XE, UK
Tel: (+44)1223 400400
Visit the new ARM Connected Community<https://urldefense.proofpoint.com/v2/url?u=http-3A__community.arm.com_welcome&d=DwMFAg&c=bKNseOHsszsRCtLFfd6XtA&r=h1JGsEgvfYBPUFak8b6Fs-oV6irVsuuFc4RBESlsTFE&m=N8x4mnijl-wwLmtz3tbzVbe9R-QRW9bVZHfjOv3YY0o&s=FTtAextvaqUli74ujYjbDxp7jK9aaA6G7NOsVfryuyw&e=>

________________________________
From: Alejandro Enedino Hernandez Samaniego <aehs29 at gmail.com<mailto:aehs29 at gmail.com>>
Sent: 18 October 2018 19:18
To: Burton, Ross
Cc: Jonathan Haigh; OE-core; Jeremy Johnson; Hugues Kamba
Subject: Re: [OE-core] Missing venv module in Python 3

Hello folks,

On Thu, Oct 18, 2018, 3:34 PM Burton, Ross <ross.burton at intel.com<mailto:ross.burton at intel.com>> wrote:
On Thu, 18 Oct 2018 at 16:19, Jonathan Haigh <Jonathan.Haigh at arm.com<mailto:Jonathan.Haigh at arm.com>> wrote:
> The venv package (https://docs.python.org/3/library/venv.html) itself is missing although its source is available within Python 3.5 (https://www.python.org/ftp/python/3.5.6/Python-3.5.6.tgz). As mentioned earlier, it seems the issue is because there is no entry in the manifest from which the various Python 3 modules are created (https://github.com/openembedded/openembedded-core/blob/master/meta/recipes-devtools/python/python3/python3-manifest.json).
>
> The manifest is generated from https://github.com/openembedded/openembedded-core/blob/master/meta/recipes-devtools/python/python3/create_manifest3.py, and it does not seem obvious to me how to go about correctly adding a new module to the manifest, using create_manifest3.py or otherwise.

The script itself should get the venv dependency automatically (my guess is that this isn't getting detected by the script because the module itself its coming from a directory) but I'd have to debug what is exactly happening, although I don't have access to my computer at the moment.

An easy fix (for now) is to add the

/usr/lib/python3.5/venv/

Path to the manifest on the FILES for the pyenv module.

After that running the create_manifest task should get its dependencies correctly and update the manifest so it can be pushed to the repo.

Cheers,

Alejandro

Ah, right.

In that case:

$ oe-pkgdata-util find-path */*venv*py
python3-tests: /usr/lib/python3.5/test/test_venv.py
python3-misc: /usr/lib/python3.5/venv/__main__.py
python3-misc: /usr/lib/python3.5/venv/__init__.py

The files are in the python3-misc package, which is the fallback for
everything which isn't explicitly put into another package.  I'd say
the best fix for this would be to edit the manifest so the venv module
goes into the python3-pyvenv package too.

Ross
IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.
IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openembedded.org/pipermail/openembedded-core/attachments/20181019/d9aa9dc5/attachment-0002.html>