[OE-core] [RFC PATCH] package.bbclass: omit .pyc and .pyo file

[Mike Looijmans]

On 01/07/2015 12:16 PM, Burton, Ross wrote:
>
> On 7 January 2015 at 09:23, Mike Looijmans <mike.looijmans at topic.nl
> <mailto:mike.looijmans at topic.nl>> wrote:
>
>     You definitely SHOULD ship the .pyc files. If they don't exist, the
>     interpreter is forced to re-compile the .py source, and will attempt to
>     write the result to the filesystem. It won't cause harm, it won't fail,
>     but it's very inefficient. It's better to let the host do the py->pyc
>     conversion anyway.
>
>     The opposite works just fine: You can omit the .py files and ship only
>     .pyc files. We do that on settopboxes that use Python for the GUI, this
>     saves several megabytes of flash space.
>     To accomplish that, we put the .py files into a $PN-src package.
>
>
> I agree with Mike, there is a use-case for just shipping .pyc.  Whether
> oe-core should do something to assist with this or not is debatable.

We currently have this in a python_2.7.%.bbappend:

PACKAGES =+ "${PN}-src"
RDEPENDS_{PN}-src = "${PN}"
FILES_${PN}-src += "${libdir}/python${PYTHON_MAJMIN}/*.py"
FILES_${PN}-src += "${libdir}/python${PYTHON_MAJMIN}/*/*.py"
FILES_${PN}-src += "${libdir}/python${PYTHON_MAJMIN}/*/*/*.py"

This moves all sources into a single package. I experimented with creating src 
packages for each python sub-package, but all that accomplished was adding a 
lot of packages to the feed that no one would ever install.

I patched the larger Python recipes (e.g. twisted) in similar ways to reduce 
the image size (and network traffic).

A generic "please remove or split all .py files" flag or class would be a 
useful addition.

> There's an open bug report about this:
> https://bugzilla.yoctoproject.org/show_bug.cgi?id=6434.
>
> https://docs.python.org/2/tutorial/modules.html#compiled-python-files has the
> details we're after.  Summary:
>
> .pyc is Python bytecode, which is an implementation detail of CPython.  As
> such it's version-dependent but explicitly architecture-independent.
> .pyo generated with -O is simply .pyc but with asserts removed.
> .pyo generated with -O -O has asserts and docstrings removed.

For horrible historic reasons OpenPLi (and its many forks) still use a patch 
that makes .pyo files the default instead of .pyc. It's been on my todo list 
for over a year now to get rid of it, but too many plugins and 3rd party stuff 
still count on this being the case.

> I think it's fair to say we should just ignore .pyo - no point generating and
> shipping it.  It does seem that if we want to ship usable .pyc we need to
> ensure that they are compiled with python-native. I guess this could be done
> by calling python-native's compileall module to recompile the sources at
> package time.

I'd expect any halfway decent recipe to have done that already in the 
do_compile step. Most, if not all, Python recipes already do so. If anything, 
the core could provide a class that provides a simple do_compile that calls 
compileall on the source dir.

Moving this to packaging will lead to unexpected failures, there are some 
situations where the source .py file must be installed and the .pyc will be 
ignored. A simple example is when the .py script is the application entry 
point, those aren't compiled into pyc at runtime, and adding a pyc would be a 
waste. Only the package's makefile (or whatever) can be expected to know that.

Mike.


Met vriendelijke groet / kind regards,

Mike Looijmans
System Expert


TOPIC Embedded Systems
Eindhovenseweg 32-C, NL-5683 KH Best
Postbus 440, NL-5680 AK Best
Telefoon: (+31) (0) 499 33 69 79
Telefax:  (+31) (0) 499 33 69 70
E-mail: mike.looijmans at topic.nl
Website: www.topic.nl

Please consider the environment before printing this e-mail

Topic zoekt gedreven (embedded) software specialisten!
http://topic.nl/vacatures/topic-zoekt-software-engineers/