[oe] Python3 bytecode is stale

[Mike Looijmans]

On 25-01-19 18:26, Stefan Agner wrote:
> On 25.01.2019 15:35, Mike Looijmans wrote:
>> Most likely the date/time stamps on the files are such that the "py" files
>> appear to be newer.
> 
> Indeed, this seems to be the issue. We are using OSTree, which removes
> mtimes. Ricardo (now on CC) also pointed me to the relevant issue in
> OSTree:
> https://github.com/ostreedev/ostree/issues/1469
> 
>>
>> A very simple fix would be to simply delete the ".py" files, they aren't
>> really needed anyway so you'll save a lot of space in the process.
>>
>> To do this compile-time, you could add something like this in a bbappend:
>> PACKAGES =+ "${PN}-src"
>> FILES_${PN}-src = "{installdir}/*.py {installdir}/*/*.py"
>>
>> That'll push the "py" files into another package and prevent them being
>> installed by default. Change "{installdir}" into something more appropriate.
>> Of just delete them in a do_install_append.
> 
> Hm, interesting idea, will give this a try.
> 
> Wouldn't be something like this be a nice OpenEmbedded addition in
> general? Optimizing embedded rootfs size by deleting py files seems to
> be a common use case.

Here's a bbappend example for python:

https://github.com/OpenPLi/openpli-oe-core/blob/develop/meta-openpli/recipes-devtools/python/python_2.7.13.bbappend#L20

> In a simple test deleting all py files on my rootfs did not go well (the
> application does not start anymore). I need to investigate further.

The "main" module (the one started from commandline) needs to be in .py 
format still, otherwise you cannot "run" it. As a workaround, you can 
run it using "python -m", or make sure that that one .py file gets 
installed.

It's actually even possible to put the pyc files into a ZIP file and 
load directly from that at runtime. This reduces the number of files. On 
a compressed filesystem, this won't make a difference of course, and is 
likely to increase the total size.

--
M.

> 
> --
> Stefan
> 
>>
>>
>> On 25-01-19 14:46, Stefan Agner wrote:
>>> Hi,
>>>
>>> Recently I noticed that Python3 programs got rather slow. Running the
>>> test application took 12.5 seconds. We are using a read-only file system
>>> and I immediately suspected the bytecode cache. After mounting the
>>> rootfs rw, from the second execution on running the test application
>>> took below 4 seconds.
>>>
>>> By simply executing a Hello World, Python declares all code objects as
>>> being stale and tries to rewrite them:
>>>
>>> # python3 -v -c 'print("Hello World")'
>>> import _frozen_importlib # frozen
>>> import _imp # builtin
>>> import sys # builtin
>>> import '_warnings' # <class '_frozen_importlib.BuiltinImporter'>
>>> import '_thread' # <class '_frozen_importlib.BuiltinImporter'>
>>> import '_weakref' # <class '_frozen_importlib.BuiltinImporter'>
>>> import '_frozen_importlib_external' # <class
>>> '_frozen_importlib.FrozenImporter'>
>>> import '_io' # <class '_frozen_importlib.BuiltinImporter'>
>>> import 'marshal' # <class '_frozen_importlib.BuiltinImporter'>
>>> import 'posix' # <class '_frozen_importlib.BuiltinImporter'>
>>> import _thread # previously loaded ('_thread')
>>> import '_thread' # <class '_frozen_importlib.BuiltinImporter'>
>>> import _weakref # previously loaded ('_weakref')
>>> import '_weakref' # <class '_frozen_importlib.BuiltinImporter'>
>>> # installing zipimport hook
>>> import 'zipimport' # <class '_frozen_importlib.BuiltinImporter'>
>>> # installed zipimport hook
>>> # bytecode is stale for 'encodings'
>>> # code object from /usr/lib/python3.5/encodings/__init__.py
>>> # could not create
>>> '/usr/lib/python3.5/encodings/__pycache__/__init__.cpython-35.pyc':
>>> OSError(30, 'Read-only file system')
>>> # wrote
>>> '/usr/lib/python3.5/encodings/__pycache__/__init__.cpython-35.pyc'
>>> # bytecode is stale for 'codecs'
>>> # code object from /usr/lib/python3.5/codecs.py
>>> # could not create
>>> '/usr/lib/python3.5/__pycache__/codecs.cpython-35.pyc': OSError(30,
>>> 'Read-only file system')
>>> # wrote '/usr/lib/python3.5/__pycache__/codecs.cpython-35.pyc'
>>> import '_codecs' # <class '_frozen_importlib.BuiltinImporter'>
>>> import 'codecs' # <_frozen_importlib_external.SourceFileLoader object at
>>> 0xb6b33130>
>>> # bytecode is stale for 'encodings.aliases'
>>> # code object from /usr/lib/python3.5/encodings/aliases.py
>>> # could not create
>>> '/usr/lib/python3.5/encodings/__pycache__/aliases.cpython-35.pyc':
>>> OSError(30, 'Read-only file system')
>>> # wrote
>>> '/usr/lib/python3.5/encodings/__pycache__/aliases.cpython-35.pyc'
>>> import 'encodings.aliases' #
>>> <_frozen_importlib_external.SourceFileLoader object at 0xb6b3d9d0>
>>> import 'encodings' # <_frozen_importlib_external.SourceFileLoader object
>>> at 0xb6b154b0>
>>> # bytecode is stale for 'encodings.ascii'
>>> # code object from /usr/lib/python3.5/encodings/ascii.py
>>> # could not create
>>> '/usr/lib/python3.5/encodings/__pycache__/ascii.cpython-35.pyc':
>>> OSError(30, 'Read-only file system')
>>> # wrote '/usr/lib/python3.5/encodings/__pycache__/ascii.cpython-35.pyc'
>>> import 'encodings.ascii' # <_frozen_importlib_external.SourceFileLoader
>>> object at 0xb6b3db10>
>>> import '_signal' # <class '_frozen_importlib.BuiltinImporter'>
>>> ....
>>>
>>>
>>> This is on a Armv7 device. OpenEmbedded layers are current master.
>>> Python version is 3.5.6. Is this a known issue? Any idea?
>>>
>>> --
>>> Stefan
>>>


-- 
Mike Looijmans