[OE-core] [PATCH 0/2] recipetool: add python buildsystem support

Mon Jan 12 18:00:56 UTC 2015

On Mon, Jan 12, 2015 at 7:15 AM, Paul Eggleton <
paul.eggleton at linux.intel.com> wrote:

> Hi Chris,
>
> On Thursday 08 January 2015 11:24:08 Christopher Larson wrote:
> > - Handles distutils & setuptools.
> > - Supports pulling metadata from PKG-INFO, .egg-info, & setup.py (via two
> >   different mechanisms).
> > - Doesn't handle python 3 yet.
> > - Optional dependencies via `extra_requires` are not currently
> >   handled.
> > - Implements scanning for dependencies by using the python ast to
> >   determine what modules/packages are being imported by the python
> >   code in question
> >
> > See https://gist.github.com/kergoth/6cca6caa4e9b22a21f75 for the
> projects
> > I used to test this against, and
> > https://gist.github.com/kergoth/7fea9494e3cf60b382ff which is the
> result of
> > running recipetool create against every recipe that inherits distutils or
> > setuptools in my build environment.
> >
> > https://gist.github.com/kergoth/7ad4b5102b2a5fe3a956 has my current
> TODO for
> > this feature.
>
> This works pretty well, thanks!
>
> I tried this on python-mako and got this:
>
> ------------- snip --------------
> ...
> # WARNING: the following rdepends are from setuptools install_requires.
> These
> # upstream names may not correspond exactly to bitbake package names.
> RDEPENDS_${PN} += "markupsafe"
>
> # WARNING: the following rdepends are determined through basic analysis of
> the
> # python sources, and might not be 100% accurate. It is your
> responsibility to
> # verify that the values are complete and correct.
> RDEPENDS_${PN} += "python-core python-textutils"
>
> # WARNING: We were unable to map the following python package/module
> # dependencies to the bitbake packages which include them:
> #    mako.lookup
> #    mako.template
> #    sys
> ------------- snip --------------
>

There was a bug which didn't properly detect a case where some of its info
from setup.py (`scripts` in this case) is available, but some of it is not
(`packages`), and as a result ended up only parsing some of the python code
available. Since the paths provided to the dependency scanning are also
passed to the packages search, it didn't pick up on the provide of mako &
mako.* either. Once that's fixed, it looks like this:
https://gist.github.com/0a641e90cdfd37fd2100.

* Shouldn't it know that sys is provided by python-core (maybe this has to
> be
> hardcoded)?
>

Yeah, there are cases where it can't pick up a provide. Built-in modules
(which is the case here), C extensions, or if a module is renamed or
similar programmatically (os.path is actually posixpath in linux).

It seems that sys.builtin_module_names lists the built-in modules. To get
at that list, we could use that from recipetool and implicitly assume that
this list is the same for the host python as the target, which isn't ideal,
or we could run the native python from the sysroot, which I'm not currently
using in any way, or we could copy the list into recipetool. Thoughts?

* Our existing recipe has "python-threading python-netclient python-html"
> for
> RDEPENDS_${PN}, I don't know if all of these are correct but it does at
> least
> try to import threading.
>

With the aforementioned bug fixed, it picks up these and many others,
thankfully.

> * Should we just assume by convention that package names should start with
> "python-" and thus make that prefix on all items brought in from setuptools
> install_requires?
>

Good question, I'd thought about this too. Whichever way we go, the user
will have to review and confirm sanity regardless, so it's a matter of
trying to optimize for the common case. One would hope that the python-
convention would be the common case, so I'm not opposed to this.

>
> * I think "It is your responsibility ..." is a little harsh for the
> dependency
> situation - you'll soon find out at runtime if the value is insufficient.
> I used
> this wording for LICENSE because if you don't check it yourself you won't
> find
> out if it's incorrect.
>

K, I'll drop that.

Earlier you mentioned it would be nice if the buildsystem handler could
> replace the existing value of LICENSE - I checked and it already can;
> you'll
> find the existing value in lines_before. It's all a little bit clunky
> API-wise
> but it is at least possible.
>

Ah! I swear I tried exactly that, but apparently something went wrong. I'll
look into adding that feature to simplify the recipe.

I also noticed when looking at this that my code wasn't replacing the
> version
> with ${PV} in S (but was in SRC_URI). I'll send a patch to fix that. The
> way
> the code works at the moment, I only replace it in SRC_URI if you've
> specified
> a file name containing the version and then we can find that same version
> in the
> SRC_URI value (I'm not sure there is any other practical way)
>

Cool, thanks. I'd noticed this too (in fact it's in the linked todo list :).

After reviewing the generated recipe for mako more closely, I see multiple
potential concerns:
- It doesn't handle find_packages() for packages in setup.py. This is
relatively minor, without knowing the list of packages being shipped, we
just scan the whole source tree -- this is why things like nose are showing
up in the new recipe, because it scanned the unit test suite. So I see this
as a potential future enhancement.
- setuptools/pkg_resources still isn't being properly mapped, even with
both python and python-distribute built. It looks like python-distribute
ships setuptools.py & friends in an egg, and the package search via pkgdata
isn't handling that. - Fixed locally.
- The detected 'beaker' dependency is an optional one, which we'd know if
we parsed extra_requires() or the optional sections of
*.egg-info/requires.txt. This is a known limitation and is already on the
todo list.
- The babel/pygments/templating deps are slightly interesting. Mako doesn't
actually depend on these itself, but it does provide extensions to those
projects via entry points. It could be worth enhancing the tool to handle
those specially, either excluding the entry point packages from the
dependency scanning, or somehow automatically splitting out the entry point
bits into separate binary packages, but that'd be slightly non-trivial, and
might be more "magic" than we want going on. Worth looking into as a
potential enhancement.
- urllib.parse/urllib.request/html.entities: these were picked up out of
compat.py, which does runtime python 2 vs 3 handling, so these are python 3
module deps. There's really no way to avoid this sort of thing, whenever we
have conditional imports going on, it's going to pick up them all.

Up next:
- Drop the duplicated LICENSE
- Drop the 'your responsibility' message
- Rebase to merge in the local fixes mentioned above
- Add python- prefixes to non-mapped deps (e.g. install-requires)
- For now, add 'sys' to the hardcoded list of modules to ignore (part of
core)

After that, I'll re-submit.

Then for the future enhancements todo:
- Handle all of sys.builtin_module_names
- Consider handling find_packages(), particularly the excludes=
- Consider handling deps detected from non-local entry points
-- 
Christopher Larson
kergoth at gmail dot com
Founder - BitBake, OpenEmbedded, OpenZaurus
Maintainer - Tslib
Senior Software Engineer, Mentor Graphics
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openembedded.org/pipermail/openembedded-core/attachments/20150112/a62fac66/attachment-0002.html>