<div dir="ltr"><div class="gmail_extra"><div class="gmail_quote">On Mon, Jan 12, 2015 at 7:15 AM, Paul Eggleton <span dir="ltr"><<a href="mailto:paul.eggleton@linux.intel.com" target="_blank">paul.eggleton@linux.intel.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">Hi Chris,<br>
<span class=""><br>
On Thursday 08 January 2015 11:24:08 Christopher Larson wrote:<br>
> - Handles distutils & setuptools.<br>
> - Supports pulling metadata from PKG-INFO, .egg-info, & setup.py (via two<br>
> different mechanisms).<br>
> - Doesn't handle python 3 yet.<br>
> - Optional dependencies via `extra_requires` are not currently<br>
> handled.<br>
> - Implements scanning for dependencies by using the python ast to<br>
> determine what modules/packages are being imported by the python<br>
> code in question<br>
><br>
> See <a href="https://gist.github.com/kergoth/6cca6caa4e9b22a21f75" target="_blank">https://gist.github.com/kergoth/6cca6caa4e9b22a21f75</a> for the projects<br>
> I used to test this against, and<br>
> <a href="https://gist.github.com/kergoth/7fea9494e3cf60b382ff" target="_blank">https://gist.github.com/kergoth/7fea9494e3cf60b382ff</a> which is the result of<br>
> running recipetool create against every recipe that inherits distutils or<br>
> setuptools in my build environment.<br>
><br>
> <a href="https://gist.github.com/kergoth/7ad4b5102b2a5fe3a956" target="_blank">https://gist.github.com/kergoth/7ad4b5102b2a5fe3a956</a> has my current TODO for<br>
> this feature.<br>
<br>
</span>This works pretty well, thanks!<br>
<br>
I tried this on python-mako and got this:<br>
<br>
------------- snip --------------<br>
...<br>
<span class=""># WARNING: the following rdepends are from setuptools install_requires. These<br>
</span><span class=""># upstream names may not correspond exactly to bitbake package names.<br>
</span>RDEPENDS_${PN} += "markupsafe"<br>
<br>
# WARNING: the following rdepends are determined through basic analysis of the<br>
# python sources, and might not be 100% accurate. It is your responsibility to<br>
# verify that the values are complete and correct.<br>
RDEPENDS_${PN} += "python-core python-textutils"<br>
<br>
# WARNING: We were unable to map the following python package/module<br>
# dependencies to the bitbake packages which include them:<br>
# mako.lookup<br>
# mako.template<br>
# sys<br>
------------- snip --------------<br></blockquote><div><br></div><div>There was a bug which didn't properly detect a case where some of its info from setup.py (`scripts` in this case) is available, but some of it is not (`packages`), and as a result ended up only parsing some of the python code available. Since the paths provided to the dependency scanning are also passed to the packages search, it didn't pick up on the provide of mako & mako.* either. Once that's fixed, it looks like this: <a href="https://gist.github.com/0a641e90cdfd37fd2100">https://gist.github.com/0a641e90cdfd37fd2100</a>.</div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
* Shouldn't it know that sys is provided by python-core (maybe this has to be<br>
hardcoded)?<br></blockquote><div><br></div><div>Yeah, there are cases where it can't pick up a provide. Built-in modules (which is the case here), C extensions, or if a module is renamed or similar programmatically (os.path is actually posixpath in linux).</div><div><br></div><div>It seems that sys.builtin_module_names lists the built-in modules. To get at that list, we could use that from recipetool and implicitly assume that this list is the same for the host python as the target, which isn't ideal, or we could run the native python from the sysroot, which I'm not currently using in any way, or we could copy the list into recipetool. Thoughts?</div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
* Our existing recipe has "python-threading python-netclient python-html" for<br>
RDEPENDS_${PN}, I don't know if all of these are correct but it does at least<br>
try to import threading.<br></blockquote><div><br></div><div>With the aforementioned bug fixed, it picks up these and many others, thankfully.</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
* Should we just assume by convention that package names should start with<br>
"python-" and thus make that prefix on all items brought in from setuptools<br>
install_requires?<br></blockquote><div><br></div><div>Good question, I'd thought about this too. Whichever way we go, the user will have to review and confirm sanity regardless, so it's a matter of trying to optimize for the common case. One would hope that the python- convention would be the common case, so I'm not opposed to this.</div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<br>
* I think "It is your responsibility ..." is a little harsh for the dependency<br>
situation - you'll soon find out at runtime if the value is insufficient. I used<br>
this wording for LICENSE because if you don't check it yourself you won't find<br>
out if it's incorrect.<br></blockquote><div><br></div><div>K, I'll drop that. </div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
Earlier you mentioned it would be nice if the buildsystem handler could<br>
replace the existing value of LICENSE - I checked and it already can; you'll<br>
find the existing value in lines_before. It's all a little bit clunky API-wise<br>
but it is at least possible.<br></blockquote><div><br></div><div>Ah! I swear I tried exactly that, but apparently something went wrong. I'll look into adding that feature to simplify the recipe.</div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
I also noticed when looking at this that my code wasn't replacing the version<br>
with ${PV} in S (but was in SRC_URI). I'll send a patch to fix that. The way<br>
the code works at the moment, I only replace it in SRC_URI if you've specified<br>
a file name containing the version and then we can find that same version in the<br>
SRC_URI value (I'm not sure there is any other practical way)<br></blockquote><div><br></div><div>Cool, thanks. I'd noticed this too (in fact it's in the linked todo list :).</div><div><br></div><div>After reviewing the generated recipe for mako more closely, I see multiple potential concerns:</div><div>- It doesn't handle find_packages() for packages in setup.py. This is relatively minor, without knowing the list of packages being shipped, we just scan the whole source tree -- this is why things like nose are showing up in the new recipe, because it scanned the unit test suite. So I see this as a potential future enhancement.</div><div>- setuptools/pkg_resources still isn't being properly mapped, even with both python and python-distribute built. It looks like python-distribute ships setuptools.py & friends in an egg, and the package search via pkgdata isn't handling that. - Fixed locally.</div><div>- The detected 'beaker' dependency is an optional one, which we'd know if we parsed extra_requires() or the optional sections of *.egg-info/requires.txt. This is a known limitation and is already on the todo list.</div><div>- The babel/pygments/templating deps are slightly interesting. Mako doesn't actually depend on these itself, but it does provide extensions to those projects via entry points. It could be worth enhancing the tool to handle those specially, either excluding the entry point packages from the dependency scanning, or somehow automatically splitting out the entry point bits into separate binary packages, but that'd be slightly non-trivial, and might be more "magic" than we want going on. Worth looking into as a potential enhancement.<br></div><div>- urllib.parse/urllib.request/html.entities: these were picked up out of compat.py, which does runtime python 2 vs 3 handling, so these are python 3 module deps. There's really no way to avoid this sort of thing, whenever we have conditional imports going on, it's going to pick up them all.</div><div><br></div><div>Up next:</div><div>- Drop the duplicated LICENSE</div><div>- Drop the 'your responsibility' message</div><div>- Rebase to merge in the local fixes mentioned above</div><div>- Add python- prefixes to non-mapped deps (e.g. install-requires)</div><div>- For now, add 'sys' to the hardcoded list of modules to ignore (part of core)</div><div><br></div><div>After that, I'll re-submit.</div><div><br></div><div>Then for the future enhancements todo:</div><div>- Handle all of sys.builtin_module_names</div><div>- Consider handling find_packages(), particularly the excludes=</div><div>- Consider handling deps detected from non-local entry points</div></div>-- <br><div class="gmail_signature"><div dir="ltr">Christopher Larson<br>kergoth at gmail dot com<br>Founder - BitBake, OpenEmbedded, OpenZaurus<br>Maintainer - Tslib<br>Senior Software Engineer, Mentor Graphics</div></div>
</div></div>