[bitbake-devel] [PATCH 0/6] fetch2: SFTP fetcher and restructure of decodeurl

[Olof Johansson]

Hi,

In our early work in migrating to poky we realized that bitbake's current SSH
fetcher doesn't really suite our needs. It suffers from some specification
conformance bugs --- some which may be easily worked around. However, this
patch series instead introduces a new "SFTP" fetcher, meant to in a way,
replace the SSH fetcher, with stricter conformance to the SECSH URI spec [1].

The SSH fetcher has its own URI parsing, and can not rely on
bb.fetch2.decodeurl. I found that decodeurl was hard to extend, with it
returning a tuple. This prompted me to write a generic URI class, with
attributes for the various URI components (and some general bitbake kludges,
like support relative file:// uris).

I realize that these changes are a bit bold and non-trivial, but I hope you'll
find time to review them and point out problems to fix. If my reasoning is
completely wrong, please let me know and I can try to solve our direct problems
with the SSH fetcher with minimal changes instead.

The changes come with unittests for the URI class; it passes all the tests in
bb.tests.fetch and all other tests under bb.tests, except
bb.tests.data.DataExpansions, but the same errors are present on master as
well. I've also tested running bitbake core-image-minimal in a new tree without
any issues.


Olof Johansson (6):
  fetch2: Remove unused code in wget fetcher
  fetch2: Add a class representing a generic URI
  fetch2: unittests for bb.fetch2.URI class
  fetch2: Adapt encode/decode url to use URI class
  fetch2: Add editor modelines for bb.tests.*
  fetch2: Add SFTP fetcher

 lib/bb/fetch2/__init__.py  |  262 +++++++++++++++++++++++++++++++++++---------
 lib/bb/fetch2/sftp.py      |  104 ++++++++++++++++++
 lib/bb/fetch2/wget.py      |    5 -
 lib/bb/tests/codeparser.py |    2 +
 lib/bb/tests/cow.py        |    2 +
 lib/bb/tests/data.py       |    2 +
 lib/bb/tests/fetch.py      |  228 ++++++++++++++++++++++++++++++++++++++
 lib/bb/tests/utils.py      |    2 +
 8 files changed, 553 insertions(+), 54 deletions(-)
 create mode 100644 lib/bb/fetch2/sftp.py


References:
1: http://tools.ietf.org/html/draft-ietf-secsh-scp-sftp-ssh-uri-04

-- 
Olof Johansson
Tools engineer
(IRC: zibri on Freenode)