[bitbake-devel] [master][PATCH] fetch2/wget: support releases from private github repositories

[André Draszik]

Hi Ross,

On Fri, 2019-11-22 at 17:01 +0000, Ross Burton wrote:
> On 14/11/2019 14:21, André Draszik wrote:
> > The wget / http fetcher currently doesn't support fetching
> > assets attached to releases on private GitHub repositories,
> > i.e. release artefacts like
> >      https://github.com/<user>/<project>/releases/download/v1.0.0/asset1.txt
> > 
> > Those are special, in that HTTP basic auth is not used / possible
> > on the URL as seen in the GitHub UI, but instead the GitHub API
> > must be used for downloading (which does support HTTP basic auth)
> > where the URL will be different.
> > 
> > To be able to access the GitHub API, opportunistic authentication
> > (auth-no-challenge) needs to be enabled. Then the API needs to
> > be queried for the real URL of the file to be downloaded, and
> > finally application/octet-stream must be specified explicitly.
> > 
> > Note that there is a slight difference in the location of the
> > REST API endpoints between GitHub.com and GitHub Enterprise.
> > 
> >      https://developer.github.com/v3/repos/releases/
> >      https://developer.github.com/enterprise/2.19/v3/enterprise-admin/
> > 
> > As it's impossible to determine if a repository is on GitHub
> > or not (considering GitHub Enterprise), and even more so if a
> > repository is private or not, a new flag is introduced that
> > should be set to "1" - "github_private_asset", e.g.
> 
> That's a lot of "special" and "unfortunately" and "but", adding a 
> site-specific option and a non-trivial site-specific chunk of code to 
> the otherwise generic HTTP fetcher.
> 
> I'd suggest implementing this as a new fetcher instead of shoe-horning 
> it into the wget fetcher.  Then you can do something like githubasset:// 
> in SRC_URI and the fetcher can do the right thing instead of jumping 
> through hoops and guessing.

Thanks for your comments. Yes, I was in two minds about it as well. I was also
contemplating a new bbclass for this instead of a new fetcher for github
private repositories.

I'll write a new fetcher and see how that will work out...


Thanks,
Andre'