[OE-core] tslib keeps failing on checksum

[Martin Jansa]

On Wed, Nov 20, 2013 at 01:43:45PM +0000, Richard Purdie wrote:
> On Wed, 2013-11-20 at 13:45 +0100, Mike Looijmans wrote:
> > On 11/20/2013 01:29 PM, Martin Jansa wrote:
> > > On Wed, Nov 20, 2013 at 01:02:27PM +0100, Mike Looijmans wrote:
> > >> On 11/20/2013 12:09 PM, Mike Looijmans wrote:
> > >>> On 11/20/2013 11:38 AM, Martin Jansa wrote:
> > >>>> On Wed, Nov 20, 2013 at 11:01:36AM +0100, Mike Looijmans wrote:
> > >>>>> I get this error every time I try t build the current oe-core master:
> > >>>>>
> > >>>>> ERROR: Checksum failure fetching
> > >>>>> https://github.com/kergoth/tslib/releases/download/1.1/tslib-1.1.tar.xz;downloadfilename=tslib-1.1.tar.xz
> > >>>>>
> > >>>>> ERROR: Function failed: Fetcher failure for URL:
> > >>>>> 'https://github.com/kergoth/tslib/releases/download/1.1/tslib-1.1.tar.xz;downloadfilename=tslib-1.1.tar.xz'.
> > >>>>>
> > >>>>> Checksum mismatch!
> > >>>>> File: '/home/mike/zynq-next/build/downloads/tslib-1.1.tar.xz' has md5 checksum
> > >>>>> d41d8cd98f00b204e9800998ecf8427e when 14771f8607b341bb4b297819d37e837d was
> > >>>>> expected
> > >>>>>
> > >>>>> Now the weird thing is, that when I fetch the thing using wget, I get ablob
> > >>>>> with the right checksum:
> > >>>>>
> > >>>>> # wget https://github.com/kergoth/tslib/releases/download/1.1/tslib-1.1.tar.xz
> > >>>>> # md5sum tslib-1.1.tar.xz
> > >>>>> 14771f8607b341bb4b297819d37e837d  tslib-1.1.tar.xz
> > >>>>>
> > >>>>> Is oe-core using some other flavour of wget or what else could cause this kind
> > >>>>> of fetch error?
> > >>>>
> > >>>> Have you tried to unpack both tarballs and compare content in them?
> > >>>
> > >>> Nope, I just assumed they're two somewhat different copies of the same
> > >>> software (e.g. one of them has a "downloaded from here" remark or so).
> > >>> Anything in particular I should look for?
> > >
> > > Any difference is important, sometimes you can find there HTML code from some
> > > proxy or server which generates HTML page instead of 404 or someone
> > > trying to sell you domain of long dead project etc
> > >
> > >> After upgrading I tried again. The file as downloaded by bitbake is completely
> > >> empty. Nothing in it. The md5 sum of this empty file is
> > >> d41d8cd98f00b204e9800998ecf8427e indeed.
> > >>
> > >> I'm now using bitbake 1.21.0 (current master) and OE rev
> > >> 0eb947454e1c92467283e6f1adeca67c7c57698b to build with, with the above results
> > >> still.
> > >
> > > OK, I was asking mostly because newer bitbake (1.19+ is new enough)
> > > renames file with bad checksum, moving corrupted download aside so it
> > > doesn't stand in way for new one.
> > >
> > >> (I know I can just move my own file into the download directory and get on
> > >> with it, but i'd rather actually solve this problem).
> > >>
> > >> There's a direct connection to the internet, no proxy (just a router) that
> > >> might have been caching things.
> > >>
> > >> Any suggestions?
> > >
> > > Check log.do_fetch in WORKDIR/temp, I'm not sure if the error shows only
> > > the original URL or also possible (PRE)MIRROR url from which it could
> > > download that empty file.
> > >
> > > You should see the exact URL in log.do_fetch.
> > 
> > The problem seems to be that I am "my own mirror". There's a http server that 
> > serves sstate-cache and downloads to the rest of the company. And via an NFS 
> > mount, that dir is also linked to my local downloads directory. So bitbake 
> > ended up executing:
> > 
> > /usr/bin/env wget -t 2 -T 30 -nv --passive-ftp --no-check-certificate -O 
> > /home/mike/zynq-next/build/downloads/tslib-1.1.tar.xz -P 
> > /home/mike/zynq-next/build/downloads 
> > 'http://192.168.80.24/sources/tslib-1.1.tar.xz'
> > 
> > This would create an empty file "tslib-1.1.tar.xz" and the http server happily 
> > returns that empty file, and then bitbake thinks it all went well.
> > 
> > I removed the link to the NFS mount, and now it works.
> > 
> > Weird that only tslib suffers from this. Then again, it's probably just the 
> > only package that wasn't readily available.
> > 
> > Is this my mistake and don't do this again, or should there be some protection 
> > against fetching your own files? (for example, by NOT using the filename 
> > itself as a target, but download with a ".part" extension and then rename when 
> > done. This also helps when multible builds share the download dir).
> 
> We certainly don't support circular references like this. As you
> mention, we can work around some parts with the .part approach but for
> things like git repositories for example things are much harder to
> ensure they work.
> 
> I'm not even sure how we'd detect this kind of situation to be honest,
> I'm open to ideas though.
> 
> Sharing a "live" DL_DIR is probably a bad idea as files may be
> incomplete, etc. The checksums do at least help catch most of the
> problems.

Aren't .lock and .done stampfiles supposed to prevent downloading
incomplete files when it's shared over NFS between multiple builders?

I haven't tried it in practise yet, but I was planing to convert all our
jenkins builders to share common live DL_DIR/SSTATE_DIR, so that first
builder to fetch or build something saves the work for other slaves.

I know that especially for sstate archives it could be too late when 2
builders are already in runqueue phase, but still it could be a bit
sooner than waiting for whole build to finish and rsync
DL_DIR/SSTATE_DIR after the build.

Cheers,

-- 
Martin 'JaMa' Jansa     jabber: Martin.Jansa at gmail.com
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 198 bytes
Desc: Digital signature
URL: <http://lists.openembedded.org/pipermail/openembedded-core/attachments/20131120/b0995b84/attachment-0002.sig>