[OE-core] Possible stale tags in the download directory

[Richard Purdie]

On Wed, 2011-12-07 at 12:27 +0100, Ulf Samuelsson wrote:
> On 2011-12-07 10:16, Richard Purdie wrote:
> > On Wed, 2011-12-07 at 09:27 +0100, Ulf Samuelsson wrote:
> >> When I look at the "downloads" location I find a lot of files called.
> >>
> >> "11_usagi_fix.patch.done"
> >>
> >> but no  "11_usagi_fix.patch" file in the directory.
> >>
> >> If this is a indication of that a patch has been downloaded,
> >> what happens if you for some reason or other delete your
> >> OE-core directory and keep the downloads directory.
> >>
> >> When you start from fresh, all the tags are then stale.
> >>
> >> (Deleting the "downloads" directory seems a bad idea)
> >>
> >> I think that if there should be tags in the "downloads" directory,
> >> they should only reflect things which are downloaded to the
> >> "downloads" directory, and nothing else.
> >>
> >> As for the tags in the directory, I think a better approach
> >> is to download a file "tarball.tar.bz2" to a different filename first
> >> I.E: "tarball.tar.bz2.in-progress"  and if the download completes
> >> then move the file to "tarball.tar.bz2".
> >> Should remove a lot of clutter from the "downloads" directory.
> > What we've tried to do is simplify the fetcher code paths in fetch2.
> > There were some many corner/special cases and different code paths it
> > was near impossible to tell what was going on. One of the side effects
> > is that local file:// urls do touch the done stamp in the same way as
> > other downloads. The main reason for those files is now checksum
> > tracking. If the done stamps were part of tmpdir, we'd keep checksumming
> > the contents of the downloads at each build rather than once after the
> > download. The way the implementation works, there is very little risk
> > from stale stamps, it would just mean the checksum code wouldn't get
> > triggered and that is likely unneeded for a local file:// url anyway.
> >
> > So yes, its not ideal but looking at the code I'd be surprised if a
> > build ever broke due to it.
> >
> Maybe I am paranoid, but I think I remember seeing tarballs developing
> bit rot when moved between different machines.

I don't understand what you mean here. If your process for copying files
between machines corrupts files, you have bigger problems. You can just
rm *.done at the same time if that is an issue.

> Could of course clean out the downloads directory, but it would be
> nice if the stamps would be in a different directory.
> Then I just delete the directory when I move the tarballs.
> 
> I tend to use the "downloads" directory for other build systems as well,
> to avoid duplication, and having stamps in the directory could have
> real bad effects.
> 
> I agree that recomputing checksums for each build is not so good,
> but doing it once after moving to a new machine, or for each new user
> is acceptable (at least to me).
> 
> The stamps could be stored in a central location.
> 
>      "~/.oe/downloads"  ?

I believe the data the .done files represent is directly related to the
files themselves and they should be kept together.

Nothing said here is convincing me there is a real world problem...

Cheers,

Richard