[OE-core] Possible stale tags in the download directory

[Ulf Samuelsson]

On 2011-12-07 10:16, Richard Purdie wrote:
> On Wed, 2011-12-07 at 09:27 +0100, Ulf Samuelsson wrote:
>> When I look at the "downloads" location I find a lot of files called.
>>
>> "11_usagi_fix.patch.done"
>>
>> but no  "11_usagi_fix.patch" file in the directory.
>>
>> If this is a indication of that a patch has been downloaded,
>> what happens if you for some reason or other delete your
>> OE-core directory and keep the downloads directory.
>>
>> When you start from fresh, all the tags are then stale.
>>
>> (Deleting the "downloads" directory seems a bad idea)
>>
>> I think that if there should be tags in the "downloads" directory,
>> they should only reflect things which are downloaded to the
>> "downloads" directory, and nothing else.
>>
>> As for the tags in the directory, I think a better approach
>> is to download a file "tarball.tar.bz2" to a different filename first
>> I.E: "tarball.tar.bz2.in-progress"  and if the download completes
>> then move the file to "tarball.tar.bz2".
>> Should remove a lot of clutter from the "downloads" directory.
> What we've tried to do is simplify the fetcher code paths in fetch2.
> There were some many corner/special cases and different code paths it
> was near impossible to tell what was going on. One of the side effects
> is that local file:// urls do touch the done stamp in the same way as
> other downloads. The main reason for those files is now checksum
> tracking. If the done stamps were part of tmpdir, we'd keep checksumming
> the contents of the downloads at each build rather than once after the
> download. The way the implementation works, there is very little risk
> from stale stamps, it would just mean the checksum code wouldn't get
> triggered and that is likely unneeded for a local file:// url anyway.
>
> So yes, its not ideal but looking at the code I'd be surprised if a
> build ever broke due to it.
>
Maybe I am paranoid, but I think I remember seeing tarballs developing
bit rot when moved between different machines.

Could of course clean out the downloads directory, but it would be
nice if the stamps would be in a different directory.
Then I just delete the directory when I move the tarballs.

I tend to use the "downloads" directory for other build systems as well,
to avoid duplication, and having stamps in the directory could have
real bad effects.

I agree that recomputing checksums for each build is not so good,
but doing it once after moving to a new machine, or for each new user
is acceptable (at least to me).

The stamps could be stored in a central location.

     "~/.oe/downloads"  ?

Con: If you have several repositories, maybe because you
forgot to configure for your central repository,
you may be in trouble.


> Cheers,
>
> Richard
>
>
>
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core at lists.openembedded.org
> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core


-- 
Best Regards
Ulf Samuelsson
eMagii