[oe] content addressable fetching of packages, through MD5SUM or SHA1SUM?

[Leon Woestenberg]

In trying to improve our package fetching robustness, a package
crawler. (I still wonder if no such thing already exists...)

A new version is inlined below, usage is as follows:

Use mode #1, generate URI checksums given a package file name:

leon at precise:/tmp$ ./crawl.sh popt-1.7.tar.gz
[GENERATED] 5988e7aeb0ae4dac8d83561265984cc9
66f3c77b87a160951b180447f4a6dce68ad2f71b
www.netsw.org/system/libs/options/popt-1.7.tar.gz
[GENERATED] 5988e7aeb0ae4dac8d83561265984cc9
66f3c77b87a160951b180447f4a6dce68ad2f71b
su-se.lunar-linux.org/lunar/mirrors/popt-1.7.tar.gz
[GENERATED] 5988e7aeb0ae4dac8d83561265984cc9
66f3c77b87a160951b180447f4a6dce68ad2f71b
www.aoloser.com/dist/popt-1.7.tar.gz
[GENERATED] 5988e7aeb0ae4dac8d83561265984cc9
66f3c77b87a160951b180447f4a6dce68ad2f71b
www.netmax.org/Download/SOURCE/packages/extras/SOURCE/popt-1.7.tar.gz
[GENERATED] 5988e7aeb0ae4dac8d83561265984cc9
66f3c77b87a160951b180447f4a6dce68ad2f71b
be.lunar-linux.org/lunar/mirrors/popt-1.7.tar.gz
[GENERATED] 5988e7aeb0ae4dac8d83561265984cc9
66f3c77b87a160951b180447f4a6dce68ad2f71b
mirrors.isc.org/pub/MidnightBSD/distfiles/popt-1.7.tar.gz
[GENERATED] 5988e7aeb0ae4dac8d83561265984cc9
66f3c77b87a160951b180447f4a6dce68ad2f71b
www.ibiblio.org/pub/packages/solaris/freeware/SOURCES/popt-1.7.tar.gz

Use mode #2, find URI's that have a matching MD5SUM and SHA1SUM:

leon at precise:/tmp$ ./crawl.sh popt-1.7.tar.gz
5988e7aeb0ae4dac8d83561265984cc9
66f3c77b87a160951b180447f4a6dce68ad2f71b
[VALIDATED] 5988e7aeb0ae4dac8d83561265984cc9
66f3c77b87a160951b180447f4a6dce68ad2f71b
www.netsw.org/system/libs/options/popt-1.7.tar.gz
[VALIDATED] 5988e7aeb0ae4dac8d83561265984cc9
66f3c77b87a160951b180447f4a6dce68ad2f71b
su-se.lunar-linux.org/lunar/mirrors/popt-1.7.tar.gz
[VALIDATED] 5988e7aeb0ae4dac8d83561265984cc9
66f3c77b87a160951b180447f4a6dce68ad2f71b
www.aoloser.com/dist/popt-1.7.tar.gz
[VALIDATED] 5988e7aeb0ae4dac8d83561265984cc9
66f3c77b87a160951b180447f4a6dce68ad2f71b
www.netmax.org/Download/SOURCE/packages/extras/SOURCE/popt-1.7.tar.gz
[VALIDATED] 5988e7aeb0ae4dac8d83561265984cc9
66f3c77b87a160951b180447f4a6dce68ad2f71b
be.lunar-linux.org/lunar/mirrors/popt-1.7.tar.gz
[VALIDATED] 5988e7aeb0ae4dac8d83561265984cc9
66f3c77b87a160951b180447f4a6dce68ad2f71b
mirrors.isc.org/pub/MidnightBSD/distfiles/popt-1.7.tar.gz
[VALIDATED] 5988e7aeb0ae4dac8d83561265984cc9
66f3c77b87a160951b180447f4a6dce68ad2f71b
www.ibiblio.org/pub/packages/solaris/freeware/SOURCES/popt-1.7.tar.gz
leon at precise:/tmp$


#!/bin/bash

# Requires links, sed, grep, md5sum, sha1sum
# $1 package file name
# $2 md5sum to check against
# $3 sha1sum to check against

if [ x$1y == xy ]; then
  echo "Provide a filename, for example: popt-1.7.tar.gz"
  exit
fi

FILE=`basename $1`

LIST=`links -dump
'http://www.google.com/search?q=intitle%3A%22index+of%22+'$FILE | grep
-e 'Cached' | sed 's@[ \t]*\(.*\)/[ ^t]\(.*\)@\1/@' | grep -v Cached`

cd /tmp

for URI in $LIST
do
  rm -rf /tmp/$FILE
  wget --timeout=2 -t1 -O$FILE $URI$FILE 2>/dev/null
  if [ $? == 0 ]; then
    FILE_MD=`md5sum $FILE | sed 's@\([0-9,a-f]*\)\(.*\)@\1@'`
    FILE_SHA=`sha1sum $FILE | sed 's@\([0-9,a-f]*\)\(.*\)@\1@'`
    if [ x$2y != xy -a x$3y != xy ]; then
      if [ $2 == $FILE_MD -a $3 == $FILE_SHA ]; then
        echo "[VALIDATED]" $FILE_MD $FILE_SHA $URI$FILE
      fi
    else
      echo "[GENERATED]" $FILE_MD $FILE_SHA $URI$FILE
    fi
  fi
done