Can anyone recommend software for searching for true duplicate files
over the network ... were running everything with ms w2k and the users
are copying sets of files all over the place, much of them duplicates
but doing it by hand would be an impossible task.

[ Post a follow-up to this message ]

m0rk :

> Can anyone recommend software for searching for true duplicate files
> over the network ... were running everything with ms w2k and the users
> are copying sets of files all over the place, much of them duplicates
> but doing it by hand would be an impossible task.

Generate a list of the files to be checked and run a checksum over
them, like sha1 or md5. Sort on the checksum and duplicates should be
listed in adjacent positions. A shortish PERL script could be used as
well.

--

Joerg Lenneis

email: lenneis@wu-wien.ac.at

[ Post a follow-up to this message ]

#! /usr/bin/perl -w
# finddups.pl
# Lists duplicates in MD5 sums
# Use with find something -type f -print0 | xargs -i -0 md5sum "{}"
use strict;
$|=1;
my %h;
while(<> )
{   chomp $_;
#    print STDERR substr($_,0,70),qq(     \r);
my @a=split / /,$_,2;
push @{$h{$a[0]}},$a[1] if @a==2;
};
foreach(keys %h)
{   print join qq(\n),'',@{$h{$_}},'' if @{$h{$_}}>
1;
}

[ Post a follow-up to this message ]