When I was using MS DOS I could specify a "verify" option with the copy
so that it would read back in the block it had written and verify that
the checksum was the same as the block it just copied and if the
checksum differed it would rewrite the block (at least I believe it
worked that way). Now I am working on various Linux and Unix flavors
(AIX 5.2 currently) where file integrity is extremely important and I
am wondering what facilities exist for both administrators and
end-users to do a "verify" on the file or batches of files copied. I
know about the "sum" command but don't have a script that can handle
doing the checksum and retrying if it fails using the usual parameters
given to "cp". Can somebody point me to a web site that covers this
issue of verified copies?

Thanks,
Roland

[ Post a follow-up to this message ]

"RolandRB" <rolandberry@hotmail.com> writes:

> When I was using MS DOS I could specify a "verify" option with the copy
> so that it would read back in the block it had written and verify that
> the checksum was the same as the block it just copied and if the
> checksum differed it would rewrite the block (at least I believe it
> worked that way). Now I am working on various Linux and Unix flavors
> (AIX 5.2 currently) where file integrity is extremely important and I
> am wondering what facilities exist for both administrators and
> end-users to do a "verify" on the file or batches of files copied. I
> know about the "sum" command but don't have a script that can handle
> doing the checksum and retrying if it fails using the usual parameters
> given to "cp". Can somebody point me to a web site that covers this
> issue of verified copies?

Look into the rsync command.  If you're copying lots of files it has
several advantages. Verification is among them, but the biggie is that
it only copies the files it needs to (i.e. skips ones that exist and
are identical to the source at the destination) , which makes it ideal
for doing backups.   rsync can also be done over an ssh link for
remote archiving.

--
Todd H.
http://www.toddh.net/

[ Post a follow-up to this message ]

RolandRB wrote:
> When I was using MS DOS I could specify a "verify" option with the copy
> so that it would read back in the block it had written and verify that
> the checksum was the same as the block it just copied and if the
> checksum differed it would rewrite the block (at least I believe it
> worked that way). Now I am working on various Linux and Unix flavors
> (AIX 5.2 currently) where file integrity is extremely important and I
> am wondering what facilities exist for both administrators and
> end-users to do a "verify" on the file or batches of files copied. I
> know about the "sum" command but don't have a script that can handle
> doing the checksum and retrying if it fails using the usual parameters
> given to "cp". Can somebody point me to a web site that covers this
> issue of verified copies?
>
> Thanks,
> Roland
>

cmp compares two files.
To do a comparison after a successful cp:

cp "$file" "$dest" && cmp "$file" "$dest" || echo "failed"

But if you trust the Unix kernel and the hardware then you can
just check cp's exit status:

cp "$file" "$dest" || echo "failed".



--
Michael Tosch @ hp : com

[ Post a follow-up to this message ]

Michael Tosch wrote:
> RolandRB wrote: 
>
> cmp compares two files.
> To do a comparison after a successful cp:
>
> cp "$file" "$dest" && cmp "$file" "$dest" || echo "failed"

That's a good idea. Thanks! The files are likely to be circa 1 Gbyte
binary files and "cmp" will work on binary files and it would be a more
thorough check but I am wordering about the speed of it compared to
doing a less thorough "sum" checksum. Most likely a number of files of
this size will be copied across in bulk.

> But if you trust the Unix kernel and the hardware then you can
> just check cp's exit status:
>
> cp "$file" "$dest" || echo "failed".

I am wondering if it is possible for "cp" to return a zero return code
but the copy not to be a perfect copy.

>
> --
> Michael Tosch @ hp : com

[ Post a follow-up to this message ]

2007-01-12, 10:39(-08), RolandRB:
[...] 
>
> I am wondering if it is possible for "cp" to return a zero return code
> but the copy not to be a perfect copy.
[...]

cp will return an error if any of the open, write or close
fails.

For instance

cp file /dev/null

will succeed.

even though cmp file /dev/null will fail (unless file is empty).

write or open may fail if the virtual filesystem interface of
the system fails. For NFS or other network filesystem, it will
check for the acknowledgement of the server that the write was
successful on the other end (I beleive).

For other filesystems (like for the write on the NFS server
side), errors are most likely to be due to filesystem space,
quotas or permissions. Then the data will be lying in kernel
pages in memory with flags saying that those pages need to be
flushed to whatever devices lays underneath (which may be a disk
partition, a mtd device, loop device... potentially other layers
before it is stored to a physical storage (possibly encrypted or
duplicated...)) at some point.

You can mount filesystems with a sync option or use the O_SYNC
flag for open(2) (I don't think cp has an option to enable
that). This will make sure all the data has been flushed to
hardware when the writes and the close are done (not sure about
NFS). That won't give you a lot more guarantee, just that you
may pull the plug now or remove the USB key.

--
Stéphane

[ Post a follow-up to this message ]

RolandRB wrote:
> 
> 
>
> I am wondering if it is possible for "cp" to return a zero return code
> but the copy not to be a perfect copy.

For hardware problems that's good enough.

If there's a chance that some other process has the fil eopen and
is writing to it that's not good enough.  It will copy what's in the
file at the time it gets to each buffer read and when it hits the
current end of the file it will complete - Wonderfull for a file that
is
not open by any other process.

[ Post a follow-up to this message ]

RolandRB wrote:
> 
> 
>
> I am wondering if it is possible for "cp" to return a zero return code
> but the copy not to be a perfect copy.

For hardware problems that's good enough.

If there's a chance that some other process has the fil eopen and
is writing to it that's not good enough.  It will copy what's in the
file at the time it gets to each buffer read and when it hits the
current end of the file it will complete - Wonderfull for a file that
is
not open by any other process.

[ Post a follow-up to this message ]

Stephane CHAZELAS wrote:
> 2007-01-12, 17:22(+01), Michael Tosch: 
>
> And again, it probably doesn't make sense. First the file will
> probably be written down to physical storage (if permanent
> storage there is) long after the cmp command has returned, and
> second, even the filesystem is configured to synchronously write
> to disk, cmp's read()s will probably not ask the system to read
> the data from disk as it will already be in cache, and even if
> it's not one of the kernel layer caches, it may be in the disk
> cache, and even if the data is not corrupted on disk at this
> very moment, it may very well become corrupted the next minute.

A most excellent point. It only makes sense to do the "cmp"s after a
whole batch of copies has completed and hope the first of the batch of
files has been flushed out of the cache. I hadn't thought of that. I'm
glad I read your post here.
 
> [...]
>
> That's the only sensible thing to do, though you forgot the
> "--".
>
> cp -- "$file" "$dest" || exit
> cp -- "$file" "$dest" || take-corrective-action
>
> (cp will already have output an error message, no need to add
> one).
>=20
> --=20
> St=E9phane

[ Post a follow-up to this message ]