Unix Shell - yet another question on file names with spaces?

Author

yet another question on file names with spaces?

russellgoyder@yahoo.co.uk

2005-02-20, 6:19 pm

Hello,

I have seen many posts on this topic but have not yet found one which
solves my problem, which is the following:

I would like to scp multiple files from my local machine to a remote
machine, each of which contains spaces in its name. If I were to do
this interactively, I would type something like:

scp first\ file.txt second\ file.txt
me@remotemachine:/destination/path/.

I have been trying to do this from a shell script, but have not yet
succeeded. Taking the simpler example of ls, if I do this:

a="first\ file.txt second\ file.txt"
echo $(ls "$a")

then ls only receives one argument and of course fails to find the file
"first\ file.txt second\ file.txt". Removing the double quotes to give
echo $(ls $a) passes first\, file.txt, second\, and file.txt as four
separate arguments to ls. The distinction between the above is all I
have been able to find discussed so far!

I can put the file names in an array and call scp multiple times. This
works fine

b[0]="first file.txt"
b[1]="second file.txt"
i=0
while [ $i -lt ${#b[*]} ]
do
echo $(ls "${b[$i]}")
let "i = $i + 1"
done

However, I don't really want to do that - it's not a password-typing
issue (I know about ssh-keygen/agent/add), it's just that one call to
scp is cleaner and what I'd do by hand anyway. Is there a way to call
scp (or ls) from a script such that I can pass it the two arguments
above, not one or four?

Thanks,

Russell

Michael Heiming

2005-02-20, 6:19 pm

In comp.unix.shell russellgoyder@yahoo.co.uk:
> Hello,

> I have seen many posts on this topic but have not yet found one which
> solves my problem, which is the following:

> I would like to scp multiple files from my local machine to a remote
> machine, each of which contains spaces in its name. If I were to do
> this interactively, I would type something like:

> scp first\ file.txt second\ file.txt
> me@remotemachine:/destination/path/.

> I have been trying to do this from a shell script, but have not yet
> succeeded. Taking the simpler example of ls, if I do this:

What about the easiest solution, using "_" in filenames instead
of " "? No spaces no more problems, which are likely to arise as
soon as you use spaces in file/directory names as you just
encountered.

--
Michael Heiming (X-PGP-Sig > GPG-Key ID: EDD27B94)
mail: echo zvpunry@urvzvat.qr | PERL -pe 'y/a-z/n-za-m/'
#bofh excuse 370: Virus due to computers having unsafe sex.

Chris F.A. Johnson

2005-02-20, 6:19 pm

On Sat, 19 Feb 2005 at 23:23 GMT, russellgoyder@yahoo.co.uk wrote:
> Hello,
>
> I have seen many posts on this topic but have not yet found one which
> solves my problem, which is the following:
>
> I would like to scp multiple files from my local machine to a remote
> machine, each of which contains spaces in its name. If I were to do
> this interactively, I would type something like:
>
> scp first\ file.txt second\ file.txt
> me@remotemachine:/destination/path/.
>
> I have been trying to do this from a shell script, but have not yet
> succeeded. Taking the simpler example of ls, if I do this:
>
> a="first\ file.txt second\ file.txt"
> echo $(ls "$a")

What's echo for? And what's wrong with:

ls first\ file.txt second\ file.txt

Or:

echo first\ file.txt second\ file.txt

Or:

printf "%s\n" first\ file.txt second\ file.txt

> then ls only receives one argument and of course fails to find the file
> "first\ file.txt second\ file.txt". Removing the double quotes to give
> echo $(ls $a) passes first\, file.txt, second\, and file.txt as four
> separate arguments to ls. The distinction between the above is all I
> have been able to find discussed so far!
>
> I can put the file names in an array and call scp multiple times. This
> works fine
>
> b[0]="first file.txt"
> b[1]="second file.txt"
> i=0
> while [ $i -lt ${#b[*]} ]
> do
> echo $(ls "${b[$i]}")

Why use echo as well as ls?

echo "${b[$i]}"

Or:

ls "${b[$i]}"

> let "i = $i + 1"
> done

Why use a loop?

printf "%s\n" "${b[@]}"

> However, I don't really want to do that - it's not a password-typing
> issue (I know about ssh-keygen/agent/add), it's just that one call to
> scp is cleaner and what I'd do by hand anyway. Is there a way to call
> scp (or ls) from a script such that I can pass it the two arguments
> above, not one or four?

How are you getting the file names?

--
Chris F.A. Johnson http://cfaj.freeshell.org/shell
========================================
===========================
My code (if any) in this post is copyright 2005, Chris F.A. Johnson
and may be copied under the terms of the GNU General Public License

rgoyder

2005-02-20, 6:19 pm

I'm afraid I don't have control over how the files are named. I
wouldn't mind placing this restriction on the file names if I was the
only user, but the people who create and use these files would regard
it as unacceptable I'm afraid!

Russell Goyder

2005-02-20, 6:19 pm

Chris, thanks for the help.

>
> What's echo for? And what's wrong with:
>
> ls first\ file.txt second\ file.txt
>
> Or:
>
> echo first\ file.txt second\ file.txt
>
> Or:
>
> printf "%s\n" first\ file.txt second\ file.txt

I agree, the echo is redudant. I posted the message after some time
spent debugging my script and was just used to echoing! All of your
three suggestions above do indeed print out the names of the two files
on stdout. However, I do not have the file names hardcoded - I don't
know what they are before the script runs - see below.

In answer to:

> Why use a loop?
>
> printf "%s\n" "${b[@]}"

and

> How are you getting the file names?

I used a loop because, if a="first\ file.txt second\ file.txt" then ls
$a and ls "$a" both fail. I am not really interested in ls, I am
interested in scp, but I am using ls because the same principle applies
(I think) and it is quicker to test. With ls $a, ls receives 4
arguments and with ls "$a" it receives 1.

I am getting the file names via the following proceedure: For a given
directory, I find the files in it, then find the files in a directory
with the same name on a remote machine. I go through the local files
finding whether they are also on the remote machine. Those that are
present locally, but not remotely, I copy to the remote machine via scp.

I am writing a very primitive form of syncing script in order to mirror
two directories. This is part of updating a web site - the text files
(html, php etc) are quick to zip up, transfer and unpack on the server
- I transfer them all every time, but there are binary files such as
images and pdfs which I only want to transfer once. Say I add another
set of images to my images directory, I would run my script on that
directory and it would upload the necessary files.

I have been doing this by hand for a number of years, but I am trying
to make it possible for the web site to be developed collaboratively by
people separated geographically. If I add some images to the site,
someone else would sync in the opposite way to that described above in
order to download the extra files before starting work, then upload any
further files introduced during that work.

I would rather use rsync for this, but that is not an option right now
(with my current hosting provider). Although crude, once working, my
scripts will be good enough for the time being, and I am glad to be
finally learning some shell programming!

This has turned into quite a long explanation! The bottom line is that
I can choose how I have the file names - I can put them in an array or
have them as a single variable in the usual space-separated list format
($a above). However, I can't see a way to supply the list to the scp
command in a way that works because scp $fileList gives one argument
per space-separated string and scp "$fileList" gives a single string,
neither of which correspond to file names.

Russell

Michael Heiming

2005-02-20, 6:19 pm

In comp.unix.shell rgoyder <russellgoyder@yahoo.co.uk>:
> I'm afraid I don't have control over how the files are named. I
> wouldn't mind placing this restriction on the file names if I was the
> only user, but the people who create and use these files would regard
> it as unacceptable I'm afraid!

Sounds like windoze (l)users.;( Telling people you couldn't
guarantee for backup/recovery if they insist on spaces in
file/directory names usually works.

--
Michael Heiming (X-PGP-Sig > GPG-Key ID: EDD27B94)
mail: echo zvpunry@urvzvat.qr | PERL -pe 'y/a-z/n-za-m/'
#bofh excuse 373: Suspicious pointer corrupted virtual machine

Chris F.A. Johnson

2005-02-20, 6:19 pm

On Sun, 20 Feb 2005 at 08:55 GMT, Russell Goyder wrote:
>
> Chris, thanks for the help.
>
>
> I agree, the echo is redudant. I posted the message after some time
> spent debugging my script and was just used to echoing! All of your
> three suggestions above do indeed print out the names of the two files
> on stdout. However, I do not have the file names hardcoded - I don't
> know what they are before the script runs - see below.
>
> In answer to:
>
>
> and
>
>
> I used a loop because, if a="first\ file.txt second\ file.txt" then ls
> $a and ls "$a" both fail. I am not really interested in ls, I am
> interested in scp, but I am using ls because the same principle applies
> (I think) and it is quicker to test. With ls $a, ls receives 4
> arguments and with ls "$a" it receives 1.

Why do you have two file names in the variable?

> I am getting the file names via the following proceedure: For a given
> directory, I find the files in it, then find the files in a directory
> with the same name on a remote machine. I go through the local files
> finding whether they are also on the remote machine. Those that are
> present locally, but not remotely, I copy to the remote machine via scp.

HOW are you doing it? Let's see the code. Working with spaces in
filenames is not hard; we need to see what you are doing wrong.

> I am writing a very primitive form of syncing script in order to mirror
> two directories. This is part of updating a web site - the text files
> (html, php etc) are quick to zip up, transfer and unpack on the server
> - I transfer them all every time, but there are binary files such as
> images and pdfs which I only want to transfer once. Say I add another
> set of images to my images directory, I would run my script on that
> directory and it would upload the necessary files.
>
> I have been doing this by hand for a number of years, but I am trying
> to make it possible for the web site to be developed collaboratively by
> people separated geographically. If I add some images to the site,
> someone else would sync in the opposite way to that described above in
> order to download the extra files before starting work, then upload any
> further files introduced during that work.
>
> I would rather use rsync for this, but that is not an option right now
> (with my current hosting provider). Although crude, once working, my
> scripts will be good enough for the time being, and I am glad to be
> finally learning some shell programming!
>
> This has turned into quite a long explanation! The bottom line is that
> I can choose how I have the file names - I can put them in an array or
> have them as a single variable in the usual space-separated list format
> ($a above).

You CANNOT put them in a space-separated list, because a space
betweeen file names is no different from a space between parts of
the name. (There are ways, but it's more trouble than you need.)

> However, I can't see a way to supply the list to the scp
> command in a way that works because scp $fileList gives one argument
> per space-separated string and scp "$fileList" gives a single string,
> neither of which correspond to file names.

Show us HOW you are getting the file names, and we'll give you an
answer.

--
Chris F.A. Johnson http://cfaj.freeshell.org/shell
========================================
===========================
My code (if any) in this post is copyright 2005, Chris F.A. Johnson
and may be copied under the terms of the GNU General Public License

Russell Goyder

2005-02-20, 6:19 pm

> Show us HOW you are getting the file names, and we'll give you an
> answer.

OK! Here it is, warts and all (perhaps only warts?).

#!/bin/sh

# ------------- define functions first ---------------

# take a space-separated list of strings and a unique string and split the former on the latter, returning an array of strings
function split {

fileList=$1
uniqueStr=$2

# build up an array of remote files, splitting on uniqueStr
count=0
for bit in $fileList
do
if [ $bit == "$uniqueStr" ]
then
# add the file name to the array removing initial characters
splitted[$count]=${tmp#}
# increment counter and blank the concatenatee
let "count += 1"
tmp=""
else
if [ "$tmp" == "" ]
then
tmp="$bit"
else
tmp="$tmp\\ $bit"
fi
fi
done

}

# ------------- main code starts here ---------------

# define debug level
DEBUG=0

# if no command line arguments, output help string
if [ -z $1 ]
then
echo "Usage: upload [-s] <directory>"
exit 1
fi

if [ $1 == "-s" ]
then
sim=1
shift
else
sim=0
fi

# generate a (fairly) unique string to use as a separator.
uniqueStr=aZbYcXxCyBzA$$
if [ $DEBUG -ge 1 ]
then
echo "Unique string:"
echo $uniqueStr
fi

# get list of files in remote directory
commandStr="cd /var/www/html; find $1 -maxdepth 1 -printf '%p $uniqueStr '"
remoteFiles=$(ssh 'admin@mywebsite.org'@www.mywebsite.org $commandStr)
if [ $DEBUG -ge 1 ]
then
echo --------------------
echo "Remote file list:"
echo $remoteFiles
fi

# split the remote file list to get an array
split "$remoteFiles" $uniqueStr
ns=${#splitted[*]}
i=0
while [ $i -lt $ns ]
do
ra[$i]=${splitted[$i]}
let "i = $i + 1"
done
if [ $DEBUG -ge 1 ]
then
echo --------------------
echo "Length of remote file array:"
echo ${#ra[*]}
echo "Contents of remote file array:"
echo ${ra[*]}
fi

# get the local file list
localFiles=`find $1 -maxdepth 1 -printf "%p $uniqueStr "`
if [ $DEBUG -ge 1 ]
then
echo --------------------
echo "Local file list:"
echo $localFiles
fi

# split the local file list
split "$localFiles" $uniqueStr
ns=${#splitted[*]}
i=0
while [ $i -lt $ns ]
do
la[$i]=${splitted[$i]}
let "i = $i + 1"
done
if [ $DEBUG -ge 1 ]
then
echo --------------------
echo "Length of local file array:"
echo ${#la[*]}
echo "Contents of local file array:"
echo ${la[*]}
fi

# match against local files
if [ $DEBUG -ge 2 ]
then
echo --------------------
fi
nl=${#la[*]}
nr=${#ra[*]}
i=0
while [ $i -lt $nl ]
do
if [ $DEBUG -ge 2 ]
then
let "k = $i + 1"
echo "Examining local file number $k (i = $i)"
fi
j=0
while [ $j -lt $nr ]
do
if [ $DEBUG -ge 3 ]
then
let "k = $j + 1"
echo "Comparing with remote file number $k (j = $j)"
fi
if [ "${la[$i]}" == "${ra[$j]}" ]
then
li="$li $i"
ri="$ri $j"
if [ $DEBUG -ge 3 ]
then
echo "Found a match."
echo "li = $li"
echo "ri = $ri"
fi
fi
let "j = $j + 1"
done
let "i = $i + 1"
done

# can now find which local files are not present in the remote directory
if [ $DEBUG -ge 1 ]
then
echo --------------------
fi
i=0
while [ $i -lt $nl ] # loop through local files
do
if [ $DEBUG -ge 2 ]
then
let "k = $i + 1"
echo "Examining local file number $k (i = $i)"
fi
j=-1 # set j to null value
for x in $li # go through local indices
do
if [ $DEBUG -ge 3 ]
then
let "k = $i + 1"
echo "li = $x"
fi
if [ $i -eq $x ] # if i'th local file is in indices list
then
j=$i # set j to be that i
fi
done
if [ $DEBUG -ge 2 ]
then
echo "j = $j"
fi
# if did not find the local file in $li then upload
if [ $j -eq -1 ] # is j equal to -1?
then
file=${la[$i]}
if [ $DEBUG -ge 2 ]
then
echo "Did not find local file $i in remote directory"
echo "File name: $file"
fi
if [ ! -d "$file" ] # if so, then if it is not a directory
then
# add file to list to be transferred
# remove space at the beginning if exists
file=${file# }
uploadList="$uploadList $file"
fi
fi
let "i = $i + 1"
done

if [ $DEBUG -ge 1 ]
then
echo --------------------
echo List of files to upload:
echo "$uploadList"
fi

# create command string and either echo it or execute it
if [ "$uploadList" ]
then
# remove initial space just to be sure
uploadList=${uploadList# }
commandStr="scp '$uploadList' 'admin@mywebsite.org'@www.mywebsite.org:/var/www/html/$1/."
if [ $sim -eq 1 ]
then
echo $commandStr
else
scp $uploadList 'admin@mywebsite.org'@www.mywebsite.org:/var/www/html/$1/.
fi
else
echo "Nothing to upload"
fi

# give a successful exit code
exit 0

Chris F.A. Johnson

2005-02-20, 6:19 pm

On Sun, 20 Feb 2005 at 20:13 GMT, Russell Goyder wrote:
>
>
> OK! Here it is, warts and all (perhaps only warts?).

All I asked for was the part that gets the file names.

[60 lines snipped]
>
> # get list of files in remote directory
> commandStr="cd /var/www/html; find $1 -maxdepth 1 -printf '%p $uniqueStr '"
> remoteFiles=$(ssh 'admin@mywebsite.org'@www.mywebsite.org $commandStr)
[166 lines snipped]

I'm assuming you are using bash (and that there are no newlines in
the file names).

IFS=$'\n'
remoteFiles=( $(ssh 'admin@mywebsite.org'@www.mywebsite.org ls /var/www/html) )
IFS=$' \t\n'

## View the list with:
printf "%s\n" "${remoteFiles[@]}"

--
Chris F.A. Johnson http://cfaj.freeshell.org/shell
========================================
===========================
My code (if any) in this post is copyright 2005, Chris F.A. Johnson
and may be copied under the terms of the GNU General Public License

Icarus Sparry

2005-02-20, 6:19 pm

On Sun, 20 Feb 2005 20:13:57 +0000, Russell Goyder wrote:

>
>
> OK! Here it is, warts and all (perhaps only warts?).

It may be slower than it need be, and is much longer!
If there are no newlines in the filenames, then the following will do it

#!/bin/sh
RF=/tmp/remote.$$
LF=/tmp/local.$$
UF=/tmp/toupload.$$
nl='
'
trap 'rm -f $RF $UF $LF; exit 0' 0
ssh 'admin@mywebsite.org'@www.mywebsite.org ls "/var/www/html/$1" > $RF
ls -C "$1" |grep -v / > $LF
comm -23 $LF $RF > $UF
if [ -s $UF ]
then
IFS=$nl
scp $(cat $UF) 'admin@mywebsite.org'@www.mywebsite.org:/var/www/html/$1/.
else
echo "Nothing to upload"
fi
exit 0

Russell Goyder

2005-02-21, 5:59 pm

> All I asked for was the part that gets the file names.
>
> [60 lines snipped]
> [166 lines snipped]

Sorry to frustrate you with too little code, and then too much.

> I'm assuming you are using bash (and that there are no newlines in
> the file names).
>
> IFS=$'\n'
> remoteFiles=( $(ssh 'admin@mywebsite.org'@www.mywebsite.org ls /var/
> www/html) ) IFS=$' \t\n'
>
> ## View the list with:
> printf "%s\n" "${remoteFiles[@]}"

Yes, there are no newlines in the file names and it is bash.

Thanks for your help.

Russell Goyder

2005-02-21, 5:59 pm

> It may be slower than it need be, and is much longer!
> If there are no newlines in the filenames, then the following will do
> it

Thanks for your help.