Unix Shell - filenames with spaces and list in a for loop

Author

filenames with spaces and list in a for loop

SiKing

2007-01-24, 1:16 pm

Hi all,

I need to process files in a loop, and I am having problems with filename
spaces, big time.

Everything is in bash 3.1.
The script in question is supposed to build a link-forest: replicate a directory
structure, but replace each file with a link pointing to the original. I decided
to do gro of the work with a recursive function:

function make_links
{
for each_item in $LIST; do
if [ -L "$1/$each_item" ]; then
# link
rm -f "$each_item" # mercilessly overwrite existing stuff
ln -s `readlink "$1/$each_item"` "$each_item"
elif [ -d "$1/$each_item" ]; then
# directory
mkdir -p "$each_item"
cd "$each_item"
make_links "$1/$each_item" &
cd ..
else
# file
rm -f "$each_item" # mercilessly overwrite existing stuff
ln -s "$1/$each_item" "$each_item"
fi
done
return 0
}

I am having significant issues with spaces in filenames, and I figure everything
revolves around the way I generate the $LIST in the above function.

My first attempt was to _replace_ the $LIST with `ls -1A $1`. I wanted this to
handle dot-files as well. Unfortunately, any file with a space would be handled
as two items. No good!
Second attempt was to double quote the "`ls -1A $1`", which generates ONE huge
argument to the for loop. Again, no good!

In the next round I experimented with `ls -mA $1`. This generates a comma
separated list, but it also brings several new problems. To break up the comma
separated list, I used IFS=','. However, this left each_item with a leading
extra space. I corrected this with a each_item=`eval echo $each_item` inside the
loop. Unfortunately, the list also generates a newline every 80 characters (or
sooner) in the $LIST, so once in a while I was left with each_item being set to
something like "\nfilename". At the first occurrence of $each_item in the
function I would get something like <filename> not found. 'man ls' hinted that
the secret is in the variable COLUMNS. Seems export COLUMNS=0 does nothing.
Right now I got COLUMNS=65000, but what if I have a directory that is longer?
This seems like the best option out of the ones I tried. However, what if I have
a directory with more than 65000 characters worth? Or, what is a better way of
getting rid of the newlines?

I also tried to use expansion. I generated the list with simple $1/*, which also
gives me the complete path, and no dot-files. The path can be corrected with
each_item=`basename "$each_item"`. However, trying to fix the dot-files with
adding $1/.* to the list, has "very interesting" effects -> DO NOT TRY THIS AT
HOME! After having to reboot the machine and purge the disk of a _lot_ of links,
the sys.admin. warned me against doing any more stupid things.

Anyone has any other _workable_ solution?

TIA, SK.

--
It may be that your sole purpose in life
is simply to serve as a warning to others.
-----
Candy for spammers:
http://members.shaw.ca/grubb/spamthis.html

Eric Moors

2007-01-24, 1:16 pm

SiKing wrote:

> Hi all,
>
> I need to process files in a loop, and I am having problems with filename
> spaces, big time.
>
> Everything is in bash 3.1.

shopt -s dotglob
for each_item in $1/*; do
...
done

Eric

Stephane CHAZELAS

2007-01-24, 1:16 pm

2007-01-24, 16:47(+01), Eric Moors:
> SiKing wrote:
>
>
> shopt -s dotglob
> for each_item in $1/*; do
> ...
> done
[...]

for each_item in "$1"/*; do

unless you want to consider the content of $1 as a globbing pattern as well.

--
Stéphane

Bill Marcum

2007-01-24, 1:16 pm

On Wed, 24 Jan 2007 15:30:20 +0000, SiKing
<nospam@noway.invalid> wrote:
>
>
> Hi all,
>
>
> I am having significant issues with spaces in filenames, and I figure
> everything revolves around the way I generate the $LIST in the above
> function.
>
Instead of 'for each_item in $LIST', try
while read each_item; do ... ; done < $FILE
or
ls -A "$1" | while read each_item ...

--
Q: Why do the police always travel in threes?
A: One to do the reading, one to do the writing, and the other keeps
an eye on the two intellectuals.

Stephane CHAZELAS

2007-01-24, 1:16 pm

2007-01-24, 10:57(-05), Bill Marcum:
> On Wed, 24 Jan 2007 15:30:20 +0000, SiKing
> <nospam@noway.invalid> wrote:
> Instead of 'for each_item in $LIST', try
> while read each_item; do ... ; done < $FILE
> or
> ls -A "$1" | while read each_item ...
[...]

To read a line of input (which is already an approximation for
reading an item from a list, given that a filename doesn't
necessarily map to one line), it is not

read each_item

it is

IFS= read -r each_item

"read each_item" means a different thing.

And to loop over the elements of a ksh/bash/zsh list type
variable, it is not

for each_item in $LIST

(except in zsh) it's is:

for each_item in "${LIST[@]}"

And if $LIST is not a list-type variable but a string type
variable containing a newline separated list of items, it is
not

for each_item in $LIST

But:

set -f
IFS='
'
for each_item in $LIST

--
Stéphane

SiKing

2007-01-25, 7:19 am

Stephane CHAZELAS wrote:
> 2007-01-24, 16:47(+01), Eric Moors:
> [...]
>
> for each_item in "$1"/*; do
>
> unless you want to consider the content of $1 as a globbing pattern as well.

This one seems to be the cleanest. I did not even know about 'shopt'. Thanx!

In case anyone is interested, the final^Wworking solution is:
function make_links
{
echo Processing $1

shopt -s dotglob # reveal dotfiles
shopt -s nullglob # skip empty directories
for each_item in "$1"/*; do
each_item=`basename "$each_item"` # extract just the filename

if [ -L "$1/$each_item" ]; then
# link
link_name=`readlink "$1/$each_item"`
rm -f "$each_item" # mercilessly overwrite existing stuff
ln -s "$link_name" "$each_item"
elif [ -d "$1/$each_item" ]; then
# directory
mkdir -p "$each_item"
cd "$each_item"
make_links "$1/$each_item" &
cd ..
else
# file
rm -f "$each_item" # mercilessly overwrite existing stuff
ln -s "$1/$each_item" "$each_item"
fi

done

return 0
}

I am sure it can still be tweaked, but I am worried about tweaking things that
already work.

SK.

--
It may be that your sole purpose in life
is simply to serve as a warning to others.
-----
Candy for spammers:
http://members.shaw.ca/grubb/spamthis.html