Unix Shell - working with .dat files

Author

working with .dat files

anju

2006-02-13, 6:04 pm

Hi,
I have the requirement as follows.

I'm trying to find the length of each rec in my .dat file. For this i'm

doing the following

for rec in `cat <filename>.dat`; do
echo ${#rec}
done

If the no of records or lines in the <filename>.dat file is suppose 10.

But, it's giving more than 10 echo messages and It's giving the wrong
output.

> cat <filename>.dat | wc -l

10

-----------------
is there any other way to deal with .dat file's or suggest me how can i

get each record in to the for loop.

How can deal with the EOL with cat.

The same is working fine with other files.

How can i solve this problem

thanks in adv,
an

Stephane CHAZELAS

2006-02-13, 6:04 pm

2006-02-13, 12:45(-08), anju:
> Hi,
> I have the requirement as follows.
>
> I'm trying to find the length of each rec in my .dat file. For this i'm
>
> doing the following
>
>
> for rec in `cat <filename>.dat`; do
> echo ${#rec}
> done
>
>
> If the no of records or lines in the <filename>.dat file is suppose 10.

You're mistaken about what `...` does.

You shouldn't use loops in shells unless you can't do otherwise.

`...`

does many things:

1- it starts a new process with its stdout redirected to a pipe
2- it reads the output of the process via the other end of the
pipe in memory.
3- it removes every trailing blank line from that (which means
that if there were blank lines at the end of the process output,
they will be discarded).
4- it splits that output according the the $IFS special
parameter. Unless you modified IFS, that means that any sequence
of space tab and newline character will separate two records and
leading and trailing blanks will be discarded.
5- last but not least, for every resulting word (from the
splitting), it will attempt to do filename generation. For
instance, if one word happens to be "*", it will expand that
into the list of files in the current directory.

So, in the end, you do end up in a list, but unless you're very
lucky, that list won't be the list of lines in the process
output.

>
> But, it's giving more than 10 echo messages and It's giving the wrong
> output.
>
>
>
>
>
> 10
>
> -----------------
> is there any other way to deal with .dat file's or suggest me how can i
[...]

Use awk or any utility that processes a file line by line.

awk '{print length}' < filename.dat

prints the length of every line in filename.dat.

--
Stéphane

Chris F.A. Johnson

2006-02-13, 6:04 pm

On 2006-02-13, anju wrote:
> Hi,
> I have the requirement as follows.
>
> I'm trying to find the length of each rec in my .dat file. For this i'm
>
> doing the following
>
>
> for rec in `cat <filename>.dat`; do

DON'T DO THAT! That is not the way to read a file line by line.
Use:

while IFS= read -r rec ## adjust or omit IFS= or omit -r as necessary
do
echo ${#rec}
done < FILENAME.dat

> echo ${#rec}
> done

Once you correct that, your problems should disappear. If not, post
your revised script.

--
Chris F.A. Johnson, author | If posting from Google Groups,
Shell Scripting Recipes: | please read:
A Problem-Solution Approach | <http://cfaj.freeshell.org/google>
2005, Apress | before replying

anju

2006-02-13, 6:04 pm

Thanks a lot for the help.

an.

anju

2006-02-13, 6:04 pm

Thanks a lot for Ur help.
an

Stephane CHAZELAS

2006-02-13, 6:04 pm

2006-02-13, 16:05(-05), Chris F.A. Johnson:
[...]
> while IFS= read -r rec ## adjust or omit IFS= or omit -r as necessary
> do
> echo ${#rec}
[...]

It should be noted that the same happens for variable expansion
as for command substitution (`...`).

When you leave ${#rec} unquoted, you ask the shell to split the
content of the variable according to IFS (which involves a
complicated algorithm), and to perform filename generation on
the resulting words. Here, luckily enough, ${#rec} will be a
number so that it shouldn't harm too much unless you modified
IFS, but it still doesn't make any sense to leave it unquoted
(unless you use zsh which fixed that behavior).

--
Stéphane