|
Home > Archive > Unix administration > January 2006 > adding up field columns in a file
You are viewing an archived Text-only version of the thread.
To view this thread in it's original format and/or if you want to reply to
this thread please [click here]
| Author |
adding up field columns in a file
|
|
| cconnell_1@lycos.com 2006-01-23, 2:55 am |
| Hello,
I have a file with some columns of numeric data e.g.
5 5
5 6
5 7
I need to total them, currently im using the following method
for i in cat <filename> | awk print column2 (for example)
do
total=3D$i
total=3D=E8xpr $total + $i`
done
However this method is really slow on large files, could someone
recommend a quicker way?
Thanks
| |
| Chris F.A. Johnson 2006-01-23, 7:52 am |
| On 2006-01-23, cconnell_1@lycos.com wrote:
> Hello,
> I have a file with some columns of numeric data e.g.
>
> 5 5
> 5 6
> 5 7
>
> I need to total them, currently im using the following method
>
>
> for i in cat <filename> | awk print column2 (for example)
> do
> total=$i
> total=èxpr $total + $i`
> done
>
> However this method is really slow on large files, could someone
> recommend a quicker way?
To add the second column:
awk -v col=2 '{ total += $col } END { print total }' FILENAME
To add the first column, change col=2 to col=1
--
Chris F.A. Johnson, author | <http://cfaj.freeshell.org>
Shell Scripting Recipes: | My code in this post, if any,
A Problem-Solution Approach | is released under the
2005, Apress | GNU General Public Licence
| |
| cconnell_1@lycos.com 2006-01-23, 7:52 am |
|
Chris F.A. Johnson wrote:
> On 2006-01-23, cconnell_1@lycos.com wrote:
>
> To add the second column:
>
> awk -v col=3D2 '{ total +=3D $col } END { print total }' FILENAME
>
> To add the first column, change col=3D2 to col=3D1
>
> --
> Chris F.A. Johnson, author | <http://cfaj.freeshell.org>
> Shell Scripting Recipes: | My code in this post, if any,
> A Problem-Solution Approach | is released under the
> 2005, Apress | GNU General Public Licence
Thanks, I think there is a typo somewhere as I get byte@osn01:/tmp-->
awk -v col=3D2 '{ total +=3D $col } END { print total }' test
awk: syntax error near line 1
awk: bailing out near line 1
Any ideas?
| |
| Chris F.A. Johnson 2006-01-23, 7:52 am |
| On 2006-01-23, cconnell_1@lycos.com wrote:
>
> Chris F.A. Johnson wrote:
>
> Thanks, I think there is a typo somewhere as I get byte@osn01:/tmp-->
> awk -v col=2 '{ total += $col } END { print total }' test
> awk: syntax error near line 1
> awk: bailing out near line 1
>
> Any ideas?
Are you using Solaris? If so, use nawk or xpg4/awk (I forget the
exact location).
--
Chris F.A. Johnson, author | <http://cfaj.freeshell.org>
Shell Scripting Recipes: | My code in this post, if any,
A Problem-Solution Approach | is released under the
2005, Apress | GNU General Public Licence
| |
| cconnell_1@lycos.com 2006-01-23, 7:52 am |
| Yes its solaris.
nawk works fantasic.Thx
| |
| Randal L. Schwartz 2006-01-23, 7:52 am |
| >>>>> "cconnell" == cconnell 1 <cconnell_1@lycos.com> writes:
cconnell> for i in cat <filename> | awk print column2 (for example)
Why the unnecessary cat?
Why not just:
cat filename | cat | cat | cat | tr | cat | awk ...
--
Randal L. Schwartz - Stonehenge Consulting Services, Inc. - +1 503 777 0095
<merlyn@stonehenge.com> <URL:http://www.stonehenge.com/merlyn/>
Perl/Unix/security consulting, Technical writing, Comedy, etc. etc.
See PerlTraining.Stonehenge.com for onsite and open-enrollment PERL training!
| |
| Chris F.A. Johnson 2006-01-23, 6:13 pm |
| On 2006-01-23, Randal L. Schwartz wrote:
>
>cconnell> for i in cat <filename> | awk print column2 (for example)
>
> Why the unnecessary cat?
>
> Why not just:
>
> cat filename | cat | cat | cat | tr | cat | awk ...
usage: tr [-csu] string1 string2
tr [-cu] -d string1
tr [-cu] -s string1
tr [-cu] -ds string1 string2
--
Chris F.A. Johnson, author | <http://cfaj.freeshell.org>
Shell Scripting Recipes: | My code in this post, if any,
A Problem-Solution Approach | is released under the
2005, Apress | GNU General Public Licence
| |
| cconnell_1@lycos.com 2006-01-24, 2:49 am |
| While on the subject, if one wants to print selective columns in a file
using awk, is it possible to do it without cat to make it faster?
i.e I always pipe the output from a file into awk from cat, can you use
awk on its own?
Thanks
| |
| Logan Shaw 2006-01-24, 7:51 am |
| cconnell_1@lycos.com wrote:
> While on the subject, if one wants to print selective columns in a file
> using awk, is it possible to do it without cat to make it faster?
> i.e I always pipe the output from a file into awk from cat, can you use
> awk on its own?
Yes. In general, you can almost always eliminate redundant "cat"s
by changing this form:
cat foo | bar
into this form:
bar < foo
With awk, there is an even shorter way, because awk can take an input
filename as a command line argument directly (just like cat does), so
you can change this
cat /etc/passwd | awk -F: '{print $1}'
into this:
awk -F: '{print $1}' /etc/passwd
You could of course use redirection so that the shell opens the file
and makes it awk's standard input:
awk -F: '{print $1}' < /etc/passwd
but that more typing.
- Logan
| |
| Chris F.A. Johnson 2006-01-24, 7:51 am |
| On 2006-01-24, cconnell_1@lycos.com wrote:
> While on the subject, if one wants to print selective columns in a file
> using awk, is it possible to do it without cat to make it faster?
> i.e I always pipe the output from a file into awk from cat, can you use
> awk on its own?
Did you look at the solutions you were given? Don't they answer
your question?
--
Chris F.A. Johnson, author | <http://cfaj.freeshell.org>
Shell Scripting Recipes: | My code in this post, if any,
A Problem-Solution Approach | is released under the
2005, Apress | GNU General Public Licence
| |
| Doug Freyburger 2006-01-24, 6:23 pm |
| cconnell_1@lycos.com wrote:
>
> While on the subject, if one wants to print selective columns in a file
> using awk, is it possible to do it without cat to make it faster?
> i.e I always pipe the output from a file into awk from cat, can you use
> awk on its own?
A large percentage of the time using cat to prime a pipeline
will end up being labelled "gratitous use of cat". You should
nearly always be able to avoid it with I/O redirection other
than a pipe, putting filenames on the command line and
so on.
And still, after decades of writing scripts I often prime a
pipeline with a cat. I especially like cat ... | while ... Why?
Stylistic reasons. I don't like while ... one < file
| |
| cconnell_1@lycos.com 2006-01-24, 6:23 pm |
| Thats great to know. Thanks for the help, I will experiment with it.
And Chris yes your answer did answer my question as there was the
filename at the end of the awk statement- doh!
| |
| Xicheng 2006-01-25, 2:58 am |
| cconnell_1@lycos.com wrote:
> Hello,
> I have a file with some columns of numeric data e.g.
>
> 5 5
> 5 6
> 5 7
>
> I need to total them, currently im using the following method
you can sum up the 2nd column by:
awk '{t+=$2}END{print t}' FILENAME
sum up two columns:
awk '{t1+=$1;t2+=$2}END{print "total:",t1,t2}' FILENAME
sum up all columns:
perl -alne '$t[$_]+=$F[$_]for(0..$#F);END{print "total: @t\n"}'
FILENAME
Xicheng
|
|
|
|
|