Unix Shell - SED and shell scripting

Author

SED and shell scripting

Xancatal

2005-02-25, 5:59 pm

Hey everybody! I need a scripting GURU to decipher this dilemma. I need
to replace a string, and append to the same line. I need to do some
kind of conditional statement before replacing test. For example on thi
file.

abcdefg rt3 #
abcdefg rt4 #
abcdefg rt5 # tke
abcdefg rt6 #
abcdefg rt7 # tke
abcdefg rt8 #

I need to find which of the lines consists of less than 4 fields (line
1), but that is easy. I did it with this nawk script:

nawk '
{if ($NF > 3){
print $0
}
}
` filename

The only problem I have is that once I find this record (rt3), I need
to replace its first field ($1) either within nawk, or use some kind of
sed script once I've obtain the record number (rt3). I have had issues
figuring out how to use the RE on sed for replacing text as in:

sed 's///g' filename

Anyone outthere knows how to use variables on a shell script for making
these changes?

For example:

a="sometext"

and then using the value of ${a} within a sed script for replacing a
string? Thanks GURUs

Chris F.A. Johnson

2005-02-25, 5:59 pm

On Fri, 25 Feb 2005 at 18:03 GMT, Xancatal wrote:
> Hey everybody! I need a scripting GURU to decipher this dilemma. I need
> to replace a string, and append to the same line. I need to do some
> kind of conditional statement before replacing test. For example on thi
> file.
>
> abcdefg rt3 #
> abcdefg rt4 #
> abcdefg rt5 # tke
> abcdefg rt6 #
> abcdefg rt7 # tke
> abcdefg rt8 #
>
> I need to find which of the lines consists of less than 4 fields (line
> 1), but that is easy. I did it with this nawk script:
>
> nawk '
> {if ($NF > 3){
> print $0
> }
> }
> ` filename

Or just:

nawk 'NF > 3' filename

> The only problem I have is that once I find this record (rt3), I need
> to replace its first field ($1) either within nawk,

nawk 'NF > 3 { $1 = newstr; print }' newstr=sometext filename

> or use some kind of sed script once I've obtain the record number
> (rt3). I have had issues figuring out how to use the RE on sed for
> replacing text as in:
>
> sed 's///g' filename
>
> Anyone outthere knows how to use variables on a shell script for making
> these changes?
>
> For example:
>
> a="sometext"
>
> and then using the value of ${a} within a sed script for replacing a
> string? Thanks GURUs

Use double quotes so the variable is expanded:

sed "s/$a/$b/g" filename

--
Chris F.A. Johnson http://cfaj.freeshell.org/shell
========================================
===========================
My code (if any) in this post is copyright 2005, Chris F.A. Johnson
and may be copied under the terms of the GNU General Public License

Xancatal

2005-02-25, 5:59 pm

It might be easiest to replace the string within nawk using sub(), but
then I need to put this replaced string back to the file. In other
words, the change I make has to be reflected on the original file, and
up until now I have only been able to make changes to a string and
display to output on the screen.

Ed Morton

2005-02-25, 5:59 pm

Xancatal wrote:

> Hey everybody! I need a scripting GURU to decipher this dilemma. I need
> to replace a string, and append to the same line. I need to do some
> kind of conditional statement before replacing test. For example on thi
> file.
>
> abcdefg rt3 #
> abcdefg rt4 #
> abcdefg rt5 # tke
> abcdefg rt6 #
> abcdefg rt7 # tke
> abcdefg rt8 #
>
> I need to find which of the lines consists of less than 4 fields (line
> 1),

Lines 1, 2, 4 and 5 all have less than 4 fields.

but that is easy. I did it with this nawk script:
>
> nawk '
> {if ($NF > 3){
> print $0
> }
> }
> ` filename

ITYM:

nawk 'NF<4' filename

> The only problem I have is that once I find this record (rt3), I need
> to replace its first field ($1) either within nawk,

This:

nawk 'NF<4{sub($1,"b");print}' filename

would replace $1 with "b" and print the result.

or use some kind of
> sed script once I've obtain the record number (rt3). I have had issues
> figuring out how to use the RE on sed for replacing text as in:
>
> sed 's///g' filename
>
> Anyone outthere knows how to use variables on a shell script for making
> these changes?
>
> For example:
>
> a="sometext"
>
> and then using the value of ${a} within a sed script for replacing a
> string? Thanks GURUs

sed "s/whatever/$a/" filename

Ed.
..
..
..
..
..
..
..
..
..

Xancatal

2005-02-25, 5:59 pm

Thanks Ed. In this case:

nawk 'NF<4{sub($1,"b");print}' filename

"b" has to be a variable like in:

read var

And once this car can be assigned to nawk via nawk -v myvar=$var, I
would then need to append a space and another entry to the end of the
string, and then write it to a file. Pretty fun ain't it? je je

Xancatal

2005-02-25, 5:59 pm

I forgot to mention the error message:

nawk: illegal field $()
input record number 1, file ethers
source line number 1

This is the result of doing the assignment as such:

nawk -v var=$adr 'NF<4{sub($1,$adr);print}' filename

Ed Morton

2005-02-25, 5:59 pm

Xancatal wrote:

> Thanks Ed. In this case:
>
> nawk 'NF<4{sub($1,"b");print}' filename
>
> "b" has to be a variable like in:
>
> read var
>
> And once this car can be assigned to nawk via nawk -v myvar=$var, I
> would then need to append a space and another entry to the end of the
> string, and then write it to a file. Pretty fun ain't it? je je
>

This would work in gawk (I don't use nawk, but it probably would there too):

awk -v var="$var" 'NF<4{sub($1,var" "otherentry);print > "newfile"}'
filename

or just this:

awk -v var="$var" 'NF<4{sub($1,var" "otherentry);print}' filename > newfile

In the above, otherentry can be a variable or a string.

Regards,

Ed.

Ed Morton

2005-02-25, 5:59 pm

Xancatal wrote:
> It might be easiest to replace the string within nawk using sub(), but
> then I need to put this replaced string back to the file. In other
> words, the change I make has to be reflected on the original file, and
> up until now I have only been able to make changes to a string and
> display to output on the screen.
>

Write the result to a tmp file, then move the tmp file to overwrite the
original. e.g.

nawk '...' file > tmp
mv tmp file

If you can't do that, you can use this approach within your awk program:

function printout(_str) { _out[++_nr] = _str }
function flushout() { close(FILENAME)
for (x=1; x<=_nr;x++)
print _out[x] > FILENAME
}
NF<4{sub($1,"b");printout( $0 ) }
END { flushout() }

i.e. store the modifcations in an array until the end of the program,
then overwrite the original file. I don't use nawk, but the above will
work for gawk and should also work for nawk, perhaps with some small tweaks.

Ed.

Ed Morton

2005-02-25, 5:59 pm

Xancatal wrote:

> I forgot to mention the error message:
>
> nawk: illegal field $()
> input record number 1, file ethers
> source line number 1
>
> This is the result of doing the assignment as such:
>
> nawk -v var=$adr 'NF<4{sub($1,$adr);print}' filename
>

nawk -v var="$adr" 'NF<4{sub($1,var);print}' filename

Ed.

Xancatal

2005-02-25, 5:59 pm

When I use this line:

read myvar

awk -v var="$myvar" 'NF<4{sub($1,$var);print}' filename > newfile

I get the following results:

nawk: illegal field $(sometexttoreplace)
input record number 1, file ethers
source line number 3

I have tried just using double quoted text as in:

awk -v var="$myvar" 'NF<4{sub($1,"sometexttoreplace");print}' filename
> newfile

and it works. The issue here is that I only need to do this replacement
of text on the first instance of NF < 4, and so I would only be sending
that one entry to a newfile. I probably need to somewhat then replace
the entire text for every instance of that record locator. So in this
case:

abcdefg rt3 #
abcdefg rt4 #
abcdefg rt5 # tke
abcdefg rt6 #
abcdefg rt7 # tke
abcdefg rt8 #

I only need to change the first field on "rt3" and bail out. I do so by
using the "exit 0" within awk which basically exits out within the
first instance of NF < 4.

I think I have a problem using the "value" of var (or $var) within the
sub() command.

Xancatal

2005-02-25, 5:59 pm

Thank Ed, This worked pretty cool. I figured after I posted that I was
not supposed to use $ bafore var. I can now edit and bail out. How'bout
concatenating another entry. Perhaps a space. I know that it can be
done as stated on the GNU pages. The only other issue is how to
incorporate this change onto a file which contains the remaining data.

Ed Morton

2005-02-25, 5:59 pm

Xancatal wrote:
> When I use this line:
>
> read myvar
>
> awk -v var="$myvar" 'NF<4{sub($1,$var);print}' filename > newfile
>
> I get the following results:
>
> nawk: illegal field $(sometexttoreplace)
> input record number 1, file ethers
> source line number 3

Yeah you will. STOP ADDING A "$" SIGN TO THE AWK VARIABLE!!!!!

<snip>
> I only need to change the first field on "rt3" and bail out. I do so by
> using the "exit 0" within awk which basically exits out within the
> first instance of NF < 4.

OK. Sounds like you're happy with that, right? Or do you want to
reproduce the other lines as-is?

> I think I have a problem using the "value" of var (or $var) within the
> sub() command.
>

Yes, the problem is that you keep adding a $ sign in front of the awk
variable name!

Ed.

Ed Morton

2005-02-25, 5:59 pm

Xancatal wrote:

> Thank Ed, This worked pretty cool. I figured after I posted that I was
> not supposed to use $ bafore var. I can now edit and bail out. How'bout
> concatenating another entry. Perhaps a space. I know that it can be
> done as stated on the GNU pages. The only other issue is how to
> incorporate this change onto a file which contains the remaining data.
>

See my other responses as I think I've answered all of that already
(depending on what you mean by "entry"). If it still isn't clear, post a
followup.

By the way, please include some context in your postings and try to
avoid starting separate sub-threads for each question, just to lessen
confusion (mine!).

Ed.

Dan Mercer

2005-02-25, 5:59 pm

"Chris F.A. Johnson" <cfajohnson@gmail.com> wrote in message news:389825F51lt52U1@individual.net...
: On Fri, 25 Feb 2005 at 18:03 GMT, Xancatal wrote:
: > Hey everybody! I need a scripting GURU to decipher this dilemma. I need
: > to replace a string, and append to the same line. I need to do some
: > kind of conditional statement before replacing test. For example on thi
: > file.
: >
: > abcdefg rt3 #
: > abcdefg rt4 #
: > abcdefg rt5 # tke
: > abcdefg rt6 #
: > abcdefg rt7 # tke
: > abcdefg rt8 #
: >
: > I need to find which of the lines consists of less than 4 fields (line
: > 1), but that is easy. I did it with this nawk script:
: >
: > nawk '
: > {if ($NF > 3){
: > print $0
: > }
: > }
: > ` filename
:
: Or just:
:
: nawk 'NF > 3' filename
:
: > The only problem I have is that once I find this record (rt3), I need
: > to replace its first field ($1) either within nawk,
:
: nawk 'NF > 3 { $1 = newstr; print }' newstr=sometext filename

This will cause reformatting of $0 compressing field delimiters:

$ awk '{print;$1 = "abc";print}'
123 456
123 456
abc 456

Dan Mercer

:
: > or use some kind of sed script once I've obtain the record number
: > (rt3). I have had issues figuring out how to use the RE on sed for
: > replacing text as in:
: >
: > sed 's///g' filename
: >
: > Anyone outthere knows how to use variables on a shell script for making
: > these changes?
: >
: > For example:
: >
: > a="sometext"
: >
: > and then using the value of ${a} within a sed script for replacing a
: > string? Thanks GURUs
:
: Use double quotes so the variable is expanded:
:
: sed "s/$a/$b/g" filename
:
:
: --
: Chris F.A. Johnson http://cfaj.freeshell.org/shell
: ========================================
===========================
: My code (if any) in this post is copyright 2005, Chris F.A. Johnson
: and may be copied under the terms of the GNU General Public License

Xancatal

2005-02-25, 5:59 pm

Yeah!! I figure that one out Ed!! je je....old scripting habit I guess.
In deed I stop at the first instance of NF < 4. I then redirect the
output to a file which only consists of the newly edited line. I now
need to make it so that that line appears on the source file. I thought
about first finding record (rt3), and then deleting it...appending the
new file to it and re-indexing the file based on its second field.

Thanks for all your help Ed. You're da man!!!

Bye the way... I don't need to reproduce the other lines, just the
first instance of a less than four fields line.