Unix Programming - Nested loops in scripts

Author

Nested loops in scripts

Norris Watkins

2004-05-17, 8:35 pm

Hello:
I have to iterate through my huge log files in the following fashion.

OUT_BLOCK_BEGIN BLOCK_ID
.....
OUT_BLOCK_END

......

INNER_BLOCK_BEGIN BLOCK_ID

INNER_BLOCK_END

The log file will have ( among other things ) outer blocks ( demarked
by OUT_BLOCK_BEGIN and OUT_BLOCK_END ).

My script should iterate through the file to find these blocks. Once
an outer block is found, there will exist another block *somewhere
below* the outer block, with an id same as the outer block. I need to
find this inner block and log certain parts of it along with the parts
from the outer blocks.

These outer blocks and inner blocks will appear pretty random, except
that the inner block with the same id will always be *after* the
occurance of the corresponding outer block. Essentially I need to do a
nested searching within the file.

Using sed or awk, I could not think of any way to do nested searchs,
as it does not have multiple cursors ( Or mechanisms to store a
current file pointer location, which can be used to come back to the
position later )

I m learning more towards perl, which is the only scripting language I
think will work. Basically I will have the search pattern, when outer
block is found, the file position will be stored and then the inner
block detecting iteration will be initiated. Once that is finished, I
will come back to the stored position and resume with the outer block
iteration.

Before delving deeper, I wanted to ask you guys, if there is a better
suited scripting language for the occasion, or a better algorithm in
perm.

Thanks for reading the long mail
--sony

Eric Moors

2004-05-18, 3:34 am

Norris Watkins wrote:

> Hello:
> I have to iterate through my huge log files in the following fashion.
>
> OUT_BLOCK_BEGIN BLOCK_ID
> ....
> OUT_BLOCK_END
>
> .....
>
> INNER_BLOCK_BEGIN BLOCK_ID
>
> INNER_BLOCK_END
>
> The log file will have ( among other things ) outer blocks ( demarked
> by OUT_BLOCK_BEGIN and OUT_BLOCK_END ).
>
> My script should iterate through the file to find these blocks. Once
> an outer block is found, there will exist another block *somewhere
> below* the outer block, with an id same as the outer block. I need to
> find this inner block and log certain parts of it along with the parts
> from the outer blocks.
>
> These outer blocks and inner blocks will appear pretty random, except
> that the inner block with the same id will always be *after* the
> occurance of the corresponding outer block. Essentially I need to do a
> nested searching within the file.
>
> Using sed or awk, I could not think of any way to do nested searchs,
> as it does not have multiple cursors ( Or mechanisms to store a
> current file pointer location, which can be used to come back to the
> position later )

You don't need to do nested searchs AFAICT.
Going just once through the file is far more effective.
And (g)awk is definitely capable of doing this.

So, adjust the following script to your needs
(I have no idea what your BLOCK_ID's look like)

#!/bin/gawk -f

{
if (match($0, /(OUT_BLOCK_BEGIN) ([[:alnum:]]*)/, ob) != 0) {
id[ob[2]] = 1
}
if (match($0, /(OUT_BLOCK_END) ([[:alnum:]]*)/, oe) != 0) {
if (oe[2] in id) {
id[oe[2]] = 2
} else {
print "End without begin? " oe[0]
exit
}
}
if (match($0, /(INNER_BLOCK_BEGIN) ([[:alnum:]]*)/, ib) != 0) {
if (ib[2] in id) {
id[ib[2]] = 3
} else {
print "Inner loop without preceding outer loop? " ib[0]
exit
}
}
if (match($0, /(INNER_BLOCK_END) ([[:alnum:]]*)/, ie) != 0) {
delete id[ie[2]]
}

for (i in id) {
if (id[i]==3) {
print "inside INNER LOOP " i
}
}
}

> I m learning more towards perl, which is the only scripting language I
> think will work. Basically I will have the search pattern, when outer
> block is found, the file position will be stored and then the inner
> block detecting iteration will be initiated. Once that is finished, I
> will come back to the stored position and resume with the outer block
> iteration.

Well you could do that, and PERL can solve this as well.
I'm not sure the algorithm you have in mind is very effective though.

Eric

rakesh sharma

2004-05-18, 11:34 pm

sonyantony@hotmail.com (Norris Watkins) wrote in message news:

>
> I have to iterate through my huge log files in the following fashion.
>
> OUT_BLOCK_BEGIN BLOCK_ID
> ....
> OUT_BLOCK_END
>
> .....
>
> INNER_BLOCK_BEGIN BLOCK_ID
>
> INNER_BLOCK_END
>
> The log file will have ( among other things ) outer blocks ( demarked
> by OUT_BLOCK_BEGIN and OUT_BLOCK_END ).
>
> My script should iterate through the file to find these blocks. Once
> an outer block is found, there will exist another block *somewhere
> below* the outer block, with an id same as the outer block. I need to
> find this inner block and log certain parts of it along with the parts
> from the outer blocks.
>
> These outer blocks and inner blocks will appear pretty random, except
> that the inner block with the same id will always be *after* the
> occurance of the corresponding outer block. Essentially I need to do a
> nested searching within the file.
>
> Using sed or awk, I could not think of any way to do nested searchs,
> as it does not have multiple cursors ( Or mechanisms to store a
> current file pointer location, which can be used to come back to the
> position later )
>

one way can be:

sed -e '
/^OUT_BLOCK_BEGIN/,/^OUT_BLOCK_END/!b
H;/^OUT_BLOCK_END/!d
/^OUTER_BLOCK_END/,/INNER_BLOCK_END/{H;/INNER_BLOCK_END/!d;}
g;# now you have the full OUTER and INNER blocks
' logfile

Sony Antony

2004-05-19, 5:40 pm

"> > Using sed or awk, I could not think of any way to do nested searchs,
>
> one way can be:
>
> sed -e '
> /^OUT_BLOCK_BEGIN/,/^OUT_BLOCK_END/!b
> H;/^OUT_BLOCK_END/!d
> /^OUTER_BLOCK_END/,/INNER_BLOCK_END/{H;/INNER_BLOCK_END/!d;}
> g;# now you have the full OUTER and INNER blocks
> ' logfile

Thanks for the reply.
I think I understood your script ( You meant "INNER_BLOCK" instead of
OUTER_BLOCK_END in the 3rd line right ? )
I doubt that it will work though, as I need to search for the
*corresponding* inner block, *once an outer block is found*.
Also there could be multiple other outer blocks appearing between an outer
block and its corresponding inner blocks.

I cannot think of any ways without 2 cursors - one iterating for the outer
blocks and the other iterating for the corresponding inner blocks.

Thanks again

--sony