Unix Programming - self-tailing output checker

Author

self-tailing output checker

Henry Townsend

2005-04-08, 5:52 pm

Where I work we do many builds that must rely on unreliable tools which
we don't control. In some cases they return an incorrect exit status, in
others there are situations which they don't consider a failure but
which we do. So there's a need to check their output and parse it for
certain user-defined patterns in addition to watching exit status.

This is normally done by diverting output to a log file, then grepping
through it later. But that's unsatisying because (a) the user can't
control where output goes via standard redirection, (b) it's harder to
keep track of progress when output isn't flowing to stdout, and (c)
there's a bunch of ugly hackery to implement in each script.

I'm trying to solve this more elegantly with a PERL module. The idea is
it would cause the program which uses it to create two pipes and fork.
One side of the fork would connect stdout and stderr to the write ends
of the pipes and continue as usual. The other process would listen on
the read ends, probably using select(). For each line read it would pass
it by a regular expression and then copy it to the real output stream
corresponding to the pipe it came out of. The exit status would be
bumped if any lines matched the REs.

A simpler alternative which might be acceptable would be to dup stdout
and stderr into one pipe which would then be re-sorted into stderr
(lines which match the RE) and stdout (the rest). This wouldn't require
select but I'd prefer to keep the "real" stdout/stderr streams.

I'm working on a proof of concept but before I post code or ask specific
questions, does anyone know of an existing implementation? Doesn't have
to be in Perl, I'd be happy to translate. In fact I originally sought
advice in a PERL newsgroup but then realized it's really a systems
programming issue; the language is beside the point.

--
Thanks,
Henry Townsend

Martin Carpenter

2005-04-08, 5:52 pm

"Henry Townsend" <henry.townsend@not.here> wrote:

> I'm working on a proof of concept but before I post
> code or ask specific questions, does anyone know of
> an existing implementation?

tee(1)

expect ?

zentara

2005-04-08, 5:52 pm

On Fri, 08 Apr 2005 10:13:34 -0400, Henry Townsend
<henry.townsend@not.here> wrote:

>
>I'm working on a proof of concept but before I post code or ask specific
>questions, does anyone know of an existing implementation? Doesn't have
>to be in Perl, I'd be happy to translate. In fact I originally sought
>advice in a PERL newsgroup but then realized it's really a systems
>programming issue; the language is beside the point.

You might want to ask on http://perlmonks.org , PERL is probably
the perfect tool to do this.

Here is a simple trick you might want to look at, where
a script can read it's own STDOUT

#!/usr/bin/perl -w
use strict;

set_it_up();
print "from set_it_up(), i got: ", scalar <STDIN>;
close(STDOUT);

sub set_it_up{
my $pid;
return if $pid = open(STDOUT,"|-");
die("Unable to attache to STDOUT by fork: $!") unless(defined $pid);
print "hello world\n";
}

There is also IPC::Open3 , which lets you control the STDIN, STDOUT,
and STDERR filehandles separately, and put them in IO::Select.

--
I'm not really a human, but I play one on earth.
http://zentara.net/japh.html

Chuck Dillon

2005-04-08, 5:52 pm

Henry Townsend wrote:
> Where I work we do many builds that must rely on unreliable tools which
> we don't control. In some cases they return an incorrect exit status, in
> others there are situations which they don't consider a failure but
> which we do. So there's a need to check their output and parse it for
> certain user-defined patterns in addition to watching exit status.
>
> This is normally done by diverting output to a log file, then grepping
> through it later. But that's unsatisying because (a) the user can't
> control where output goes via standard redirection, (b) it's harder to
> keep track of progress when output isn't flowing to stdout, and (c)
> there's a bunch of ugly hackery to implement in each script.
>
> I'm trying to solve this more elegantly with a PERL module. The idea is
> it would cause the program which uses it to create two pipes and fork.
> One side of the fork would connect stdout and stderr to the write ends
> of the pipes and continue as usual. The other process would listen on
> the read ends, probably using select(). For each line read it would pass
> it by a regular expression and then copy it to the real output stream
> corresponding to the pipe it came out of. The exit status would be
> bumped if any lines matched the REs.
>
> A simpler alternative which might be acceptable would be to dup stdout
> and stderr into one pipe which would then be re-sorted into stderr
> (lines which match the RE) and stdout (the rest). This wouldn't require
> select but I'd prefer to keep the "real" stdout/stderr streams.
>
> I'm working on a proof of concept but before I post code or ask specific
> questions, does anyone know of an existing implementation? Doesn't have
> to be in Perl, I'd be happy to translate. In fact I originally sought
> advice in a PERL newsgroup but then realized it's really a systems
> programming issue; the language is beside the point.
>

You might want to consider using Expect(http://expect.nist.gov/) and
maybe a little Tcl?

-- ced

--
Chuck Dillon
Senior Software Engineer
NimbleGen Systems Inc.

Moshe Jacobson

2005-04-12, 2:51 am

Henry Townsend <henry.townsend@not.here> wrote:
> I'm trying to solve this more elegantly with a PERL module. The idea is
> it would cause the program which uses it to create two pipes and fork.
> One side of the fork would connect stdout and stderr to the write ends
> of the pipes and continue as usual. The other process would listen on
> the read ends, probably using select(). For each line read it would pass
> it by a regular expression and then copy it to the real output stream
> corresponding to the pipe it came out of. The exit status would be
> bumped if any lines matched the REs.

Forgive me if I misunderstand your needs, but I think a simple shell
script, combined with the use of tee(1) should help significantly.

Moshe

--
*** SPAM BLOCK: Remove bra before replying! ***
http://runslinux.net :: moshe at runslinux dot net :: AIM: Jehsom

Henry Townsend

2005-04-12, 7:49 am

Moshe Jacobson wrote:
> Forgive me if I misunderstand your needs, but I think a simple shell
> script, combined with the use of tee(1) should help significantly.

No, tee is not appropriate. I want to keep stdout and stderr separately
redirectable and I want to not throw away the exit status of the
monitored command. Tee would not allow either of these.

The fact is, I've gotten it working in PERL to my satisfaction now. The
only problem is that I want it to work on Windows too before I release
it, and I'm having some problems there due to the funky Win32 Perl
emulations of fork() and select(). But I haven't brought those problems
here for obvious reasons.

--
Henry Townsend

zentara

2005-04-13, 8:03 am

On Tue, 12 Apr 2005 07:57:50 -0400, Henry Townsend
<henry.townsend@not.here> wrote:

>Moshe Jacobson wrote:
>
>No, tee is not appropriate. I want to keep stdout and stderr separately
>redirectable and I want to not throw away the exit status of the
>monitored command. Tee would not allow either of these.
>
>The fact is, I've gotten it working in PERL to my satisfaction now. The
>only problem is that I want it to work on Windows too before I release
>it, and I'm having some problems there due to the funky Win32 Perl
>emulations of fork() and select(). But I haven't brought those problems
>here for obvious reasons.

For Win32 forking with Perl, use IPC::Run, which works on unix or win32.

--
I'm not really a human, but I play one on earth.
http://zentara.net/japh.html