Hi all,

I'm writing a program which needs to launch two other programs, which will
then communicate via a pipe - one reads a file, translates it, and writes
to standard output, and the other reads standard output, translates it,
and writes to a file.

If i was writing a shell script, i'd say:

program1 inputfile | program2 outputfile

How do i do this from a real programming language? I'm using python, but i
think an approach that would work in C is usable here.

I've come up with two options so far.

The first is to use system(), passing in a string containing the command
as i'd write it in a shell script. This is simple and easy, but it feels
icky; i have to munge strings to build the command, it's vulnerable to
bugs related to escaping the command string, and it's a potential security
hole (although i won't be passing in arbitrary strings from untrusted
users or anything). Also, i was under the impression that system() worked
by spawning a subshell, although ps shows no sign of such a thing when i
do it; is that right?

The second is to use fork/exec, which in Python are wrapped up into a
'spawn' function (well, a family of functions [1]), to run the programs
separately, and have them communicate via a named pipe. For this to work,
both programs have to be able to use a file instead of standard in/out,
but in this case, they can, so that's okay. I then do something like:

char *pipename = inputfile + ".pipe" ; /* not valid C, but YKWIM */
mkfifo(pipename) ;
char *argv1[] = {"program1", "inputfile", pipename} ;
int pid1 = spawnv("/path/to/program1", argv1) ;
char *argv2[] = {"program1", pipename, "outputfile"} ;
int pid2 = spawnv("/path/to/program2", argv2) ;
int status ;
waitpid(pid1, &status, 0) ;
waitpid(pid2, &status, 0) ;
remove(pipename) ;

Only with some error checking. Now, i'm happier about the spawning than
the system call, since there's no escaping and no potential for injection
to worry about, but the stuff with the named pipe is terrible - i'm
polluting the globally visible filesystem with my internal workings, and
exposing those workings to random other programs, i'm risking leaving
garbage pipes around if i crash, i have to worry about my pipe name
conflicting with an existing file, etc.

I suppose i could do something like fork, then have the child popen
program 2 and exec program 1, (provided i didn't need to do anything with
the stdout of program 2). However, popen has an interface like system
rather than exec, so that doesn't help much.

So, what do i do? Is there any way to launch two linked subprograms like
this? How do shells do it?

tom

[1] That look a bit like:

int spawnv(const char *path, char *const argv[])
{
int pid = fork() ;
if (pid == 0) execv(path, argv) ;
else return pid ;
}

--
i'm prepared to do anything as long as someone else works out how to do
it and gives me simple instructions... -- Sean

[ Post a follow-up to this message ]

Tom Anderson wrote:
> Hi all,
>
> I'm writing a program which needs to launch two other programs, which
> will then communicate via a pipe - one reads a file, translates it, and
> writes to standard output, and the other reads standard output,
> translates it, and writes to a file.
>
> If i was writing a shell script, i'd say:
>
> program1 inputfile | program2 outputfile
>
> How do i do this from a real programming language? I'm using python, but
> i think an approach that would work in C is usable here.
>
> I've come up with two options so far.
>
> The first is to use system(), passing in a string containing the command
> as i'd write it in a shell script. This is simple and easy, but it feels
> icky; i have to munge strings to build the command, it's vulnerable to
> bugs related to escaping the command string, and it's a potential
> security hole (although i won't be passing in arbitrary strings from
> untrusted users or anything). Also, i was under the impression that
> system() worked by spawning a subshell, although ps shows no sign of
> such a thing when i do it; is that right?
>
> The second is to use fork/exec, which in Python are wrapped up into a
> 'spawn' function (well, a family of functions [1]), to run the program
s
> separately, and have them communicate via a named pipe. For this to
> work, both programs have to be able to use a file instead of standard
> in/out, but in this case, they can, so that's okay. I then do something
> like:
>
> char *pipename = inputfile + ".pipe" ; /* not valid C, but YKWIM */
> mkfifo(pipename) ;
> char *argv1[] = {"program1", "inputfile", pipename} ;
> int pid1 = spawnv("/path/to/program1", argv1) ;
> char *argv2[] = {"program1", pipename, "outputfile"} ;
> int pid2 = spawnv("/path/to/program2", argv2) ;
> int status ;
> waitpid(pid1, &status, 0) ;
> waitpid(pid2, &status, 0) ;
> remove(pipename) ;
>
> Only with some error checking. Now, i'm happier about the spawning than
> the system call, since there's no escaping and no potential for
> injection to worry about, but the stuff with the named pipe is terrible
> - i'm polluting the globally visible filesystem with my internal
> workings, and exposing those workings to random other programs, i'm
> risking leaving garbage pipes around if i crash, i have to worry about
> my pipe name conflicting with an existing file, etc.
>
> I suppose i could do something like fork, then have the child popen
> program 2 and exec program 1, (provided i didn't need to do anything
> with the stdout of program 2). However, popen has an interface like
> system rather than exec, so that doesn't help much.
>
> So, what do i do? Is there any way to launch two linked subprograms like
> this? How do shells do it?
>
> tom
>
> [1] That look a bit like:
>
> int spawnv(const char *path, char *const argv[])
> {
>     int pid = fork() ;
>     if (pid == 0) execv(path, argv) ;
>     else return pid ;
> }
>

read the following man pages:
pipe(2)
dup2(3c)

Then you should be able to solve this problem.

HTH,
Tom

[ Post a follow-up to this message ]

Tom Anderson wrote:

> I'm writing a program which needs to launch two other programs, which
> will then communicate via a pipe - one reads a file, translates it, and
> writes to standard output, and the other reads standard output,
> translates it, and writes to a file.
>
> If i was writing a shell script, i'd say:
>
> program1 inputfile | program2 outputfile
>
> How do i do this from a real programming language? I'm using python, but
> i think an approach that would work in C is usable here.
>
> I've come up with two options so far.
>
> The first is to use system(), passing in a string containing the command
> as i'd write it in a shell script. This is simple and easy, but it feels
> icky; i have to munge strings to build the command, it's vulnerable to
> bugs related to escaping the command string, and it's a potential
> security hole (although i won't be passing in arbitrary strings from
> untrusted users or anything). Also, i was under the impression that
> system() worked by spawning a subshell, although ps shows no sign of
> such a thing when i do it; is that right?
>
> The second is to use fork/exec, which in Python are wrapped up into a
> 'spawn' function (well, a family of functions [1]), to run the program
s
> separately, and have them communicate via a named pipe. For this to
> work, both programs have to be able to use a file instead of standard
> in/out, but in this case, they can, so that's okay. I then do something
> like:
>
> char *pipename = inputfile + ".pipe" ; /* not valid C, but YKWIM */
> mkfifo(pipename) ;
> char *argv1[] = {"program1", "inputfile", pipename} ;
> int pid1 = spawnv("/path/to/program1", argv1) ;
> char *argv2[] = {"program1", pipename, "outputfile"} ;
> int pid2 = spawnv("/path/to/program2", argv2) ;
> int status ;
> waitpid(pid1, &status, 0) ;
> waitpid(pid2, &status, 0) ;
> remove(pipename) ;
>
> Only with some error checking. Now, i'm happier about the spawning than
> the system call, since there's no escaping and no potential for
> injection to worry about, but the stuff with the named pipe is terrible
> - i'm polluting the globally visible filesystem with my internal
> workings, and exposing those workings to random other programs, i'm
> risking leaving garbage pipes around if i crash, i have to worry about
> my pipe name conflicting with an existing file, etc.

Why not use an anonymous pipe?

man 2 pipe

[ Post a follow-up to this message ]

Tom Anderson wrote:
>
> I'm writing a program which needs to launch two other programs, which
> will then communicate via a pipe - one reads a file, translates it, and
> writes to standard output, and the other reads standard output,
> translates it, and writes to a file.
>
> If i was writing a shell script, i'd say:
>
> program1 inputfile | program2 outputfile
>
> How do i do this from a real programming language?

#!/usr/bin/perl

open IN, '-|', 'program1', 'inputfile'
or die "Cannot open pipe from 'program1' $!";

open OUT, '|-', 'program2', 'outputfile'
or die "Cannot open pipe to 'program2' $!";

print OUT while <IN>;

close OUT or warn $! ? "Error closing 'program2' pipe: $!"
: "Exit status $? from 'program2'";

close IN or warn $! ? "Error closing 'program1' pipe: $!"
: "Exit status $? from 'program1'";

__END__



John
--
use Perl;
program
fulfillment

[ Post a follow-up to this message ]

Tom Anderson wrote:
>
> I'm writing a program which needs to launch two other programs, which
> will then communicate via a pipe - one reads a file, translates it, and
> writes to standard output, and the other reads standard output,
> translates it, and writes to a file.
>
> If i was writing a shell script, i'd say:
>
> program1 inputfile | program2 outputfile
>
> How do i do this from a real programming language?

Ummm, the shell is a real programming language.
Perhaps you mean a language in which the shell (and Perl) can be written?

--
Chris.




#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>

//	gcc -std=c99 -Wall -pedantic -Werror -o pipes pipes.c

int main(int argc, char *argv[])
{
int p[2];

if(pipe(p) < 0) {
perror("pipe()");
exit(1);
}

/*  We will have	who (the child process) | sort (the parent)

The child will write its stdout to p[1] and the parent will
read its stdin from p[0].
*/
switch (fork()) {
case -1 :
perror("fork()");
exit(1);
break;
case 0:					/* child process */
close(0);
dup2(p[1], 1);
close(p[0]);
close(p[1]);
execl("/usr/bin/who", "who", NULL);
perror("execl(who)");
exit(1);
break;
default:					/* original, parent process */
dup2(p[0], 0);
close(p[0]);
close(p[1]);
execl("/bin/sort", "sort", NULL);
perror("execl(sort)");
exit(1);
break;
}
return 0;
}

[ Post a follow-up to this message ]

On Fri, 9 Jun 2006, John W. Krahn wrote:

> Tom Anderson wrote:
> 
>
> #!/usr/bin/perl

Resisting obvious and entirely unjustified joke .

> print OUT while <IN>;

I was hoping to avoid doing that.

I should probably consider this as a portable fallback, though; a
pipe/dup2 approach will work on unix, but i have no idea about windows,
whereas this should. Not that i'm planning to run my program on windows,
but, you know, it's nice to have the option.

tom

--
Osteoclasts = monsters from the DEEP -- Andrew

[ Post a follow-up to this message ]

On Fri, 9 Jun 2006, Thomas Maier-Komor wrote:

> Tom Anderson wrote:
> 
>
> read the following man pages:
> pipe(2)
> dup2(3c)
>
> Then you should be able to solve this problem.

Aaah, i think i get it. And, having read Chris's very helpful post, yes, i
do. I didn't know about pipe(); that makes everything really simple.

Thanks to everyone for their replies.

tom

--
Osteoclasts = monsters from the DEEP -- Andrew

[ Post a follow-up to this message ]