Hey,

I need to have 4 processes: Producer, Filter1, Filter2 and Consumer.
Producer needs to read data from an input file line by line and pass
each line via a pipe to Filter1 who will replace all " " w/ "*". Once
Filter1 is done it will pass the newly modified line to Filter2.
Filter2 converts all lowercase letters to uppercases. Once Filter2 is
done it will pass the newly modified line to Consumer who simply
writes the line to an output file. This is repeated until the input
file is completely scanned.

I can get this to work w/ one line (no loops), so the first line of
text is completely converted but how do I get a while() loop to work
with this pipe synchronization so that the entire input file is
scanned?

I'd appreciate it if someone can help.

[ Post a follow-up to this message ]

In article <1174862813.738202.285660@y80g2000hsf.googlegroups.com>,
"Arturo" <arturo.rotondo@gmail.com> wrote:

> Hey,
>
> I need to have 4 processes: Producer, Filter1, Filter2 and Consumer.
> Producer needs to read data from an input file line by line and pass
> each line via a pipe to Filter1 who will replace all " " w/ "*". Once
> Filter1 is done it will pass the newly modified line to Filter2.
> Filter2 converts all lowercase letters to uppercases. Once Filter2 is
> done it will pass the newly modified line to Consumer who simply
> writes the line to an output file. This is repeated until the input
> file is completely scanned.
>
> I can get this to work w/ one line (no loops), so the first line of
> text is completely converted but how do I get a while() loop to work
> with this pipe synchronization so that the entire input file is
> scanned?
>
> I'd appreciate it if someone can help.

I'm not sure what the problem is.  As long as all the processes read
their input line by line until EOF, you should be able to do:

Producer | Filter1 | Filter2 | Consumer

Are you saying that some of these processes only process one line and
then exit?  You can change that process to:

{ while read line; do echo "$line" | <process>; done; }

--
Barry Margolin, barmar@alum.mit.edu
Arlington, MA
*** PLEASE post questions in newsgroups, not directly to me ***
*** PLEASE don't copy me on replies, I'll read them in the group ***

[ Post a follow-up to this message ]

On Mar 25, 8:45 pm, Barry Margolin <bar...@alum.mit.edu> wrote:
> In article <1174862813.738202.285...@y80g2000hsf.googlegroups.com>,
>
>
>
>  "Arturo" <arturo.roto...@gmail.com> wrote: 
> 
> 
> 
>
> I'm not sure what the problem is.  As long as all the processes read
> their input line by line until EOF, you should be able to do:
>
> Producer | Filter1 | Filter2 | Consumer
>
> Are you saying that some of these processes only process one line and
> then exit?  You can change that process to:
>
> { while read line; do echo "$line" | <process>; done; }
>
> --
> Barry Margolin, bar...@alum.mit.edu
> Arlington, MA
> *** PLEASE post questions in newsgroups, not directly to me ***
> *** PLEASE don't copy me on replies, I'll read them in the group ***


I should have mentioned I'm using C/C++ not shell script. OK so I'll
post the code so you can see how I'm approaching this. I'm a beginner
at this, the code may look a little ugly, but it's just a simple chain
where Producer forks to Filter1 whom forks to Filter2 whom forks to
Consumer.

The problem is ... The while loop seems to only execute once. I don't
understand why, I think it may have to do with close() cause I've
reading about how it deallocates the file descriptor. If this is the
case then how can I solve this?

#include <sys/types.h>
#include <sys/wait.h>
#include <unistd.h>
#include <iostream>
#include <string.h>
#include <ctype.h>
using namespace std;

int main ( int argc , char* argv[] )
{
/* status variables are used with _exit() and wait(), allowing parent
processes
to wait() for its child process to signal it; using "_exit()", when
the
child process is done. There are three status variables one for each
child process */

int status1 , status2 , status3;

/* pipe() creates a pair of file descriptors, pointing to a pipe
inode,
and places them in the array pointed to by fd arrays. fd[0] is for
reading, fd[1] is for writing. */

int fd1[2] , fd2[2] , fd3[2];

/* data from text will be read and written to/from pipelines and
files
using the following char pointers, and the maximum number of
characters is set to 500 */

const int maxChar = 500;
char buffer1[maxChar];
char buffer2[maxChar];

/* The four processes used to synchronize the file manipulation
process */

pid_t Filter1;
pid_t Filter2;
pid_t Consumer;

/* Producer opens an input file */

FILE* filePtr = fopen( "test.txt" , "r" );
if( filePtr == NULL ) perror( "Error opening file");

/* create pipe used for Producer | Filter1 data transfer */

if(	pipe( fd1 ) ) perror( "Error creating pipeline" );

while( !feof( filePtr ) )
{
/* Producer forks to Filter1 */

switch( Filter1 = vfork() )
{
/* Parent Process: Producer */

case -1 :

cout << "ERROR FORKING" << endl;
exit( -1 );

/* Child Process: Filter1 */

case 0 :

/* receive data sent by Producer */

close( fd1[1] );
read( fd1[0], buffer2, maxChar );

/* replace all " " w/ "*" */

for( int i = 0 ; i < strlen( buffer2 ) ; i++ ){
if( buffer2[i] == ' ' )
buffer2[i] = '*';}

/* create pipe used for Filter1 | Filter2 data transfer */

if(	pipe( fd2 ) ) perror( "Error creating pipeline" );

/* Filter1 forks to Filter2 */

switch( Filter2 = vfork() )
{
/* Parent Process: Filter1 */

case -1 :

cout << "ERROR FORKING" << endl;
exit( -1 );

/* Child Process: Filter2 */

case 0 :

/* receive data sent by Filter1 */

close( fd2[1] );
read( fd2[0], buffer1 , maxChar );

/* convert all lowercases to uppercases */

for( int i = 0 ; i < strlen( buffer1 ) ; i++ ){
if( isalpha( buffer1[i] ) ){
if( islower( buffer1[i] ) )
buffer1[i] = toupper( buffer1[i] );}}

/* create pipe used for Filter2 | Consumer data transfer */

if(	pipe( fd3 ) ) perror( "Error creating pipeline" );

/* Filter2 forks to Consumer */

switch( Consumer = vfork() )
{
/* Parent Process: Filter2 */

case -1 :

cout << "ERROR FORKING" << endl;
exit( -1 );

/* Child Process: Consumer */

case 0 :

/* receive data sent by Filter2 */
close( fd3[1] );
read( fd3[0], buffer2 , maxChar );

/* store data received data into an output file */
FILE *outputFile = fopen( "modifiedTest.txt" , "a" );
fputs( buffer2 , outputFile );
fclose( outputFile );
_exit( status3 );

/* Parent Process: Filter2 */

default :

/* send data to Consumer via fd3 */

close( fd3[0] );
write( fd3[1] , buffer1 , strlen( buffer1 ) + 1 );

/* wait for Consumer */

wait( &status3 );
}

_exit( status2 );

/* Parent Process: Filter1 */

default :

/* send data to Filter2 */

close( fd2[0] );
write( fd2[1] , buffer2 , strlen( buffer2 ) + 1 );

/* wait for Filter2 */

wait( &status2 );
}

_exit( status1 );

/* Parent Process: Producer */

default :

/* send data to Filter1  */

close( fd1[0] );
fgets( buffer1 , maxChar , filePtr );
write( fd1[1] , buffer1 , strlen( buffer1 ) + 1 );

/* wait for Filter1 */

wait( &status1 );
}

fclose( filePtr );

}

return 0;

}

[ Post a follow-up to this message ]

On 25 Mar 2007 19:33:32 -0700 "Arturo" <arturo.rotondo@gmail.com> wrote:
> 	while( !feof( filePtr ) )

First of all, look at <http://c-faq.com/stdio/feof.html>.  I don't know
if that's your problem but it's someplace to start.

-frank

[ Post a follow-up to this message ]

On Mar 25, 11:21 pm, Frank Cusack <fcus...@fcusack.com> wrote:
> On 25 Mar 2007 19:33:32 -0700 "Arturo" <arturo.roto...@gmail.com> wrote:
> 
>
> First of all, look at <http://c-faq.com/stdio/feof.html>.  I don't know
> if that's your problem but it's someplace to start.
>
> -frank

Thanks Frank, but same problem I just tried it. At least I reduced a
line of code.

[ Post a follow-up to this message ]

On Sun, 25 Mar 2007 19:33:32 -0700, Arturo wrote:

> On Mar 25, 8:45 pm, Barry Margolin <bar...@alum.mit.edu> wrote: 
>
>
> I should have mentioned I'm using C/C++ not shell script. OK so I'll
> post the code so you can see how I'm approaching this. I'm a beginner at
> this, the code may look a little ugly, but it's just a simple chain
> where Producer forks to Filter1 whom forks to Filter2 whom forks to
> Consumer.
>
> The problem is ... The while loop seems to only execute once. I don't
> understand why, I think it may have to do with close() cause I've
> reading about how it deallocates the file descriptor. If this is the
> case then how can I solve this?

I can see three major problems:

> 	while( !feof( filePtr ) )
> 	{
> 		/* Producer forks to Filter1 */
>
> 		switch( Filter1 = vfork() )

Don't use vfork().

> 				/* Filter1 forks to Filter2 */
>
> 				switch( Filter2 = vfork() )

Don't use vfork().

> 						switch( Consumer = vfork
() )
> 						{

Don't use vfork().

> 				fgets( buffer1 , maxChar , filePtr );
>                               write( fd1[1] , buffer1 ,
> 				strlen( buffer1 ) + 1 );

You probably don't want the +1, and you aren't checking for errors on
most of the IO calls.
Also you really want to only do the fork() calls once, and then loop in
the correct places ... instead of looping around the entire thing and
creating three processes for each line.
Also create some functions.

--
James Antill -- james@and.org
http://www.and.org/and-httpd/ -- $2,000 security guarantee
http://www.and.org/vstr/

[ Post a follow-up to this message ]

On Mar 26, 10:20 am, James Antill <james-netn...@and.org> wrote:
> On Sun, 25 Mar 2007 19:33:32 -0700, Arturo wrote: 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
>
>  I can see three major problems:
> 
> 
>
>  Don't use vfork().
> 
> 
>
>  Don't use vfork().
>
>
> 
> () ) 
>
>  Don't use vfork().
> 
>
>  You probably don't want the +1, and you aren't checking for errors on
> most of the IO calls.
>  Also you really want to only do the fork() calls once, and then loop in
> the correct places ... instead of looping around the entire thing and
> creating three processes for each line.
>  Also create some functions.
>
> --
> James Antill -- j...@and.orghttp://www.and.org/and-httpd/-- $2,000 security guaran
teehttp://www.and.org/vstr/

I am trying to accomplish that but I can't. From what I understand,
after I fork() to a child process the parent will wait() for the child
process to complete, and _exit() is used by the child process to
signal the parent process that it has completed its task. This is
simple and I get it to work with one line from the input file. So in
order to get all the lines from the input file processed, I need a
loop in the Producer code that will read the file line by line. After
each line is read the producer fork() to its child process and wait()
for the child process to finish. The Producer should then resume, it
loops and gets the next line, but this time around the close() seems
to not wanna work properly.

[ Post a follow-up to this message ]

Arturo <arturo.rotondo@gmail.com> wrote:
> I should have mentioned I'm using C/C++

Please devide if you use C or C++, they are different languages.

> not shell script. OK so I'll
> post the code so you can see how I'm approaching this. I'm a beginner
> at this, the code may look a little ugly, but it's just a simple chain
> where Producer forks to Filter1 whom forks to Filter2 whom forks to
> Consumer.

> The problem is ... The while loop seems to only execute once. I don't
> understand why, I think it may have to do with close() cause I've
> reading about how it deallocates the file descriptor. If this is the
> case then how can I solve this?

There are some many things broken with your program that I don't see
any way to correct them. So instead here's a probably incomplete list:

0) Your main() isn't 'int main(void)' even though you never use
argc and argv.
1) You continue even if you can't open the input file (note that
perror() doesn't calll exit()).
2) You continue even if you are unable to create a pipe.
3) You don't seem to understand what feof() is good for (and that
you don't need it at all if you use fgets(), which returns NULL
when EOF is reached).
4) You use vfork().
5) You create a set new processes plus new pipes in each child for
each line you read in while a single set would do perfectly well
(or probaly better!) for all the lines if each child would loop
until it doesn't get any more input.
6) You're writing spaghetti code instead of a set of functions
that deal with the sub-tasks.
7) You don't check and retain the return value of a read() call
but instead seem to think that read() would treat a '\0'
character similarly to the way fgets() treats a '\n'. But that
isn't the case. To read() all character are just the same. If
you use write() and read() you usually need some kind of "pro-
tocol" that by which the sender informs the receiver of how
many bytes to expect, e.g. by first sending the number of bytes
and only then the data.
8) You seem to believe tht toupper() only would work with alphabe-
tical, lower-case characters.
9) You don't test if fopen() succeeds for the output file
10) You're closing the read end of the first pipe each time you
read a new line and then you start a new process for the next
line which is going to try to read from this already closed
pipe descriptor. That never will work, once it's closed it
it's closed and won't suddenly be opened again just because
you try to write to it. So already when you start dealing
with the second line the top-most parent process will die
on a SIGPIPE signal the moment it's going to try to write to
the pipe since it's child only has a closed pipe handle and
so there's actually no receiver.

Do yourself a favor and start with a simpler task, e.g. reading
from a file in the parent process, passing the data to the child
process via the pipe and there just write the data to another file.
The basic problem is the same but you at least will have a fighting
chance to get it right after some time (and you can simply test if
the output is ok by doing a 'diff' on the input and output file).
To get to that point read the man pages of the functions you use
very carefully, especially those for the low level I/O functions
like read().

Especially note that the number of bytes to read you pass to
read() is an upper limit only and that you never can assume that
you will get that many - read may return anything between zero bytes
and the upper limit you called it with (or even -1 if an error
occurred). For this reason read() is normally always embedded in a
loop to be able to restart read() if less bytes arrived than you
asked for. For that reason it probably will be simpler if you
restrict yourself to the use of the higher level I/O function like
fgets and fputs() for a start (there's that nice function called
fdopen() that gives you a FILE pointer when you pass it a file or
pipe handle as its argument).

And, BTW, if you post code it would be really helpful if you would
keep the line length down to say 72 characters and replace tabs by
spaces.
Regards, Jens
--
\   Jens Thoms Toerring  ___      jt@toerring.de
\__________________________      http://toerring.de

[ Post a follow-up to this message ]

On Mar 26, 7:41 pm, j...@toerring.de (Jens Thoms Toerring) wrote:
> Arturo <arturo.roto...@gmail.com> wrote: 
>
> Please devide if you use C or C++, they are different languages.
> 
>
> There are some many things broken with your program that I don't see
> any way to correct them. So instead here's a probably incomplete list:
>
> 0) Your main() isn't 'int main(void)' even though you never use
>    argc and argv.
> 1) You continue even if you can't open the input file (note that
>    perror() doesn't calll exit()).
> 2) You continue even if you are unable to create a pipe.
> 3) You don't seem to understand what feof() is good for (and that
>    you don't need it at all if you use fgets(), which returns NULL
>    when EOF is reached).
> 4) You use vfork().
> 5) You create a set new processes plus new pipes in each child for
>    each line you read in while a single set would do perfectly well
>    (or probaly better!) for all the lines if each child would loop
>    until it doesn't get any more input.
> 6) You're writing spaghetti code instead of a set of functions
>    that deal with the sub-tasks.
> 7) You don't check and retain the return value of a read() call
>    but instead seem to think that read() would treat a '\0'
>    character similarly to the way fgets() treats a '\n'. But that
>    isn't the case. To read() all character are just the same. If
>    you use write() and read() you usually need some kind of "pro-
>    tocol" that by which the sender informs the receiver of how
>    many bytes to expect, e.g. by first sending the number of bytes
>    and only then the data.
> 8) You seem to believe tht toupper() only would work with alphabe-
>    tical, lower-case characters.
> 9) You don't test if fopen() succeeds for the output file
> 10) You're closing the read end of the first pipe each time you
>     read a new line and then you start a newprocessfor the next
>     line which is going to try to read from this already closed
>     pipe descriptor. That never will work, once it's closed it
>     it's closed and won't suddenly be opened again just because
>     you try to write to it. So already when you start dealing
>     with the second line the top-most parentprocesswill die
>     on a SIGPIPE signal the moment it's going to try to write to
>     the pipe since it's child only has a closed pipe handle and
>     so there's actually no receiver.
>
> Do yourself a favor and start with a simpler task, e.g. reading
> from a file in the parentprocess, passing the data to the childprocessvia 
the pipe and there just write the data to another file.
> The basic problem is the same but you at least will have a fighting
> chance to get it right after some time (and you can simply test if
> the output is ok by doing a 'diff' on the input and output file).
> To get to that point read the man pages of the functions you use
> very carefully, especially those for the low level I/O functions
> like read().
>
> Especially note that the number of bytes to read you pass to
> read() is an upper limit only and that you never can assume that
> you will get that many - read may return anything between zero bytes
> and the upper limit you called it with (or even -1 if an error
> occurred). For this reason read() is normally always embedded in a
> loop to be able to restart read() if less bytes arrived than you
> asked for. For that reason it probably will be simpler if you
> restrict yourself to the use of the higher level I/O function like
> fgets and fputs() for a start (there's that nice function called
> fdopen() that gives you a FILE pointer when you pass it a file or
> pipe handle as its argument).
>
> And, BTW, if you post code it would be really helpful if you would
> keep the line length down to say 72 characters and replace tabs by
> spaces.
>                              Regards, Jens
> --
>   \   Jens Thoms Toerring  ___      j...@toerring.de
>    \__________________________      http://toerring.de

Thanks to all, I got it to work finally.

Jens that was pretty harsh, but I took your advice about starting with
simply reading from a file and then writing the content to another
file,
and after getting that to work, I built on it and finally got it all
to work together.

[ Post a follow-up to this message ]

Arturo <arturo.rotondo@gmail.com> wrote:
> Jens that was pretty harsh,

Sorry, it wasn't my intention to offend you.

> but I took your advice about starting with
> simply reading from a file and then writing the content to another
> file,

Thanks for not getting too angry with me for doing that;-)

> and after getting that to work, I built on it and finally got it all
> to work together.

Congratulations!
Regards, Jens
--
\   Jens Thoms Toerring  ___      jt@toerring.de
\__________________________      http://toerring.de

[ Post a follow-up to this message ]