Unix Programming - newbie question about fork/pipe/stdin/stdout

Author

newbie question about fork/pipe/stdin/stdout

vladimir.plotnikov@gmail.com

2007-10-26, 1:29 pm

Hello!

I'm writing small C++ under Linux application with will be transfer
email from some IMAP server and put it to local database.
I have one problem:
I need to pass message to anti virus filter and spam filer.
When mail exceeded some limit, I my program is stopped on "waitpid()"
and wait when spamc script ends. but this script never ends. From
other side, another filter (anti virus) working fine on same code.
There is the part of code:
-----------------------------------------------------------------------------------------------------
pipetest = pipe(pipeStdin);
pipe(pipeStdout);
fcntl(pipeStdout[1], F_SETFD, fcntl(pipeStdout[1], F_GETFD) & ~1);
if(pipetest != 0) {
std::cerr << " unable top open pipe!!! exitting "<<std::endl;
exit(20);
}
{
if( (pid = fork()) == 0) { // child
DEBUG ("Settign UID/GID for child process to
"<<uid<<"/"<<gid);
if (setgid(gid)!=0) {
std::cerr << "Unable to set GID "<<gid<<std::endl;
throw;
}
if (setuid(uid)!=0) {
std::cerr << "Unable to set UID "<<uid<<std::endl;
throw;
}

dup2(pipeStdin[0], STDIN_FILENO);
dup2(pipeStdout[1], STDOUT_FILENO);
close(pipeStdin[1]);
close(pipeStdin[0]);
close(pipeStdout[0]);
close(pipeStdout[1]);
int rc;
rc=execlp(exec.c_str(),exec.c_str(),NULL);
close(0);
close(1);
_exit (rc);
}
close(pipeStdout[1]);

int output;
char c;
DEBUG("Send "<<msg.size()<<" bytes");
FILE* stream;
close (pipeStdin[0]);
stream = fdopen (pipeStdin[1], "w");
// for (unsigned int i = 0; i<msg.size(); i++) {
// fputc(msg[i],stream);
// }
fprintf( stream, msg.c_str());
fflush (stream);
fclose (stream);
close(pipeStdin[1]);
int status;
DEBUG ("Waiting for PID");
waitpid (pid, &status, 0);
DEBUG ("Waiting END");
stream = fdopen (pipeStdout[0], "r");
std::string out;
while(!feof(stream)) {
if (fread(&c,1, 1,stream)==1)
// std::cerr << c;
out = out+ c;
std::cout <<c;
}
close (pipeStdout[0]);
close (pipeStdout[1]);
std::cout << "child end with " << WEXITSTATUS(status) <<"
"<<errno<< std::endl;
---------------------------------------------------------------------------
where I wrong?
User (uid/gid variables) haves next limits:
#limit
cputime unlimited
filesize unlimited
datasize unlimited
stacksize 8192 kbytes
coredumpsize 0 kbytes
memoryuse unlimited
vmemoryuse unlimited
descriptors 1024
memorylocked 32 kbytes
maxproc 8190

Thank you. Vladimir.

vladimir.plotnikov@gmail.com

2007-10-26, 1:29 pm

I forgot one detail:
118kb message cannot be passed to filter.
13kb - passed normal.

Alex Fraser

2007-10-26, 7:20 pm

<vladimir.plotnikov@gmail.com> wrote in message
news:1193405143.360453.36370@57g2000hsv.googlegroups.com...
> I'm writing small C++ under Linux application with will be transfer
> email from some IMAP server and put it to local database.
> I have one problem:
> I need to pass message to anti virus filter and spam filer.
> When mail exceeded some limit, I my program is stopped on "waitpid()"
> and wait when spamc script ends. but this script never ends. From
> other side, another filter (anti virus) working fine on same code.

If the child consumes all input before producing any output, the parent
should write all the input, read the output until EOF, and only then call
wait(). Using the code you provided, if there is "too much" output, the
child will block in write() before it can exit, and the parent is waiting
for it to exit before calling read(): this is a deadlock.

If the child may produce output before consuming all input, you will have to
do something different (more complicated). You could use non-blocking IO and
select()/poll() or equivalent in the parent to move data between the
processes whenever possible (again, calling wait() only after you have read
EOF from the child). Or you could use a third process with some more
"plumbing" to create something like a shell pipeline. That is, after
creating the first child process (in which you exec), fork() again in the
parent to create another child which will read the output from the first
child.

HTH,
Alex

vladimir.plotnikov@gmail.com

2007-10-26, 7:20 pm

Alex,
Thank you,
put "wait" after read partially help me.
so, one "big" message passed to child and comes back.
But in some cases (toooooooo biiiiiiiig messages) child will output
any data to stdout. so, my program was locked on "while(!feof())".

vladimir
czech republic.

Alex Fraser

2007-10-27, 1:32 am

<vladimir.plotnikov@gmail.com> wrote in message
news:1193436864.842202.78810@o38g2000hse.googlegroups.com...
> Alex,
> Thank you,
> put "wait" after read partially help me.
> so, one "big" message passed to child and comes back.
> But in some cases (toooooooo biiiiiiiig messages) child will output
> any data to stdout. so, my program was locked on "while(!feof())".

I would expect the process to block inside fprintf(stream, msg.c_str())
(specifically, in a call to write()), if the child writes "too much" before
you get to reading it. This is another deadlock (both processes are trying
to write more before they read), which the second paragraph in my previous
reply addresses.

Alex

David Schwartz

2007-10-28, 7:32 am

On Oct 26, 3:14 pm, vladimir.plotni...@gmail.com wrote:

> put "wait" after read partially help me.
> so, one "big" message passed to child and comes back.
> But in some cases (toooooooo biiiiiiiig messages) child will output
> any data to stdout. so, my program was locked on "while(!feof())".

Your proxy should make forward progress any time it can. Waiting for
one thing to finish before starting another is a recipe for disaster
if something else waits for the second thing to finish before
finishing the first.

DS

vladimir.plotnikov@gmail.com

2007-10-29, 7:32 am

hello agin,
Thank you for replies ;-)

I changed my code to next:
.....
fcntl(pipeStdout[0], F_SETFD, fcntl(pipeStdout[0], F_GETFD) |
O_NONBLOCK);
fcntl(pipeStdin[1], F_SETFD, fcntl(pipeStdin[1], F_GETFD) |
O_NONBLOCK);
....
stream = fdopen (pipeStdin[1], "w");
streamIn = fdopen (pipeStdout[0], "r");
DEBUG ("output start");
for (unsigned int i = 0; i<msg.size();
i++) {
while(select(pipeStdout[0]+1,
&rfds, NULL, NULL, &tv)) {
DEBUG ("add input");
if (fread(&c,1,
1,streamIn)==1)
out = out+ c;
}

if(select(pipeStdin[1]+1,
NULL, &wfds, NULL, &tv)) {
if (!fputc(msg[i],stream))
{DEBUG("output failed"); break; }
DEBUG(i);
} else {
DEBUG ("send data
problem"); i--;
}
}
DEBUG("output end");
2nd "select" catch for "send data problem" when child cannot receive
more data.
In this time child should send me data back. but 1st "select" never
return value other than zero.
Where I wrong again?

Sorry for stupid questions, I never working with select/pipes before.

Alex Fraser

2007-10-30, 7:22 pm

<vladimir.plotnikov@gmail.com> wrote in message
news:1193649642.329918.170420@d55g2000hsg.googlegroups.com...
> hello agin,
> Thank you for replies ;-)
>
> I changed my code to next:
> ....
> fcntl(pipeStdout[0], F_SETFD, fcntl(pipeStdout[0], F_GETFD) |
> O_NONBLOCK);
> fcntl(pipeStdin[1], F_SETFD, fcntl(pipeStdin[1], F_GETFD) |
> O_NONBLOCK);

You must use F_GETFL/F_SETFL here.

> ....
> stream = fdopen (pipeStdin[1], "w");
> streamIn = fdopen (pipeStdout[0], "r");

You can't (reliably, at least) use stdio with descriptors set to
non-blocking mode; use read() and write().

[snip rest of code]
> Where I wrong again?

I would use a loop which does something like this (I've ignored error
conditions etc which obscure the basic logic):

1. Call select() to wait until it reports pipeStdout[0] is readable or
pipeStdin[1] is writeable. (The latter only if you still have data to
write to the child.)
2a. If pipeStdout[0] is readable, call read().
2b. If read() returns 0 (indicating EOF), exit the loop.
2c. Do whatever you need to with the data just read.
3a. If you have data to write to the child and pipeStdin[1] is writeable,
call write() to try to write some more data. For efficiency, try to
write as much data as you have available.
3b. If you succeed in writing the last byte, close pipeStdin[1] so the
child will see EOF, and make it so that in #1 you no longer check the
status of pipeStdin[1] (for obvious reasons).

After you exit the loop, call wait() to wait for the child process to exit
(if it hasn't already) and collect its status.

Alex