I have a pair of programs that communicate using standard pipes
(i.e., pipe(2)).  The process with the write-end of the pipe
downloads data from the net (or a local file), and dumps it into
the pipe.  The read-end gets data from this pipe, never caring
where it came from, and feeds it into an MPEG decoder.

It works great until I want to seek in the stream.  Here's an
outline of how I do seeking:

(1) read-end (from now on referred to as "decoder") signals the
write-end (from now on referred to as "streamer"), telling
it the position it would like to seek to;  immediately after
doing this, the decoder sleeps, waiting for the streamer to
wake it up.

(2) the streamer, having just been signaled that the decoder wants
to seek, drops whatever it was doing and seeks to the requested
position;  once it's done, the decoder is woken up and this
time the streamer goes to sleep.

(3) the decoder immediately read()s the pipe until it returns
EAGAIN, so that data prior to the seek should be discarded;
once it's done, the streamer is woken up again.

After these three steps are done, the programs return to their
normal life where the streamer write()s into the pipe, and the
decoder read()s it.  The only problem is that it doesn't work.

The first bytes read by the decoder is NOT the first bytes written
by the streamer.  The first 8192 bytes of data that appears
in the read-end is data that was written PRIOR to the seek.

I can't understand how this is possible -- isn't the pipe empty
when read() returns EAGAIN?  If not, how can I fix this while
still using pipe(2)?  I guess I could use named FIFOs instead,
because then I could close() and open() the pipe as a part of
the three-steps shown above.  It just feels a bit cleaner to use
regular pipes, and I would like to understand why it fails.

Finally, here's some info on my system: Linux 2.4.22 (x86),
glibc 2.3.2, and gcc 3.3.2.

--
Haakon

[ Post a follow-up to this message ]

Haakon Riiser <hakonrk@fys.uio.no> wrote:
# I have a pair of programs that communicate using standard pipes
# (i.e., pipe(2)).  The process with the write-end of the pipe
# downloads data from the net (or a local file), and dumps it into
# the pipe.  The read-end gets data from this pipe, never caring
# where it came from, and feeds it into an MPEG decoder.

I suppose if you flagellate your box and yourself long enough, you can get
pipes to do what you want. Or else you can try a more appropriate technique.
With IPC messages you have more direct control over what is in the queue,
you can fake up asynchronous signals, and you can put your process to sleep
on queue empty. You can set up a shared memory for the data transfer and use
the queue to transfer available buffer addresses and control messages.
You can also use semaphores to manage shared buffers.

--
Derk Gwen http://derkgwen.250free.com/html/index.html
TEMPORARILY CLOSED
BE OPENED AFTER FIRST PERIOD

[ Post a follow-up to this message ]

[Derk Gwen]
quote:

> Haakon Riiser <hakonrk@fys.uio.no> wrote:
> # I have a pair of programs that communicate using standard pipes
> # (i.e., pipe(2)).  The process with the write-end of the pipe
> # downloads data from the net (or a local file), and dumps it into
> # the pipe.  The read-end gets data from this pipe, never caring
> # where it came from, and feeds it into an MPEG decoder.
>
> I suppose if you flagellate your box and yourself long enough, you can get
> pipes to do what you want. Or else you can try a more appropriate techniqu
e.
> With IPC messages you have more direct control over what is in the queue,
> you can fake up asynchronous signals, and you can put your process to slee
p
> on queue empty. You can set up a shared memory for the data transfer and u
se
> the queue to transfer available buffer addresses and control messages.
> You can also use semaphores to manage shared buffers.


Yes, I know all about the alternatives, but if there were an
easy way to discard buffered data from pipes, they would give
me everything I need right out of the box.  I'd like to avoid
reinventing pipes using other forms of IPC, if at all possible.
(Actually, I have already done that once, but that implementation
was designed for IPC between threads, so it won't work here.
And it wasn't exactly painless to write either -- there's little I
hate more than race conditions.)

Anyway, I'm fairly sure that named pipes (using close/open to
flush old data) would work but, as I said, I am curious as to
_why_ my current solution doesn't work.  How come there's still
leftovers in the pipe after read() returns EAGAIN?

--
Haakon

[ Post a follow-up to this message ]

* Haakon Riiser <hakonrk@fys.uio.no>
| The first bytes read by the decoder is NOT the first bytes written
| by the streamer.  The first 8192 bytes of data that appears in the
| read-end is data that was written PRIOR to the seek.

Maybe fsync() et al. or fcntl() with the appropriate flags might be of
help here.  They are described in the context of regular disc files,
so I don't know whether they apply to pipe()s, too.

R'

[ Post a follow-up to this message ]

[Ralf Fassel]
quote:

> * Haakon Riiser <hakonrk@fys.uio.no> 
[QUOTE]
> Maybe fsync() et al. or fcntl() with the appropriate flags might
> be of help here.  They are described in the context of regular
> disc files, so I don't know whether they apply to pipe()s, too.


I tried fsync(), but it returned EINVAL (Invalid argument).
As for fcntl(), I don't know.  I skimmed the manual, and only
O_DIRECT sounds like something that might work.  Unfortunately,
it also says there might be a performance penalty with this flag,
so I don't think I would use it even if it works.

But thanks for your suggestions!

--
Haakon

[ Post a follow-up to this message ]

# Yes, I know all about the alternatives, but if there were an
# easy way to discard buffered data from pipes, they would give
# me everything I need right out of the box.  I'd like to avoid
# reinventing pipes using other forms of IPC, if at all possible.

The problem is you're not reinventing pipes with other forms of IPC;
rather you're reinventing other forms of IPC with pipes.

--
Derk Gwen http://derkgwen.250free.com/html/index.html
Why are we here?
whrp

[ Post a follow-up to this message ]

[Derk Gwen]
quote:

> # Yes, I know all about the alternatives, but if there were an
> # easy way to discard buffered data from pipes, they would give
> # me everything I need right out of the box.  I'd like to avoid
> # reinventing pipes using other forms of IPC, if at all possible.
>
> The problem is you're not reinventing pipes with other forms of IPC;
> rather you're reinventing other forms of IPC with pipes.


Doesn't matter now anyway.  Just tried named pipes, and the same
thing happened, even after closing and reopening -- unbelievable!
So now I don't even have a choice, I _have_ to use shared memory
and semaphores. :-(

--
Haakon

[ Post a follow-up to this message ]

Haakon Riiser writes:
quote:

> It works great until I want to seek in the stream.  Here's an
> outline of how I do seeking:
>
>  (1) read-end (from now on referred to as "decoder") signals the
>      write-end (from now on referred to as "streamer"), telling
>      it the position it would like to seek to;  immediately after
>      doing this, the decoder sleeps, waiting for the streamer to
>      wake it up.
>
>  (2) the streamer, having just been signaled that the decoder wants
>      to seek, drops whatever it was doing and seeks to the requested
>      position;  once it's done, the decoder is woken up and this
>      time the streamer goes to sleep.
>
>  (3) the decoder immediately read()s the pipe until it returns
>      EAGAIN, so that data prior to the seek should be discarded;
>      once it's done, the streamer is woken up again.
>
> After these three steps are done, the programs return to their
> normal life where the streamer write()s into the pipe, and the
> decoder read()s it.  The only problem is that it doesn't work.


The problem seemed strange to me, so I did some experiments.
Bottom line is: my implementation works, or it looks like this. The code
is far from clean (too much PERL recently), and usage of signals is
awful (should have used signalling pipe or smth like this), but it does
the thing, and effectively empties the pipe.


--
Alexey 'Kaa the Snake' Kiritchun
mailto:kaa AT nightmail.ru

[ Post a follow-up to this message ]

[Alexey A. Kiritchun]
quote:

> The problem seemed strange to me, so I did some experiments.
> Bottom line is: my implementation works, or it looks like this. The code
> is far from clean (too much PERL recently), and usage of signals is
> awful (should have used signalling pipe or smth like this), but it does
> the thing, and effectively empties the pipe.


I just created a simple test program myself, using almost exactly
the same technique as the real program, and it too emptied the pipe
correctly.  This is starting to drive me crazy -- I'll investigate
it further and post the results here.  (If I learn something. :-)

--
Haakon

[ Post a follow-up to this message ]

I finally found out what causes this problem -- signals.
It's probably not a good idea to use system calls from inside a
signal handler. :-) Here's the test program that reproduces it:

http://folk.uio.no/hakonrk/pipes.c

Compile two versions, one with signals and one without:

gcc pipes.c -o sig
gcc -DNO_SIGNAL pipes.c -o nosig

Find a reasonably big file (around 1 MB is good; I hardcoded in
some big constants for SEEK_FROM and SEEK_TO, and I didn't bother
to see how small I could make them and still reproduce this).
Then do the following:

./sig < BIG-INPUT-FILE
./nosig < BIG-INPUT-FILE

It should print something like this:

PARENT: first bytes after seek: 7b7981adab945313a8f8d5f9ef62b353
CHILD: first after seek: 7b7981adab945313a8f8d5f9ef62b353

Obviously, the 16 bytes listed should be the same for both the
child and the parent.  When compiled with NO_SIGNALS defined,
it always works.  If compiled without it, it's a race condition.
(Never works on my computer for some reason, but I tested it on
some others, and it occasionally works on them.)

--
Haakon

[ Post a follow-up to this message ]