Hi,

I want to program some queue facilities for independent processes
communication. I think this is quite typical --- one message queue, a
read end and a write end, numerous reader processes and writer
processes adding/picking items.

My initial idea is to use fifo, since it's first in first out just
like queue, and it's named pipe since I have to deal with independent
processes. I've read chapters on FIFO of APUE, and I've coded locking
mechanics to ensure atomic read/write,  but there are still two
difficulties I don't know how to deal with (at least with fifo).

- The queue I want must be asynchronous, i.e. readers/writers come and
go, pick/add items to the queue and continute to do other thing,  but
fifo is synchronous by default. It's not difficult for the readers
side, they can open the fifo with the nonblocking arguments and poll;
but on the writers side, how can they submit item and leave? How about
if no one is listening on the reader side? One possiblity is to open
the write end with O_RDWR, but then the problem turns to be -- you
won't be able to close the opened fd (or allow some process to exit
early), otherwise the data you've written but not been read, may get
lost. This become unmanagble if the number of queues or add/pick
actions is unlimited.

- The size of fifo is limited by BUF_SIZE, though I'm not clear it is
the strict limitation of the whole fifo or each  opened fd connecte to
the fifo.

It seems that fifo may not be the right tool to implement such a
function, though I really like its simplicity. Maybe some persistent
storage like normal file is necessay, however I don't know how to use
a normal file as a queue (how to efficiently remove a header part).
I'd like to hear what is the standard way to handel such a
requirement. BTW, I can only use a subset of the classical POSIX
routine as those implemented by the OCaml Unix module [1].

Thanks!

[1] http://caml.inria.fr/pub/docs/manua...ibref/Unix.html

[ Post a follow-up to this message ]

code17@gmail.com wrote On 08/06/07 15:41,:
> Hi,
>
> I want to program some queue facilities for independent processes
> communication. I think this is quite typical --- one message queue, a
> read end and a write end, numerous reader processes and writer
> processes adding/picking items.

Have you considered using, well, message queues?
See "man msgctl", "man msgsnd", "man msgrcv" to learn
about System V queues; there are also POSIX queues but
I'm less familiar with them.

--
Eric.Sosman@sun.com

[ Post a follow-up to this message ]

Hi

On 8 6 ,   10 22 , Eric Sosman <Eric.Sos...@sun.com> wrote:
>     Have you considered using, well, message queues?
> See "man msgctl", "man msgsnd", "man msgrcv" to learn
> about System V queues; there are also POSIX queues but
> I'm less familiar with them.

Thanks for the suggestion.

I'm aware of the System V queues, however they are not present in the
Unix module of OCaml (a very conservative selection of Unix system
calls).  Instead of wrapping a FFI myself, I'd like to hear some other
work arounds.

Thanks anyway.

[ Post a follow-up to this message ]

code17@gmail.com wrote:

# I'm aware of the System V queues, however they are not present in the
# Unix module of OCaml (a very conservative selection of Unix system
# calls).  Instead of wrapping a FFI myself, I'd like to hear some other
# work arounds.

Bufferred but undeliverred data has to be stored somewhere. Pipes and
such do not have unlimitted bufferring because that would require the
kernel have unlimitted bufferring; I suppose it could do it somehow
but it seems like a big chunk of responsibility to drop on the kernel.

What you want is similar to email: asynchronous senders and receivers,
unknown amounts data bufferred indefinitely. I would suggest going
ahead and using an email-like system: If everything is on a single
machine, create a directory or file read/writable to all processes
and a method to add messages (as independent files in the directory
or as appended bytes in a file) and the extract and delete messages.
There are obvious variants of this such as using a database.

If you have multiple machines without some kind file sharing like
NFS, you can set up a message server on one machine that everyone
else connects to send and receive messages over the network. If
you store messages in database like MySQL, the database server
can act as the message repository and the message server.

You can also hijack existing mail servers for this, using suitably
encoded mail messages.

--
SM Ryan http://www.rawbw.com/~wyrmwif/
But I do believe in this.

[ Post a follow-up to this message ]

On 6 Aug, 20:41, cod...@gmail.com wrote:


> - The size of fifo is limited by BUF_SIZE, though I'm not clear it is
> the strict limitation of the whole fifo or each  opened fd connecte to
> the fifo.

This is probably less of a problem than you think, unless by
"asynchronous" (snipped) you meant that none of your writers
may block.  If/when the pipe is full, any process may write
to it, but the write will block and the process will be
put to sleep until there is room for the data.  You'll
have to explicitly set the write side of the pipe non-blocking
if you want to drop any data.

[ Post a follow-up to this message ]

On Aug 6, 6:39 pm, SM Ryan <wyrm...@tango-sierra-oscar-foxtrot-
tango.fake.org> wrote:
> cod...@gmail.com wrote:
>
> # I'm aware of the System V queues, however they are not present in the
> # Unix module of OCaml (a very conservative selection of Unix system
> # calls).  Instead of wrapping a FFI myself, I'd like to hear some other
> # work arounds.
>
> Bufferred but undeliverred data has to be stored somewhere. Pipes and
> such do not have unlimitted bufferring because that would require the
> kernel have unlimitted bufferring; I suppose it could do it somehow
> but it seems like a big chunk of responsibility to drop on the kernel.
>
> What you want is similar to email: asynchronous senders and receivers,
> unknown amounts data bufferred indefinitely. I would suggest going
> ahead and using an email-like system: If everything is on a single
> machine, create a directory or file read/writable to all processes
> and a method to add messages (as independent files in the directory
> or as appended bytes in a file) and the extract and delete messages.
> There are obvious variants of this such as using a database.
>
> If you have multiple machines without some kind file sharing like
> NFS, you can set up a message server on one machine that everyone
> else connects to send and receive messages over the network.

I've heard of this technique; I expect it's already merited an article
or two on thedailywtf.com...

> If
> you store messages in database like MySQL, the database server
> can act as the message repository and the message server.
>
> You can also hijack existing mail servers for this, using suitably
> encoded mail messages.

Jabber/XMPP might be a lot easier.

>
> --
> SM Ryanhttp://www.rawbw.com/~wyrmwif/
> But I do believe in this.

[ Post a follow-up to this message ]

On Aug 6, 4:22 pm, Eric Sosman <Eric.Sos...@sun.com> wrote:
> cod...@gmail.com wrote On 08/06/07 15:41,:
> 
> 
>
>     Have you considered using, well, message queues?
> See "man msgctl", "man msgsnd", "man msgrcv" to learn
> about System V queues; there are also POSIX queues but
> I'm less familiar with them.


(to OP)  I think message queues are the way to go, but I don't use
built-ins so I can have transparency between shared memory and fifos.
In my particular implementation, each sub-process has a set of fifos
and/or a page of shared memory to communicate with the parent.  The
parent has a thread for each sub-processes' input and reads in non-
blocking mode.  As long as there is input, it reads it.  When the
input stops, it checks back every 1/4 second or so with a "standby
mode" of 1s+ after a certain elapsed time.  The threads parse input
into requests and add them to a central request queue.  The main
thread of the parent process processes the messages in order of
priority and sends output back to the sub-processes as necessary.  The
sub-processes have an input monitor loop, also, and their main threads
(ideally) deals with the process' incoming messages.

The reason I don't have a single "distributor thread" is because not
all sub-processes are equal.  Some will have requests which go
straight to the top, and some of those requests will be to disconnect
other sub-processes or to exit the main process itself.

The main reason I use non-blocking reads is that the parsing logic
needs to be able to discard garble.  If it receives input that doesn't
make sense, it starts by discarding until it finds something
coherent.  If e.g. .01s after the last successful read nothing else is
there, the discard function exits and returns input control to the
parser, which polls again every Xs for valid input.  This also
provides timing transparency when using shared memory instead of pipes
and allows selective input operations to time out.  The reason the
discard function can't block is because 1) an error response needs to
be sent to the sub-process, 2) the sub-process needs to be
disconnected if it starts bogging down the system with garbage.

If you are talking about a server, which my application is, message
queues give you a lot more control over the server itself.  If you are
just using a specific number of sub-processes with specific purposes,
I recommend you elaborate so we can see what the real issue is.
Kevin Barry

[ Post a follow-up to this message ]