Unix Programming - IPC message-queues: msgrcv() vs pipes/sockets? ACE?

Author

IPC message-queues: msgrcv() vs pipes/sockets? ACE?

Matt

2005-07-23, 2:48 am

I want to implement a complex, multi-process (and multiple threads per
process) network server system. I suspect that socket- or pipe-based
interprocess communication (IPC) will incur too much overhead.

Will the following message-queueing mechanisms (via the msg*()
functions) support the same capability (as sockets or pipes) with less
overhead (ie, make the overall system go faster)?

http://www.opengroup.org/onlinepubs...99/xsh/ipc.html

Any other suggestions/alternatives?

Free/open-source alternatives work best. I also prefer a
cross-platform solution between Linux and BSD solutions, possibly
Windows as well (but we can live without Windows support).

My software development team is also trying to get a similar mechanism
to work with ACE ( http://www.cs.wustl.edu/~schmidt/ACE.html ), and
while ACE's thread-pool management seems to work ok, we do not as of
yet see a means to manage shared-memory message queues in ACE.

Thanks for any help,
-Matt
--
Remove the "downwithspammers-" text to email me.

Gianni Mariani

2005-07-23, 2:48 am

Matt wrote:
> I want to implement a complex, multi-process (and multiple threads per
> process) network server system. I suspect that socket- or pipe-based
> interprocess communication (IPC) will incur too much overhead.
>
> Will the following message-queueing mechanisms (via the msg*()
> functions) support the same capability (as sockets or pipes) with less
> overhead (ie, make the overall system go faster)?
>
> http://www.opengroup.org/onlinepubs...99/xsh/ipc.html

The golden rule is "test it".

I suspect that pipes will be fastest since they are the most used.

>
> Any other suggestions/alternatives?

Shared memory and pipes.

>
> Free/open-source alternatives work best. I also prefer a
> cross-platform solution between Linux and BSD solutions, possibly
> Windows as well (but we can live without Windows support).
>
> My software development team is also trying to get a similar mechanism
> to work with ACE ( http://www.cs.wustl.edu/~schmidt/ACE.html ), and
> while ACE's thread-pool management seems to work ok, we do not as of
> yet see a means to manage shared-memory message queues in ACE.

Barry Margolin

2005-07-23, 7:50 am

In article <55g3e1d59f0jdf0tsq2e03dpn7lku0g4bv@4ax.com>,
Matt <matt@downwithspammers-mengland.net> wrote:

> I want to implement a complex, multi-process (and multiple threads per
> process) network server system. I suspect that socket- or pipe-based
> interprocess communication (IPC) will incur too much overhead.
>
> Will the following message-queueing mechanisms (via the msg*()
> functions) support the same capability (as sockets or pipes) with less
> overhead (ie, make the overall system go faster)?

Message queues are beneficial if you'll have multiple receiving
processes/threads. They're automatically delimited, so you don't have
to worry about one receiver getting part of a message intended for
another. You can also use the msgtype field to indicate which receiver
a message is intended for.

--
Barry Margolin, barmar@alum.mit.edu
Arlington, MA
*** PLEASE post questions in newsgroups, not directly to me ***

Loic Domaigne

2005-07-24, 5:52 pm

Salut Matt,

> I want to implement a complex, multi-process (and multiple threads per
> process) network server system. I suspect that socket- or pipe-based
> interprocess communication (IPC) will incur too much overhead.
>
> Will the following message-queueing mechanisms (via the msg*()
> functions) support the same capability (as sockets or pipes) with less
> overhead (ie, make the overall system go faster)?
>
> http://www.opengroup.org/onlinepubs...99/xsh/ipc.html

You can't really tell before your experiment it, actually. The
performance of an IPC (latency, bandwith) largely depends on the
underlying kernel implementation, and the results varies from systems to
systems.

I suggest to run some benchmarks first on your target system(s) in order
to make a clear statement. Ideas how to write such a benchmark can be
found in Stevens, UNIX Network Programming, Vol 2 (Interprocess
communication), 2nd Edition, Appendix A pp 457-499, ISBN 0-13-081081-9

> Any other suggestions/alternatives?

As quoted by Donald Knuth "premature optimization is the root of all
evil.". Choose rather the IPC that makes the program simpler. Get a
working program first. Optimize only if your performance requirements
are not met. Of course, if you're dealing with a big project, you might
want to consider agile methodology like prototyping and test driven
development. Obviously, you don't want to wait 3 years the code, to
finally realized that - gee - the solution is not efficient enough.

If traditional IPC (socket, fifo, message queue) fails, then you can
always roll own your own IPC using shared memory. Shared memory is the
fastest form of IPC available. But this implies to take care about
things that usually the kernel does for you (like synchronization etc.)

Ah - and a last note about System V message queue. This is a not well
recognized fact, but implementation having the so called MSGTQL limits
(AFAIK, only Linux and AIX doesn't have such a limit) are subject to a
Denial of Service attack. Any non root users of the system may
eventually block your server.

HTH,
Loic.