Scalable Network Server.

I am writing multithreaded web (actually something else, ) server.

1. one thread will accept connection, after connect success (accept
return) it will create a thread and pass connection id, and go back to
accept new connection
newly created thread

now due to high load more number of threads are created, I am thinking
of following approach

2. One Producer, Thread pool of  Multiple Consumers

-->Producer :
-One thread (producer) will Accept and Read- Request on Socket
-After Reading it will put Request into Queue.

-->Consumer :
-Multiple Consumer Threads will Read from the queue, procee req, send
response, close connection and again Read from Queue.
-Synchronization done using Condition  Var or Semaphor.(something like
Event Driver approach)

Problem : Single thread doing accept and read (actually recieve) if
client get connected but not Write request,one thread is block on
Read , Unwanted.

Solution : multiple producer instead of single

Disadvantages :
doing all this stuff from 1 to 2 to improve performance , but i think
in case of multiple producer and multiple consumer the Synchronizatoin
overhead will come to play. It should after all outperform Thread
Creation and Thread Destroy time.



note :
1. i am mostly implementing everything in C (no C++/Java)
2. I have also implemented Prethreaded server, Everythread block on
Accept,
3. in situation 2 , how to decide number of Consumer Thread  ? ( I
think i need to do real performance Test of 2, collect some statistics
and then decide)
4. anyone is having/knowing  any performance measurement of approach 1
or 2
5. Is Asynchronous IO is helpful here, I am knwoing very much less
about this , but heard that it is not portable,  and Conversion from 1
(which i already implemented) is complex and costly.


Any Idea how to cop with this common problem  and  to  very very good
performance to scale the server very well. ?  or any other Scalable
Server Design. ?

any link to online resouce/statistics (or opensource library/
framework) available for this ?

anyother suggestion ?

Thanks,


--Raxit

[ Post a follow-up to this message ]

Sheth Raxit wrote:
> Scalable Network Server.
>
> I am writing multithreaded web (actually something else, ) server.
>
[snip]

http://www.kegel.com/c10k.html is a must-read and also very interesting.
Start there.

HTH
Bjørn

--
Looking for an embeddable web server?
http://www.metasystems.no/products/...nder/index.html

[ Post a follow-up to this message ]

I've spent the past 12 years working on this issue. Which OS(es) are
you interested in? The techniques are very different.

Speaking generally, the approach that I've found works best works like
this:

1) You use the OS' most efficient means of discovering which sockets
are ready to do I/O.

2) For each I/O job that is ready, you queue a job to your I/O queue.

3) You have a pool of threads that pulls I/O jobs from the queue.

4) You keep a count of how many I/O jobs need to be done but have not
been done. As soon as the I/O is finished, each thread decrements that
count. (So, if you're doing a read job, you decrement the count as
soon as you have read all the data but before you fully process it.)

5) The first thread to finish an I/O job and find the count at zero
calls the OS' socket discovery function.

On some OSes, you need more than one thread in socket discovery
functions. In that case, each I/O thread works like this:

1) Is there an I/O job to do (read/write/shutdown/setup new connection/
whatever)? If so, do it and go to step 1.

2) Does the OS' socket discovery function need to be called? If so,
call it, queue any discovered jobs and go to step 1.

3) Wait until the queue is non-empty.

There are several different ways to decide how many threads to use.
One way is to add up the number of CPUs, the number of threads you
need for socket discovery, and the number of I/O jobs you can usefully
pend concurrently and use a "few more" than that. There are also
various dynamic techniques (add a thread if the queue starts to get
larger). It's tricky, you can get in to a situation where you think
you are thread starved and keep creating more and more threads even
though all the threads are really CPU starved.

DS

[ Post a follow-up to this message ]

David Schwartz <davids@webmaster.com> writes:

[...]

> Speaking generally, the approach that I've found works best works like
> this:
>
> 1) You use the OS' most efficient means of discovering which sockets
> are ready to do I/O.
>
> 2) For each I/O job that is ready, you queue a job to your I/O queue.
>
> 3) You have a pool of threads that pulls I/O jobs from the queue.
>
> 4) You keep a count of how many I/O jobs need to be done but have not
> been done. As soon as the I/O is finished, each thread decrements that
> count. (So, if you're doing a read job, you decrement the count as
> soon as you have read all the data but before you fully process it.)
>
> 5) The first thread to finish an I/O job and find the count at zero
> calls the OS' socket discovery function.

Assuming the 'socket discovery routine' returns exactly one job, a
single thread will process that to I/O completion and then call the
discovery routine again, while everything else sits around idling (or
I am missing something here).

Worth a look: <URL:http://pl.atyp.us/content/tech/servers.html>

Excerpt:

The simplest conceptual model of a multi-threaded
event-driven server has a queue at its center; requests are
read by one or more "listener" threads and put on queues,
from which one or more "worker" threads will remove and
process them. Conceptually, this is a good model, but all too
often people actually implement their code this way. Why is
this wrong? Because the #2 cause of context switches is
transferring work from one thread to another. Some people
even compound the error by requiring that the response to a
request be sent by the original thread - guaranteeing not one
but two context switches per request. It's very important to
use a "symmetric" approach in which a given thread can go
from being a listener to a worker to a listener again without
ever changing context. Whether this involves partitioning
connections between threads or having all threads take turns
being listener for the entire set of connections seems to
matter a lot less.

[ Post a follow-up to this message ]

On May 13, 11:16 am, Rainer Weikusat <rweiku...@mssgmbh.com> wrote:

> Assuming the 'socket discovery routine' returns exactly one job, a
> single thread will process that to I/O completion and then call the
> discovery routine again, while everything else sits around idling (or
> I am missing something here).

The only socket discovery routine that returns exactly one job that I
know of is IOCP. In that case, every thread should block on
GetOverlappedResult. I don't know of any UNIX that has a socket
discovery routine that always returns one result. Linux has epoll,
which does not. FreeBSD has kqueue, which does not. Generic UNIX has
select or poll which do not.

> Worth a look: <URL:http://pl.atyp.us/content/tech/servers.html>
>
> Excerpt:
>

>          Because the #2 cause of context switches is
>          transferring work from one thread to another. Some people
>          even compound the error by requiring that the response to a
>          request be sent by the original thread - guaranteeing not one
>          but two context switches per request. It's very important to
>          use a "symmetric" approach in which a given thread can go
>          from being a listener to a worker to a listener again without
>          ever changing context. Whether this involves partitioning
>          connections between threads or having all threads take turns
>          being listener for the entire set of connections seems to
>          matter a lot less.

In my example, context switches are never forced. The thread that
calls the socket discovery routine immediately starts pulling jobs
from the queue. The first thread that happens to discover an empty
queue calls the socket discovery routine. My threads become 'listener'
threads when there is no work to do and 'worker' threads as soon as
there is.

DS

[ Post a follow-up to this message ]

On May 13, 11:16 pm, Rainer Weikusat <rweiku...@mssgmbh.com> wrote:
> David Schwartz <dav...@webmaster.com> writes:
>
> [...]
>
>
>
>
> 
> 
mostly select (or epoll etc., i need to study kegel.com )[vbcol=seagreen]
> 
> 
> 
> 
Intresting !
Any re-usable stuff (or opensource example) available for these ?[vbcol=seagreen]
>
> Assuming the 'socket discovery routine' returns exactly one job, a
> single thread will process that to I/O completion and then call the
> discovery routine again, while everything else sits around idling (or
> I am missing something here).
>
> Worth a look: <URL:http://pl.atyp.us/content/tech/servers.html>
V. Good link,

i blooged the link before a year or 2, but i think i need to do more
in these,

while reading the link i think User level Threading - Instead of
pthread (which i think most probably maps to 1:1 model) (Correct me if
I am wrong !)  may helpful.

Currently most of my code is using PThread, anyone knowing any Posix
compatible User Level Thread library (for Linux,Sol 8,10) ? I think
user level thread libraries are doing simillar stuff like thread
pooling  (but may not like Queue and worker thread) so that

1. Thread creation and destroy time is less
2. Context Switch time is less.

>
> Excerpt:
>
>          The simplest conceptual model of a multi-threaded
>          event-driven server has a queue at its center; requests are
>          read by one or more "listener" threads and put on queues,
>          from which one or more "worker" threads will remove and
>          process them. Conceptually, this is a good model, but all too
>          often people actually implement their code this way. Why is
>          this wrong? Because the #2 cause of context switches is
>          transferring work from one thread to another. Some people
>          even compound the error by requiring that the response to a
>          request be sent by the original thread - guaranteeing not one
>          but two context switches per request. It's very important to
>          use a "symmetric" approach in which a given thread can go
>          from being a listener to a worker to a listener again without
>          ever changing context. Whether this involves partitioning
>          connections between threads or having all threads take turns

I think i can pass connection-fd to queue so the Workerthread can send
the response,

>          being listener for the entire set of connections seems to
>          matter a lot less.- Hide quoted text -
>
> - Show quoted text -

--Raxit

[ Post a follow-up to this message ]

On May 14, 1:48 am, Sheth Raxit <raxitsheth2...@yahoo.co.in> wrote:

> Currently most of my code is using PThread, anyone knowing any Posix
> compatible User Level Thread library (for Linux,Sol 8,10) ? I think
> user level thread libraries are doing simillar stuff like thread
> pooling  (but may not like Queue and worker thread) so that

User level thread libraries are essentially useless for I/O and server
type applications. The two main reasons for using threads are to keep
multiple CPUs busy and to prevent a stall (page fault, kernel thread
hijack, etcetera) from stalling the entire process. User level thread
libraries can't do either of these two things.

> 1. Thread creation and destroy time is less

So what? Say you need 100 threads. You create 100 threads in the first
second and then you never have to create or destroy a thread ever
again. You create and destroy threads (other than those you initially
need) as an optimization. Don't do it if it costs more than it saves.

> 2. Context Switch time is less.

You have to do something really wrong to create an application where
the cost of context switches is significant. Just don't do any of the
obviously stupid things that cause this problem.

DS

[ Post a follow-up to this message ]

On May 14, 2:13 pm, David Schwartz <dav...@webmaster.com> wrote:
> On May 14, 1:48 am, Sheth Raxit <raxitsheth2...@yahoo.co.in> wrote:
> 
>
> User level thread libraries are essentially useless for I/O and server
> type applications. The two main reasons for using threads are to keep
> multiple CPUs busy and to prevent a stall (page fault, kernel thread
> hijack, etcetera) from stalling the entire process. User level thread
> libraries can't do either of these two things.
> 
>
> So what? Say you need 100 threads. You create 100 threads in the first
> second and then you never have to create or destroy a thread ever
> again. You create and destroy threads (other than those you initially
> need) as an optimization. Don't do it if it costs more than it saves.
> 
>
> You have to do something really wrong to create an application where
> the cost of context switches is significant. Just don't do any of the
> obviously stupid things that cause this problem.
I think, you are prompting about over-optimization i am trying to
do, ?

--Raxit

>
> DS

[ Post a follow-up to this message ]

On May 14, 3:01 am, Sheth Raxit <raxitsheth2...@yahoo.co.in> wrote:

> I think, you are prompting about over-optimization i am trying to
> do, ?

Whenever someone says something like "high-load" or "high-
performance", I assume they mean somewhere near the limit. If "high
load" to you means a couple of hundred concurrent connections or
something, then almost anything will work. (Except one-thread-per-
client.)

DS

[ Post a follow-up to this message ]

On May 14, 3:11 pm, David Schwartz <dav...@webmaster.com> wrote:
> On May 14, 3:01 am, Sheth Raxit <raxitsheth2...@yahoo.co.in> wrote:
> 
>
> Whenever someone says something like "high-load" or "high-
> performance", I assume they mean somewhere near the limit. If "high
> load" to you means a couple of hundred concurrent connections or
> something, then almost anything will work. (Except one-thread-per-
> client.)

I mean approx.  1000 concurrent user can do,

My TCP Server actually doing this per request (not per client or not
per connection. !) (It serves one Request per connection -- I think
this may be bottleneck, Socket open/close is more more costly then
thread create/destroy !)

1. accept connection
2. create a thread this thread will do following
3. read request
4. process it ( disk i/o, read write record to files)
5. write response
6. close connection
7. exit thread

in other approach i tried on Pre Threaded Server with block on Accept,
but don't help much.


Thanks for  your insightful comment on this.

--Raxit
>
> DS

[ Post a follow-up to this message ]