|
Home > Archive > Apache Directory Project > December 2005 > [MINA] 0.9.0 - Deadlock in SocketIoProcessor
You are viewing an archived Text-only version of the thread.
To view this thread in it's original format and/or if you want to reply to
this thread please [click here]
| Author |
[MINA] 0.9.0 - Deadlock in SocketIoProcessor
|
|
| Srikanth Veeramachaneni 2005-12-16, 8:45 pm |
| Subject: [MINA] 0.9.0 - Deadlock in SocketIoProcessor
Hello,
I am trying to build a prototype using MINA for a proxy-like server.
The role of the proxy server is to communicate with the client using SSL
and act as a proxy to the application server communicating with it using
plain TCP.
TCP/SSL TCP
Client -----------------> Proxy Server -----------> Application Server
I initially started with MINA 0.8.1 and was successful doing so. I recently
upgraded to 0.9.0 and am facing the following problem.
Terms used:
-----------
Source Session - The SSL io session created when a client connects.
Source Handler - The handler for source sessions.
Destination Session - The io session created to the application server
Destination Handler - The handler for destination sessions.
Implementation Details:
-----------------------
I haven't customized any of the MINA settings and so am mostly using the
defaults. So the number of threads in the SocketIoProcessor is 1.
Once the handshake is completed for the source session a destination sessio=
n
is created to the application server using SocketConnector and attributes a=
re
set so that the source and destination sessions have references to each oth=
er.
Data is piped to each other in the source and destination handlers.
Also in each sessionClosed() method of the handlers the other session is cl=
osed.
The Problem Scenario:
---------------------
- the destination session is closed for some reason by the application ser=
ver
- the sessionClosed() method of the DestinationHandler is invoked by the
SocketIoProcessor worker thread as designed.
- The DestinationHandler.sessionClosed() method invokes sourceSession.clos=
e()
at which point a deadlock is occurring.
I have tried to debug what the problem is and here is what I found.
- Since the source session is an SSL session, SSLFilter is in the filter c=
hain.
- the close() invocation gets propagated down the chain to
SSLFilter.filterClose()
- SSLFilter.filterClose() invokes initiateClosure()
- SSLFilter.initiateClosure() invokes SSLHandler.closeOutbound() which cre=
ates
SSL close_notify message to be written to the session
- SSLFilter.initiateClosure() then invokes SSLHandler.writeNetBuffer() whi=
ch
creates a WriteFuture event for the data that needs to be written.
- SSLFilter.filterClose() then invokes WriteFuture.join() to wait for the
WriteFuture to complete.
- Since the worker thread of the SocketIoProcessor is the one that invoked
SSLFilter.filterClose() in the first place it waits here and never gets
to process the scheduled write, there by waiting for it selves and causi=
ng
a deadlock.
Potential solutions that I thought about :
-----------------------------------------
- Use the features in SocketIoProcessor to increase the number of worker
threads. But I realized that this will not solve my problem because ther=
e is
no guarantee that the Source Session and the Destination Session will be
allocated to different worker threads.
- Create an implementation of SSLFilter overriding the filterClose() metho=
d not
to do the join(). I haven't researched as to why the join was being done=
in
the first place. Assuming that there is a good reason to do so, this sol=
ution
might break something else.
- Create an implementation of ThreadPoolFilter in which filterClose() even=
ts
will also be handled by the worker threads of the thread pool.
I am hoping that there is something simple that I am not doing,
that is causing this problem.
Please advise the best way to overcome this problem.
thanks,
Srikanth
| |
| Niklas Therning 2005-12-17, 2:45 am |
| Hi,
I think there is a simple solution to your problem. Try to add a
ThreadPoolFilter to your SocketConnector's filter chain builder. That
should make the sessionClosed() call on your DestinationHandler be
handled in a different thread (not in SocketIoProcessor's worker thread).
Of course, SSLFilter shouldn't block the thread like it does right now
in initiateClosure. Your approach should work without a
ThreadPoolFilter. I don't know enough about the implementation details
of SSLFilter to say why it does block and if it could be fixed. Trustin?
HTH
/Niklas
Srikanth Veeramachaneni wrote:
> Subject: [MINA] 0.9.0 - Deadlock in SocketIoProcessor
>
> Hello,
>
> I am trying to build a prototype using MINA for a proxy-like server.
> The role of the proxy server is to communicate with the client using SSL
> and act as a proxy to the application server communicating with it using
> plain TCP.
>
> TCP/SSL TCP
> Client -----------------> Proxy Server -----------> Application Server
>
> I initially started with MINA 0.8.1 and was successful doing so. I recently
> upgraded to 0.9.0 and am facing the following problem.
>
> Terms used:
> -----------
>
> Source Session - The SSL io session created when a client connects.
> Source Handler - The handler for source sessions.
> Destination Session - The io session created to the application server
> Destination Handler - The handler for destination sessions.
>
> Implementation Details:
> -----------------------
>
> I haven't customized any of the MINA settings and so am mostly using the
> defaults. So the number of threads in the SocketIoProcessor is 1.
>
> Once the handshake is completed for the source session a destination session
> is created to the application server using SocketConnector and attributes are
> set so that the source and destination sessions have references to each other.
>
> Data is piped to each other in the source and destination handlers.
> Also in each sessionClosed() method of the handlers the other session is closed.
>
> The Problem Scenario:
> ---------------------
>
> - the destination session is closed for some reason by the application server
> - the sessionClosed() method of the DestinationHandler is invoked by the
> SocketIoProcessor worker thread as designed.
> - The DestinationHandler.sessionClosed() method invokes sourceSession.close()
> at which point a deadlock is occurring.
>
> I have tried to debug what the problem is and here is what I found.
>
> - Since the source session is an SSL session, SSLFilter is in the filter chain.
> - the close() invocation gets propagated down the chain to
> SSLFilter.filterClose()
> - SSLFilter.filterClose() invokes initiateClosure()
> - SSLFilter.initiateClosure() invokes SSLHandler.closeOutbound() which creates
> SSL close_notify message to be written to the session
> - SSLFilter.initiateClosure() then invokes SSLHandler.writeNetBuffer() which
> creates a WriteFuture event for the data that needs to be written.
> - SSLFilter.filterClose() then invokes WriteFuture.join() to wait for the
> WriteFuture to complete.
> - Since the worker thread of the SocketIoProcessor is the one that invoked
> SSLFilter.filterClose() in the first place it waits here and never gets
> to process the scheduled write, there by waiting for it selves and causing
> a deadlock.
>
> Potential solutions that I thought about :
> -----------------------------------------
>
> - Use the features in SocketIoProcessor to increase the number of worker
> threads. But I realized that this will not solve my problem because there is
> no guarantee that the Source Session and the Destination Session will be
> allocated to different worker threads.
> - Create an implementation of SSLFilter overriding the filterClose() method not
> to do the join(). I haven't researched as to why the join was being done in
> the first place. Assuming that there is a good reason to do so, this solution
> might break something else.
> - Create an implementation of ThreadPoolFilter in which filterClose() events
> will also be handled by the worker threads of the thread pool.
>
> I am hoping that there is something simple that I am not doing,
> that is causing this problem.
>
> Please advise the best way to overcome this problem.
>
> thanks,
> Srikanth
>
| |
| Trustin Lee 2005-12-21, 7:45 am |
| 2005/12/17, Niklas Therning <niklas-8FIgwK2HfyIwFerOooGFRg@public.gmane.org>:
>
> I think there is a simple solution to your problem. Try to add a
> ThreadPoolFilter to your SocketConnector's filter chain builder. That
> should make the sessionClosed() call on your DestinationHandler be
> handled in a different thread (not in SocketIoProcessor's worker thread).
Exactly.
Of course, SSLFilter shouldn't block the thread like it does right now
> in initiateClosure. Your approach should work without a
> ThreadPoolFilter. I don't know enough about the implementation details
> of SSLFilter to say why it does block and if it could be fixed. Trustin?
filterClose is invoked when user called IoSession.close(). As specified in
TLS specification, each side should send close_notify message before the
connection is closed. So, we have to ensure that the message is written
before the connection close request is passed to the IoSessionManager. To
do so, initiateClosure() waits for WriteFuture for the close_notify message,
by calling join(). The bottom line is that we should not call join() in
filters. We could sometimes use IoFuture.Callback, but I'm not sure if we
can always use this workaround.
There are two join() calls in SSLFilter. I can fix the case for
filterClose, but not for onPreRemove because the connection is not going to
be closed and we cannot prevent users from calling IoSession.close() before
close_notify message is written out not using join().
WDYT?
Trustin
--
what we call human nature is actually human habit
--
http://gleamynode.net/
PGP Key ID: 0x854B996C
| |
| Trustin Lee 2005-12-21, 7:45 am |
| I removed all join() calls anyway. Srikanth, could you let me know if your
issue is resolved?
Thanks,
Trustin
--
what we call human nature is actually human habit
--
http://gleamynode.net/
PGP Key ID: 0x854B996C
| |
| Srikanth Veeramachaneni 2005-12-21, 5:45 pm |
| Thanks, I will try your latest changes and let you know. Meanwhile I
went with Niklas's solution. It worked and it is really simple  .
thanks,
Srikanth
On 12/21/05, Trustin Lee <trustin-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
> I removed all join() calls anyway. Srikanth, could you let me know if yo=
ur
> issue is resolved?
>
> Thanks,
>
> Trustin
> --
> what we call human nature is actually human habit
> --
> http://gleamynode.net/
> PGP Key ID: 0x854B996C
| |
| Srikanth Veeramachaneni 2005-12-21, 8:45 pm |
| I tested my use case with the latest changes and my issue is resolved.
Thanks for the speedy resolution.
-Srikanth
On 12/21/05, Srikanth Veeramachaneni <sveerama-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
> Thanks, I will try your latest changes and let you know. Meanwhile I
> went with Niklas's solution. It worked and it is really simple .
>
> thanks,
> Srikanth
>
>
> On 12/21/05, Trustin Lee <trustin-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
your[vbcol=seagreen]
>
|
|
|
|
|