Subject: [MINA] 0.9.0 - Deadlock in SocketIoProcessor

Hello,

I am trying to build a prototype using MINA for a proxy-like server.
The role of the proxy server is to communicate with the client using SSL
and act as a proxy to the application server communicating with it using
plain TCP.

TCP/SSL                           TCP
Client -----------------> Proxy Server -----------> Application Server

I initially started with MINA 0.8.1 and was successful doing so. I recently
upgraded to 0.9.0 and am facing the following problem.

Terms used:
-----------

Source Session      - The SSL io session created when a client connects.
Source Handler      - The handler for source sessions.
Destination Session - The io session created to the application server
Destination Handler - The handler for destination sessions.

Implementation Details:
-----------------------

I haven't customized any of the MINA settings and so am mostly using the
defaults. So the number of threads in the SocketIoProcessor is 1.

Once the handshake is completed for the source session a destination sessio=
n
is created to the application server using SocketConnector and attributes a=
re
set so that the source and destination sessions have references to each oth=
er.

Data is piped to each other in the source and destination handlers.
Also in each sessionClosed() method of the handlers the other session is cl=
osed.

The Problem Scenario:
---------------------

- the destination session is closed for some reason by the application ser=
ver
- the sessionClosed() method of the DestinationHandler is invoked by the
SocketIoProcessor worker thread as designed.
- The DestinationHandler.sessionClosed() method invokes sourceSession.clos=
e()
at which point a deadlock is occurring.

I have tried to debug what the problem is and here is what I found.

- Since the source session is an SSL session, SSLFilter is in the filter c=
hain.
- the close() invocation gets propagated down the chain to
SSLFilter.filterClose()
- SSLFilter.filterClose() invokes initiateClosure()
- SSLFilter.initiateClosure() invokes SSLHandler.closeOutbound() which cre=
ates
SSL close_notify message to be written to the session
- SSLFilter.initiateClosure() then invokes SSLHandler.writeNetBuffer() whi=
ch
creates a WriteFuture event for the data that needs to be written.
- SSLFilter.filterClose() then invokes WriteFuture.join() to wait for the
WriteFuture to complete.
- Since the worker thread of the SocketIoProcessor is the one that invoked
SSLFilter.filterClose() in the first place it waits here and never gets
to process the scheduled write, there by waiting for it selves and causi=
ng
a deadlock.

Potential solutions that I thought about :
-----------------------------------------

- Use the features in SocketIoProcessor to increase the number of worker
threads. But I realized that this will not solve my problem because ther=
e is
no guarantee that the Source Session and the Destination Session will be
allocated to different worker threads.
- Create an implementation of SSLFilter overriding the filterClose() metho=
d not
to do the join(). I haven't researched as to why the join was being done=
in
the first place. Assuming that there is a good reason to do so, this sol=
ution
might break something else.
- Create an implementation of ThreadPoolFilter in which filterClose() even=
ts
will also be handled by the worker threads of the thread pool.

I am hoping that there is something simple that I am not doing,
that is causing this problem.

Please advise the best way to overcome this problem.

thanks,
Srikanth

[ Post a follow-up to this message ]

Hi,

I think there is a simple solution to your problem. Try to add a
ThreadPoolFilter to your SocketConnector's filter chain builder. That
should make the sessionClosed() call on your DestinationHandler be
handled in a different thread (not in SocketIoProcessor's worker thread).

Of course, SSLFilter shouldn't block the thread like it does right now
in initiateClosure. Your approach should work without a
ThreadPoolFilter. I don't know enough about the implementation details
of SSLFilter to say why it does block and if it could be fixed. Trustin?

HTH

/Niklas

Srikanth Veeramachaneni wrote:
> Subject: [MINA] 0.9.0 - Deadlock in SocketIoProcessor
>
> Hello,
>
> I am trying to build a prototype using MINA for a proxy-like server.
> The role of the proxy server is to communicate with the client using SSL
> and act as a proxy to the application server communicating with it using
> plain TCP.
>
>              TCP/SSL                           TCP
>   Client -----------------> Proxy Server -----------> Application Server
>
> I initially started with MINA 0.8.1 and was successful doing so. I recentl
y
> upgraded to 0.9.0 and am facing the following problem.
>
> Terms used:
> -----------
>
> Source Session      - The SSL io session created when a client connects.
> Source Handler      - The handler for source sessions.
> Destination Session - The io session created to the application server
> Destination Handler - The handler for destination sessions.
>
> Implementation Details:
> -----------------------
>
> I haven't customized any of the MINA settings and so am mostly using the
> defaults. So the number of threads in the SocketIoProcessor is 1.
>
> Once the handshake is completed for the source session a destination sessi
on
> is created to the application server using SocketConnector and attributes 
are
> set so that the source and destination sessions have references to each ot
her.
>
> Data is piped to each other in the source and destination handlers.
> Also in each sessionClosed() method of the handlers the other session is c
losed.
>
> The Problem Scenario:
> ---------------------
>
>  - the destination session is closed for some reason by the application se
rver
>  - the sessionClosed() method of the DestinationHandler is invoked by the
>    SocketIoProcessor worker thread as designed.
>  - The DestinationHandler.sessionClosed() method invokes sourceSession.clo
se()
>    at which point a deadlock is occurring.
>
> I have tried to debug what the problem is and here is what I found.
>
>  - Since the source session is an SSL session, SSLFilter is in the filter 
chain.
>  - the close() invocation gets propagated down the chain to
>    SSLFilter.filterClose()
>  - SSLFilter.filterClose() invokes initiateClosure()
>  - SSLFilter.initiateClosure() invokes SSLHandler.closeOutbound() which cr
eates
>    SSL close_notify message to be written to the session
>  - SSLFilter.initiateClosure() then invokes SSLHandler.writeNetBuffer() wh
ich
>    creates a WriteFuture event for the data that needs to be written.
>  - SSLFilter.filterClose() then invokes WriteFuture.join() to wait for the
>    WriteFuture to complete.
>  - Since the worker thread of the SocketIoProcessor is the one that invoke
d
>    SSLFilter.filterClose() in the first place it waits here and never gets
>    to process the scheduled write, there by waiting for it selves and caus
ing
>    a deadlock.
>
> Potential solutions that I thought about :
> -----------------------------------------
>
>  - Use the features in SocketIoProcessor to increase the number of worker
>    threads. But I realized that this will not solve my problem because the
re is
>    no guarantee that the Source Session and the Destination Session will b
e
>    allocated to different worker threads.
>  - Create an implementation of SSLFilter overriding the filterClose() meth
od not
>    to do the join(). I haven't researched as to why the join was being don
e in
>    the first place. Assuming that there is a good reason to do so, this so
lution
>    might break something else.
>  - Create an implementation of ThreadPoolFilter in which filterClose() eve
nts
>    will also be handled by the worker threads of the thread pool.
>
> I am hoping that there is something simple that I am not doing,
> that is causing this problem.
>
> Please advise the best way to overcome this problem.
>
> thanks,
> Srikanth
>

[ Post a follow-up to this message ]

2005/12/17, Niklas Therning <niklas-8FIgwK2HfyIwFerOooGFRg@public.gmane.org>:
>
> I think there is a simple solution to your problem. Try to add a
> ThreadPoolFilter to your SocketConnector's filter chain builder. That
> should make the sessionClosed() call on your DestinationHandler be
> handled in a different thread (not in SocketIoProcessor's worker thread).


Exactly.

Of course, SSLFilter shouldn't block the thread like it does right now
> in initiateClosure. Your approach should work without a
> ThreadPoolFilter. I don't know enough about the implementation details
> of SSLFilter to say why it does block and if it could be fixed. Trustin?


filterClose is invoked when user called IoSession.close().  As specified in
TLS specification, each side should send close_notify message before the
connection is closed.  So, we have to ensure that the message is written
before the connection close request is passed to the IoSessionManager.  To
do so, initiateClosure() waits for WriteFuture for the close_notify message,
by calling join().  The bottom line is that we should not call join() in
filters.  We could sometimes use IoFuture.Callback, but I'm not sure if we
can always use this workaround.

There are two join() calls in SSLFilter.  I can fix the case for
filterClose, but not for onPreRemove because the connection is not going to
be closed and we cannot prevent users from calling IoSession.close() before
close_notify message is written out not using join().

WDYT?

Trustin
--
what we call human nature is actually human habit
--
http://gleamynode.net/
PGP Key ID: 0x854B996C

[ Post a follow-up to this message ]

I removed all join() calls anyway.  Srikanth, could you let me know if your
issue is resolved?

Thanks,
Trustin
--
what we call human nature is actually human habit
--
http://gleamynode.net/
PGP Key ID: 0x854B996C

[ Post a follow-up to this message ]

Thanks, I will try your latest changes and let you know. Meanwhile I
went with Niklas's solution. It worked and it is really simple .

thanks,
Srikanth


On 12/21/05, Trustin Lee <trustin-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
> I removed all join() calls anyway.  Srikanth, could you let me know if yo=
ur
> issue is resolved?
>
> Thanks,
>
> Trustin
> --
> what we call human nature is actually human habit
> --
>  http://gleamynode.net/
> PGP Key ID: 0x854B996C

[ Post a follow-up to this message ]

I tested my use case with the latest changes and my issue is resolved.
Thanks for the speedy resolution.

-Srikanth

On 12/21/05, Srikanth Veeramachaneni <sveerama-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrot
e:
> Thanks, I will try your latest changes and let you know. Meanwhile I
> went with Niklas's solution. It worked and it is really simple .
>
> thanks,
> Srikanth
>
>
> On 12/21/05, Trustin Lee <trustin-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>
 wrote: 
your[vbcol=seagreen] 
>

[ Post a follow-up to this message ]