AOL Webserver - crash in AOLServer 4.5.0, after exiting threads

Author

crash in AOLServer 4.5.0, after exiting threads

Alex

2006-11-01, 7:11 pm

Hi,

I am getting yet another crash in AOLServer 4.5.0.
This time it crashes after exiting from threads started with
"ns_thread begin" or "ns_thread begindetached".

Any Suggestions?

Thanks,
~ Alex.

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 1086359904 (LWP 19612)]
0x00002aaaaae6c2a7 in Tcl_AsyncDelete (async=0x54e6c0) at
/srv/DIST/tcl8.4.13/unix/../generic/tclAsync.c:297
297 while (prevPtr->nextPtr != asyncPtr) {
(gdb) back
#0 0x00002aaaaae6c2a7 in Tcl_AsyncDelete (async=0x54e6c0) at
/srv/DIST/tcl8.4.13/unix/../generic/tclAsync.c:297
#1 0x00002aaaad48190e in TclX_SelectInit () from /usr/lib/libtclx8.4.so.1
#2 0x00002aaaaae6cfe2 in DeleteInterpProc (interp=0x647420) at
/srv/DIST/tcl8.4.13/unix/../generic/tclBasic.c:1037
#3 0x00002aaaaaee8416 in Tcl_EventuallyFree (clientData=0x647420,
freeProc=0x2aaaaae6ce5f <DeleteInterpProc> )
at /srv/DIST/tcl8.4.13/unix/../generic/tclPreserve.c:302
#4 0x00002aaaaae6ce5d in Tcl_DeleteInterp (interp=0x647420) at
/srv/DIST/tcl8.4.13/unix/../generic/tclBasic.c:924
#5 0x00002aaaaabf863f in NsConnThread (arg=<value optimized out> ) at
queue.c:590
#6 0x00002aaaaad36492 in ThreadMain (arg=0x515fc0) at pthread.c:469
#7 0x00002aaaab141b1c in start_thread () from /lib/libpthread.so.0
#8 0x00002aaaab5b09b2 in clone () from /lib/libc.so.6
#9 0x0000000000000000 in ?? ()
(gdb) print prevPtr
$1 = (AsyncHandler *) 0x0
(gdb) print asyncPtr
$2 = (AsyncHandler *) 0x54e6c0
(gdb) print tsdPtr
$3 = (ThreadSpecificData *) 0x7569c0
(gdb) print tsdPtr->firstHandler
$4 = (AsyncHandler *) 0x662b00

Here is the bit of code that fails. <<<<<<<<<< marks line 297 which is
probably the cause

void
Tcl_AsyncDelete(async)
Tcl_AsyncHandler async; /* Token for handler to delete. */
{
ThreadSpecificData *tsdPtr = TCL_TSD_INIT(&dataKey);
AsyncHandler *asyncPtr = (AsyncHandler *) async;
AsyncHandler *prevPtr;

Tcl_MutexLock(&tsdPtr->asyncMutex);
if (tsdPtr->firstHandler == asyncPtr) {
tsdPtr->firstHandler = asyncPtr->nextPtr;
if (tsdPtr->firstHandler == NULL) {
tsdPtr->lastHandler = NULL;
}
} else {
prevPtr = tsdPtr->firstHandler;
while (prevPtr->nextPtr != asyncPtr) { <<<<<<<<<<<<<<<<<
prevPtr = prevPtr->nextPtr;
}
prevPtr->nextPtr = asyncPtr->nextPtr;
if (tsdPtr->lastHandler == asyncPtr) {
tsdPtr->lastHandler = prevPtr;
}
}
Tcl_MutexUnlock(&tsdPtr->asyncMutex);
ckfree((char *) asyncPtr);
}

Zoran Vasiljevic

2006-11-01, 7:11 pm

On 01.11.2006, at 23:27, Alex wrote:

> Hi,
>
> I am getting yet another crash in AOLServer 4.5.0.
> This time it crashes after exiting from threads started with
> "ns_thread begin" or "ns_thread begindetached".
>
> Any Suggestions?
>
> Thanks,
> ~ Alex.
>
> Program received signal SIGSEGV, Segmentation fault.
> [Switching to Thread 1086359904 (LWP 19612)]
> 0x00002aaaaae6c2a7 in Tcl_AsyncDelete (async=0x54e6c0) at
> /srv/DIST/tcl8.4.13/unix/../generic/tclAsync.c:297
> 297 while (prevPtr->nextPtr != asyncPtr) {
> (gdb) back
> #0 0x00002aaaaae6c2a7 in Tcl_AsyncDelete (async=0x54e6c0) at
> /srv/DIST/tcl8.4.13/unix/../generic/tclAsync.c:297
> #1 0x00002aaaad48190e in TclX_SelectInit () from /usr/lib/
> libtclx8.4.so.1

I'm not sure TclX is thread-safe...

Cheers
Zoran

Jim Davidson

2006-11-01, 7:11 pm

Hi,

I think this is related to the comment I added to the RELEASE notes:

* Loading libnsd into a tclsh and then creating new threads with
the ns_thread command will result in a crash when those threads
exit. The issues has to do with finalization of the async-cancel
context used to support the new "ns_ictl cancel" feature. This bug
is not present when using the "nstclsh" binary.

The issue above, where Tcl is initialized before AOLserver by loading
libnsd into tclsh, results in Tcl thread local storage being
finalized before AOLserver's context which includes a pointer to an
async handler.

Now, that's not what you're doing here but perhaps TclX is having the
same effect. I haven't looked at TclX for sometime so I can't recall
what it would be using an async handler for -- perhaps you could dig
through the code and comment it out as the async handler stuff was
really designed for Unix signal-related things which aren't common in
multi-threaded AOLserver.

Alternatively, Tcl could be fixed to avoid freeing itself before
AOLserver or any other extension. Unfortunately, that could be a big
job -- the Tcl core is already riddled with a lot of code to try to
manage the order of finalization.

-Jim

On Nov 1, 2006, at 5:35 PM, Zoran Vasiljevic wrote:

> On 01.11.2006, at 23:27, Alex wrote:
>
>
> I'm not sure TclX is thread-safe...
>
> Cheers
> Zoran
>
>
> --
> AOLserver - http://www.aolserver.com/
>
> To Remove yourself from this list, simply send an email to
> <listserv@listserv.aol.com> with the
> body of "SIGNOFF AOLSERVER" in the email message. You can leave the
> Subject: field of your email blank.

Alex

2006-11-02, 1:11 am

Jim,

I tried in on the command line, seems to be my case

However, I run aolserver on debian, via /etc/init.d/aolserver,
Which basically invokes /usr/lib/aolserver4/bin/nsd.
How do I make it use nstclsh instead of tclsh ?
I don't see any options for that.

Thanks,
~ Alex.

On 11/1/06, Jim Davidson <jgdavidson@mac.com> wrote:
> Hi,
>
> I think this is related to the comment I added to the RELEASE notes:
>
> * Loading libnsd into a tclsh and then creating new threads with
> the ns_thread command will result in a crash when those threads
> exit. The issues has to do with finalization of the async-cancel
> context used to support the new "ns_ictl cancel" feature. This bug
> is not present when using the "nstclsh" binary.
>
>
> The issue above, where Tcl is initialized before AOLserver by loading
> libnsd into tclsh, results in Tcl thread local storage being
> finalized before AOLserver's context which includes a pointer to an
> async handler.
>
> Now, that's not what you're doing here but perhaps TclX is having the
> same effect. I haven't looked at TclX for sometime so I can't recall
> what it would be using an async handler for -- perhaps you could dig
> through the code and comment it out as the async handler stuff was
> really designed for Unix signal-related things which aren't common in
> multi-threaded AOLserver.
>
> Alternatively, Tcl could be fixed to avoid freeing itself before
> AOLserver or any other extension. Unfortunately, that could be a big
> job -- the Tcl core is already riddled with a lot of code to try to
> manage the order of finalization.
>
> -Jim
>
>
>
>
>
>
> On Nov 1, 2006, at 5:35 PM, Zoran Vasiljevic wrote:
>
>
>
> --
> AOLserver - http://www.aolserver.com/
>
> To Remove yourself from this list, simply send an email to <listserv@listserv.aol.com> with the
> body of "SIGNOFF AOLSERVER" in the email message. You can leave the Subject: field of your email blank.
>

Alex

2006-11-02, 1:11 am

Zoran, Jim

thanks very much for suggestions!
I think I figured it out.
The code which was executing in the thread concluded with "exit" tcl command.
I got it replaced with "return" and it seems not to be crashing anymore.

However, it would be probably a good idea to disable/rename "exit" for
the code executed in threads created by ns_thread. Not sure if this
shall be submitted as an "enhancement"-level bug.

Thanks,
~ Alex.

On 11/1/06, Alex <alexsott@gmail.com> wrote:
> Jim,
>
> I tried in on the command line, seems to be my case

Jim Davidson

2006-11-02, 1:11 pm

Hi,

To clarify, the "nstclsh" is another binary which is the same as
"tclsh" but includes the AOLserver commands, e.g.:

[jimbo@localhost aolserver]$ bin/nstclsh
% ns_info version
4.5
%

The idea was that some of those ns_* commands could be useful in non-
AOLserver shell scripts. Nstclsh doesn't yet have the capability to
be a full AOLserver so you still need nsd which still does all sorts
of weird process init (chroot, pre-bind, etc.), evals a config, and
starts web-server related threads.

BTW: Some day it should be possible to do:

$ tclsh
% package require aolserver
% nsd::do various config things...
% nsd::start
% vwait nsd

But that'll take some time -- it would still require threaded builds
of Tcl which aren't standard and the startup/init/config process
would be non-backwards compatible although quite a bit cleaner.

-Jim

On Nov 1, 2006, at 8:26 PM, Alex wrote:

> Jim,
>
> I tried in on the command line, seems to be my case

>
> However, I run aolserver on debian, via /etc/init.d/aolserver,
> Which basically invokes /usr/lib/aolserver4/bin/nsd.
> How do I make it use nstclsh instead of tclsh ?
> I don't see any options for that.
>
> Thanks,
> ~ Alex.
>
> On 11/1/06, Jim Davidson <jgdavidson@mac.com> wrote:
>
>
> --
> AOLserver - http://www.aolserver.com/
>
> To Remove yourself from this list, simply send an email to
> <listserv@listserv.aol.com> with the
> body of "SIGNOFF AOLSERVER" in the email message. You can leave the
> Subject: field of your email blank.

Alex

2006-11-03, 1:11 pm

Oh, well

so I guess it was too early to celebrate. Now I am getting the same
crashes again, even without "exit" command in the tcl code executed in
thread.

Seems to me that the same problem now discussed in
bug 1589968
https://sourceforge.net/tracker/?fu...8&group_id=3152

and

bug 1582671 http://sourceforge.net/tracker/?fun...&group_id=10894

Thanks,
~ Alex.

On 11/1/06, Alex <alexsott@gmail.com> wrote:
> Zoran, Jim
>
> thanks very much for suggestions!
> I think I figured it out.
> The code which was executing in the thread concluded with "exit" tcl command.
> I got it replaced with "return" and it seems not to be crashing anymore.
>
> However, it would be probably a good idea to disable/rename "exit" for
> the code executed in threads created by ns_thread. Not sure if this
> shall be submitted as an "enhancement"-level bug.
>
> Thanks,
> ~ Alex.
>
> On 11/1/06, Alex <alexsott@gmail.com> wrote:
>

Tom Jackson

2006-11-03, 7:11 pm

There have been several reports of AOLserver crashing, but very little data on
what the conditions are that cause the crash.

Is it possible that the problem is in a Tcl or AOLserver module and not in
aolserver? The folks helping out with the problem appear to understand what
is going on, but it leaves me (at least) wondering if there is some nasty
bug.

When bugs of this nature are reported is it possible to provide more
information so that other users can distinguish this behavior from other
similar problems, or be able to figure out if it might affect a particular
configuration? Heck, maybe it is an old operating system clashing with new
code, but these reports only create doubt for other users.

Also, I noticed that TclX is being used in one system with a bug, which may
not be thread safe. Does ns_thread* work well with a plain AOLserver?

tom jackson

On Friday 03 November 2006 10:31, Alex wrote:
> Oh, well
>
> so I guess it was too early to celebrate. Now I am getting the same
> crashes again, even without "exit" command in the tcl code executed in
> thread.
>
> Seems to me that the same problem now discussed in
> bug 1589968
> https://sourceforge.net/tracker/?fu...=1589968&group_
>id=3152
>
> and
>
> bug 1582671
> http://sourceforge.net/tracker/?fun...1582671&group_i
>d=10894
>
>
> Thanks,
> ~ Alex.
>
> On 11/1/06, Alex <alexsott@gmail.com> wrote:
>
> --
> AOLserver - http://www.aolserver.com/
>
> To Remove yourself from this list, simply send an email to
> <listserv@listserv.aol.com> with the body of "SIGNOFF AOLSERVER" in the
> email message. You can leave the Subject: field of your email blank.