Unix True 64 - Thread safe malloc in Tru64?

Author

Thread safe malloc in Tru64?

Ioannis E. Venetis

2004-01-23, 6:50 pm

Hello,

I am trying to create a library that will run, among other architectures, on
Tru64/Alpha systems.
Of course, this library has a machine independant (the same for all
architectures) and a machine
dependant part. For some reasons, I chose to use the
thread_create/thread_get_state/
thread_set_state/thread_resume/thread_terminate interface in the machine
dependant part of the
library to manage kernel-level threads on Tru64 Unix systems. I am
mentioning it in case this has
something to do with the problem I will describe.

Now, the problem is that all my applications that use this library crash on
Tru64 Unix systems
(but not on other systems the library compiles on). After spending hours on
debugging and
inserting printf's on several places I found what was going wrong. The
malloc() function, when
called by all threads of the application, returns most of the time the same
starting address of the
allocated memory for different threads. This seems to be an issue with
thread-safety of the
malloc() function. I searched the Internet and found that I had to link my
applications with a few
libraries, in order to make them thread-safe. Some of the possibilities I
found are:

-lpthreads -lmach -lc_r
-lpthread -lmach -lexc -lc

I also tried other combinations
(-lpthread -lmach -lc_r, -lpthread -lmach -lc, -lpthreads -lmach -lc,...)
but none of them solved the problem. I also tried to use -D_REENTRANT and/or
-D_THREAD_SAFE when compiling the library and the applications, but the
problem still remains.

Finally, I replaced malloc() with mmap(), and the applications run perfectly
in this case, so I doubt
that something else is wrong in the library.

I would appreciate it if someone could give me some directions on how to
link my libray and applications
with thread-safe versions of all necessary libraries, both for shared and
static versions.

Some system specific information:

OS: Compaq Tru64 UNIX V5.1 (Rev. 732)
Compiler: gcc -v
Reading specs from /usr/local/lib/gcc-lib/alphaev6-dec-osf4.0e/2.95.2/specs
gcc version 2.95.2 19991024 (release)

Thank you in advance,

Ioannis E. Venetis
PhD Student - Research Assistant
High Performance Information Systems Laboratory
Computer Engineering and Informatics Department
University of Patras, Greece
URL: http://www.hpclab.ceid.upatras.gr/pgroup/members/iev

Jerry Feldman

2004-01-23, 6:50 pm

On Thu, 20 Nov 2003 16:11:01 +0200
"Ioannis E. Venetis" <iev@hpclab.ceid.upatras.gr> wrote:

quote:

> Hello,
>
> I am trying to create a library that will run, among other
> architectures, on Tru64/Alpha systems.
> Of course, this library has a machine independant (the same for all
> architectures) and a machine
> dependant part. For some reasons, I chose to use the
> thread_create/thread_get_state/
> thread_set_state/thread_resume/thread_terminate interface in the
> machine dependant part of the
> library to manage kernel-level threads on Tru64 Unix systems. I am
> mentioning it in case this has
> something to do with the problem I will describe.

While I am not purporting to solve your problem, but all libc functions,
including malloc(3), were made threadsafe (I believe) as of Tru64
(Digital Unix) 4.0.
When compiling use -pthread rather than -lpthread or -lpthreads.
Additionally, libc_r is nothing more than a symlink to libc.

Also be aware that the Tru64 thread package uses a combination of kernel
threads and user threads.

Try using HP's native compiler for your testing. Part of your problem
might be that gcc 2.95 is out of date.
--
Jerry Feldman <gaf-nospam-at-blu.org>
Boston Linux and Unix user group
http://www.blu.org PGP key id:C5061EA9
PGP Key fingerprint:053C 73EC 3AC1 5C44 3E14 9245 FB00 3ED5 C506 1EA9

David Butenhof

2004-01-23, 6:50 pm

Ioannis E. Venetis wrote:

quote:

> I am trying to create a library that will run, among other architectures,
> on Tru64/Alpha systems.
> Of course, this library has a machine independant (the same for all
> architectures) and a machine
> dependant part. For some reasons, I chose to use the
> thread_create/thread_get_state/
> thread_set_state/thread_resume/thread_terminate interface in the machine
> dependant part of the
> library to manage kernel-level threads on Tru64 Unix systems. I am
> mentioning it in case this has
> something to do with the problem I will describe.

Yes, that's extremely relevant. In fact, explanation is exactly and
exclusively the cause of your problems.

Direct use of Mach threading primitives is not SUPPORTED for user mode
applications. That is, nobody says it'll work, it's not documented, and if
you insist on doing it anyway whatever you get is your own responsibility.

In particular, in this case, the thread safety of all of libc (and other
user-space libraries) is a result of cooperation between those libraries
and the SUPPORTED/DOCUMENTED user mode thread implementation in libpthread.
Without libpthread, there's no thread safety.

quote:

> I searched the Internet and found that I had to link my
> applications with a few
> libraries, in order to make them thread-safe. Some of the possibilities I
> found are:
>
> -lpthreads -lmach -lc_r
> -lpthread -lmach -lexc -lc

You're already linking with -lmach, if you're using the UNSUPPORTED Mach
thread interfaces. Because libpthreads has a dependency on libpthread, and
both depend on libexc, much of this is redundant. (And while very old
versions of libpthread used Mach operations directly and required libmach,
this hasn't been true since V4.0D.)

The proper way to build a threaded application (using POSIX threads) is with
the -pthread switch (not -lpthread). If you must compile and link
separately, you can compile with -D_REENTRANT and link with -lpthread -lexc
-lc. (libc_r is nothing but a link to libc; once upon a time, in the days
of the threadosaurus, they were distinct, and the old name remains for
compatibility; but it no longer means anything.) If you were to use the
obsolete DCE thread or the archaic CMA interfaces, you'd use -threads,
which translates to linking with -lpthreads in addition to the others.

Furthermore, using POSIX thread synchronization in a thread created using
the Mach interfaces isn't supported, and that's really what you're doing
when you add -lpthread to your binary. The process is mostly prepared to
pretend to be threaded -- but it's not, really. All of your Mach threads
will think they're the same POSIX thread, and, well, it's not going to work
very well.

quote:

> Finally, I replaced malloc() with mmap(), and the applications run
> perfectly in this case, so I doubt
> that something else is wrong in the library.

Because mmap() is a kernel syscall, not user space code. You could "get
away" with using Mach threads if you didn't rely on any user-space thread
safety. If you're content to do that, go for it... but your application is
still going to be technically unsupportable and in my opinion it's not a
good idea.

quote:

> I would appreciate it if someone could give me some directions on how to
> link my libray and applications
> with thread-safe versions of all necessary libraries, both for shared and
> static versions.

Use POSIX thread operations instead of the unsupported Mach interfaces.
Build with -pthread, if the version of gcc you're using supports it; or use
-D_REENTRANT -lpthread -lexc -lc. Note that gcc doesn't integrate with
libexc (the Tru64 UNIX exception package); if you mix C and C++, C++
destructors won't run on thread cancellation and TRY/FINALLY/CATCH_ALL
macros won't work with C++ exceptions propagated through a frame. (This
will work with HP's compiler.)

--
/--------------------[ David.Butenhof@hp.com ]--------------------\
| Hewlett-Packard Company Tru64 UNIX & VMS Thread Architect |
| My book: http://www.awl.com/cseng/titles/0-201-63392-2/ |
\----[ http://homepage.mac.com/dbutenhof/Threads/Threads.html ]---/

Ioannis E. Venetis

2004-01-23, 6:50 pm

First of all, thank you for your very detailed answer.

It seems that my choice to use the Mach interface on the Tru64 Unix
system was unfortunate. Anyway, it is quite trivial to change the machine
dependant part of the library to use POSIX threads for this architecture
and I will do it. But there still remains a very important problem, at least
for me :-)

This library is also to be run on a simulator. The simulator is based on
an Alpha processor and it has some extensions to support Simultaneous
Multithreading (SMT). It executes static binaries created on Tru64
Unix systems and it explicitly does NOT support POSIX Threads.
Instead, the machine dependant part of the library HAS to use the
primitives of the simulator to start and manage hardware threads, if
parallel applications are to be run on the simulator (there is no OS, so
hardware threads can be thought of as equivalent entities
to kernel threads when an OS is present). The problem of course is
that malloc() returns the same addresses for allocated regions when
executed concurrently on more hardware threads, since the linked in
C library is not thread safe. Moreover, the simulator does not provide
mmap() (which would perhaps help me overcome the problem, as on
the real Tru64 system). This means that I still need to find another way
to link thread safe versions of libraries with my applications, in order to
successfully run these applications on the simulator. It seems that it will
not be possible for me to avoid some kind of hacks, at least to make
the library run on the simulator.

The very easy solution to use a lock around each malloc() in the
library did not help. Of course I did not use POSIX primitives
for the locks, but hand-written assembly code (which runs correctly,
both on the Tru64 system and the simulator).

Any other suggestion on this issue is greatly appreciated.

Ioannis E. Venetis
PhD Student - Research Assistant
High Performance Information Systems Laboratory
Computer Engineering and Informatics Department
University of Patras, Greece
URL: http://www.hpclab.ceid.upatras.gr/pgroup/members/iev