 |
|
 |
|
|
 |
Poor web serving performance |
 |
 |
|
|
01-26-07 12:12 AM
hi,
i'm having some problems getting decent performance out of the web
serving portion of perlbal.
i have a custom selector service, which then dispatches to a
web_server service, using the following code:
my $new_svc = Perlbal->service('my_web_server');
$new_svc->adopt_base_client($cb);
return 1;
this all works as expected, but starts to crumble under high request
rates (500/s is fine, 600/s is not). using http_load i start to get
alot of "Operation already in progress" kernel errors:
150 EINPROGRESS Operation now in progress
An operation which takes a long time to complete (such as a
connect(2)) was attempted on a non-blocking object [see ioctl(2)].
and around half of the requests to perlbal fail (perlbal never seems
to see them). if i swap out the web server hand-off with this code:
$cb->_simple_response(201, 'Fake thingy!');
return 1;
then it works as expected. if i substitute it with this code:
if (-e $req->{uri}){ 0; }
$cb->_simple_response(201, 'Fake thingy!');
return 1;
then it starts to fail again (in the same way), since it blocks on the
non-async stat call. the reason i tried this is because the time
*seems* to be spent doing stat's.
states
Perlbal::ClientHTTP wait_open 1
Perlbal::ClientHTTP wait_stat 128
Perlbal::ClientHTTP xfer_disk 262
Perlbal::ClientHTTPBase persist_wait 2
Perlbal::ClientHTTPBase reading_headers 320
.
states
Perlbal::ClientHTTP closed 1
Perlbal::ClientHTTP wait_stat 142
Perlbal::ClientHTTP xfer_disk 364
Perlbal::ClientHTTPBase persist_wait 2
Perlbal::ClientHTTPBase reading_headers 375
but maybe i'm reading it wrong. it might be something else between
setting 'wait_stat' and 'wait_open'. hmm, nope - not the case. when i
change the state to something else directly after the async stat call
comes back, it still has hundreds of clients in the 'wait_stat' state.
some other management console outputs:
fd
max 16384
cur 855
.
prof data
Perlbal::ClientHTTP-err 0.19197 0.03599 2839 0.0000676 0.0000127
Perlbal::ClientHTTP-write 0.83087 0.22297 8880 0.0000936 0.0000251
Perlbal::ClientHTTPBase-read 2.76559 0.34394 7187 0.0003848 0.0000479
Perlbal::TCPListener-read 1.09684 0.12698 60 0.0182806 0.0021163
.
the disks the files reside on are mounted over NFS and are relativly
slow, but Apache can easily serve 1000 req/s for the same files on the
same box.
i'm running this on RHEL 4, d-core d-proc 2.8Ghz w/ 3GB.
[calh@pc12 Perlbal-1.53]$ uname -a
Linux pc12.xxxxxxx.com 2.6.9-22.ELsmp #1 SMP Mon Sep 19 18:32:14 EDT
2005 i686 i686 i386 GNU/Linux
strace says:
Process 29684 detached
% time seconds usecs/call calls errors syscall
------ ----------- ----------- --------- --------- ----------------
21.51 0.354831 10 35806 epoll_ctl
20.46 0.337470 10 33806 37 futex
7.93 0.130758 12 11134 1720 close
7.21 0.118962 8 14335 14331 ioctl
6.79 0.112034 6 19230 fcntl64
5.43 0.089521 6 14400 9931 _llseek
5.30 0.087388 13 6663 1223 read
5.28 0.087031 10 8686 setsockopt
4.70 0.077592 8 9384 time
4.26 0.070197 14 4981 20 accept
3.72 0.061351 25 2425 201 sendfile
2.88 0.047494 11 4371 write
2.14 0.035248 8 4367 fstat64
0.91 0.014963 365 41 epoll_wait
0.60 0.009821 13 729 mprotect
0.27 0.004423 14 326 286 stat64
0.25 0.004080 13 325 210 open
0.13 0.002194 13 170 brk
0.08 0.001278 128 10 clone
0.05 0.000749 12 60 old_mmap
0.03 0.000463 10 47 gettimeofday
0.02 0.000369 12 30 mmap2
0.01 0.000230 14 17 munmap
0.01 0.000173 173 1 execve
0.01 0.000171 8 21 rt_sigprocmask
0.01 0.000153 9 18 rt_sigaction
0.00 0.000081 16 5 socket
0.00 0.000077 15 5 pipe
0.00 0.000046 15 3 2 connect
0.00 0.000038 38 1 readlink
0.00 0.000038 19 2 listen
0.00 0.000032 16 2 waitpid
0.00 0.000023 12 2 bind
0.00 0.000020 20 1 send
0.00 0.000017 9 2 uname
0.00 0.000016 16 1 _sysctl
0.00 0.000014 14 1 epoll_create
0.00 0.000013 7 2 poll
0.00 0.000010 10 1 1 access
0.00 0.000010 10 1 recvfrom
0.00 0.000005 5 1 set_thread_area
0.00 0.000004 4 1 getpid
0.00 0.000004 4 1 getuid32
0.00 0.000004 4 1 getgid32
0.00 0.000004 4 1 getegid32
0.00 0.000004 4 1 set_tid_address
0.00 0.000003 3 1 getrlimit
0.00 0.000003 3 1 geteuid32
------ ----------- ----------- --------- --------- ----------------
100.00 1.649410 171420 27962 total
anyone have any ideas?
thanks,
--cal
[ Post a follow-up to this message ]
|
|
|
 |
|
 |
|
 |
|
|
|
Re: Poor web serving performance |
|
|
|
|
01-26-07 12:12 AM
What's your:
SERVER aio_threads = nnnn
Set to in your config?
And can you verify that the same number of perlbal child processes are
hanging off the parent process?
On Thu, 25 Jan 2007, Cal Henderson wrote:
> hi,
>
> i'm having some problems getting decent performance out of the web
> serving portion of perlbal.
>
> i have a custom selector service, which then dispatches to a
> web_server service, using the following code:
>
> my $new_svc = Perlbal->service('my_web_server');
> $new_svc->adopt_base_client($cb);
> return 1;
>
> this all works as expected, but starts to crumble under high request
> rates (500/s is fine, 600/s is not). using http_load i start to get
> alot of "Operation already in progress" kernel errors:
>
> 150 EINPROGRESS Operation now in progress
> An operation which takes a long time to complete (such as a
> connect(2)) was attempted on a non-blocking object [see ioctl(2)].
>
> and around half of the requests to perlbal fail (perlbal never seems
> to see them). if i swap out the web server hand-off with this code:
>
> $cb->_simple_response(201, 'Fake thingy!');
> return 1;
>
> then it works as expected. if i substitute it with this code:
>
> if (-e $req->{uri}){ 0; }
> $cb->_simple_response(201, 'Fake thingy!');
> return 1;
>
> then it starts to fail again (in the same way), since it blocks on the
> non-async stat call. the reason i tried this is because the time
> *seems* to be spent doing stat's.
>
> states
> Perlbal::ClientHTTP wait_open 1
> Perlbal::ClientHTTP wait_stat 128
> Perlbal::ClientHTTP xfer_disk 262
> Perlbal::ClientHTTPBase persist_wait 2
> Perlbal::ClientHTTPBase reading_headers 320
> .
> states
> Perlbal::ClientHTTP closed 1
> Perlbal::ClientHTTP wait_stat 142
> Perlbal::ClientHTTP xfer_disk 364
> Perlbal::ClientHTTPBase persist_wait 2
> Perlbal::ClientHTTPBase reading_headers 375
>
> but maybe i'm reading it wrong. it might be something else between
> setting 'wait_stat' and 'wait_open'. hmm, nope - not the case. when i
> change the state to something else directly after the async stat call
> comes back, it still has hundreds of clients in the 'wait_stat' state.
>
> some other management console outputs:
>
> fd
> max 16384
> cur 855
> .
>
> prof data
> Perlbal::ClientHTTP-err 0.19197 0.03599 2839 0.0000676 0.0000127
> Perlbal::ClientHTTP-write 0.83087 0.22297 8880 0.0000936 0.0000251
> Perlbal::ClientHTTPBase-read 2.76559 0.34394 7187 0.0003848 0.0000479
> Perlbal::TCPListener-read 1.09684 0.12698 60 0.0182806 0.0021163
> .
>
> the disks the files reside on are mounted over NFS and are relativly
> slow, but Apache can easily serve 1000 req/s for the same files on the
> same box.
>
> i'm running this on RHEL 4, d-core d-proc 2.8Ghz w/ 3GB.
>
> [calh@pc12 Perlbal-1.53]$ uname -a
> Linux pc12.xxxxxxx.com 2.6.9-22.ELsmp #1 SMP Mon Sep 19 18:32:14 EDT
> 2005 i686 i686 i386 GNU/Linux
>
> strace says:
>
> Process 29684 detached
> % time seconds usecs/call calls errors syscall
> ------ ----------- ----------- --------- --------- ----------------
> 21.51 0.354831 10 35806 epoll_ctl
> 20.46 0.337470 10 33806 37 futex
> 7.93 0.130758 12 11134 1720 close
> 7.21 0.118962 8 14335 14331 ioctl
> 6.79 0.112034 6 19230 fcntl64
> 5.43 0.089521 6 14400 9931 _llseek
> 5.30 0.087388 13 6663 1223 read
> 5.28 0.087031 10 8686 setsockopt
> 4.70 0.077592 8 9384 time
> 4.26 0.070197 14 4981 20 accept
> 3.72 0.061351 25 2425 201 sendfile
> 2.88 0.047494 11 4371 write
> 2.14 0.035248 8 4367 fstat64
> 0.91 0.014963 365 41 epoll_wait
> 0.60 0.009821 13 729 mprotect
> 0.27 0.004423 14 326 286 stat64
> 0.25 0.004080 13 325 210 open
> 0.13 0.002194 13 170 brk
> 0.08 0.001278 128 10 clone
> 0.05 0.000749 12 60 old_mmap
> 0.03 0.000463 10 47 gettimeofday
> 0.02 0.000369 12 30 mmap2
> 0.01 0.000230 14 17 munmap
> 0.01 0.000173 173 1 execve
> 0.01 0.000171 8 21 rt_sigprocmask
> 0.01 0.000153 9 18 rt_sigaction
> 0.00 0.000081 16 5 socket
> 0.00 0.000077 15 5 pipe
> 0.00 0.000046 15 3 2 connect
> 0.00 0.000038 38 1 readlink
> 0.00 0.000038 19 2 listen
> 0.00 0.000032 16 2 waitpid
> 0.00 0.000023 12 2 bind
> 0.00 0.000020 20 1 send
> 0.00 0.000017 9 2 uname
> 0.00 0.000016 16 1 _sysctl
> 0.00 0.000014 14 1 epoll_create
> 0.00 0.000013 7 2 poll
> 0.00 0.000010 10 1 1 access
> 0.00 0.000010 10 1 recvfrom
> 0.00 0.000005 5 1 set_thread_area
> 0.00 0.000004 4 1 getpid
> 0.00 0.000004 4 1 getuid32
> 0.00 0.000004 4 1 getgid32
> 0.00 0.000004 4 1 getegid32
> 0.00 0.000004 4 1 set_tid_address
> 0.00 0.000003 3 1 getrlimit
> 0.00 0.000003 3 1 geteuid32
> ------ ----------- ----------- --------- --------- ----------------
> 100.00 1.649410 171420 27962 total
>
> anyone have any ideas?
>
>
> thanks,
>
> --cal
>
>
>
[ Post a follow-up to this message ]
|
|
|
|
|
|
|
|
|
|
|
Re: Poor web serving performance |
|
|
|
|
01-26-07 12:12 AM
Brad Fitzpatrick <brad@danga.com> wrote:
: What's your:
:
: SERVER aio_threads = nnnn
:
: Set to in your config?
:
: And can you verify that the same number of perlbal child processes
are
: hanging off the parent process?
SERVER aio_mode = ioaio
SERVER aio_threads = 30
i've tried with values from 2-50 and some higher ones too - all give
the same result. for any number of aio threads, i can only find a
single child processes - ps, pstree and top all just show the two
processes (i'm using IO::AIO).
16201 pts/0 Sl+ 0:00 /usr/local/bin/perl
./perlbal --config=my.conf
16202 pts/0 S+ 0:00 /usr/local/bin/perl
./perlbal --config=my.conf
as far as i can see, IO::AIO::min_parallel(30) is being called
sucessfully.
Linux::AIO wont pass the test suite so i haven't tried that. it fails
the usual tests:
[calh@pc12 Linux-AIO-1.9]$ make test
PERL_DL_NONLAZY=1 /home/y/bin/perl5.8.6 "-MExtUtils::Command::MM" "-e"
"test_harness(0, 'blib/lib', 'blib/arch')" t/*.t
t/00_load......ok
t/01_stat......ok
t/02_read......ok
t/03_errors....# Failed test 1 in t/03_errors.t at line 21
# t/03_errors.t line 21 is: ok($_[0] < 0 && $! == ENOENT);
t/03_errors....ok 3/10# Failed test 7 in t/03_errors.t at line 40
# t/03_errors.t line 40 is: ok($! == ENOENT);
t/03_errors....ok 9/10# Failed test 10 in t/03_errors.t at line 49
# t/03_errors.t line 49 is: ok($! == EBADF);
t/03_errors....FAILED tests 1, 7, 10
Failed 3/10 tests, 70.00% okay
Failed Test Stat Wstat Total Fail Failed List of Failed
----------------------------------------------------------------------------
---
t/03_errors.t 10 3 30.00% 1 7 10
Failed 1/4 test scripts, 75.00% okay. 3/26 subtests failed, 88.46%
okay.
make: *** [test_dynamic] Error 255
thanks,
--cal
[ Post a follow-up to this message ]
|
|
|
|
|
|
|
|
|
|
|
Re: Poor web serving performance |
|
|
|
|
01-26-07 12:12 AM
> Linux::AIO wont pass the test suite so i haven't tried that. it fails
> the usual tests:
Yes, this is a known bug in Linux::AIO. I emailed the author about it
and, iirc, he confirmed that RedHat's poor thread support is to blame.
There really isn't a work around, other than to use IO::AIO instead,
which you probably want anyway.
This is a wild guess, but I suspect that this issue (redhat's poor
thread support) is to blame for your performance problems as well. I
would switch to Debian if feasible.
Jay
[ Post a follow-up to this message ]
|
|
|
|
|
|
|
|
|
|
|
Re: Poor web serving performance |
|
|
|
|
01-26-07 12:12 AM
On Thu, 25 Jan 2007, Cal Henderson wrote:
> : And can you verify that the same number of perlbal child processes > are
> : hanging off the parent process?
>
> SERVER aio_mode = ioaio
> SERVER aio_threads = 30
>
> i've tried with values from 2-50 and some higher ones too - all give
> the same result. for any number of aio threads, i can only find a
> single child processes - ps, pstree and top all just show the two
> processes (i'm using IO::AIO).
>
> 16201 pts/0 Sl+ 0:00 /usr/local/bin/perl ./perlbal --config=my.conf
> 16202 pts/0 S+ 0:00 /usr/local/bin/perl ./perlbal --config=my.conf
>
>
> as far as i can see, IO::AIO::min_parallel(30) is being called
> sucessfully.
Sure doesn't look like it's working! Unless it's using NPTL? What
arch/kernel you using?
I'm on a 32-bit 2.6.11.12-xenU, and with IO::AIO with SERVER aio_threads
of 20, I see 20 child processes.
- Brad
[ Post a follow-up to this message ]
|
|
|
|
|
|
|
|
|
|
|
Re: Poor web serving performance |
|
|
|
|
01-26-07 12:12 AM
On Thu, 25 Jan 2007, Cal Henderson wrote:
> i've tried with values from 2-50 and some higher ones too - all give
> the same result. for any number of aio threads, i can only find a
> single child processes - ps, pstree and top all just show the two
> processes (i'm using IO::AIO).
>
> 16201 pts/0 Sl+ 0:00 /usr/local/bin/perl
> ./perlbal --config=my.conf
> 16202 pts/0 S+ 0:00 /usr/local/bin/perl
> ./perlbal --config=my.conf
But btw, with only one AIO thread working, that would definitely explain
the low aio_stat() throughput!
- Brad
[ Post a follow-up to this message ]
|
|
|
|
|
|
|
|
|
|
|
Re: Poor web serving performance |
|
|
|
|
01-26-07 12:12 AM
> >Linux::AIO wont pass the test suite so i haven't tried that. it fails
>
> Yes, this is a known bug in Linux::AIO. I emailed the author about it
> and, iirc, he confirmed that RedHat's poor thread support is to blame.
That's kinda difficult to fathom (Redhat's poor thread support)
considering that Ulrich Drepper/Ingo Molnar of NPTL fame work for Redhat.
Can he point to any bugzilla reports which detail more of this ?
Regards, Yusuf
[ Post a follow-up to this message ]
|
|
|
|
|
|
|
|
|
|
|
Re: Poor web serving performance |
|
|
|
|
01-26-07 12:12 AM
Brad Fitzpatrick <brad@danga.com> wrote:
: Sure doesn't look like it's working! Unless it's using NPTL? What
: arch/kernel you using?
[calh@pc12 Perlbal-1.53]$ cat /etc/redhat-release
Red Hat Enterprise Linux AS release 4 (Nahant Update 2)
[calh@pc12 Perlbal-1.53]$ uname -a
Linux pc12.xxxxxxx.com 2.6.9-22.ELsmp #1 SMP Mon Sep 19 18:32:14 EDT
2005 i686 i686 i386 GNU/Linux
[calh@pc12 Perlbal-1.53]$ PERL -V | grep archname
osname=linux, osvers=linux, archname=i686-linux-64int
[calh@pc12 Perlbal-1.53]$ getconf GNU_LIBPTHREAD_VERSION
NPTL 2.3.4
so it /is/ using NPTL threads. if i switch to linuxthreads:
[calh@pc12 Perlbal-1.53]$ export LD_ASSUME_KERNEL=2.4.19
[calh@pc12 Perlbal-1.53]$ getconf GNU_LIBPTHREAD_VERSION
linuxthreads-0.10
then i can see the additional processes spawning. however, on startup,
i only see 2 processes:
[calh@pc12 Perlbal-1.53]$ pstree -aclp | grep perlbal
| --perl,25351 ./perlbal --config=my.conf
| --perl,25352 ./perlbal --config=my.conf
and then when i throw some load at it, more processes get created:
[calh@pc12 Perlbal-1.53]$ pstree -aclp | grep perlbal
| --perl,25351 ./perlbal --config=my.conf
| --perl,25352 ./perlbal --config=my.conf
| --perl,25362 ./perlbal --config=my.conf
| --perl,25363 ./perlbal --config=my.conf
| --perl,25364 ./perlbal --config=my.conf
| --perl,25365 ./perlbal --config=my.conf
| --perl,25366 ./perlbal --config=my.conf
but never the amount i specified using SERVER aio_threads = 40 (and i
don't get better performance).
any ideas?
thanks,
--cal
[ Post a follow-up to this message ]
|
|
|
|
|
|
|
|
|
|
|
Re: Poor web serving performance |
|
|
|
|
01-26-07 12:12 AM
> That's kinda difficult to fathom (Redhat's poor thread support)
> considering that Ulrich Drepper/Ingo Molnar of NPTL fame work for Redhat.
> Can he point to any bugzilla reports which detail more of this ?
I dug up that email thread where I emailed the Linux::AIO author about
the same test failures that Cal is experiencing. Two big differences
between my problem and Cal's is that I was using a very old RHEL
version (2.1) and my failures were for IO::AIO, not Linux::AIO.
Here's the gist of our exchange that we had Nov. 2005. My comments
are prefaced with a >, his are not:
> [jay@lassen IO-AIO-1.61]$ make test
> PERL_DL_NONLAZY=1 /usr/local/bin/perl "-MExtUtils::Command::MM" "-e"
> "test_harness(0, 'blib/lib', 'blib/arch')" t/*.t
> t/00_load.......ok
> t/01_stat.......ok
> t/02_read.......ok
> t/03_errors.....# Failed test 1 in t/03_errors.t at line 22
> # t/03_errors.t line 22 is: ok((!defined $_[0]) && $! == ENOENT);
> t/03_errors.....ok 3/13# Failed test 6 in t/03_errors.t at line 40
> # t/03_errors.t line 40 is: ok($! == ENOENT);
> t/03_errors.....ok 8/13# Failed test 9 in t/03_errors.t at line 49
> # t/03_errors.t line 49 is: ok($! == EBADF);
> t/03_errors.....FAILED tests 1, 6, 9
> Failed 3/13 tests, 76.92% okay
> t/04_fork.......ok 2/8
>
> And it stops (never gets to 3/8). I went ahead and killed it and did
> make install. Everything seemed to work, but now I've been
Well, RHEL has always had severly broken threads support, which manifests
itself in various problems, but I haven't see these symptoms.
> [jay@lassen IO-AIO-1.61]$ cat /etc/redhat-release
> Red Hat Enterprise Linux WS release 2.1 (Tampa)
I just noticed the version. The rhel release you are using is
*extremely* old (four years!), and *completely* unusable with regards
to threads.
> [jay@lassen apache]$ PERL -V
> *snip*
> edt 2003 i686 unknown '
> config_args='-de'
> hint=recommended, useposix=true, d_sigaction=define
> usethreads=undef use5005threads=undef useithreads=undef
It is likely that your platform requires a threaded PERL (which I cannot
recommend, and is not necesary with newer threading libraries).
> snippet from gdb of running test:
> 0x400b3ca5 in sigsuspend () from /lib/libc.so.6
> (gdb) bt
> #0 0x400b3ca5 in sigsuspend () from /lib/libc.so.6
> #1 0x40217c09 in pthread_setconcurrency () from /lib/libpthread.so.0
> #2 0x40213dcc in pthread_cond_wait () from /lib/libpthread.so.0
Very much looks like a threading bug.
[ Post a follow-up to this message ]
|
|
|
|
|
|
|
|
|
|
|
Re: Poor web serving performance |
|
|
|
|
01-27-07 06:11 AM
And the IO::AIO test suite passes? (I assume, since you said the
Linux::AIO one failed...)
Definitely looks like it's not working. I wonder if the tests aren't
aggressive enough... can you tell if they exercise the set_minimum()
function at all and verify somehow that it works? (counting
threads/processes, or testing max outstanding AIOs... though the latter
seems hard without being able to fake and slow down the filesystem...)
On Thu, 25 Jan 2007, Cal Henderson wrote:
> Brad Fitzpatrick <brad@danga.com> wrote:
> : Sure doesn't look like it's working! Unless it's using NPTL? What
> : arch/kernel you using?
>
> [calh@pc12 Perlbal-1.53]$ cat /etc/redhat-release
> Red Hat Enterprise Linux AS release 4 (Nahant Update 2)
>
> [calh@pc12 Perlbal-1.53]$ uname -a
> Linux pc12.xxxxxxx.com 2.6.9-22.ELsmp #1 SMP Mon Sep 19 18:32:14 EDT
> 2005 i686 i686 i386 GNU/Linux
>
> [calh@pc12 Perlbal-1.53]$ PERL -V | grep archname
> osname=linux, osvers=linux, archname=i686-linux-64int
>
> [calh@pc12 Perlbal-1.53]$ getconf GNU_LIBPTHREAD_VERSION
> NPTL 2.3.4
>
>
> so it /is/ using NPTL threads. if i switch to linuxthreads:
>
> [calh@pc12 Perlbal-1.53]$ export LD_ASSUME_KERNEL=2.4.19
> [calh@pc12 Perlbal-1.53]$ getconf GNU_LIBPTHREAD_VERSION
> linuxthreads-0.10
>
> then i can see the additional processes spawning. however, on startup,
> i only see 2 processes:
>
> [calh@pc12 Perlbal-1.53]$ pstree -aclp | grep perlbal
> | --perl,25351 ./perlbal --config=my.conf
> | --perl,25352 ./perlbal --config=my.conf
>
> and then when i throw some load at it, more processes get created:
>
> [calh@pc12 Perlbal-1.53]$ pstree -aclp | grep perlbal
> | --perl,25351 ./perlbal --config=my.conf
> | --perl,25352 ./perlbal --config=my.conf
> | --perl,25362 ./perlbal --config=my.conf
> | --perl,25363 ./perlbal --config=my.conf
> | --perl,25364 ./perlbal --config=my.conf
> | --perl,25365 ./perlbal --config=my.conf
> | --perl,25366 ./perlbal --config=my.conf
>
> but never the amount i specified using SERVER aio_threads = 40 (and i
> don't get better performance).
>
> any ideas?
>
>
> thanks,
>
> --cal
>
>
[ Post a follow-up to this message ]
|
|
|
|
|
|
|
|
|
|
|
Sponsored Links |
|
|
|
|
 |
All times are GMT. The time now is 12:51 PM. |
 |
|
|
 |
|
 |
|
|
 |
|
Forum Rules:
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts
|
HTML code is OFF
vB code is ON
Smilies are ON
[IMG] code is OFF
|
|
|
|
Medical and Health forum | Computer Games Reviews | Graphics design forum
|
 |
|
 |
|
