Hello everyone,

we are currently looking to solve one of our Borland CaliberRM Server's
major bottlenecks, namely our IO-Subsystem.

We are using a dual Xeon Server with lot's of ram and overall we never
see -high- a cpu usage, but pretty often long write wait times / disk
queues. Operating System is Windows 2003, the actual database backend
is a versant database with ~5gb size. The Server itself is currently
serving the database from a Raid-10, but our old Raid-5 had the same
issues.


We recently got the chance to attach a test server to a NetApp Filer
(FAS 3020, 2x14hdds, connected via Fibre Channel) and we suddenly saw a
-good- increase in responsiveness and overall performance of the
server.

We'll have the next couple days to do more thorough benchmarks on the
NetApp System, but I was wondering whether someone has experience with
other solutions (SSD, ..?) in an environment where the database
consists of one big chunk (3 x 2gb volume files, unsplittable).

We've tried various performance tweaks on the OS-Level, as well as had
some versant expert look at the database indices etc and -software-
wise we're in perfect state, but nevertheless have really severe IO
problems (LOT's of reading and writing activites - each of very small
size).

Any ideas, suggestions, hints?

Best regards and thanks in advance,
-Joerg Battermann

[ Post a follow-up to this message ]

joerg.battermann@gmail.com wrote:
> Hello everyone,
>
> we are currently looking to solve one of our Borland CaliberRM Server's
> major bottlenecks, namely our IO-Subsystem.
>
> We are using a dual Xeon Server with lot's of ram and overall we never
> see -high- a cpu usage, but pretty often long write wait times / disk
> queues. Operating System is Windows 2003, the actual database backend
> is a versant database with ~5gb size. The Server itself is currently
> serving the database from a Raid-10, but our old Raid-5 had the same
> issues.
>
>
> We recently got the chance to attach a test server to a NetApp Filer
> (FAS 3020, 2x14hdds, connected via Fibre Channel) and we suddenly saw a
> -good- increase in responsiveness and overall performance of the
> server.
>
> We'll have the next couple days to do more thorough benchmarks on the
> NetApp System, but I was wondering whether someone has experience with
> other solutions (SSD, ..?) in an environment where the database
> consists of one big chunk (3 x 2gb volume files, unsplittable).
>
> We've tried various performance tweaks on the OS-Level, as well as had
> some versant expert look at the database indices etc and -software-
> wise we're in perfect state, but nevertheless have really severe IO
> problems (LOT's of reading and writing activites - each of very small
> size).
>
> Any ideas, suggestions, hints?

If you're seeing lots of small reads, then there are really only a
couple of ways to improve their performance:  lots of cache (if they're
not too random to benefit from it), and/or lots of disks (if they're
sufficiently parallel to benefit from them).  If the NetApp box has
considerably more of either or both than your server does, that's part
of the explanation of why it's faster (unless for some reason your
RAID-10 server storage isn't striping the database across all the
storage or is doing so inefficiently, such that each small access
involves multiple disks because they're not well aligned).

(Hmmm - if your server *really* has 'lots of ram', then you should be
able to keep the entire database memory-resident and avoid *any*
small-read problems, no?  Unless Versant either can't benefit from that
much ram, or isn't handling its own caching and Win2K3's file cache is
limited by its virtual-address constraints - though running a 64-bit
Windows version should fix the latter...)

For lots of small writes, the NetApp box is ideal, since it not only
captures them initially in NVRAM at near-zero latency but also destages
them back to disk in batches with its WAFL no-overwrite file system.  If
the writes are pretty randomly distributed, then the only way Versant
could handle them well would be if it itself used a no-overwrite update
mechanism (I'm guessing that it doesn't); if they're more concentrated,
then Versant *could* do a lot of aggregation in its transaction log to
help matters (by deferring in-place updates until many had accumulated
for each page), but I have no idea whether it actually has this level of
internal optimization (if it does, it *might* be being constrained by
having too small a log defined to defer the updates long enough for this
strategy to be effective).

- bill

[ Post a follow-up to this message ]

On 2 Nov 2006 13:41:45 -0800, joerg.battermann@gmail.com wrote:

>Hello everyone,
>
>we are currently looking to solve one of our Borland CaliberRM Server's
>major bottlenecks, namely our IO-Subsystem.
>
>We are using a dual Xeon Server with lot's of ram and overall we never
>see -high- a cpu usage, but pretty often long write wait times / disk
>queues. Operating System is Windows 2003, the actual database backend
>is a versant database with ~5gb size. The Server itself is currently
>serving the database from a Raid-10, but our old Raid-5 had the same
>issues.
>
>
>We recently got the chance to attach a test server to a NetApp Filer
>(FAS 3020, 2x14hdds, connected via Fibre Channel) and we suddenly saw a
>-good- increase in responsiveness and overall performance of the
>server.
>
>We'll have the next couple days to do more thorough benchmarks on the
>NetApp System, but I was wondering whether someone has experience with
>other solutions (SSD, ..?) in an environment where the database
>consists of one big chunk (3 x 2gb volume files, unsplittable).
>
>We've tried various performance tweaks on the OS-Level, as well as had
>some versant expert look at the database indices etc and -software-
>wise we're in perfect state, but nevertheless have really severe IO
>problems (LOT's of reading and writing activites - each of very small
>size).
>
>Any ideas, suggestions, hints?
>
>Best regards and thanks in advance,
>-Joerg Battermann


As Bill basically says, you're likely benefitting from the NVRAM in
the filer.

Another option might be some NVRAM cards in your server.  I forget the
names atm but I know there are a few people out there who make battery
backed ram for servers.  Plopping one of those in the server and
dropping the database on it may be a very good option.

~F

[ Post a follow-up to this message ]

Thanks Bill & Faeandar,

I did some more Tests on an FAS 3020 yesterday and the results were
more or less dissappointing (compared to the normal Raid). We had some
improvements when the user concurrency increased, but somehow the
overall increase wasn't astonishing.

We ran some syntetic tests using iometer using configurations similiar
to typical versant read/writes (16k block size etc) and the raid outran
the iscsi attached netapp by far.

In our original/first NetApp tests, we had the NetApp Filer cluster
attached 'locally' via fibre-channel in contrast to gbe now, but our
NetApp rep. said gbe would be "sufficient"... is this true?


What we are seeing on the performance monitor/iometer logs is that the
cpu-usage is very low (15-30%) while IOPS with 16k max out at around
10.000 with the NetApp installation (raid gives us about twice the
IOPS).

In case someone knows some vendors for such battery-backup'ed NVRAM
cards, I'd be really grateful if you could post them here too.

(Our CaliberRM installation currently -hinders- our engineers instead
of helping them and everyone's getting a tendency to let CaliberRM go,
which isn't really an option... but letting the users wait a couple
minutes after doing the simplest operations is a no-go)


I've also contacted a SSD/Texas Memory dealer here in Germany, hoping
that a sd-ram driven storage device will further improve performance...

*sigh* Glad it's weekend and I don't have to care tooo much about that
for another 2 days...

Thanks for your help so far,
-Joerg 

Faeandar wrote:
> On 2 Nov 2006 13:41:45 -0800, joerg.battermann@gmail.com wrote:
> 
>
>
> As Bill basically says, you're likely benefitting from the NVRAM in
> the filer.
>
> Another option might be some NVRAM cards in your server.  I forget the
> names atm but I know there are a few people out there who make battery
> backed ram for servers.  Plopping one of those in the server and
> dropping the database on it may be a very good option.
>
> ~F

[ Post a follow-up to this message ]

Joerg Battermann wrote:
> Thanks Bill & Faeandar,
>
> I did some more Tests on an FAS 3020 yesterday and the results were
> more or less dissappointing (compared to the normal Raid). We had some
> improvements when the user concurrency increased, but somehow the
> overall increase wasn't astonishing.
>
> We ran some syntetic tests using iometer using configurations similiar
> to typical versant read/writes (16k block size etc) and the raid outran
> the iscsi attached netapp by far.
>
> In our original/first NetApp tests, we had the NetApp Filer cluster
> attached 'locally' via fibre-channel in contrast to gbe now, but our
> NetApp rep. said gbe would be "sufficient"... is this true?
>
>
> What we are seeing on the performance monitor/iometer logs is that the
> cpu-usage is very low (15-30%) while IOPS with 16k max out at around
> 10.000 with the NetApp installation

160 MB/sec is *excellent* performance from a Gigabit Ethernet connection
(since it maxes out at only about 120 MB/sec in each direction) -
suggesting that the NetApp box is handling almost all reads and writes
in its cache (unless your workload is sufficiently parallel that it can
involve enough disks concurrently to generate that kind of bandwidth
with only 16 KB transfers - unless, of course, those transfers are
sequential rather than random, which presumably wouldn't really reflect
your Versant workload very well).  Presumably your Fibre Channel link
was 2 Gb or 4 Gb to have out-performed it significantly.

(raid gives us about twice the
> IOPS).

That's not bad either (again, assuming that the transfers weren't
sequential).

>
> In case someone knows some vendors for such battery-backup'ed NVRAM
> cards, I'd be really grateful if you could post them here too.
>
> (Our CaliberRM installation currently -hinders- our engineers instead
> of helping them and everyone's getting a tendency to let CaliberRM go,
> which isn't really an option... but letting the users wait a couple
> minutes after doing the simplest operations is a no-go)
>
>
> I've also contacted a SSD/Texas Memory dealer here in Germany, hoping
> that a sd-ram driven storage device will further improve performance...

For relatively small (few GB) sizes you might be able to find something
at a reasonable price.  You didn't say whether you were running 64-bit
versions of Windows and Versant:  if not, doing so (with, say, at least
8 GB of RAM on the server) might also be effective (i.e., you might want
to compare the cost of this to that of SSD or NVRAM).

- bill

[ Post a follow-up to this message ]

Bill Todd wrote:

...
 
>
> For relatively small (few GB) sizes you might be able to find something
> at a reasonable price.  You didn't say whether you were running 64-bit
> versions of Windows and Versant:  if not, doing so (with, say, at least
> 8 GB of RAM on the server) might also be effective (i.e., you might want
> to compare the cost of this to that of SSD or NVRAM).

Duh - I forgot to ask what might be the most obvious question of all:
is your current RAID controller maxed out with controller NVRAM?  Your
database is small enough that using your current (or a replacement) RAID
controller with a few GB of (mirrored) controller NVRAM might be the
least expensive solution (and would also probably require the least
change to your current set-up).

- bill

[ Post a follow-up to this message ]

Bill,

CaliberRM is currently 32bit only and so is the Versant-Version they
are using (and supporting). We know that Caliber can only handly 2gb
properly (that's what Borland told us).

The database itself is currently 4.4.gb with a growth of about
100-250mb/month.

Our Raid-Controller supports up to 256mb of ram and that's what we are
having installed on it. I had look for a raid-controller that allows
more ram 2 weeks are so ago, but never found anything that allowed e.g.
8gb or even 16 :-/ Do you know any controllers that do allow more than
that?

-j

Bill Todd wrote:
> Bill Todd wrote:
>
> ...
> 
>
> Duh - I forgot to ask what might be the most obvious question of all:
> is your current RAID controller maxed out with controller NVRAM?  Your
> database is small enough that using your current (or a replacement) RAID
> controller with a few GB of (mirrored) controller NVRAM might be the
> least expensive solution (and would also probably require the least
> change to your current set-up).
>
> - bill

[ Post a follow-up to this message ]

If you really want to see a performance Jump, bring in an EMC CX3 and
do raid 1/0.  It will out perform the NetApp 3050 at FCP in everyway.
You will see a performance gain in a magnitude of 3.  Remember that
Ontap is Raid 4 which is great for NAS because you can not saturate the
backend.  FCP requires a much more robust raid type and backend then
what NetApp delivers.

Also remember that Ontap is a software Raid not a hardware Raid.  That
alone is going to have its own performance implications.


Joerg Battermann wrote:[vbcol=seagreen]
> Bill,
>
> CaliberRM is currently 32bit only and so is the Versant-Version they
> are using (and supporting). We know that Caliber can only handly 2gb
> properly (that's what Borland told us).
>
> The database itself is currently 4.4.gb with a growth of about
> 100-250mb/month.
>
> Our Raid-Controller supports up to 256mb of ram and that's what we are
> having installed on it. I had look for a raid-controller that allows
> more ram 2 weeks are so ago, but never found anything that allowed e.g.
> 8gb or even 16 :-/ Do you know any controllers that do allow more than
> that?
>
> -j
>
> Bill Todd wrote: 

[ Post a follow-up to this message ]