|
Home > Archive > Data Storage > October 2005 > Hitachi 9990 & MPXIO Performance
You are viewing an archived Text-only version of the thread.
To view this thread in it's original format and/or if you want to reply to
this thread please [click here]
| Author |
Hitachi 9990 & MPXIO Performance
|
|
| Greg Brown 2005-10-24, 9:43 am |
| Hello,
Just a quick info gathering. I am at a customer site installing a new
HDS 9990. The high level config overview:
HDS 9990 (Open V 40GB LUNS)
HDS 9200 (Legacy Array)
Sun Fire v880
Brocade 4100's (2 Fabrics)
QLogic 2GB Cards (375-3102) to new SAN
JNI FCE2-6412 cards to old HDS 9200 Array
MPXIO enabled and configured for Round Robin
VxVM 4.1
Oracle 9i
During this phased implementation, we are in the date migration stage.
We are mirroring the old storage, which is from a HDS 9200 to the new
LUNS on the TS (9990).
Once the mirroring is complete, we will break of the plexes from the
old array and be fully migrated to the new Hitachi.
The customer decided not to break the mirrors yet. We have noticed a
decrease in write and read performance on all the volumes on the host.
I would expect a slight decrease in write performance, however, we are
seeing upto a 1/5 milli-second increase in read time as well on each of
the volumes. My assumption is that because of the double writes to two
different (types) of LUNS, that is impacting our reads.
Suggestions?
| |
| Faeandar 2005-10-24, 9:44 am |
| On 17 Oct 2005 13:42:10 -0700, "Greg Brown" <gregnbrown@hotmail.com>
wrote:
>Hello,
>
>Just a quick info gathering. I am at a customer site installing a new
>HDS 9990. The high level config overview:
>
>HDS 9990 (Open V 40GB LUNS)
>HDS 9200 (Legacy Array)
>Sun Fire v880
>Brocade 4100's (2 Fabrics)
>QLogic 2GB Cards (375-3102) to new SAN
>JNI FCE2-6412 cards to old HDS 9200 Array
>MPXIO enabled and configured for Round Robin
>VxVM 4.1
>Oracle 9i
>
>During this phased implementation, we are in the date migration stage.
>We are mirroring the old storage, which is from a HDS 9200 to the new
>LUNS on the TS (9990).
>Once the mirroring is complete, we will break of the plexes from the
>old array and be fully migrated to the new Hitachi.
>
>The customer decided not to break the mirrors yet. We have noticed a
>decrease in write and read performance on all the volumes on the host.
>I would expect a slight decrease in write performance, however, we are
>seeing upto a 1/5 milli-second increase in read time as well on each of
>the volumes. My assumption is that because of the double writes to two
>different (types) of LUNS, that is impacting our reads.
>
>Suggestions?
What do you mean "types of LUN's"? Is one raid 5 and the other 1+0?
For 1/5 millisecond (isn't that like 20 microseconds?) could it be the
switch? A V880 will throw out alot of data.
It could also be cache getting in your way. If the algorithm is not
suited to your access you could be spending time seeking on spindles
and loading into cache data that doesn't matter to you.
Also, doesn't the Lightning line have an Oracle verification built in
somewhere? Like firmware or something. If so, and it's enabled, it
could be introducing latency.
~F
| |
| victor.engle@gmail.com 2005-10-24, 9:44 am |
| The thing that comes to mind for me is the fact that you are migrating
from an old technology midrange array to a very highly scalable new
technology enterprise array. I imagine this means that the workload
previously supported by the 9200 is relatively low and probably not
enough to even register on the 9990. The 9990 should be capable of
supporting many times the workload of the 9200.
What tools are you using to measure disk I/O performance? Do you have
any tool to measure array performance? It would be interesting to know
the IO/s load and some cache utilization statistics.
Vic
| |
| Greg Brown 2005-10-24, 9:44 am |
| That's a good point. I will check into the Ora Verification today as
well as the cache and update this posting. The type of LUNS are both
RAID-5, however the LUNS from the 9200 are spread across multiple
columns (6-8 disks per LUN) as opposed to the LDEVS from the TagmaStore
(9990) which comes from only 4 disks in the raid group. That could be
part (not all) of the issue.
Thanks for your help.
Greg
| |
| Greg Brown 2005-10-24, 9:44 am |
| Hello Vic,
I am using vxstat to gather this information on a per Veritas volume
basis. I do not have any tool that I am using to gather performance
metrics from the TagmaStore 9990.
Thanks,
Greg
| |
| Greg Brown 2005-10-24, 9:44 am |
| That's a good point. I will check into the Ora Verification today as
well as the cache and update this posting. The type of LUNS are both
RAID-5, however the LUNS from the 9200 are spread across multiple
columns (6-8 disks per LUN) as opposed to the LDEVS from the TagmaStore
(9990) which comes from only 4 disks in the raid group. That could be
part (not all) of the issue.
Thanks for your help.
Greg
| |
| Greg Brown 2005-10-24, 9:44 am |
| That's a good point. I will check into the Ora Verification today as
well as the cache and update this posting. The type of LUNS are both
RAID-5, however the LUNS from the 9200 are spread across multiple
columns (6-8 disks per LUN) as opposed to the LDEVS from the TagmaStore
(9990) which comes from only 4 disks in the raid group. That could be
part (not all) of the issue.
Thanks for your help.
Greg
| |
| Greg Brown 2005-10-24, 9:44 am |
| Hello Faeandar,
That's a good point. I will check into the Ora Verification today as
well as the cache and update this posting. The type of LUNS are both
RAID-5, however the LUNS from the 9200 are spread across multiple
columns (6-8 disks per LUN) as opposed to the LDEVS from the TagmaStore
(9990) which comes from only 4 disks in the raid group. That could be
part (not all) of the issue.
Thanks for your help.
Greg
| |
| Greg Brown 2005-10-24, 9:44 am |
| Something also to note is my Queue Depth setting we set on the host.
Currently I set the Queue Depth to 20 (/etc/system file set
ssd:ssd_max_throttle=20, rebooted of course). I calculated this number
by using the formula for the Queue Depth per fiber port on the
TagmaStore 9990, which is 1024, divided by the total number of LUNS
mapped to that port on the FED (Front End Director). On the Thunder
(9200) it is my understanding that the Queue Depth per fiber port is
256, so before our changes, it was set to: sd:sd_max_throttle setting
was 8. I am not sure why we decided to change the sd driver in addition
adding the ssd setting. The 9200 uses the sd driver and the 9990 uses
the ssd driver. What I would like to do is set the sd:sd_max_throttle
setting back to 8 and leave the ssd setting at 20 and see if this
improves performance. I think what we are seeing is that changing the
sd parameter, we are over queuing the JNI HBA's for the 9200 and that
is causing out read & write latency. This is one avenue I am exploring
at the moment.
Greg
| |
| Greg Brown 2005-10-24, 9:44 am |
| Something also to note is my Queue Depth setting we set on the host.
Currently I set the Queue Depth to 20 (/etc/system file set
ssd:ssd_max_throttle=20, rebooted of course). I calculated this number
by using the formula for the Queue Depth per fiber port on the
TagmaStore 9990, which is 1024, divided by the total number of LUNS
mapped to that port on the FED (Front End Director). In this case: 1024
Queue Depth for the Fiber port divided by 58 total LUNS mapped to ports
7A & 8A which gives us 17.65 rounded up to 20. On the Thunder
(9200) it is my understanding that the Queue Depth per fiber port is
256, so before our changes, it was set to: sd:sd_max_throttle setting
was 8. I am not sure why we decided to change the sd driver in addition
adding the ssd setting. The 9200 uses the sd driver and the 9990 uses
the ssd driver. What I would like to do is set the sd:sd_max_throttle
setting back to 8 and leave the ssd setting at 20 and see if this
improves performance. I think what we are seeing is that changing the
sd parameter, we are over queuing the JNI HBA's for the 9200 and that
is causing out read & write latency. This is one avenue I am exploring
at the moment.
Greg
| |
| victor.engle@gmail.com 2005-10-24, 9:44 am |
| This paragraph is from a Qlogic HBA doc but I believe it applies also
in your case because it talks about the effect of rejected scsi
commands on the sd driver. If you could view the current kernel
parameters you might be able to confirm that sd_max_throttle has been
reduced to 1.
sd_max_throttle
The sd_max_throttle variable is the maximum number of commands
that the SCSI sd driver will attempt to queue to the HBA
driver(qla2100). The default value is 256. This variable should
be set to a value less than or equal to the maximum queue depth of
each lun connected to each instance of the sd driver. If this is
not done, then commands may be rejected because of a full queue
condition and the sd driver instance that receives the queue full
message will throttle down sd_max_throttle to 1. This will
result in degraded performance.
Greg Brown wrote:
> Something also to note is my Queue Depth setting we set on the host.
> Currently I set the Queue Depth to 20 (/etc/system file set
> ssd:ssd_max_throttle=20, rebooted of course). I calculated this number
> by using the formula for the Queue Depth per fiber port on the
> TagmaStore 9990, which is 1024, divided by the total number of LUNS
> mapped to that port on the FED (Front End Director). In this case: 1024
> Queue Depth for the Fiber port divided by 58 total LUNS mapped to ports
> 7A & 8A which gives us 17.65 rounded up to 20. On the Thunder
> (9200) it is my understanding that the Queue Depth per fiber port is
> 256, so before our changes, it was set to: sd:sd_max_throttle setting
> was 8. I am not sure why we decided to change the sd driver in addition
> adding the ssd setting. The 9200 uses the sd driver and the 9990 uses
> the ssd driver. What I would like to do is set the sd:sd_max_throttle
> setting back to 8 and leave the ssd setting at 20 and see if this
> improves performance. I think what we are seeing is that changing the
> sd parameter, we are over queuing the JNI HBA's for the 9200 and that
> is causing out read & write latency. This is one avenue I am exploring
> at the moment.
>
> Greg
| |
| Faeandar 2005-10-24, 9:44 am |
| On 18 Oct 2005 06:49:51 -0700, "Greg Brown" <gregnbrown@hotmail.com>
wrote:
>That's a good point. I will check into the Ora Verification today as
>well as the cache and update this posting. The type of LUNS are both
>RAID-5, however the LUNS from the 9200 are spread across multiple
>columns (6-8 disks per LUN) as opposed to the LDEVS from the TagmaStore
>(9990) which comes from only 4 disks in the raid group. That could be
>part (not all) of the issue.
>
>Thanks for your help.
>
>Greg
It could be all of the issue. 1/5 of a millisecond? That's a very
small time, and could easily be explained by the loss of 2 drives in a
stripe.
~F
| |
| victor.engle@gmail.com 2005-10-24, 9:44 am |
| Greg,
What did you learn regarding the Oracle verification question? Also
what kind of drives are you using in both arrays?
Thanks,
Vic
| |
|
|
| Greg Brown 2005-10-24, 9:44 am |
|
victor.engle@gmail.com wrote:
> Greg,
>
> What did you learn regarding the Oracle verification question? Also
> what kind of drives are you using in both arrays?
>
> Thanks,
> Vic
Thanks Victor,
Regarding the Ora verification, we do not have that enabled. The drives
are a mixture of the Seagate 72gb, and Fujitsu 72gb. I have proposed
this action plan moving forward; We will break the mirrors off of two
volumes on one of there systems and then monitor the I/O off that plex
(attached to the new array). Tommorrow we are then going to compare
that data with the data collected when the two arrays were mirrored for
that particular volume. This will help us eliminate if it is a queue
depth issue for the sd driver on the old array (the 9990 uses the ssd
driver and if it is an issue between mpxio and dmp. I will keep you
posted. Thanks again for all your help and thanks for the two links.
Greg
| |
| victor.engle@gmail.com 2005-10-24, 9:44 am |
| Greg,
Something else occurred to me. Reading your original post I noticed you
said you were doing round-robin load balancing. I think this can cause
problems with the TagmaStor. At least that is what we were told by HP
storage engineering regarding the XP12000 which is a rebranded
TagmaStor. The problem they say is that if a host comes in on CL1-A for
a read, the CHIP processors for CL1-A own the IO and manage the stripe
of data read from disk into cache. Then if the next IO from the host
comes in on CL2-A looking for data blocks in the stripe previously read
through CL1-A there is overhead with the channel processors that may
result in some latency.
We were using VxVM "Balanced Path" which is a bit better than
round-robin and ended up switching to "Preferred Path" and manually
balanced each lun in our 3TB volumes by alternating the preferred path.
Something to consider.
Vic
|
|
|
|
|