|
Home > Archive > Data Storage > October 2004 > Bad disk IO performance problem during reads and writes
You are viewing an archived Text-only version of the thread.
To view this thread in it's original format and/or if you want to reply to
this thread please [click here]
| Author |
Bad disk IO performance problem during reads and writes
|
|
| Beeblebrox 2004-10-19, 5:45 pm |
| I'm using V480s with A1000s as storage using Veritas DB edition,
solaris 8 running oracle.
We're seeing poor IO performance and I'm having trouble pinpointing
the problem. IO looks fine when it's mostly writes, but tanks the
moment reads increase:
Lookin' good:
r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 c2t5d0
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 c5t4d0
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 c0t0d0
5.1 124.5 52.0 1154.4 0.2 0.8 1.5 6.3 2 23 c2t5d1
5.7 120.7 54.3 1124.0 0.4 0.8 2.9 6.4 2 22 c2t5d2
5.0 131.9 51.2 1212.8 0.2 0.9 1.4 6.3 2 24 c2t5d3
5.1 128.8 52.0 1188.8 0.4 0.9 2.8 6.4 3 24 c2t5d4
6.2 127.4 65.6 1177.6 0.5 1.0 3.6 7.7 3 27 c5t4d1
4.9 128.0 44.9 1180.2 0.2 1.0 1.4 7.5 2 26 c5t4d2
5.9 122.8 52.8 1139.2 0.3 0.9 2.4 7.1 3 26 c5t4d3
4.9 124.1 44.8 1151.2 0.2 0.8 1.6 6.5 2 24 c5t4d4
Crap-o-rific:
r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device
0.4 0.0 0.2 0.0 0.0 0.0 0.0 0.0 0 0 c2t5d0
0.4 0.0 0.2 0.0 0.0 0.0 0.0 0.0 0 0 c5t4d0
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 c0t0d0
47.8 5.4 531.3 172.8 0.9 4.9 17.6 92.5 16 51 c2t5d1
51.8 5.4 559.3 172.8 1.4 5.3 24.1 93.3 17 54 c2t5d2
48.1 5.8 532.9 176.0 0.9 4.7 16.1 87.0 16 52 c2t5d3
46.6 5.4 515.9 172.8 0.5 4.4 9.7 84.4 10 50 c2t5d4
51.3 6.0 545.7 182.4 2.5 8.3 43.2 144.8 32 84 c5t4d1
49.6 5.7 535.3 179.3 1.8 7.9 32.2 142.0 30 79 c5t4d2
52.5 6.2 560.9 183.2 1.9 6.9 32.2 117.1 26 74 c5t4d3
50.0 5.6 552.1 179.2 2.5 7.3 44.3 131.8 27 78 c5t4d4
The examples above are from the standby, and we see the same IO
profile on the primary.
I've checked write-cache, prefetch, read-ahead. I've looked for HW
problems, too.
Neither of these hosts have been touched for months -- I'm very
certain of that.
Thoughts appreciated.
| |
| Michael Vilain 2004-10-19, 5:45 pm |
| In article <c976cb73.0410190928.53bc2f3@posting.google.com>,
sl1433@hotmail.com (Beeblebrox) wrote:
> I'm using V480s with A1000s as storage using Veritas DB edition,
> solaris 8 running oracle.
>
> We're seeing poor IO performance and I'm having trouble pinpointing
> the problem. IO looks fine when it's mostly writes, but tanks the
> moment reads increase:
>
>
>
> Lookin' good:
>
> r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device
> 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 c2t5d0
> 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 c5t4d0
> 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 c0t0d0
> 5.1 124.5 52.0 1154.4 0.2 0.8 1.5 6.3 2 23 c2t5d1
> 5.7 120.7 54.3 1124.0 0.4 0.8 2.9 6.4 2 22 c2t5d2
> 5.0 131.9 51.2 1212.8 0.2 0.9 1.4 6.3 2 24 c2t5d3
> 5.1 128.8 52.0 1188.8 0.4 0.9 2.8 6.4 3 24 c2t5d4
> 6.2 127.4 65.6 1177.6 0.5 1.0 3.6 7.7 3 27 c5t4d1
> 4.9 128.0 44.9 1180.2 0.2 1.0 1.4 7.5 2 26 c5t4d2
> 5.9 122.8 52.8 1139.2 0.3 0.9 2.4 7.1 3 26 c5t4d3
> 4.9 124.1 44.8 1151.2 0.2 0.8 1.6 6.5 2 24 c5t4d4
>
>
> Crap-o-rific:
>
> r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device
> 0.4 0.0 0.2 0.0 0.0 0.0 0.0 0.0 0 0 c2t5d0
> 0.4 0.0 0.2 0.0 0.0 0.0 0.0 0.0 0 0 c5t4d0
> 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 c0t0d0
> 47.8 5.4 531.3 172.8 0.9 4.9 17.6 92.5 16 51 c2t5d1
> 51.8 5.4 559.3 172.8 1.4 5.3 24.1 93.3 17 54 c2t5d2
> 48.1 5.8 532.9 176.0 0.9 4.7 16.1 87.0 16 52 c2t5d3
> 46.6 5.4 515.9 172.8 0.5 4.4 9.7 84.4 10 50 c2t5d4
> 51.3 6.0 545.7 182.4 2.5 8.3 43.2 144.8 32 84 c5t4d1
> 49.6 5.7 535.3 179.3 1.8 7.9 32.2 142.0 30 79 c5t4d2
> 52.5 6.2 560.9 183.2 1.9 6.9 32.2 117.1 26 74 c5t4d3
> 50.0 5.6 552.1 179.2 2.5 7.3 44.3 131.8 27 78 c5t4d4
>
>
> The examples above are from the standby, and we see the same IO
> profile on the primary.
>
> I've checked write-cache, prefetch, read-ahead. I've looked for HW
> problems, too.
>
> Neither of these hosts have been touched for months -- I'm very
> certain of that.
>
>
> Thoughts appreciated.
What are the volumes in Veritas? You've probably been told that RAID5
is a Bad Idea(tm) for write-heavy applications.
--
DeeDee, don't press that button! DeeDee! NO! Dee...
|
|
|
|
|