07-29-04 10:45 PM
Hello
I am having a peculiar problem with my storage and am using iostat to
try identify the bottleneck. Situation: A Server E450 (4 CPUs, Solaris
7) is connectet to a tape library StorageTek L20 (with 2 LTO drives)
and to a disk storage Dell/EMC CX400 (10 HDs in Raid 5) via fibre
channel (2Gbit/s). A HSM Software (samfs) monitors a file system on
the Raid and archives files to tape automatically. At the same time,
other servers write to the same LUN through the network (1 GB) under
Veritas control.
So the file system behaves like a buffer to the tape device.
The issue is that transfer rates of both devices slow down
periodically and are generally to low.
I don't understand why the service time of the disks rises to
astronomical values of several seconds, why there are never more that
20 outstanding write request on the disks, why the device is 100% busy
when no data transfer is taking place...
Obviously, if one of the ten harddisks in the Raid is specified to be
able to transfer 30 MB/s and we have 10 (effectively 9) of them, then
an average of 4,5 MB/s is miserable low. The files written to tape at
any point in time are NOT the same files being written to the
harddisks. The tape drive is not waiting for files to write, and it is
able to write at >20MB/s natively.
I do have one suspicion: Both devices are connected to a double port
HBA (Emulex 9820DC, driver V. 6.0g). Could be both channels are
interfering with each other? But I don't know how to investigate that.
Thanks in advance
Rick Denoire
iostat -xM 5 (output im MByte, every 5 seconds)
st383 = tape drive
sd915 = Raid 5 LUN
extended device statistics
device r/s w/s Mr/s Mw/s wait actv svc_t %w %b
sd915 35.2 17.9 1.9 0.6 5.9 2.1 149.7 9 17
st383 0.1 1.9 0.0 0.5 0.0 0.0 12.7 0 2
extended device statistics
device r/s w/s Mr/s Mw/s wait actv svc_t %w %b
sd915 125.0 131.4 4.0 4.0 61.4 18.9 312.9 89 100
st383 0.0 16.0 0.0 4.0 0.0 0.2 9.4 0 15
extended device statistics
device r/s w/s Mr/s Mw/s wait actv svc_t %w %b
sd915 123.4 100.2 3.8 3.1 70.5 19.0 400.4 91 100
st383 0.0 12.8 0.0 3.2 0.0 0.1 9.3 0 12
extended device statistics
device r/s w/s Mr/s Mw/s wait actv svc_t %w %b
sd915 110.4 207.2 3.5 6.3 68.8 17.8 272.9 82 100
st383 0.0 16.0 0.0 4.0 0.0 0.1 9.2 0 15
extended device statistics
device r/s w/s Mr/s Mw/s wait actv svc_t %w %b
sd915 79.8 126.2 2.5 3.8 68.6 18.7 423.8 89 100
st383 0.0 9.6 0.0 2.4 0.0 0.1 9.5 0 9
extended device statistics
device r/s w/s Mr/s Mw/s wait actv svc_t %w %b
sd915 86.8 162.2 2.7 4.9 68.1 19.4 351.1 91 100
st383 0.0 9.6 0.0 2.4 0.0 0.1 9.2 0 9
extended device statistics
device r/s w/s Mr/s Mw/s wait actv svc_t %w %b
sd915 107.0 280.0 3.6 8.5 47.8 17.1 167.6 76 100
st383 0.0 16.0 0.0 4.0 0.0 0.1 9.3 0 15
extended device statistics
device r/s w/s Mr/s Mw/s wait actv svc_t %w %b
sd915 25.4 109.8 0.9 3.3 66.3 18.9 630.4 89 100
st383 0.0 3.2 0.0 0.8 0.0 0.0 9.2 0 3
extended device statistics
device r/s w/s Mr/s Mw/s wait actv svc_t %w %b
sd915 54.4 113.6 2.2 3.5 40.0 18.5 348.0 90 100
st383 0.0 9.2 0.0 2.3 0.0 0.1 9.3 0 9
extended device statistics
device r/s w/s Mr/s Mw/s wait actv svc_t %w %b
sd915 66.2 233.4 2.6 7.1 54.2 17.9 240.7 81 100
st383 0.0 10.0 0.0 2.5 0.0 0.1 9.4 0 9
extended device statistics
device r/s w/s Mr/s Mw/s wait actv svc_t %w %b
sd915 95.4 92.2 3.0 2.7 68.4 19.4 467.9 93 100
st383 0.0 12.8 0.0 3.2 0.0 0.1 9.2 0 12
extended device statistics
device r/s w/s Mr/s Mw/s wait actv svc_t %w %b
sd915 375.4 292.8 12.1 9.0 59.6 16.3 113.6 76 99
st383 0.0 48.0 0.0 12.0 0.0 0.5 10.1 0 48
extended device statistics
device r/s w/s Mr/s Mw/s wait actv svc_t %w %b
sd915 21.6 20.8 0.7 0.6 62.1 19.2 1916.5 92 100
st383 0.0 3.2 0.0 0.8 0.0 0.0 9.2 0 3
extended device statistics
device r/s w/s Mr/s Mw/s wait actv svc_t %w %b
sd915 331.2 269.8 10.5 8.2 52.6 16.7 115.3 78 100
st383 0.0 41.6 0.0 10.4 0.0 0.4 10.5 0 44
extended device statistics
device r/s w/s Mr/s Mw/s wait actv svc_t %w %b
sd915 21.6 26.8 0.7 0.8 58.1 18.4 1579.8 87 100
st383 0.0 3.2 0.0 0.8 0.0 0.0 9.3 0 3
extended device statistics
device r/s w/s Mr/s Mw/s wait actv svc_t %w %b
sd915 156.6 177.8 4.8 5.3 64.7 18.4 248.5 86 100
st383 0.0 19.2 0.0 4.8 0.0 0.2 9.5 0 18
extended device statistics
device r/s w/s Mr/s Mw/s wait actv svc_t %w %b
sd915 118.6 78.8 3.7 2.4 72.0 19.5 463.5 94 100
st383 0.0 12.8 0.0 3.2 0.0 0.1 9.3 0 12
extended device statistics
device r/s w/s Mr/s Mw/s wait actv svc_t %w %b
sd915 414.0 304.8 13.2 9.3 55.2 16.2 99.4 75 99
st383 0.0 54.4 0.0 13.6 0.0 0.5 10.0 0 54
extended device statistics
device r/s w/s Mr/s Mw/s wait actv svc_t %w %b
sd915 16.0 0.0 0.5 0.0 72.4 20.0 5778.1 100 100
st383 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0
extended device statistics
device r/s w/s Mr/s Mw/s wait actv svc_t %w %b
sd915 84.8 174.4 2.7 5.4 55.6 18.3 284.9 84 100
st383 0.0 12.8 0.0 3.2 0.0 0.1 9.5 0 12
extended device statistics
device r/s w/s Mr/s Mw/s wait actv svc_t %w %b
sd915 98.4 160.6 3.1 4.9 67.6 19.0 334.4 91 100
st383 0.0 12.8 0.0 3.2 0.0 0.1 9.3 0 12
extended device statistics
device r/s w/s Mr/s Mw/s wait actv svc_t %w %b
sd915 110.6 137.6 3.5 4.1 71.1 18.5 361.3 89 100
st383 0.0 12.8 0.0 3.2 0.0 0.1 9.5 0 12
extended device statistics
device r/s w/s Mr/s Mw/s wait actv svc_t %w %b
sd915 216.2 150.8 6.9 4.6 63.6 18.7 224.2 88 100
st383 0.0 24.8 0.0 6.2 0.0 0.2 9.6 0 24
extended device statistics
device r/s w/s Mr/s Mw/s wait actv svc_t %w %b
sd915 254.6 219.8 8.1 6.7 72.8 17.2 189.7 82 100
st383 0.0 36.0 0.0 9.0 0.0 0.4 10.2 0 37
extended device statistics
device r/s w/s Mr/s Mw/s wait actv svc_t %w %b
sd915 26.0 113.8 0.8 3.5 66.2 19.2 610.5 90 100
st383 0.0 3.2 0.0 0.8 0.0 0.0 9.3 0 3
extended device statistics
device r/s w/s Mr/s Mw/s wait actv svc_t %w %b
sd915 81.0 414.4 2.5 12.5 33.7 12.4 93.0 54 99
st383 0.0 10.6 0.0 2.7 0.0 0.1 9.7 0 10
extended device statistics
device r/s w/s Mr/s Mw/s wait actv svc_t %w %b
sd915 0.0 182.6 0.0 5.6 0.0 3.9 21.2 0 100
st383 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0
extended device statistics
device r/s w/s Mr/s Mw/s wait actv svc_t %w %b
sd915 0.0 59.8 0.0 1.8 0.0 1.8 30.7 0 100
st383 0.0 0.2 0.0 0.0 0.0 0.2 774.7 0 15
extended device statistics
device r/s w/s Mr/s Mw/s wait actv svc_t %w %b
sd915 0.0 0.0 0.0 0.0 0.0 1.0 0.0 0 100
st383 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0
extended device statistics
device r/s w/s Mr/s Mw/s wait actv svc_t %w %b
sd915 0.0 0.2 0.0 0.0 0.0 1.1 5534.6 0 100
st383 0.0 0.0 0.0 0.0 0.0 0.1 0.0 0 10
extended device statistics
device r/s w/s Mr/s Mw/s wait actv svc_t %w %b
sd915 0.0 0.0 0.0 0.0 0.0 13.8 0.0 0 100
st383 0.2 0.0 0.0 0.0 0.0 0.0 73.3 0 1
extended device statistics
device r/s w/s Mr/s Mw/s wait actv svc_t %w %b
sd915 40.4 688.2 0.8 21.1 1.2 5.4 9.1 2 91
st383 0.0 0.0 0.0 0.0 0.0 0.7 0.0 0 70
extended device statistics
device r/s w/s Mr/s Mw/s wait actv svc_t %w %b
sd915 0.0 234.2 0.0 7.2 0.0 3.7 15.6 0 98
st383 0.0 0.0 0.0 0.0 0.0 1.0 0.0 0 100
extended device statistics
device r/s w/s Mr/s Mw/s wait actv svc_t %w %b
sd915 0.0 137.0 0.0 3.8 0.0 2.7 20.0 0 100
st383 0.0 0.0 0.0 0.0 0.0 1.0 0.0 0 100
extended device statistics
device r/s w/s Mr/s Mw/s wait actv svc_t %w %b
sd915 0.0 0.0 0.0 0.0 66.5 13.2 0.0 64 100
st383 0.0 3.2 0.0 0.8 0.0 0.5 149.3 0 48
extended device statistics
device r/s w/s Mr/s Mw/s wait actv svc_t %w %b
sd915 0.0 0.0 0.0 0.0 104.0 20.0 0.0 100 100
st383 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0
extended device statistics
device r/s w/s Mr/s Mw/s wait actv svc_t %w %b
sd915 0.0 0.0 0.0 0.0 104.0 20.0 0.0 100 100
st383 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0
extended device statistics
device r/s w/s Mr/s Mw/s wait actv svc_t %w %b
extended device statistics
device r/s w/s Mr/s Mw/s wait actv svc_t %w %b
sd915 35.2 18.0 1.9 0.6 5.9 2.1 150.1 9 17
st383 0.1 1.9 0.0 0.5 0.0 0.0 12.7 0 2
extended device statistics
device r/s w/s Mr/s Mw/s wait actv svc_t %w %b
sd915 393.8 521.6 12.4 15.8 16.0 15.0 33.9 35 99
st383 0.0 50.4 0.0 12.6 0.0 0.6 11.4 0 57
extended device statistics
device r/s w/s Mr/s Mw/s wait actv svc_t %w %b
sd915 3.2 33.8 0.1 1.0 80.4 19.9 2709.6 69 100
st383 0.0 3.2 0.0 0.8 0.0 0.0 9.2 0 3
extended device statistics
device r/s w/s Mr/s Mw/s wait actv svc_t %w %b
sd915 0.0 0.0 0.0 0.0 125.0 20.0 0.0 100 100
st383 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0
extended device statistics
device r/s w/s Mr/s Mw/s wait actv svc_t %w %b
sd915 337.4 270.6 10.6 8.2 58.8 16.7 124.2 79 99
st383 0.0 41.6 0.0 10.4 0.0 0.4 10.2 0 42
extended device statistics
device r/s w/s Mr/s Mw/s wait actv svc_t %w %b
sd915 22.8 75.6 0.7 2.3 60.7 19.5 815.1 92 100
st383 0.0 3.2 0.0 0.8 0.0 0.0 9.2 0 3
extended device statistics
device r/s w/s Mr/s Mw/s wait actv svc_t %w %b
sd915 353.6 294.0 11.2 8.9 54.7 16.7 110.3 76 100
st383 0.0 44.8 0.0 11.2 0.0 0.5 10.2 0 45
extended device statistics
device r/s w/s Mr/s Mw/s wait actv svc_t %w %b
sd915 24.2 21.2 0.8 0.6 78.8 19.9 2172.4 99 100
st383 0.0 3.2 0.0 0.8 0.0 0.0 9.2 0 3
extended device statistics
device r/s w/s Mr/s Mw/s wait actv svc_t %w %b
sd915 113.0 97.2 3.6 3.0 62.2 19.0 386.1 91 100
st383 0.0 12.8 0.0 3.2 0.0 0.1 9.2 0 12
extended device statistics
device r/s w/s Mr/s Mw/s wait actv svc_t %w %b
sd915 56.4 180.4 1.8 5.5 65.9 18.7 357.2 86 100
st383 0.0 6.4 0.0 1.6 0.0 0.1 9.6 0 6
extended device statistics
device r/s w/s Mr/s Mw/s wait actv svc_t %w %b
sd915 234.2 246.6 7.3 7.4 59.6 17.8 161.1 84 100
st383 0.0 28.6 0.0 7.1 0.0 0.3 9.7 0 28
extended device statistics
device r/s w/s Mr/s Mw/s wait actv svc_t %w %b
sd915 186.0 141.4 5.9 4.3 74.9 18.1 284.1 88 100
st383 0.0 25.8 0.0 6.5 0.0 0.3 10.0 0 26
extended device statistics
device r/s w/s Mr/s Mw/s wait actv svc_t %w %b
sd915 29.4 97.8 0.9 3.0 62.5 19.0 640.8 91 100
st383 0.0 3.2 0.0 0.8 0.0 0.0 9.2 0 3
extended device statistics
device r/s w/s Mr/s Mw/s wait actv svc_t %w %b
sd915 324.8 290.2 10.2 8.9 51.8 15.9 110.1 74 99
st383 0.0 41.6 0.0 10.4 0.0 0.5 10.9 0 45
extended device statistics
device r/s w/s Mr/s Mw/s wait actv svc_t %w %b
sd915 20.0 38.4 0.6 1.2 49.6 19.8 1188.9 97 100
st383 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0
extended device statistics
device r/s w/s Mr/s Mw/s wait actv svc_t %w %b
sd915 7.0 305.2 0.2 9.3 60.0 16.5 245.2 68 100
st383 0.0 3.2 0.0 0.8 0.0 0.0 10.0 0 3
extended device statistics
device r/s w/s Mr/s Mw/s wait actv svc_t %w %b
sd915 17.6 79.4 0.6 2.3 60.7 19.0 822.0 92 100
st383 0.0 3.2 0.0 0.8 0.0 0.0 9.2 0 3
extended device statistics
device r/s w/s Mr/s Mw/s wait actv svc_t %w %b
sd915 263.0 154.4 8.5 4.7 57.7 18.3 182.1 88 100
st383 0.0 30.0 0.0 7.5 0.0 0.3 9.9 0 30
extended device statistics
device r/s w/s Mr/s Mw/s wait actv svc_t %w %b
sd915 168.6 150.4 5.4 4.6 76.4 18.1 296.2 88 100
st383 0.0 24.4 0.0 6.1 0.0 0.2 10.1 0 25
extended device statistics
device r/s w/s Mr/s Mw/s wait actv svc_t %w %b
sd915 25.8 119.6 0.8 3.7 64.6 19.1 576.1 90 100
st383 0.0 3.2 0.0 0.8 0.0 0.0 9.3 0 3
extended device statistics
device r/s w/s Mr/s Mw/s wait actv svc_t %w %b
sd915 110.2 117.4 3.4 3.6 65.4 18.9 370.1 91 100
st383 0.0 12.8 0.0 3.2 0.0 0.1 9.2 0 12
extended device statistics
device r/s w/s Mr/s Mw/s wait actv svc_t %w %b
sd915 264.0 359.6 8.3 11.0 48.2 15.7 102.5 69 98
st383 0.0 30.8 0.0 7.7 0.0 0.4 12.4 0 38
extended device statistics
[ Post a follow-up to this message ]
| 
