【大话IT】服务器IO瓶颈，原始是什么啊？

显示全部楼层 · 2015-7-16 14:18:24

有6台服务器（IBM），全部接在一台存储上（IBM）。现在6台服务器上接到存储的分区上的IO都到了或接近100%。
下面是其中一台服务器状况。帮忙看看是硬件IO性能差问题，还是应用的问题？谢谢！！！
IOTOP--------------------------------------------------------Total DISK READ: 0.00 B/s | Total DISK WRITE: 17.01 M/sTIDPRIOUSER DISK READDISK WRITESWAPIN IO>COMMAND
20919 be/4 root0.00 B/s100.85 K/s0.00 % 93.67 % [flush-253:2]16377 be/4 timesten0.00 B/s119.53 K/s0.00 % 93.46 % timestensubd -verbose -userlog tterrors.log -supportlog ttmesg.log -id 1000003 -facility user16374 be/4 timesten0.00 B/s0.00 B/s0.00 % 38.52 % timestensubd -verbose -userlog tterrors.log -supportlog ttmesg.log -id 1000003 -facility user 2486 be/3 root0.00 B/s0.00 B/s0.00 % 20.88 % [jbd2/dm-2-8]29362 be/4 root0.00 B/s3.74 K/s0.00 %0.00 % java -Djava.util.logging.config.file=/opt/tompolicyweb/conf/logging.~r=/opt/tompolicyweb/temp org.apache.catalina.startup.Bootstrap start29365 be/4 root0.00 B/s7.47 K/s0.00 %0.00 % java -Djava.util.TOP--------------------------------------------------------top - 14:46:20 up 80 days, 36 min,4 users,load average: 5.14, 6.16, 6.28Tasks: 349 total, 1 running, 348 sleeping, 0 stopped, 0 zombieCpu(s): 17.2%us,0.6%sy,0.0%ni, 70.2%id, 11.8%wa,0.0%hi,0.3%si,0.0%stMem:99054872k total, 97538640k used,1516232k free, 165656k buffersSwap: 10239996k total,2801584k used,7438412k free, 80852504k cachedPID USERPRNIVIRTRESSHR S %CPU %MEMTIME+COMMAND
11941 root20 0 80.2g13g11g S 101.3 14.7 2013:13 java
3912 root20 0 81.5g10g 8.8g S 95.0 11.5 6575:22 java
2749 root20 0 79.3g34g32g S 55.1 36.2 3928:56 java
26729 timesten20 0 69.6g20g19g S 32.9 21.7 4258:25 timestenrepd
16020 timesten20 0 68.8g33g33g S3.3 35.0 1107:39 timestensubd
IOSTAT--------------------------------------------------------Linux 2.6.32-504.el6.x86_64 07/02/2015_x86_64_(16 CPU)Device: rrqm/s wrqm/s r/s w/srkB/swkB/s avgrq-sz avgqu-sz awaitsvctm%utilsda
0.0638.120.073.04 3.01 164.64 107.97 0.18 57.27 3.79 1.18sdb
0.001903.380.20 75.2113.817888.39 209.59 0.354.68 3.1523.76dm-0
0.00 0.000.08 40.85 2.80 163.40 8.12 0.174.14 0.29 1.17dm-1
0.00 0.000.050.31 0.21 1.24 8.00 0.01 17.74 0.33 0.01dm-2
0.00 0.000.20 1972.1013.817888.40 8.01 0.350.14 0.1223.76sdc
0.00 0.000.000.00 0.00 0.00 8.92 0.00551.04 550.42 0.01Device: rrqm/s wrqm/s r/s w/srkB/swkB/s avgrq-sz avgqu-sz awaitsvctm%utilsda
0.0033.000.004.00 0.00 148.0074.00 0.05 13.25 7.50 3.00sdb
0.00 10215.000.00102.00 0.00 27172.00 532.78 148.68 2063.45 9.81 100.10dm-0
0.00 0.000.00 37.00 0.00 148.00 8.00 0.287.68 0.81 3.00dm-1
0.00 0.000.000.00 0.00 0.00 0.00 0.000.00 0.00 0.00dm-2
0.00 0.000.00 10358.00 0.00 41432.00 8.009294.36 1126.79 0.10 100.10sdc
0.00 0.000.000.00 0.00 0.00 0.00 0.000.00 0.00 0.00复制代码

千问 · 2015-7-16 14:18:24

sda sdb是内置盘吧？呵呵

千问 · 2015-7-16 14:18:24

wolfop 发表于 2015-7-2 20:28
sda sdb是内置盘吧？呵呵
都是挂的存储
[root@zhengcetongbu ~]# pvscan
/dev/sdc: read failed after 0 of 4096 at 0: Input/output error
/dev/sdc: read failed after 0 of 4096 at 322122481664: Input/output error
/dev/sdc: read failed after 0 of 4096 at 322122539008: Input/output error
/dev/sdc: read failed after 0 of 4096 at 4096: Input/output error
PV /dev/sdb1 VG zhengcetongbu lvm2 [299.99 GiB / 1016.00 MiB free]
PV /dev/sda3 VG VolGrouplvm2 [278.07 GiB / 0free]
Total: 2 [578.06 GiB] / in use: 2 [578.06 GiB] / in no VG: 0 [0 ]
[root@zhengcetongbu ~]# df -h
df: `/media/cdrom': Input/output error
Filesystem
SizeUsed Avail Use% Mounted on
/dev/mapper/VolGroup-gen

264G 57G194G23% /
tmpfs
48G 0 48G 0% /dev/shm
/dev/sda2
190M 32M149M18% /boot
/dev/sda1
200M260K200M 1% /boot/efi
/dev/mapper/zhengcetongbu-zhengcetongbu

295G139G142G50% /opt/TimesTen

千问 · 2015-7-16 14:18:24

首先确认光纤线缆的连接方式没有问题，光纤交换机的ZONE划分没有问题，
然后到这个上面去对比兼容性列表吧
http://www-03.ibm.com/systems/su ... nteroperability.wss

千问 · 2015-7-16 14:18:24

本帖最后由 wolfop 于 2015-7-3 16:31 编辑
alibull 发表于 2015-7-3 00:13
都是挂的存储
[root@zhengcetongbu ~]# pvscan
/dev/sdc: read failed after 0 of 4096 at 0: Input ...lsscsi输出给我看。
timesten做check point对存储的IOPS很高的，低端基本顶不住。你的存储什么配置？

千问 · 2015-7-16 14:18:24

本帖最后由 alibull 于 2015-7-3 18:37 编辑
wolfop 发表于 2015-7-3 16:21
lsscsi输出给我看。
timesten做check point对存储的IOPS很高的，低端基本顶不住。你的存储什么配置？现在确实遇到检查点文件写不及写了。日志文件有堆积。
Command> call ttlogholds;

换了一台机器，和上面的主机一样，都有IO瓶颈，主机和存储都是IBM。这是下这台的信息，帮看下吧[root@zhengcetoufang124 ~]# df -hdf: `/media/cdrom': Input/output errorFilesystem
SizeUsed Avail Use% Mounted on/dev/mapper/VolGroup-gen
264G 46G206G18% /tmpfs
64G 80K 64G 1% /dev/shm/dev/sda2
190M 32M149M18% /boot/dev/sda1
200M260K200M 1% /boot/efi/dev/mapper/zhengcetoufang124-zhengcetoufang124
335G162G157G51% /opt/TimesTen[root@zhengcetoufang124 ~]# lsscsi[0:2:0:0]diskIBMServeRAID-MR10i1.40/dev/sda [5:0:0:0]diskIBM1746FAStT1070- [5:0:0:31] diskIBMUniversal Xport1070- [5:0:1:0]diskIBM1726-4xxFAStT0617/dev/sdb [5:0:1:1]diskIBM1726-4xxFAStT0617/dev/sdc [5:0:1:2]diskIBM1726-4xxFAStT0617/dev/sdd [5:0:1:3]diskIBM1726-4xxFAStT0617/dev/sde [5:0:1:4]diskIBM1726-4xxFAStT0617/dev/sdk [5:0:1:31] diskIBMUniversal Xport0617- [5:0:2:0]diskIBM1746FAStT1070- [5:0:2:31] diskIBMUniversal Xport1070- [5:0:3:0]diskIBM1726-4xxFAStT0617/dev/sdf [5:0:3:1]diskIBM1726-4xxFAStT0617/dev/sdg [5:0:3:2]diskIBM1726-4xxFAStT0617/dev/sdh [5:0:3:3]diskIBM1726-4xxFAStT0617/dev/sdi [5:0:3:4]diskIBM1726-4xxFAStT0617/dev/sdl [5:0:3:31] diskIBMUniversal Xport0617- [5:0:4:0]diskIBM1726-4xxFAStT0617- [5:0:5:0]diskIBM1726-4xxFAStT0617- [5:0:6:0]diskIBM1746FAStT1070- [5:0:6:31] diskIBMUniversal Xport1070- [5:0:7:0]diskIBM1746FAStT1070- [5:0:7:31] diskIBMUniversal Xport1070- [root@zhengcetoufang124 ~]# df -hdf: `/media/cdrom': Input/output errorFilesystem
SizeUsed Avail Use% Mounted on/dev/mapper/VolGroup-gen
264G 46G206G19% /tmpfs
64G 80K 64G 1% /dev/shm/dev/sda2
190M 32M149M18% /boot/dev/sda1
200M260K200M 1% /boot/efi/dev/mapper/zhengcetoufang124-zhengcetoufang124
335G162G157G51% /opt/TimesTen[root@zhengcetoufang124 ~]# pvscan /dev/sdc: read failed after 0 of 4096 at 0: Input/output error/dev/sdc: read failed after 0 of 4096 at 300647645184: Input/output error/dev/sdc: read failed after 0 of 4096 at 300647702528: Input/output error/dev/sdc: read failed after 0 of 4096 at 4096: Input/output error/dev/sdf: read failed after 0 of 4096 at 0: Input/output error/dev/sdf: read failed after 0 of 4096 at 343597318144: Input/output error/dev/sdf: read failed after 0 of 4096 at 343597375488: Input/output error/dev/sdf: read failed after 0 of 4096 at 4096: Input/output error/dev/sdh: read failed after 0 of 4096 at 0: Input/output error/dev/sdh: read failed after 0 of 4096 at 322122481664: Input/output error/dev/sdh: read failed after 0 of 4096 at 322122539008: Input/output error/dev/sdh: read failed after 0 of 4096 at 4096: Input/output error/dev/sdi: read failed after 0 of 4096 at 0: Input/output error/dev/sdi: read failed after 0 of 4096 at 257697972224: Input/output error/dev/sdi: read failed after 0 of 4096 at 257698029568: Input/output error/dev/sdi: read failed after 0 of 4096 at 4096: Input/output error/dev/sdk: read failed after 0 of 4096 at 0: Input/output error/dev/sdk: read failed after 0 of 4096 at 107374116864: Input/output error/dev/sdk: read failed after 0 of 4096 at 107374174208: Input/output error/dev/sdk: read failed after 0 of 4096 at 4096: Input/output errorPV /dev/sdg1 VG nfdshuju
lvm2 [279.99 GiB / 1016.00 MiB free]PV /dev/sde1 VG zhengcetoufang124 lvm2 [240.00 GiB / 0free]PV /dev/sdl1 VG zhengcetoufang124 lvm2 [100.00 GiB / 16.00 MiB free]PV /dev/sdd1 VG toufangshujulvm2 [299.99 GiB / 1016.00 MiB free]PV /dev/sdb1 VG czhangshuju lvm2 [320.00 GiB / 1020.00 MiB free]PV /dev/sda3 VG VolGroup
lvm2 [278.07 GiB / 0free]Total: 6 [1.48 TiB] / in use: 6 [1.48 TiB] / in no VG: 0 [0 ]复制代码

千问 · 2015-7-16 14:18:24

dong3488 发表于 2015-7-3 13:57
首先确认光纤线缆的连接方式没有问题，光纤交换机的ZONE划分没有问题，
然后到这个上面去对比兼容性列表吧 ...
恩，谢谢！

千问 · 2015-7-16 14:18:24

[root@zhengcetoufang124 ~]# iostat -kdx 1Linux 2.6.32-504.el6.x86_64 (zhengcetoufang124) 07/03/2015_x86_64_(24 CPU)Device: rrqm/s wrqm/s r/s w/srkB/swkB/s avgrq-sz avgqu-sz awaitsvctm%utilsda
0.31 316.890.54 20.7913.801350.33 127.86 0.104.74 2.68 5.73dm-0
0.00 0.000.67337.3513.011349.38 8.06 0.651.91 0.17 5.71dm-1
0.00 0.000.200.24 0.79 0.95 8.00 0.01 24.81 0.90 0.04sdb
0.00 0.000.000.00 0.00 0.00 6.59 0.00 10.15 9.99 0.00sdc
0.00 0.000.000.00 0.00 0.00 8.74 0.00517.18 517.18 0.01sdd
0.00 0.000.000.00 0.00 0.00 5.80 0.00 13.9213.83 0.00sde
0.002640.890.88 74.2219.89 10840.27 289.22 0.425.55 7.0853.15sdf
0.00 0.000.000.00 0.00 0.00 8.74 0.00516.81 516.66 0.01sdg
0.00 0.000.000.00 0.00 0.00 9.14 0.001.80 1.80 0.00sdh
0.00 0.000.000.00 0.00 0.00 8.74 0.00517.96 517.76 0.01sdi
0.00 0.000.000.00 0.00 0.00 8.74 0.00516.86 516.86 0.01dm-2
0.00 0.000.88 2749.1819.89 10997.32 8.01 1.130.35 0.1953.25sdl
0.0038.740.000.45 0.01 157.04 698.34 0.09196.60 4.73 0.21sdk
0.00 0.000.000.00 0.00 0.00 8.22 0.00543.59 526.07 0.00Device: rrqm/s wrqm/s r/s w/srkB/swkB/s avgrq-sz avgqu-sz awaitsvctm%utilsda
0.00 0.001.000.00 8.00 0.0016.00 0.003.00 3.00 0.30dm-0
0.00 0.000.000.00 0.00 0.00 0.00 0.000.00 0.00 0.00dm-1
0.00 0.000.000.00 0.00 0.00 0.00 0.000.00 0.00 0.00sdb
0.00 0.000.000.00 0.00 0.00 0.00 0.000.00 0.00 0.00sdc
0.00 0.000.000.00 0.00 0.00 0.00 0.000.00 0.00 0.00sdd
0.00 0.000.000.00 0.00 0.00 0.00 0.000.00 0.00 0.00sde
0.004912.000.00 71.00 0.00 16152.00 454.99 151.55 1116.2114.0499.70sdf
0.00 0.000.000.00 0.00 0.00 0.00 0.000.00 0.00 0.00sdg
0.00 0.000.000.00 0.00 0.00 0.00 0.000.00 0.00 0.00sdh
0.00 0.000.000.00 0.00 0.00 0.00 0.000.00 0.00 0.00sdi
0.00 0.000.000.00 0.00 0.00 0.00 0.000.00 0.00 0.00dm-2
0.00 0.000.00 4998.00 0.00 19992.00 8.007201.07781.57 0.2099.70sdl
0.00 0.000.000.00 0.00 0.00 0.00 0.000.00 0.00 0.00sdk
0.00 0.000.000.00 0.00 0.00 0.00 0.000.00 0.00 0.00[root@zhengcetoufang124 ~]# vmstat 1procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu----- rb swpd free buffcache si sobibo in cs us sy id wa st 3369968 43748441424 9740422400 1 51510 111 8630 1369968 43770841444 9740407200 0 45936 17218 2855181 82 100 3269968 43746041452 9740457600 0 13072 17067 2380981 81 100 3269968 43714841460 9740457600 09464 16874 2209581 8290 3269968 43778441468 9740486400 0 16404 16160 2064881 81 100 [root@zhengcetoufang124 ~]# toptop - 18:40:06 up 31 days, 19:50,9 users,load average: 9.99, 12.83, 14.51Tasks: 525 total, 1 running, 523 sleeping, 0 stopped, 1 zombieCpu(s):6.1%us,0.2%sy,0.0%ni, 83.3%id, 10.4%wa,0.0%hi,0.0%si,0.0%stMem:132147792k total, 131641444k used, 506348k free,41784k buffersSwap: 10239996k total,69968k used, 10170028k free, 97187048k cachedPID USERPRNIVIRTRESSHR S %CPU %MEMTIME+COMMAND
6658 root20 0109g43g40g S 94.2 34.8 271:38.28 java
6201 root20 0109g44g40g S 35.2 34.9 263:20.10 java
5792 root20 0109g43g40g S 15.6 34.6 294:02.87 java
7239 root20 0109g25g22g S1.3 20.096:02.72 java
6404 root20 0109g44g40g S0.7 35.1 269:52.84 java
6779 root20 0109g43g40g S0.7 34.9 263:04.28 java
7568 root20 0109g25g22g S0.7 20.1 122:47.54 java
7667 root20 0109g25g22g S0.7 20.3 104:05.74 java
13433 timesten20 0 92.8g49g49g S0.7 39.3 101:27.68 timestensubd
14456 timesten20 0 92.7g32g31g S0.7 25.5 155:29.84 timestenrepd
16423 root20 0 16.1g 1.5g19m S0.71.2 100:53.94 java
112 root20 0 000 S0.30.016:55.35 events/13
116 root20 0 000 S0.30.0 7:32.05 events/17
4708 root20 0 15404 1680988 R0.30.0 0:00.04 top
6914 root20 0109g43g40g S0.3 34.6 289:06.25 java
13127 root20 0 000 D0.30.0 7:34.76 flush-253:2
18818 root20 0 15400 1696992 S0.30.0 0:31.49 top
1 root20 0 19232 1056856 S0.00.0 0:03.88 init
2 root20 0 000 S0.00.0 0:00.05 kthreadd
3 rootRT 0 000 S0.00.014:58.54 migration/0
4 root20 0 000 S0.00.0 2:58.05 ksoftirqd/0
复制代码

千问 · 2015-7-16 14:18:24

从存储端再看一下I/O流量、响应时间。如果也慢，哪就是存储的问题，如果不慢，哪就是光纤卡、FC交换机、光纤线的问题，也有可能是多路径软件的设置问题，既然是IBM的机器和存储，让IBM的人过来看看啊

千问 · 2015-7-16 14:18:24

找IBM,高达上的。。。