RAC 一个节点不能启动，报ORA-27506

显示全部楼层 · 2008-1-2 17:35:53

aix 5.3hacmp5.3 DB: 10g 10.2.0.1
startup nomount在alert日志中报下面的错误，不能启动
Communication channels reestablished
Mon May 18 14:07:18 2009
IPC Send timeout detected.Sender: ospid 164132
Receiver: inst 2 binc -926319099 ospid 1016164
Mon May 18 14:07:21 2009
Errors in file /oracle/admin/mct/bdump/mct1_lmd0_164132.trc:
ORA-27506: IPC error connecting to a port
Mon May 18 14:07:21 2009
Errors in file /oracle/admin/mct/bdump/mct1_lmd0_164132.trc:
ORA-07445: exception encountered: core dump [] [] [] [] [] []
Mon May 18 14:07:22 2009
Trace dumping is performing id=[cdmp_20090518140722]
Mon May 18 14:07:26 2009
Errors in file /oracle/admin/mct/bdump/mct1_pmon_315432.trc:
ORA-00482: LMD* process terminated with error
Mon May 18 14:07:26 2009
PMON: terminating instance due to error 482
Mon May 18 14:07:26 2009
Errors in file /oracle/admin/mct/bdump/mct1_pmon_315432.trc:
ORA-27300: OS system dependent operation:invalid_process failed with status: 0
ORA-27301: OS failure message: Error 0
ORA-27302: failure occurred at: skgpstime3
ORA-27144: attempt to kill process failed
ORA-00482: LMD* process terminated with error
谁遇到过呢，请指点谢谢

千问 · 2008-1-2 17:35:53

检查一下rac的心跳网通是不是通的。

千问 · 2008-1-2 17:35:53

这种情况应该是BUG，10201 RAC相当的不稳定，建议升级吧

千问 · 2008-1-2 17:35:53

测试过，是通的，系统一直运行很好，突然down了，再startup nomount出现上面的错误

千问 · 2008-1-2 17:35:53

从报错上看是通信有问题
oifcfg getif，ping看一下是不是通的
另看一下crs的状态：
crs_stat -t
/oracle/admin/mct/bdump/mct1_lmd0_164132.trc取有用的部分放上来。

千问 · 2008-1-2 17:35:53

把down的这个结点所有oracle的进程都重启一下，crs可能存在进程占用相应的端口。
10.2.0.1这一版oracle及crs的bug非常多，可以的话建议升一下级。

千问 · 2008-1-2 17:35:53

机器都重启过了，
oifcfg getif
en0178.17.20.0globalpublic
en1192.168.3.0globalcluster_interconnect
crs_stat -t 只有数据库instance mct1不能启动

lmd_trc文件内容
*** SERVICE NAME

) 2009-05-18 13:31:18.874
*** SESSION ID

3301.1) 2009-05-18 13:31:18.874
syncr inc 72 lvl 1 from 1 rcvd (my inc,lvl: 0, 0) (72/4.0.0)
*** 2009-05-18 13:36:30.679
*** 2009-05-18 13:36:30.679
IPC Send timeout detected.Sender: ospid 209158
Receiver: inst 2 binc -926319099 ospid 1016164
SKGXPCTX: 0x110541d38 ctx
wait delta 0 sec (1 msec) ctx ts 0x4ce83 last ts 0x4cea7
user cpu time since last wait 0 sec 0 ticks
system cpu time since last wait 0 sec 0 ticks
locked 1
blocked 3
timed wait receives 0
admno 0x643ed9c4 admport:
SSKGXPT 0x10542084 flagsactive network 0
info for network 0
socket no 9 IP 192.168.3.14 UDP 33761
HACMP network_id 0sflags SSKGXPT_UP
context timestamp 0x4ce83
buffers queued on port 11058c690
sconno accono erttstate seq# sentasync sync rtrans acks
ach accono sconnoadmnostate seq#rcv rtrans acks
0x1114a5630 0x3a741c01 0x5350342c 0x3ce8514e 4032763000
0x1114a59f0 0x3a741c02 0x401cdc7b 0x1df3403b 4032764101
Submitting synchronized dump request [268435460]
ORA-27506: IPC error connecting to a port
Assertion failed: 0, fileskgxpu.c, line 8568

千问 · 2008-1-2 17:35:53

也想升级到10.2.0.4呢，但是得找一个能停止服务的时间。现在想的是用什么办法可以先让这个节点起来
[ 本帖最后由 lhlzjx 于 2009-5-18 15:28 编辑 ]

千问 · 2008-1-2 17:35:53

/usr/sbin/cluster/utilities/cldump
看下HA的状态我怀疑是不是HA那边有问题啊

千问 · 2008-1-2 17:35:53

非常感谢，这个语句的输出显示HA运行正常