求大神帮忙，集群报错 ora-07445

显示全部楼层 · 2014-1-20 11:36:59

Mon Jan 20 11:36:59 2014
Exception [type: SIGSEGV, Address not mapped to object] [ADDR:0x110CF403A] [PC:0x1000B2570, kghinf_cmt()+48] [flags: 0x0, count: 1]
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Errors in file /u01/app/oracle/diag/rdbms/epms/EPMS2/trace/EPMS2_ora_35848298.trc(incident=303632):
ORA-07445: exception encountered: core dump [kghinf_cmt()+48] [SIGSEGV] [ADDR:0x110CF403A] [PC:0x1000B2570] [Address not mapped to object] []
Errors in file /u01/app/oracle/diag/rdbms/epms/EPMS2/trace/EPMS2_pz86_43974944.trc(incident=303640):
ORA-07445: exception encountered: core dump [kghinf_cmt()+48] [SIGSEGV] [ADDR:0x110CF403A] [PC:0x1000B2570] [Address not mapped to object] []
Incident details in: /u01/app/oracle/diag/rdbms/epms/EPMS2/incident/incdir_303632/EPMS2_ora_35848298_i303632.trc
Incident details in: /u01/app/oracle/diag/rdbms/epms/EPMS2/incident/incdir_303640/EPMS2_pz86_43974944_i303640.trc
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Mon Jan 20 11:36:59 2014
Exception [type: SIGSEGV, Address not mapped to object] [ADDR:0x110CF403A] [PC:0x1000B2570, kghinf_cmt()+48] [flags: 0x0, count: 1]
Mon Jan 20 11:36:59 2014
Exception [type: SIGSEGV, Address not mapped to object] [ADDR:0x110CF403A] [PC:0x1000B2570, kghinf_cmt()+48] [flags: 0x0, count: 1]
Errors in file /u01/app/oracle/diag/rdbms/epms/EPMS2/trace/EPMS2_ora_45416674.trc(incident=303648):
ORA-07445: exception encountered: core dump [kghinf_cmt()+48] [SIGSEGV] [ADDR:0x110CF403A] [PC:0x1000B2570] [Address not mapped to object] []
Errors in file /u01/app/oracle/diag/rdbms/epms/EPMS2/trace/EPMS2_ora_48365926.trc(incident=303656):
ORA-07445: exception encountered: core dump [kghinf_cmt()+48] [SIGSEGV] [ADDR:0x110CF403A] [PC:0x1000B2570] [Address not mapped to object] []
Incident details in: /u01/app/oracle/diag/rdbms/epms/EPMS2/incident/incdir_303648/EPMS2_ora_45416674_i303648.trc
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Incident details in: /u01/app/oracle/diag/rdbms/epms/EPMS2/incident/incdir_303656/EPMS2_ora_48365926_i303656.trc
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Mon Jan 20 11:37:00 2014
Exception [type: SIGSEGV, Address not mapped to object] [ADDR:0x110CF403A] [PC:0x1000B2570, kghinf_cmt()+48] [flags: 0x0, count: 1]
Mon Jan 20 11:37:00 2014
Exception [type: SIGSEGV, Address not mapped to object] [ADDR:0x110CF403A] [PC:0x1000B2570, kghinf_cmt()+48] [flags: 0x0, count: 1]
Errors in file /u01/app/oracle/diag/rdbms/epms/EPMS2/trace/EPMS2_ora_40436142.trc(incident=303664):
ORA-07445: exception encountered: core dump [kghinf_cmt()+48] [SIGSEGV] [ADDR:0x110CF403A] [PC:0x1000B2570] [Address not mapped to object] []
Incident details in: /u01/app/oracle/diag/rdbms/epms/EPMS2/incident/incdir_303664/EPMS2_ora_40436142_i303664.trc
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Errors in file /u01/app/oracle/diag/rdbms/epms/EPMS2/trace/EPMS2_ora_30212212.trc(incident=303672):
ORA-07445: exception encountered: core dump [kghinf_cmt()+48] [SIGSEGV] [ADDR:0x110CF403A] [PC:0x1000B2570] [Address not mapped to object] []
Mon Jan 20 11:37:00 2014
Exception [type: SIGSEGV, Address not mapped to object] [ADDR:0x110CF403A] [PC:0x1000B2570, kghinf_cmt()+48] [flags: 0x0, count: 1]
Errors in file /u01/app/oracle/diag/rdbms/epms/EPMS2/trace/EPMS2_ora_40566908.trc(incident=303680):
ORA-07445: exception encountered: core dump [kghinf_cmt()+48] [SIGSEGV] [ADDR:0x110CF403A] [PC:0x1000B2570] [Address not mapped to object] []
Incident details in: /u01/app/oracle/diag/rdbms/epms/EPMS2/incident/incdir_303672/EPMS2_ora_30212212_i303672.trc
Mon Jan 20 11:37:00 2014
Exception [type: SIGSEGV, Address not mapped to object] [ADDR:0x110CF403A] [PC:0x1000B2570, kghinf_cmt()+48] [flags: 0x0, count: 1]
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Mon Jan 20 11:37:00 2014
Exception [type: SIGSEGV, Address not mapped to object] [ADDR:0x110CF403A] [PC:0x1000B2570, kghinf_cmt()+48] [flags: 0x0, count: 1]
Errors in file /u01/app/oracle/diag/rdbms/epms/EPMS2/trace/EPMS2_ora_31261176.trc(incident=303688):
ORA-07445: exception encountered: core dump [kghinf_cmt()+48] [SIGSEGV] [ADDR:0x110CF403A] [PC:0x1000B2570] [Address not mapped to object] []
Mon Jan 20 11:37:00 2014
Exception [type: SIGSEGV, Address not mapped to object] [ADDR:0x110CF403A] [PC:0x1000B2570, kghinf_cmt()+48] [flags: 0x0, count: 1]
Incident details in: /u01/app/oracle/diag/rdbms/epms/EPMS2/incident/incdir_303680/EPMS2_ora_40566908_i303680.trc
Errors in file /u01/app/oracle/diag/rdbms/epms/EPMS2/trace/EPMS2_ora_50593926.trc(incident=303704):
ORA-07445: exception encountered: core dump [kghinf_cmt()+48] [SIGSEGV] [ADDR:0x110CF403A] [PC:0x1000B2570] [Address not mapped to object] []
Errors in file /u01/app/oracle/diag/rdbms/epms/EPMS2/trace/EPMS2_ora_48693492.trc(incident=303696):
ORA-07445: exception encountered: core dump [kghinf_cmt()+48] [SIGSEGV] [ADDR:0x110CF403A] [PC:0x1000B2570] [Address not mapped to object] []
Incident details in: /u01/app/oracle/diag/rdbms/epms/EPMS2/incident/incdir_303688/EPMS2_ora_31261176_i303688.trc
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.

千问 · 2014-1-20 11:36:59

本帖最后由 jinyoufu12345 于 2014-1-22 13:13 编辑
大量用户反映应用程序无法连接某些用户锁定，然后尝试用PL/SQL DEVELOPER 连接提示：tns连接已关闭（节点1，和节点2，均连接不上）
经初步检查集群运行正常，节点1的alert日志没有异常，节点2的alert日志提示异常，产生大量的trc和trm文件和incident目录下的incdir文件，很快将/u01 使用率从64%到100%突增18G，sqlplus 无法连接

千问 · 2014-1-20 11:36:59

Dump file /u01/app/oracle/diag/rdbms/epms/EPMS2/incident/incdir_303632/EPMS2_ora_35848298_i303632.trc
Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64bit Production
With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP,
Data Mining and Real Application Testing options
ORACLE_HOME = /u01/app/oracle/product/11.2.0/db_1
System name:
AIX
Node name:
P750B
Release:
1
Version:
6
Machine:
00F775404C00
Instance name: EPMS2
Redo thread mounted by this instance: 2
Oracle process number: 0
Unix process pid: 35848298, image: oracle@P750B

*** 2014-01-20 11:36:59.578
Dump continued from file: /u01/app/oracle/diag/rdbms/epms/EPMS2/trace/EPMS2_ora_35848298.trc
ORA-07445: exception encountered: core dump [kghinf_cmt()+48] [SIGSEGV] [ADDR:0x110CF403A] [PC:0x1000B2570] [Address not mapped to object] []
========= Dump for incident 303632 (ORA 7445 [kghinf_cmt()+48]) ========
----- Beginning of Customized Incident Dump(s) -----
Exception [type: SIGSEGV, Address not mapped to object] [ADDR:0x110CF403A] [PC:0x1000B2570, kghinf_cmt()+48] [flags: 0x0, count: 1]
Registers:
iar: 0x00000001000b2570lr: 0x00000001021f4468
msr: 0xa00000000000d032cr: 0x0000000044244484
r00: 0x00000001021f462c r01: 0x0fffffffffffb920 r02: 0x00000001106598b0
r03: 0x0000000110c2a048 r04: 0x0000000110cf4000 r05: 0x0000000110cf4000
r06: 0x00000001100045d0 r07: 0x0000000109d7b92c r08: 0x0000000000000010
r09: 0x0000000000000020 r10: 0x00000000000000b0 r11: 0x0000000000000000
r12: 0x00000001021ecde4 r13: 0x00000001106d31e0 r14: 0x000000010af62da8
r15: 0x0000000110661648 r16: 0x000000010af99218 r17: 0x000000010af991e8
r18: 0x000000010af991d8 r19: 0x000000010af98000 r20: 0x000000010af98028
r21: 0x000000010af97840 r22: 0x000000010a2c53c8 r23: 0x000000010a2c53a8
r24: 0x000000010a2c5388 r25: 0x0000000110c31938 r26: 0x0000000109d7b92c
r27: 0x0000000000000000 r28: 0x0000000110000240 r29: 0x00000001100045d0
r30: 0x0000000109d7b92c r31: 0x0000000110c2a048
*** 2014-01-20 11:36:59.655
dbkedDefDump(): Starting a non-incident diagnostic dump (flags=0x3, level=3, mask=0x0)
----- SQL Statement (None) -----
Current SQL information unavailable - no session.
----- Call Stack Trace -----
calling
call entry
argument values in hex
location
type point
(? means dubious value)
-------------------- -------- -------------------- ----------------------------
skdstdst()+40bl 107c53a08
000000000 ? 000000000 ?

000000003 ? 000000001 ?

000000000 ? 000000001 ?

000000003 ? 000000001 ?
ksedst1()+112call skdstdst() 000000000 ?

4824384100000000 ?

110C08E68 ? 000002004 ?

110A72B80 ? 10A396AEC ?

000000000 ? 110A72B80 ?
ksedst()+40
call ksedst1()
30352DC99DB ? 002050033 ?

10A396AE0 ? 7000000000262 ?

000000000 ? 000000000 ?

10A3960E0 ? 000000000 ?
dbkedDefDump()+1516call ksedst()
000000000 ? 110A740E4 ?

000000000 ? 110A740E4 ?

3A00000000000000 ?

000000000 ? 000000000 ?

300000003 ?
ksedmp()+72
call dbkedDefDump() 300000000 ? 030633261 ?

1000B2570 ? 000000001 ?

00000000B ? 110C09B10 ?

110C09DC0 ? 000000001 ?
ssexhd()+2672call ksedmp()
109E1B058 ? 200000002 ?

000000004 ? 000000000 ?

000000004 ? 000000001 ?

3537302C206B6768 ?

000000004 ?
47dc
call ssexhd()
000000000 ? 000000000 ?

000000000 ? 000000000 ?

千问 · 2014-1-20 11:36:59

本帖最后由 heu 于 2014-1-22 14:21 编辑
初步判断可能需要打补丁，需要进一步提供日志进行分析和确认。

千问 · 2014-1-20 11:36:59

把集群状态贴出来
alert log 贴出来看下