oracle集群断电重启,Oracle10.2.0.4 RAC 2节点集群节点被驱逐导致系统异常重启
oracle 10.2.0.4
redhat 5.4??linux
OCSS.LOG日志：
1.正常的節(jié)點2
[? ? CSSD]2013-12-03 15:07:52.100 [1258428736] >WARNING: clssnmPollingThread: node rac1 (1) at 50 1.625521e-315artbeat fatal, eviction in 29.240 seconds
[? ? CSSD]2013-12-03 15:07:52.100 [1258428736] >TRACE:? ?clssnmPollingThread: node rac1 (1) is impending reconfig, flag 1039, misstime 30760
[? ? CSSD]2013-12-03 15:07:52.100 [1258428736] >TRACE:? ?clssnmPollingThread: diskTimeout set to (57000)ms impending reconfig status(1)
[? ? CSSD]2013-12-03 15:08:07.101 [1258428736] >WARNING: clssnmPollingThread: node rac1 (1) at 75 1.625521e-315artbeat fatal, eviction in 14.240 seconds
[? ? CSSD]2013-12-03 15:08:16.109 [1258428736] >WARNING: clssnmPollingThread: node rac1 (1) at 90 1.625521e-315artbeat fatal, eviction in 5.230 seconds
[? ? CSSD]2013-12-03 15:08:17.101 [1258428736] >WARNING: clssnmPollingThread: node rac1 (1) at 90 1.625521e-315artbeat fatal, eviction in 4.240 seconds
[? ? CSSD]2013-12-03 15:08:18.104 [1258428736] >WARNING: clssnmPollingThread: node rac1 (1) at 90 1.482197e-323artbeat fatal, eviction in 3.240 seconds
[? ? CSSD]2013-12-03 15:08:19.106 [1258428736] >WARNING: clssnmPollingThread: node rac1 (1) at 90 1.482197e-323artbeat fatal, eviction in 2.240 seconds
[? ? CSSD]2013-12-03 15:08:20.108 [1258428736] >WARNING: clssnmPollingThread: node rac1 (1) at 90 1.482197e-323artbeat fatal, eviction in 1.240 seconds
[? ? CSSD]2013-12-03 15:08:21.110 [1258428736] >WARNING: clssnmPollingThread: node rac1 (1) at 90 1.482197e-323artbeat fatal, eviction in 0.230 seconds
[? ? CSSD]2013-12-03 15:08:21.343 [1258428736] >TRACE:? ?clssnmPollingThread: Eviction started for node rac1 (1), flags 0x040f, state 3, wt4c 0
[? ? CSSD]2013-12-03 15:08:21.343 [1279408448] >TRACE:? ?clssnmDoSyncUpdate: Initiating sync 11
[? ? CSSD]2013-12-03 15:08:21.343 [1279408448] >TRACE:? ?clssnmDoSyncUpdate: diskTimeout set to (57000)ms
[? ? CSSD]2013-12-03 15:08:21.343 [1279408448] >TRACE:? ?clssnmSetupAckWait: Ack message type (11)
[? ? CSSD]2013-12-03 15:08:21.343 [1279408448] >TRACE:? ?clssnmSetupAckWait: node(2) is ALIVE
[? ? CSSD]2013-12-03 15:08:21.343 [1279408448] >TRACE:? ?clssnmSendSync: syncSeqNo(11)
[? ? CSSD]2013-12-03 15:08:21.343 [1279408448] >TRACE:? ?clssnmWaitForAcks: Ack message type(11), ackCount(1)
[? ? CSSD]2013-12-03 15:08:21.343 [1205979456] >TRACE:? ?clssnmHandleSync: diskTimeout set to (57000)ms
[? ? CSSD]2013-12-03 15:08:21.343 [1205979456] >TRACE:? ?clssnmHandleSync: Acknowledging sync: src[2] srcName[jzjyrac2] seq[1] sync[11]
[? ? CSSD]2013-12-03 15:08:21.343 [1279408448] >TRACE:? ?clssnmWaitForAcks: done, msg type(11)
[? ? CSSD]2013-12-03 15:08:21.343 [1279408448] >TRACE:? ?clssnmDoSyncUpdate: Terminating node 1, rac1, misstime(60000) state(5)
[? ? CSSD]2013-12-03 15:08:21.343 [1279408448] >TRACE:? ?clssnmSetupAckWait: Ack message type (13)
[? ? CSSD]2013-12-03 15:08:21.343 [1279408448] >TRACE:? ?clssnmSetupAckWait: node(2) is ACTIVE
[? ? CSSD]2013-12-03 15:08:21.343 [3410396480] >USER:? ? NMEVENT_SUSPEND [00][00][00][06]
[? ? CSSD]2013-12-03 15:08:21.343 [1279408448] >TRACE:? ?clssnmWaitForAcks: Ack message type(13), ackCount(1)
[? ? CSSD]2013-12-03 15:08:21.344 [1205979456] >TRACE:? ?clssnmSendVoteInfo: node(2) syncSeqNo(11)
[? ? CSSD]2013-12-03 15:08:21.344 [1279408448] >TRACE:? ?clssnmWaitForAcks: done, msg type(13)
[? ? CSSD]2013-12-03 15:08:21.344 [1279408448] >TRACE:? ?clssnmCheckDskInfo: Checking disk info...
[? ? CSSD]2013-12-03 15:08:21.344 [1279408448] >TRACE:? ?clssnmEvict: Start
[? ? CSSD]2013-12-03 15:08:21.344 [1279408448] >TRACE:? ?clssnmEvict: Evicting node 1, rac1, birth 1, death 11, impendingrcfg 1, stateflags 0x40f
[? ? CSSD]2013-12-03 15:08:21.344 [1279408448] >TRACE:? ?clssnmSendShutdown: req to node 1, kill time 279059068
[? ? CSSD]2013-12-03 15:08:21.344 [1279408448] >TRACE:? ?clssnmDiscHelper: rac1, node(1) connection failed, con (0x13b34eb0), probe((nil))
[? ? CSSD]2013-12-03 15:08:21.344 [1279408448] >TRACE:? ?clssnmWaitOnEvictions: Start
[? ? CSSD]2013-12-03 15:08:21.344 [1279408448] >TRACE:? ?clssnmWaitOnEvictions: node 1, rac1, undead 0
[? ? CSSD]2013-12-03 15:08:21.344 [1279408448] >TRACE:? ?clssnmSetupAckWait: Ack message type (15)
[? ? CSSD]2013-12-03 15:08:21.344 [1279408448] >TRACE:? ?clssnmSetupAckWait: node(2) is ACTIVE
[? ? CSSD]2013-12-03 15:08:21.344 [1279408448] >TRACE:? ?clssnmSendUpdate: syncSeqNo(11)
[? ? CSSD]2013-12-03 15:08:21.344 [1279408448] >TRACE:? ?clssnmWaitForAcks: Ack message type(15), ackCount(1)
[? ? CSSD]2013-12-03 15:08:21.345 [1205979456] >TRACE:? ?clssnmUpdateNodeState: node 0, state (0/0) unique (0/0) prevConuni(0) birth (0/0) (old/new)
[? ? CSSD]2013-12-03 15:08:21.345 [1205979456] >TRACE:? ?clssnmUpdateNodeState: node 1, state (5/0) unique (1378888098/1378888098) prevConuni(1378888098) birth (1/1) (old/new)
[? ? CSSD]2013-12-03 15:08:21.345 [1205979456] >TRACE:? ?clssnmDeactivateNode: node 1 (rac1) left cluster
[? ? CSSD]2013-12-03 15:08:21.345 [1205979456] >TRACE:? ?clssnmUpdateNodeState: node 2, state (3/3) unique (1381478833/1381478833) prevConuni(0) birth (10/10) (old/new)
[? ? CSSD]2013-12-03 15:08:21.345 [1205979456] >USER:? ? clssnmHandleUpdate: SYNC(11) from node(2) completed
[? ? CSSD]2013-12-03 15:08:21.345 [1205979456] >USER:? ? clssnmHandleUpdate: NODE 2 (jzjyrac2) IS ACTIVE MEMBER OF CLUSTER
[? ? CSSD]2013-12-03 15:08:21.345 [1205979456] >TRACE:? ?clssnmHandleUpdate: diskTimeout set to (200000)ms
[? ? CSSD]2013-12-03 15:08:21.345 [1279408448] >TRACE:? ?clssnmWaitForAcks: done, msg type(15)
[? ? CSSD]2013-12-03 15:08:21.345 [1279408448] >TRACE:? ?clssnmDoSyncUpdate: Sync 11 complete!
[? ? CSSD]2013-12-03 15:08:21.345 [1289898304] >TRACE:? ?clssgmReconfigThread:??started for reconfig (11)
[? ? CSSD]2013-12-03 15:08:21.346 [1289898304] >USER:? ? NMEVENT_RECONFIG [00][00][00][04]
[? ? CSSD]2013-12-03 15:08:21.346 [1289898304] >TRACE:? ?clssgmCleanupGrocks: cleaning up grock _ORA_CRS_MEMBER_rac1 type 3
[? ? CSSD]2013-12-03 15:08:21.346 [1289898304] >TRACE:? ?clssgmCleanupOrphanMembers: cleaning up remote mbr(0) grock(_ORA_CRS_MEMBER_rac1) birth(1/1)
[? ? CSSD]2013-12-03 15:08:21.346 [1289898304] >TRACE:? ?clssgmCleanupGrocks: cleaning up grock _ORA_CRS_MEMBER_jzjyrac2 type 3
[? ? CSSD]2013-12-03 15:08:21.346 [1289898304] >TRACE:? ?clssgmCleanupGrocks: cleaning up grock crs_version type 2
[? ? CSSD]2013-12-03 15:08:21.346 [1289898304] >TRACE:? ?clssgmCleanupOrphanMembers: cleaning up remote mbr(0) grock(crs_version) birth(1/1)
[? ? CSSD]2013-12-03 15:08:21.346 [1289898304] >TRACE:? ?clssgmCleanupGrocks: cleaning up grock DAALL_DB type 2
[? ? CSSD]2013-12-03 15:08:21.346 [1289898304] >TRACE:? ?clssgmCleanupOrphanMembers: cleaning up remote mbr(0) grock(DAALL_DB) birth(1/1)
[? ? CSSD]2013-12-03 15:08:21.346 [1289898304] >TRACE:? ?clssgmCleanupGrocks: cleaning up grock EVMDMAIN type 2
[? ? CSSD]2013-12-03 15:08:21.346 [1289898304] >TRACE:? ?clssgmCleanupOrphanMembers: cleaning up remote mbr(1) grock(EVMDMAIN) birth(1/1)
[? ? CSSD]2013-12-03 15:08:21.346 [1289898304] >TRACE:? ?clssgmCleanupGrocks: cleaning up grock CRSDMAIN type 2
[? ? CSSD]2013-12-03 15:08:21.346 [1289898304] >TRACE:? ?clssgmCleanupOrphanMembers: cleaning up remote mbr(1) grock(CRSDMAIN) birth(1/1)
[? ? CSSD]2013-12-03 15:08:21.346 [1289898304] >TRACE:? ?clssgmCleanupGrocks: cleaning up grock DBJZJYRAC type 2
[? ? CSSD]2013-12-03 15:08:21.346 [1289898304] >TRACE:? ?clssgmCleanupOrphanMembers: cleaning up remote mbr(0) grock(DBJZJYRAC) birth(1/1)
[? ? CSSD]2013-12-03 15:08:21.347 [1289898304] >TRACE:? ?clssgmCleanupGrocks: cleaning up grock ORA_CLSRD_1_jzjyrac type 3
[? ? CSSD]2013-12-03 15:08:21.347 [1289898304] >TRACE:? ?clssgmCleanupOrphanMembers: cleaning up remote mbr(0) grock(ORA_CLSRD_1_jzjyrac) birth(1/1)
[? ? CSSD]2013-12-03 15:08:21.347 [1289898304] >TRACE:? ?clssgmCleanupGrocks: cleaning up grock ORA_CLSRD_1_jzjyrac type 2
[? ? CSSD]2013-12-03 15:08:21.347 [1289898304] >TRACE:? ?clssgmCleanupOrphanMembers: cleaning up remote mbr(0) grock(ORA_CLSRD_1_jzjyrac) birth(1/1)
[? ? CSSD]2013-12-03 15:08:21.347 [1289898304] >TRACE:? ?clssgmCleanupGrocks: cleaning up grock ORA_CLSRD_2_jzjyrac type 2
[? ? CSSD]2013-12-03 15:08:21.347 [1289898304] >TRACE:? ?clssgmCleanupGrocks: cleaning up grock ORA_CLSRD_2_jzjyrac type 3
[? ? CSSD]2013-12-03 15:08:21.347 [1289898304] >TRACE:? ?clssgmCleanupGrocks: cleaning up grock DGJZJYRAC type 2
[? ? CSSD]2013-12-03 15:08:21.347 [1289898304] >TRACE:? ?clssgmCleanupOrphanMembers: cleaning up remote mbr(0) grock(DGJZJYRAC) birth(1/1)
[? ? CSSD]2013-12-03 15:08:21.347 [1289898304] >TRACE:? ?clssgmCleanupGrocks: cleaning up grock ocr_crs type 2
[? ? CSSD]2013-12-03 15:08:21.347 [1289898304] >TRACE:? ?clssgmCleanupOrphanMembers: cleaning up remote mbr(1) grock(ocr_crs) birth(1/1)
[? ? CSSD]2013-12-03 15:08:21.347 [1289898304] >TRACE:? ?clssgmCleanupGrocks: cleaning up grock IGJZJYRACALL type 2
[? ? CSSD]2013-12-03 15:08:21.347 [1289898304] >TRACE:? ?clssgmCleanupOrphanMembers: cleaning up remote mbr(1) grock(IGJZJYRACALL) birth(1/1)
[? ? CSSD]2013-12-03 15:08:21.347 [1289898304] >TRACE:? ?clssgmCleanupGrocks: cleaning up grock #CSS_CLSSOMON type 2
[? ? CSSD]2013-12-03 15:08:21.347 [1289898304] >TRACE:? ?clssgmCleanupOrphanMembers: cleaning up remote mbr(1) grock(#CSS_CLSSOMON) birth(1/1)
[? ? CSSD]2013-12-03 15:08:21.347 [1289898304] >TRACE:? ?clssgmEstablishConnections: 1 nodes in cluster incarn 11
[? ? CSSD]2013-12-03 15:08:21.348 [1247938880] >TRACE:? ?clssgmPeerDeactivate: node 1 (rac1), death 11, state 0x80000000 connstate 0xa
[? ? CSSD]2013-12-03 15:08:21.348 [1247938880] >TRACE:? ?clssgmPeerListener: connects done (1/1)
[? ? CSSD]2013-12-03 15:08:21.348 [1289898304] >TRACE:? ?clssgmEstablishMasterNode: MASTER for 11 is node(2) birth(10)
[? ? CSSD]2013-12-03 15:08:21.348 [1289898304] >TRACE:? ?clssgmChangeMasterNode: requeued 0 RPCs
[? ? CSSD]2013-12-03 15:08:21.348 [1289898304] >TRACE:? ?clssgmMasterCMSync: Synchronizing group/lock status
[? ? CSSD]2013-12-03 15:08:21.348 [1289898304] >TRACE:? ?clssgmMasterSendDBDone: group/lock status synchronization complete
[? ? CSSD]CLSS-3000: reconfiguration successful, incarnation 11 with 1 nodes
[? ? CSSD]CLSS-3001: local node number 2, master node number 2
[? ? CSSD]2013-12-03 15:08:21.349 [1289898304] >TRACE:? ?clssgmReconfigThread:??completed for reconfig(11), with status(1)
[? ? CSSD]2013-12-03 15:27:09.357 [1205979456] >TRACE:? ?clssnmConnComplete: probe from node 1, your version: 10.2.1.2
, support PENDINA: 1
[? ? CSSD]2013-12-03 15:27:09.357 [1205979456] >TRACE:? ?clssnmConnComplete: MSGSRC 1, type 5, node 1, flags 0x0001, con 0x2aaab0001100, probe (nil)
[? ? CSSD]2013-12-03 15:27:09.357 [1205979456] >TRACE:? ?clssnmConnComplete: node 1, rac1, con(0x2aaab0001100), probcon((nil)), ninfcon((nil)), node unique 1386055667, prev unique 0, msg unique 1386055667 node state 0
[? ? CSSD]2013-12-03 15:27:09.357 [1205979456] >TRACE:? ?clssnmsendConnAck: node 1, node state 0
[? ? CSSD]2013-12-03 15:27:09.357 [1205979456] >TRACE:? ?clssnmConnComplete: connecting to node 1 (con 0x2aaab0001100), ninfcon (0x2aaab0001100), state (0)
[? ? CSSD]2013-12-03 15:27:09.357 [1205979456] >TRACE:? ?clssnmConnComplete: connected to node 1 (con 0x2aaab0001100), ninfcon (0x2aaab0001100), state (0), flag (1037)
[? ? CSSD]2013-12-03 15:27:10.279 [1279408448] >TRACE:? ?clssnmDoSyncUpdate: Initiating sync 12
[? ? CSSD]2013-12-03 15:27:10.279 [1279408448] >TRACE:? ?clssnmDoSyncUpdate: diskTimeout set to (57000)ms
[? ? CSSD]2013-12-03 15:27:10.279 [1279408448] >TRACE:? ?clssnmSetupAckWait: Ack message type (11)
[? ? CSSD]2013-12-03 15:27:10.279 [1279408448] >TRACE:? ?clssnmSetupAckWait: node(1) is ALIVE
[? ? CSSD]2013-12-03 15:27:10.279 [1279408448] >TRACE:? ?clssnmSetupAckWait: node(2) is ALIVE
[? ? CSSD]2013-12-03 15:27:10.279 [1279408448] >TRACE:? ?clssnmSendSync: syncSeqNo(12)
[? ? CSSD]2013-12-03 15:27:10.279 [1205979456] >TRACE:? ?clssnmHandleSync: diskTimeout set to (57000)ms
[? ? CSSD]2013-12-03 15:27:10.279 [1279408448] >TRACE:? ?clssnmWaitForAcks: Ack message type(11), ackCount(2)
[? ? CSSD]2013-12-03 15:27:10.279 [1205979456] >TRACE:? ?clssnmHandleSync: Acknowledging sync: src[2] srcName[jzjyrac2] seq[5] sync[12]
[? ? CSSD]2013-12-03 15:27:10.280 [3410396480] >USER:? ? NMEVENT_SUSPEND [00][00][00][04]
[? ? CSSD]2013-12-03 15:27:10.280 [1279408448] >TRACE:? ?clssnmWaitForAcks: done, msg type(11)
[? ? CSSD]2013-12-03 15:27:10.280 [1279408448] >TRACE:? ?clssnmDoSyncUpdate: node(1) is transitioning from joining state to active state
[? ? CSSD]2013-12-03 15:27:10.280 [1279408448] >TRACE:? ?clssnmSetupAckWait: Ack message type (13)
[? ? CSSD]2013-12-03 15:27:10.280 [1279408448] >TRACE:? ?clssnmSetupAckWait: node(1) is ACTIVE
[? ? CSSD]2013-12-03 15:27:10.280 [1279408448] >TRACE:? ?clssnmSetupAckWait: node(2) is ACTIVE
[? ? CSSD]2013-12-03 15:27:10.280 [1205979456] >TRACE:? ?clssnmSendVoteInfo: node(2) syncSeqNo(12)
[? ? CSSD]2013-12-03 15:27:10.280 [1279408448] >TRACE:? ?clssnmWaitForAcks: Ack message type(13), ackCount(1)
[? ? CSSD]2013-12-03 15:27:10.280 [1279408448] >TRACE:? ?clssnmWaitForAcks: done, msg type(13)
[? ? CSSD]2013-12-03 15:27:10.280 [1279408448] >TRACE:? ?clssnmCheckDskInfo: Checking disk info...
[? ? CSSD]2013-12-03 15:27:10.281 [1279408448] >TRACE:? ?clssnmEvict: Start
[? ? CSSD]2013-12-03 15:27:10.281 [1279408448] >TRACE:? ?clssnmWaitOnEvictions: Start
[? ? CSSD]2013-12-03 15:27:10.281 [1279408448] >TRACE:? ?clssnmSetupAckWait: Ack message type (15)
[? ? CSSD]2013-12-03 15:27:10.281 [1279408448] >TRACE:? ?clssnmSetupAckWait: node(1) is ACTIVE
[? ? CSSD]2013-12-03 15:27:10.281 [1279408448] >TRACE:? ?clssnmSetupAckWait: node(2) is ACTIVE
[? ? CSSD]2013-12-03 15:27:10.281 [1279408448] >TRACE:? ?clssnmSendUpdate: syncSeqNo(12)
[? ? CSSD]2013-12-03 15:27:10.281 [1205979456] >TRACE:? ?clssnmUpdateNodeState: node 0, state (0/0) unique (0/0) prevConuni(0) birth (0/0) (old/new)
[? ? CSSD]2013-12-03 15:27:10.281 [1205979456] >TRACE:? ?clssnmUpdateNodeState: node 1, state (2/3) unique (1386055667/1386055667) prevConuni(0) birth (12/12) (old/new)
[? ? CSSD]2013-12-03 15:27:10.281 [1205979456] >TRACE:? ?clssnmUpdateNodeState: node 2, state (3/3) unique (1381478833/1381478833) prevConuni(0) birth (10/10) (old/new)
[? ? CSSD]2013-12-03 15:27:10.281 [1205979456] >USER:? ? clssnmHandleUpdate: SYNC(12) from node(2) completed
[? ? CSSD]2013-12-03 15:27:10.281 [1205979456] >USER:? ? clssnmHandleUpdate: NODE 1 (rac1) IS ACTIVE MEMBER OF CLUSTER
[? ? CSSD]2013-12-03 15:27:10.281 [1205979456] >USER:? ? clssnmHandleUpdate: NODE 2 (jzjyrac2) IS ACTIVE MEMBER OF CLUSTER
[? ? CSSD]2013-12-03 15:27:10.281 [1205979456] >TRACE:? ?clssnmHandleUpdate: diskTimeout set to (200000)ms
[? ? CSSD]2013-12-03 15:27:10.281 [1279408448] >TRACE:? ?clssnmWaitForAcks: Ack message type(15), ackCount(1)
[? ? CSSD]2013-12-03 15:27:10.281 [1279408448] >TRACE:? ?clssnmWaitForAcks: done, msg type(15)
[? ? CSSD]2013-12-03 15:27:10.281 [1279408448] >TRACE:? ?clssnmDoSyncUpdate: Sync 12 complete!
[? ? CSSD]2013-12-03 15:27:10.281 [1289898304] >TRACE:? ?clssgmReconfigThread:??started for reconfig (12)
[? ? CSSD]2013-12-03 15:27:10.282 [1289898304] >USER:? ? NMEVENT_RECONFIG [00][00][00][06]
[? ? CSSD]2013-12-03 15:27:10.283 [1289898304] >TRACE:? ?clssgmEstablishConnections: 2 nodes in cluster incarn 12
[? ? CSSD]2013-12-03 15:27:10.379 [1247938880] >TRACE:? ?clssgmInitialRecv: (0x13ba6880) accepted a new connection from node 1 born at 12 active (2, 2), vers (10,3,1,2)
[? ? CSSD]2013-12-03 15:27:10.379 [1247938880] >TRACE:? ?clssgmInitialRecv: conns done (2/2)
[? ? CSSD]2013-12-03 15:27:10.380 [1289898304] >TRACE:? ?clssgmEstablishMasterNode: MASTER for 12 is node(2) birth(10)
[? ? CSSD]2013-12-03 15:27:10.380 [1289898304] >TRACE:? ?clssgmMasterCMSync: Synchronizing group/lock status
[? ? CSSD]2013-12-03 15:27:10.383 [1289898304] >TRACE:? ?clssgmMasterSendDBDone: group/lock status synchronization complete
[? ? CSSD]CLSS-3000: reconfiguration successful, incarnation 12 with 2 nodes
[? ? CSSD]CLSS-3001: local node number 2, master node number 2
[? ? CSSD]2013-12-03 15:27:10.383 [1289898304] >TRACE:? ?clssgmReconfigThread:??completed for reconfig(12), with status(1)
2. 被重啟的節(jié)點1
[? ? CSSD]2013-12-03 15:27:47.299 >USER:? ? Copyright 2013, Oracle version 10.2.0.4.0
[? ? CSSD]2013-12-03 15:27:47.299 >USER:? ? CSS daemon log for node rac1, number 1, in cluster crs
[??clsdmt]Listening to (ADDRESS=(PROTOCOL=ipc)(KEY=rac1DBG_CSSD))
[? ? CSSD]2013-12-03 15:27:47.318 [1587963200] >TRACE:? ?clssscmain: local-only set to false
[? ? CSSD]2013-12-03 15:27:47.324 [1587963200] >TRACE:? ?clssnmReadNodeInfo: added node 1 (rac1) to cluster
[? ? CSSD]2013-12-03 15:27:47.332 [1587963200] >TRACE:? ?clssnmReadNodeInfo: added node 2 (rac2) to cluster
[? ? CSSD]2013-12-03 15:27:47.341 [1127602496] >TRACE:? ?clssnm_skgxninit: Compatible vendor clusterware not in use
[? ? CSSD]2013-12-03 15:27:47.341 [1127602496] >TRACE:? ?clssnm_skgxnmon: skgxn init failed
[? ? CSSD]2013-12-03 15:27:47.343 [1587963200] >TRACE:? ?clssnmNMInitialize: misscount set to (60)
[? ? CSSD]2013-12-03 15:27:47.344 [1587963200] >TRACE:? ?clssnmNMInitialize: Network heartbeat thresholds are: impending reconfig 30000 ms, reconfig start (misscount) 60000 ms
[? ? CSSD]2013-12-03 15:27:47.347 [1587963200] >TRACE:? ?clssnmDiskStateChange: state from 1 to 2 disk (0//oradata/jzjyrac/vote1)
[? ? CSSD]2013-12-03 15:27:47.347 [1127602496] >TRACE:? ?clssnmvDPT: spawned for disk 0 (/oradata/jzjyrac/vote1)
[? ? CSSD]2013-12-03 15:27:47.348 [1587963200] >TRACE:? ?clssnmDiskStateChange: state from 1 to 2 disk (1//oradata/jzjyrac/vote2)
[? ? CSSD]2013-12-03 15:27:47.354 [1138092352] >TRACE:? ?clssnmvDPT: spawned for disk 1 (/oradata/jzjyrac/vote2)
[? ? CSSD]2013-12-03 15:27:47.355 [1587963200] >TRACE:? ?clssnmDiskStateChange: state from 1 to 2 disk (2//oradata/jzjyrac/vote3)
[? ? CSSD]2013-12-03 15:27:47.361 [1148582208] >TRACE:? ?clssnmvDPT: spawned for disk 2 (/oradata/jzjyrac/vote3)
[? ? CSSD]2013-12-03 15:27:49.363 [1127602496] >TRACE:? ?clssnmDiskStateChange: state from 2 to 4 disk (0//oradata/jzjyrac/vote1)
[? ? CSSD]2013-12-03 15:27:49.368 [1148582208] >TRACE:? ?clssnmDiskStateChange: state from 2 to 4 disk (2//oradata/jzjyrac/vote3)
[? ? CSSD]2013-12-03 15:27:49.369 [1159072064] >TRACE:? ?clssnmvKillBlockThread: spawned for disk 0 (/oradata/jzjyrac/vote1) initial sleep interval (1000)ms
[? ? CSSD]2013-12-03 15:27:49.369 [1127602496] >TRACE:? ?clssnmReadDskHeartbeat: node(2) is down. rcfg(12) wrtcnt(4564413) LATS(4294225120) Disk lastSeqNo(4564413)
[? ? CSSD]2013-12-03 15:27:49.374 [1169561920] >TRACE:? ?clssnmvKillBlockThread: spawned for disk 2 (/oradata/jzjyrac/vote3) initial sleep interval (1000)ms
[? ? CSSD]2013-12-03 15:27:49.374 [1138092352] >TRACE:? ?clssnmDiskStateChange: state from 2 to 4 disk (1//oradata/jzjyrac/vote2)
[? ? CSSD]2013-12-03 15:27:49.374 [1148582208] >TRACE:? ?clssnmReadDskHeartbeat: node(2) is down. rcfg(12) wrtcnt(4564413) LATS(4294225130) Disk lastSeqNo(4564413)
[? ? CSSD]2013-12-03 15:27:49.384 [1180051776] >TRACE:? ?clssnmvKillBlockThread: spawned for disk 1 (/oradata/jzjyrac/vote2) initial sleep interval (1000)ms
[? ? CSSD]2013-12-03 15:27:49.385 [1587963200] >TRACE:? ?clssnmFatalInit: fatal mode enabled
[? ? CSSD]2013-12-03 15:27:49.390 [1138092352] >TRACE:? ?clssnmReadDskHeartbeat: node(2) is down. rcfg(12) wrtcnt(4564413) LATS(4294225150) Disk lastSeqNo(4564413)
[? ? CSSD]2013-12-03 15:27:49.405 [1201031488] >TRACE:? ?clssnmClusterListener: Listening on (ADDRESS=(PROTOCOL=tcp)(HOST=rac1-priv)(PORT=49895))
[? ? CSSD]2013-12-03 15:27:49.412 [1201031488] >TRACE:? ?clssnmClusterListener: Probing node rac2 (2), probcon(0xed55800)
[? ? CSSD]2013-12-03 15:27:49.413 [1201031488] >TRACE:? ?clssnmConnComplete: MSGSRC 2, type 6, node 2, flags 0x0001, con 0xed55800, probe 0xed55800
[? ? CSSD]2013-12-03 15:27:49.413 [1201031488] >TRACE:? ?clssnmConnComplete: node 2, rac2, con(0xed55800), probcon(0xed55800), ninfcon((nil)), node unique 1381478833, prev unique 0, msg unique 1381478833 node state 0
[? ? CSSD]2013-12-03 15:27:49.413 [1201031488] >TRACE:? ?clssnmConnComplete: connected to node 2 (con 0xed55800), ninfcon (0xed55800), state (0), flag (1037)
[? ? CSSD]2013-12-03 15:27:49.489 [1211521344] >TRACE:? ?clssgmclientlsnr: listening on (ADDRESS=(PROTOCOL=ipc)(KEY=Oracle_CSS_LclLstnr_crs_1))
[? ? CSSD]2013-12-03 15:27:49.489 [1211521344] >TRACE:? ?clssgmclientlsnr: listening on (ADDRESS=(PROTOCOL=ipc)(KEY=OCSSD_LL_rac1_crs))
[? ? CSSD]2013-12-03 15:27:49.495 [1242990912] >TRACE:? ?clssgmPeerListener: Listening on (ADDRESS=(PROTOCOL=tcp)(DEV=25)(HOST=192.168.168.80)(PORT=61103))
[? ? CSSD]2013-12-03 15:27:50.334 [1201031488] >TRACE:? ?clssnmHandleSync: diskTimeout set to (57000)ms
[? ? CSSD]2013-12-03 15:27:50.335 [1201031488] >TRACE:? ?clssnmHandleSync: Acknowledging sync: src[2] srcName[rac2] seq[5] sync[12]
[? ? CSSD]2013-12-03 15:27:50.335 [1274460480] >TRACE:? ?clssnmRcfgMgrThread: initial lastleader(2) unique(1381478833)
[? ? CSSD]2013-12-03 15:27:50.335 [1201031488] >TRACE:? ?clssnmSendVoteInfo: node(2) syncSeqNo(12)
[? ? CSSD]2013-12-03 15:27:50.336 [1201031488] >TRACE:? ?clssnmUpdateNodeState: node 0, state (0/0) unique (0/0) prevConuni(0) birth (0/0) (old/new)
[? ? CSSD]2013-12-03 15:27:50.336 [1201031488] >TRACE:? ?clssnmUpdateNodeState: node 1, state (1/3) unique (1386055667/1386055667) prevConuni(0) birth (0/12) (old/new)
[? ? CSSD]2013-12-03 15:27:50.336 [1201031488] >TRACE:? ?clssnmUpdateNodeState: node 2, state (4/3) unique (1381478833/1381478833) prevConuni(0) birth (0/10) (old/new)
[? ? CSSD]2013-12-03 15:27:50.336 [1201031488] >USER:? ? clssnmHandleUpdate: SYNC(12) from node(2) completed
[? ? CSSD]2013-12-03 15:27:50.336 [1201031488] >USER:? ? clssnmHandleUpdate: NODE 1 (rac1) IS ACTIVE MEMBER OF CLUSTER
[? ? CSSD]2013-12-03 15:27:50.336 [1201031488] >USER:? ? clssnmHandleUpdate: NODE 2 (rac2) IS ACTIVE MEMBER OF CLUSTER
[? ? CSSD]2013-12-03 15:27:50.336 [1201031488] >TRACE:? ?clssnmHandleUpdate: diskTimeout set to (200000)ms
[? ? CSSD]2013-12-03 15:27:50.427 [1587963200] >USER:? ? NMEVENT_SUSPEND [00][00][00][00]
[? ? CSSD]2013-12-03 15:27:50.432 [1284950336] >TRACE:? ?clssgmReconfigThread:??started for reconfig (12)
[? ? CSSD]2013-12-03 15:27:50.432 [1284950336] >USER:? ? NMEVENT_RECONFIG [00][00][00][06]
[? ? CSSD]2013-12-03 15:27:50.433 [1284950336] >TRACE:? ?clssgmEstablishConnections: 2 nodes in cluster incarn 12
[? ? CSSD]2013-12-03 15:27:50.434 [1242990912] >TRACE:? ?clssgmInitialRecv: (0xede5f10) accepted a new connection from node 2 born at 10 active (2, 2), vers (10,3,1,2)
[? ? CSSD]2013-12-03 15:27:50.434 [1242990912] >TRACE:? ?clssgmInitialRecv: conns done (2/2)
[? ? CSSD]2013-12-03 15:27:50.434 [1284950336] >TRACE:? ?clssgmEstablishMasterNode: MASTER for 12 is node(2) birth(10)
[? ? CSSD]2013-12-03 15:27:50.434 [1284950336] >TRACE:? ?clssgmChangeMasterNode: requeued 0 RPCs
[? ? CSSD]2013-12-03 15:27:50.438 [1242990912] >TRACE:? ?clssgmHandleDBDone(): src/dest (2/65535) size(72) incarn 12
[? ? CSSD]CLSS-3000: reconfiguration successful, incarnation 12 with 2 nodes
[? ? CSSD]CLSS-3001: local node number 1, master node number 2
[? ? CSSD]2013-12-03 15:27:50.438 [1284950336] >TRACE:? ?clssgmReconfigThread:??completed for reconfig(12), with status(1)
[? ? CSSD]2013-12-03 15:27:50.511 [1242990912] >TRACE:? ?clssgmCommonAddMember: clsomon joined (1/0x1000000/#CSS_CLSSOMON)
rac 的節(jié)點被驅逐，最常見的是網(wǎng)絡心跳異常 或 磁盤心跳異常導致節(jié)點被驅逐，從而導致系統(tǒng)異常重啟。
從上面的日志來看，是屬于網(wǎng)絡心跳異常？還是磁盤心跳異常？
謝謝。
總結
以上是生活随笔為你收集整理的oracle集群断电重启,Oracle10.2.0.4 RAC 2节点集群节点被驱逐导致系统异常重启的全部內容，希望文章能夠幫你解決所遇到的問題。
- 上一篇： #读书笔记# 《人类简史》Chapter
- 下一篇： bzoj 3375: [Usaco200