ASM磁盘组冗余的三种类型：external、normal、high,这里恢复的是normal状态,模拟OCR磁盘或votedisk不可用时，RAC会出现什么现象？给出故障定位的整个过程。在11.2.0.3中表决盘是放到了ocr中，所以OCR磁盘或votedisk不可用的两个实验一起做。在11.2.0.3中可ASM磁盘组冗余的三种类型：external、normal、high,这里恢复的是normal状态,模拟OCR磁盘或votedisk不可用时，RAC会出现什么现象？给出故障定位的整个过程。在11.2.0.3中表决盘是放到了ocr中，所以OCR磁盘或votedisk不可用的两个实验一起做。在11.2.0.3中可以手动备份OCR，但手动备份是无效的。ocrconfig-export/u01/ocr.exp检查OCR有哪些备份：[root@rac1~]#ocrconfig-showbackuprac12013/07/2205:39:51/u01/grid/crs/cdata/rac/backup00.ocrrac12013/07/2201:39:51/u01/grid/crs/cdata/rac/backup01.ocrrac12013/07/2121:39:50/u01/grid/crs/cdata/rac/backup02.ocrrac22013/07/2101:52:54/u01/grid/crs/cdata/rac/day.ocrrac22013/07/0901:52:25/u01/grid/crs/cdata/rac/week.ocrPROT-25:ManualbackupsfortheOracleClusterRegistryarenotavailable注意：orcle明确给出了手动备份是无效的！查看表决盘信息：[root@rac1~]#crsctlquerycssvotedisk##STATEFileUniversalIdFileNameDiskgroup------------------------------------------1.ONLINE745716af7e5b4faebfc8d948d096aa55(/dev/oracleasm/disks/OCR_VOT1)[OCR_VOT]2.ONLINE7092079f66c04f9dbf65974d0dcc611a(/dev/oracleasm/disks/OCR_VOT2)[OCR_VOT]3.ONLINE6510631353284f5fbf3d4c8839822dbd(/dev/oracleasm/disks/OCR_VOT3)[OCR_VOT]Located3votingdisk(s).#停库：[root@rac1~]#srvctlstopdatabase-dorcl-oimmediate#停集群：[root@rac1~]#crsctlstopcluster-all-f#破坏OCR和VOT：[root@rac1~]#ddif=/dev/zerof=/dev/mapper/mpathap1bs=1024Kcount=1记录了1+0的读入记录了1+0的写出1048576字节(1.0MB)已复制，0.0160613秒，65.3MB/秒[root@rac1~]#ddif=/dev/zerof=/dev/mapper/mpathap2bs=1024Kcount=1记录了1+0的读入记录了1+0的写出1048576字节(1.0MB)已复制，0.00800275秒，131MB/秒[root@rac1~]#ddif=/dev/zerof=/dev/mapper/mpathap3bs=1024Kcount=1记录了1+0的读入记录了1+0的写出1048576字节(1.0MB)已复制，0.00927389秒，113MB/秒注意：破坏后，各节点服务一切正常：[root@rac1~]#crs_stat-tNameTypeTargetStateHost------------------------------------------------------------ora.DATA.dgora.up.typeONLINEONLINErac1ora.FRA.dgora.up.typeONLINEONLINErac1ora.ER.lsnrora.er.typeONLINEONLINErac1ora.N1.lsnrora.er.typeONLINEONLINErac1ora.OCR_VOT.dgora.up.typeONLINEONLINErac1ora.asmora.asm.typeONLINEONLINErac1ora.orcl.dbora.se.typeONLINEONLINErac1ora.cvuora.cvu.typeONLINEONLINErac1ora.SM1.asmapplicationONLINEONLINErac1ora.C1.lsnrapplicationONLINEONLINErac1ora.ac1.gsdapplicationOFFLINEOFFLINEora.ac1.onsapplicationONLINEONLINErac1ora.ac1.vipora.t1.typeONLINEONLINErac1ora.SM2.asmapplicationONLINEONLINErac2ora.C2.lsnrapplicationONLINEONLINErac2ora.ac2.gsdapplicationOFFLINEOFFLINEora.ac2.onsapplicationONLINEONLINErac2ora.ac2.vipora.t1.typeONLINEONLINErac2ora.gsdora.gsd.typeOFFLINEOFFLINEora.networkora.rk.typeONLINEONLINErac1ora.oc4jora.oc4j.typeONLINEONLINErac1ora.onsora.ons.typeONLINEONLINErac1ora.ry.acfsora.fs.typeONLINEONLINErac1ora.scan1.vipora.ip.typeONLINEONLINErac1所有节点重启操作系统后集群服务启不来了：[root@rac1~]#reboot如果只是停止集群服务，后面的重新创建ASM磁盘组会失败，但重启操作系统后，就可以创建成功。检查CRS：[grid@rac1~]$crsctlcheckcrsCRS-4638:OracleHighAvailabilityServicesisonlineCRS-4535:CannotcommunicatewithClusterReadyServicesCRS-4530:CommunicationsfailurecontactingClusterSynchronizationServicesdaemonCRS-4534:CannotcommunicatewithEventManager启动集群服务：[root@rac1~]#crsctlstartcluster-allCRS-2672:尝试启动'ora.cssdmonitor'(在'rac1'上)CRS-2672:尝试启动'ora.cssdmonitor'(在'rac2'上)CRS-2676:成功启动'ora.cssdmonitor'(在'rac1'上)CRS-2676:成功启动'ora.cssdmonitor'(在'rac2'上)CRS-2672:尝试启动'ora.cssd'(在'rac1'上)CRS-2672:尝试启动'ora.diskmon'(在'rac1'上)CRS-2672:尝试启动'ora.cssd'(在'rac2'上)CRS-2672:尝试启动'ora.diskmon'(在'rac2'上)CRS-2676:成功启动'ora.diskmon'(在'rac1'上)CRS-2676:成功启动'ora.diskmon'(在'rac2'上)#直停在这里其他终端使用其他命令启动集群服务：[root@rac1~]#crsctlstartcrsCRS-4640:OracleHighAvailabilityServicesisalreadyactiveCRS-4000:CommandStartfailed,orcompletedwitherrors.操作系统及crs日志中没看到特别有用的信息：[root@rac1~]#vi/var/log/messages[grid@rac1~]#vi$ORACLE_HOME/log/rac1/crsd/crsd.logocss日志中提示：vi$ORACLE_HOME/log/rac1/cssd/ocssd.log2013-07-2121:15:08.550:[CSSD][1095031104]clssnmvFindInitialConfigs:Novotingfilesfound发现部分ASM磁盘没有了：[root@rac1~]#/etc/init.d/oracleasmscandisksScanningthesystemforOracleASMLibdisks:[OK][root@rac1~]#/etc/init.d/oracleasmlistdisksDATAFRA依照RAC安装文档重建ASM磁盘：[root@rac1~]#/etc/init.d/oracleasmcreatediskOCR_VOT1/dev/mapper/mpathap1Markingdisk"OCR_VOT1"asanASMdisk:[OK][root@rac1~]#/etc/init.d/oracleasmcreatediskOCR_VOT2/dev/mapper/mpathap2Markingdisk"OCR_VOT2"asanASMdisk:[OK][root@rac1~]#/etc/init.d/oracleasmcreatediskOCR_VOT3/dev/mapper/mpathap3Markingdisk"OCR_VOT3"asanASMdisk:[OK]停掉集群服务：要加-f，否则可能停止非常慢[root@rac1~]#crsctlstopcrs-fCRS-2791:StartingshutdownofOracleHighAvailabilityServices-managedresourceson'rac1'CRS-2673:Attemptingtostop'ora.mdnsd'on'rac1'CRS-2673:Attemptingtostop'ora.crf'on'rac1'CRS-2677:Stopof'ora.mdnsd'on'rac1'succeededCRS-2677:Stopof'ora.crf'on'rac1'succeededCRS-2673:Attemptingtostop'ora.gipcd'on'rac1'CRS-2677:Stopof'ora.gipcd'on'rac1'succeededCRS-2673:Attemptingtostop'ora.gpnpd'on'rac1'CRS-2677:Stopof'ora.gpnpd'on'rac1'succeededCRS-2793:ShutdownofOracleHighAvailabilityServices-managedresourceson'rac1'hascompletedCRS-4133:OracleHighAvailabilityServiceshasbeenstopped.以-excl-nocrs方式启动集群，这将启动ASM实例但不启动CRS[root@rac1~]#crsctlstartcrs-excl-nocrsCRS-4123:OracleHighAvailabilityServiceshasbeenstarted.CRS-2672:Attemptingtostart'ora.mdnsd'on'rac1'CRS-2676:Startof'ora.mdnsd'on'rac1'succeededCRS-2672:Attemptingtostart'ora.gpnpd'on'rac1'CRS-2676:Startof'ora.gpnpd'on'rac1'succeededCRS-2672:Attemptingtostart'ora.cssdmonitor'on'rac1'CRS-2672:Attemptingtostart'ora.gipcd'on'rac1'CRS-2676:Startof'ora.cssdmonitor'on'rac1'succeededCRS-2676:Startof'ora.gipcd'on'rac1'succeededCRS-2672:Attemptingtostart'ora.cssd'on'rac1'CRS-2672:Attemptingtostart'ora.diskmon'on'rac1'CRS-2676:Startof'ora.diskmon'on'rac1'succeededCRS-2676:Startof'ora.cssd'on'rac1'succeededCRS-2672:Attemptingtostart'ora.drivers.acfs'on'rac1'CRS-2679:Attemptingtoclean'ora.cluster_interconnect.haip'on'rac1'CRS-2672:Attemptingtostart'ora.ctssd'on'rac1'CRS-2681:Cleanof'ora.cluster_interconnect.haip'on'rac1'succeededCRS-2672:Attemptingtostart'ora.cluster_interconnect.haip'on'rac1'CRS-2676:Startof'ora.drivers.acfs'on'rac1'succeededCRS-2676:Startof'ora.ctssd'on'rac1'succeededCRS-2676:Startof'ora.cluster_interconnect.haip'on'rac1'succeededCRS-2672:Attemptingtostart'ora.asm'on'rac1'CRS-2676:Startof'ora.asm'on'rac1'succeeded此时crs仍然报错：[root@rac1~]#crs_stat-tCRS-0184:CannotcommunicatewiththeCRSdaemon.[root@rac1~]#crsctlcheckcrsCRS-4638:OracleHighAvailabilityServicesisonlineCRS-4535:CannotcommunicatewithClusterReadyServicesCRS-4530:CommunicationsfailurecontactingClusterSynchronizationServicesdaemonCRS-4534:CannotcommunicatewithEventManager重建原ocr和votedisk所在磁盘组：注意：这里是在grid用户下SQL>colpathfora50SQL>setlines300SQL>selectpath,header_statusfromv$asm_disk;SQL>creatediskgroupOCR_VOTnormalredundancydisk'/dev/oracleasm/disks/OCR_VOT1','/dev/oracleasm/disks/OCR_VOT2','/dev/oracleasm/disks/OCR_VOT3'attribute'compatible.rdbms'='11.2','compatible.asm'='11.2';ASM磁盘组冗余的三种类型：external、normal、high，我这里之前用的是normal。从ocrbackup中恢复OCR：在每个节点grid用户下：cd$ORACLE_HOME/cdata/racocrconfig-restore/u01/grid/crs/cdata/rac/backup00.ocr恢复表决盘的准备工作：showparameterasm_diskstring如果asm_diskstring没有值，表示ASM磁盘用的是默认ASM磁盘搜索路径。修改成实际的ASM磁盘搜索路径：altersystemsetasm_diskstring='/dev/oracleasm/disks/*';恢复表决盘：[root@rac1~]#crsctlreplacevotedisk+OCR_VOTSuccessfuladditionofvotingdisk4ad2b9cc0a754fffbf1515281199a78f.Successfuladditionofvotingdisk9f8dc1c013df4f39bfd85c64051a0bc1.Successfuladditionofvotingdiska4aea7a1aa434fb3bff161f6ea8ce102.Successfullyreplacedvotingdiskgroupwith+OCR_VOT.CRS-4266:Votingfile(s)successfullyreplacedocr和vot恢复后，crs等服务就会自动起来了。[root@rac1~]#crsctlcheckcrsCRS-4638:OracleHighAvailabilityServicesisonlineCRS-4535:CannotcommunicatewithClusterReadyServicesCRS-4529:ClusterSynchronizationServicesisonlineCRS-4534:CannotcommunicatewithEventManager[root@rac1~]#crsctlquerycssvotedisk##STATEFileUniversalIdFileNameDiskgroup------------------------------------------1.ONLINE4ad2b9cc0a754fffbf1515281199a78f(/dev/oracleasm/disks/OCR_VOT1)[OCR_VOT]2.ONLINE9f8dc1c013df4f39bfd85c64051a0bc1(/dev/oracleasm/disks/OCR_VOT2)[OCR_VOT]3.ONLINEa4aea7a1aa434fb3bff161f6ea8ce102(/dev/oracleasm/disks/OCR_VOT3)[OCR_VOT]Located3votingdisk(s).重启集群服务，检查是否已经恢复正常：[root@rac1~]#crsctlstopcrs[root@rac1~]#crsctlstartcrs
某用户增加LUN到ASM DISKGROUP发现某个ASM Disk header KFBTYP_DISKHEAD被意外清除掉，导致该Diskgroup无法mount的问题， 后续DBA采用kfed merge等手法修复了KFBTYP_DISKHEAD block，但仍无法mount diskgroup，alert.log中出现如下的日志：
NOTE: F1X0 found on disk 0 fcn 0.0
NOTE: cache opening disk 1 of grp 1: VOL2 label:VOL2
NOTE: cache opening disk 2 of grp 1: VOL3 label:VOL3
NOTE: cache opening disk 3 of grp 1: VOL4 label:VOL4
NOTE: cache opening disk 4 of grp 1: VOL5 label:VOL5
NOTE: cache opening disk 5 of grp 1: VOL6 label:VOL6
NOTE: cache opening disk 6 of grp 1: VOL7 label:VOL7
NOTE: cache opening disk 7 of grp 1: VOL8 label:VOL8
NOTE: cache opening disk 8 of grp 1: VOL9 label:VOL9
NOTE: cache opening disk 9 of grp 1: VOL10 label:VOL10
NOTE: cache opening disk 10 of grp 1: VOL11 label:VOL11
NOTE: cache mounting (first) group 1/0x3A2C35D6 (DG)
* allocate domain 1, invalid = TRUE
kjbdomatt send to node 0
kjbdomatt send to node 2
Mon Jan 27 02:18:51 CST 2014
NOTE: attached to recovery domain 1
Mon Jan 27 02:18:51 CST 2014
NOTE: starting recovery of thread=1 ckpt=1712.152 group=1
NOTE: advancing ckpt for thread=1 ckpt=1712.153
NOTE: cache recovered group 1 to fcn 0.491275704
Mon Jan 27 02:18:51 CST 2014
NOTE: LGWR attempting to mount thread 1 for disk group 1
NOTE: LGWR mounted thread 1 for disk group 1
NOTE: opening chunk 1 at fcn 0.491275704 ABA
NOTE: seq=1713 blk=154
Mon Jan 27 02:18:51 CST 2014
NOTE: cache mounting group 1/0x3A2C35D6 (DG) succeeded
SUCCESS: diskgroup DG was mounted
Mon Jan 27 02:18:53 CST 2014
NOTE: recovering COD for group 1/0x3a2c35d6 (DG)
WARNING: cache read a corrupted block gn=1 dsk=0 blk=2817 from disk 0
NOTE: a corrupted block was dumped to the trace file
ERROR: cache failed to read dsk=0 blk=2817 from disk(s): 0
ORA-15196: invalid ASM block header [kfc.c:8281] [endian_kfbh] [2147483648] [2817] [173 != 1]
System State dumped to trace file /u01/app/oracle/admin/+ASM/bdump/+asm2_rbal_31204.trc
NOTE: cache initiating offline of disk 0 group 1
WARNING: process 31204 initiating offline of disk 0.3913073997 (VOL1) with mask 0x3 in group 1
WARNING: Disk 0 in group 1 in mode: 0x7,state: 0x2 will be taken offline
NOTE: PST update: grp = 1, dsk = 0, mode = 0x6
Mon Jan 27 02:18:54 CST 2014
ERROR: too many offline disks in PST (grp 1)
Mon Jan 27 02:18:54 CST 2014
WARNING: Disk 0 in group 1 in mode: 0x7,state: 0x2 was taken offline
Mon Jan 27 02:18:54 CST 2014
NOTE: halting all I/Os to diskgroup DG
NOTE: active pin found: 0x0x65faff60
NOTE: active pin found: 0x0x65fb0170
NOTE: active pin found: 0x0x65fb0010
NOTE: active pin found: 0x0x65fb0220
NOTE: active pin found: 0x0x65fb02d0
NOTE: active pin found: 0x0x65fb00c0
NOTE: active pin found: 0x0x65fb0380
Mon Jan 27 02:18:54 CST 2014
ERROR: ORA-15130 in COD recovery for diskgroup 1/0x3a2c35d6 (DG)
ERROR: ORA-15130 thrown in RBAL for group number 1
Mon Jan 27 02:18:54 CST 2014
Errors in file /u01/app/oracle/admin/+ASM/bdump/+asm2_rbal_31204.trc:
ORA-15130: diskgroup "DG" is being dismounted
Mon Jan 27 02:18:54 CST 2014
ERROR: PST-initiated MANDATORY DISMOUNT of group DG
NOTE: cache dismounting group 1/0x3A2C35D6 (DG)
Mon Jan 27 02:18:57 CST 2014
kjbdomdet send to node 0
detach from dom 1, sending detach message to node 0
kjbdomdet send to node 2
detach from dom 1, sending detach message to node 2
Mon Jan 27 02:18:57 CST 2014
Dirty detach reconfiguration started (old inc 23, new inc 23)
List of nodes:
0 1 2
Global Resource Directory partially frozen for dirty detach
* dirty detach - domain 1 invalid = TRUE
138 GCS resources traversed, 0 cancelled
6104 GCS resources on freelist, 6124 on array, 6124 allocated
Dirty Detach Reconfiguration complete
Mon Jan 27 02:18:57 CST 2014
freeing rdom 1
Mon Jan 27 02:18:57 CST 2014
WARNING: dirty detached from domain 1
Mon Jan 27 02:18:57 CST 2014
SUCCESS: diskgroup DG was dismounted
Mon Jan 27 02:18:57 CST 2014
WARNING: PST-initiated MANDATORY DISMOUNT of group DG not performed - group not mounted
Mon Jan 27 02:18:57 CST 2014
Errors in file /u01/app/oracle/admin/+ASM/bdump/+asm2_b001_31755.trc:
ORA-15001: diskgroup "DG" does not exist or is not mounted
ORA-15001: diskgroup "DG" does not exist or is not mounted
ORA-15001: diskgroup "DG" does not exist or is not mounted
Mon Jan 27 02:31:00 CST 2014
这里可以看到Diskgroup mount到了recovering COD for group 1/0x3a2c35d6 (DG)阶段时，发现了一个逻辑坏块WARNING: cache read a corrupted block gn=1 dsk=0 blk=2817 from disk 0 NOTE: a corrupted block was dumped to the trace file ERROR: cache failed to read dsk=0 blk=2817 from disk(s): 0，并因为该坏块引起了ORA-15196: invalid ASM block header [kfc.c:8281] [endian_kfbh] [2147483648] [2817] [173 != 1]。
这里2817是出错的ASM metadata的block number，173是实际从endian_kfbh位置读出的值，173!=1 这里的1是该位置理论上该有的值，由于读取到block中错误的字节序endian_kfbh信息，所以这里出现了ASM ORA-600错误。
这里recovering COD for group 1/0x3a2c35d6 (DG) 里的COD 指asm metadata file number 4 COD, Continuing Operation Directory (COD) 该metadata file 4 中记录的是在单个metadata block中无法完成的操作记录到COD中，这样当ASM instance crash时可以恢复这些操作。例如创建 删除和resize文件，这其中file number 4 blkn=1为KFBTYP_COD_RB 即回滚rollback数据，后面的数据为KFBTYP_COD_DATA。
可回滚的操作opcodes包括：
1 - Create a file
2 - Delete a file
3 - Resize a file
4 - Drop alias entry
5 - Rename alias entry
6 - Rebalance space COD
7 - Drop disks force
8 - Attribute drop
9 - Disk Resync
10 - Disk Repair Time
11 - Volume create
12 - Volume delete
13 - Attribute directory creation
14 - Set zone attributes
15 - User drop
每次ASM diskgroup 尝试mount时都会读取FILE number 4 COD中的数据来保证操作要么完成、要么回滚。
对于此类ASM file number 4 COD出现了源数据坏块的情况， 一般需要手动设置内部事件，并尝试手动Patch ASM metadata的手法才能修复。
建议遇到此类事件第一时间备份ASM disk header 100M的数据，保护现场，以便专业恢复人员介入恢复时现场不被破坏。
如果自己搞不定可以找ASKMACLEAN专业ORACLE数据库修复团队成员帮您恢复!
