Skip to main content

Adding Quorum Disks in Exadata


<<Back to Exadata Main Page

What is Quorum Disk

A quorum disk is same as voting disk holds same information and used for the same cause. The only difference is in the syntax of its administration and management. You can identify the quorum disk by querying FAILGROUP_TYPE column of  v$asm_disk.
Login to asm instance as sysasm
SQL> select a.GROUP_NUMBER, b.name group_name, a.DISK_NUMBER, a.PATH, a.name, a.TOTAL_MB, a.FREE_MB, a.failgroup_type from v$asm_disk a, v$asm_diskgroup b where a.group_number = b.group_number and a.group_number = 1;

 GROUP_NUMBER    GROUP_NAME   DISK_NUMBER   NAME                                              FAILGROUP_TYPE
------------                      ----------                 -----------              ------------------------------                     ---------------------
           1                         DATAC3                19                       QD_DATAC3_DB02                      QUORUM
           1                         DATAC3                18                       QD_DATAC3_DB01                      QUORUM

Exadata- Adding Quorum Disks to Database Nodes

If you have Oracle Exadata rack which has fewer than five storage servers and you have 2 database nodes and you want to configure HIGH redundancy diskgroup for Voting disk then you can use Quorum Disk Manager utility introduced in Oracle Exadata release 12.1.2.3.0 to create and maintain additional required quorum disks

Software Requirements for Quorum Disk Manager

  • Oracle Exadata software release 12.1.2.3.0 and above
  • Patch 23200778 for all Database homes
  • Oracle Grid Infrastructure 12.1.0.2.160119 with patches 22722476 and 22682752, or Oracle Grid Infrastructure 12.1.0.2.160419 and above
Step1> Find out the network interfaces to be used for communication with the iSCSI devices using the following command
[oracle@host]$ oifcfg getif | grep cluster_interconnect | awk '{print $1}'
ib0
ib1
Step2> Find the IP address of each interface using the following command. The IP address in my example is 192.168.10.4[5-8]
ip addr show interface_name
ON DB01
[oracle@host1]$ ip addr show ib0
192.168.10.45
192.168.10.46
ON DB02
[oracle@host2]$ ip addr show ib1
192.168.10.47
192.168.10.48
On Both Nodes DB01 and DB02 as root User
Step3> Run the quorumdiskmgr command with the --create --config options to create quorum disk configurations on both db01 and db02
[root@DB01]#/opt/oracle.SupportTools/quorumdiskmgr --create --config --owner=oracle --group=dba --network-iface-list="ib0, ib1"
[Info] Successfully created iface exadata_ib0 with iface.net_ifacename ib0
[Info] Successfully created iface exadata_ib1 with iface.net_ifacename ib1
[Success] Successfully created quorum disk configurations

[root@DB02]#/opt/oracle.SupportTools/quorumdiskmgr --create --config --owner=oracle --group=dba --network-iface-list="ib0, ib1"
[Info] Successfully created iface exadata_ib0 with iface.net_ifacename ib0
[Info] Successfully created iface exadata_ib1 with iface.net_ifacename ib1
[Success] Successfully created quorum disk configurations
Step4>Run the quorumdiskmgr command with the --list --config options to verify that the configurations have been successfully created on both db01 and db02
[root@DB01]#/opt/oracle.SupportTools/quorumdiskmgr --list --config
Owner: oracle
Group: dba
ifaces: exadata_ib0 exadata_ib1
[root@DB02]#/opt/oracle.SupportTools/quorumdiskmgr --list --config
Owner: oracle
Group: dba
ifaces: exadata_ib0 exadata_ib1
Step5>Run the quorumdiskmgr command with the --create --target options to create a target on both db01 and db02 for Oracle ASM disk group DATAC1 and make the target visible to both db01 and db02
[root@DB01]# /opt/oracle.SupportTools/quorumdiskmgr --create --target --asm-disk-group=DATAC3 --visible-to="192.168.10.45, 192.168.10.46, 192.168.10.47, 192.168.10.48"
[Success] Successfully created target iqn.2015-05.com.oracle:QD_DATAC3_DB01.
[root@DB02]#/opt/oracle.SupportTools/quorumdiskmgr --create --target --asm-disk-group=DATAC3 --visible-to="192.168.10.45, 192.168.10.46, 192.168.10.47, 192.168.10.48"
[Success] Successfully created target iqn.2015-05.com.oracle:QD_DATAC3_DB02.
Step6> Run the quorumdiskmgr command with the --list --target options to verify the target has been successfully created on both db01 and db02
[root@DB01]# /opt/oracle.SupportTools/quorumdiskmgr --list --target
Name: iqn.2015-05.com.oracle:QD_DATAC3_DB01
Host name: DB01
ASM disk group name: DATAC3
Size: 128 MB
Visible to: 192.168.10.45, 192.168.10.46, 192.168.10.47, 192.168.10.48
Discovered by:
[root@DB02]# /opt/oracle.SupportTools/quorumdiskmgr --list --target
Name: iqn.2015-05.com.oracle:QD_DATAC3_DB02
Host name: DB02
ASM disk group name: DATAC3
Size: 128 MB
Visible to: 192.168.10.45, 192.168.10.46, 192.168.10.47, 192.168.10.48
Discovered by:
Step7> Run the quorumdiskmgr command with the --create --device options to create devices on both db01 and db02 from targets on both db01 and db02.
[root@DB01]# /opt/oracle.SupportTools/quorumdiskmgr --create --device --target-ip-list="192.168.10.45, 192.168.10.46, 192.168.10.47, 192.168.10.48"
[Success] Successfully created all device(s) from target(s) on machine with IP address 192.168.10.45
[Success] Successfully created all device(s) from target(s) on machine with IP address 192.168.10.46
[Success] Successfully created all device(s) from target(s) on machine with IP address 192.168.10.47
[Success] Successfully created all device(s) from target(s) on machine with IP address 192.168.10.48
[root@DB02]# /opt/oracle.SupportTools/quorumdiskmgr --create --device --target-ip-list="192.168.10.45, 192.168.10.46, 192.168.10.47, 192.168.10.48"
[Success] Successfully created all device(s) from target(s) on machine with IP address 192.168.10.45
[Success] Successfully created all device(s) from target(s) on machine with IP address 192.168.10.46
[Success] Successfully created all device(s) from target(s) on machine with IP address 192.168.10.47
[Success] Successfully created all device(s) from target(s) on machine with IP address 192.168.10.48
Step8> Run the quorumdiskmgr command with the --list --device options to verify the devices have been successfully created on both db01 and db02 
[root@DB01~]# /opt/oracle.SupportTools/quorumdiskmgr --list --device
Device path: /dev/exadata_quorum/QD_DATAC3_DB01
Host name: DB01
ASM disk group name: DATAC3
Size: 128 MB
Device path: /dev/exadata_quorum/QD_DATAC3_DB02
Host name: DB02
ASM disk group name: DATAC3
Size: 128 MB
[root@DB02~]# /opt/oracle.SupportTools/quorumdiskmgr --list --device
Device path: /dev/exadata_quorum/QD_DATAC3_DB01
Host name: DB01
ASM disk group name: DATAC3
Size: 128 MB
Device path: /dev/exadata_quorum/QD_DATAC3_DB02
Host name: DB02
ASM disk group name: DATAC3
Size: 128 MB
Step9> login to asm instance and verify the disk
On DB01or DB02 as oracle user
SQL> show parameter asm_diskstring
NOTE: Adjust the asm_diskstring parameter in case it is required to discover the quorum disks /dev/exadata_quorum/*
SQL>set linesize 200
col path format a50
col path format a30
select inst_id, label, path, mode_status, header_status from gv$asm_disk where path like '/dev/exadata_quorum/%'; 
  INST_ID LABEL                          PATH                                               MODE_STATUS           HEADER_STATUS
---------- ------------------------------ -------------------------------------------------- --------------------- ------------------------------------
         1 QD_DATAC3_DB02          /dev/exadata_quorum/QD_DATAC3_DB02          ONLINE                CANDIDATE
         1 QD_DATAC3_DB01          /dev/exadata_quorum/QD_DATAC3_DB01          ONLINE                CANDIDATE
         2 QD_DATAC3_DB02          /dev/exadata_quorum/QD_DATAC3_DB02          ONLINE                CANDIDATE
         2 QD_DATAC3_DB01          /dev/exadata_quorum/QD_DATAC3_DB01          ONLINE                CANDIDATE
Step10> Add the discovered candidate disks as Quorumdisk or create a new diskgroup as applicable
SQL> alter diskgroup DATAC3 add quorum failgroup "DB01" disk '/dev/exadata_quorum/QD_DATAC3_DB01' quorum failgroup "DB02" disk '/dev/exadata_quorum/QD_DATAC3_DB02';
OR
SQL> create diskgroup DATAC1 high redundancy quorum failgroup db01 disk '/dev/exadata_quorum/QD_ DATAC1_DB01' quorum failgroup db02 disk '/dev/exadata_quorum/QD_ DATAC1_DB02' ...
Step11> Wait for the rebalancing operation to complete
SQL> select * from v$asm_operation;
Step12> Check the votingdisk status
[oracle@DB01]$ crsctl query css votedisk;
##  STATE    File Universal Id                File Name Disk group
................................................................................................................................................................

................................................................................................................................................................
 4. ONLINE   ff198b7b1ff74fbcbf8a7baedd669b22 (/dev/exadata_quorum/QD_DATAC3_DB02) [DATAC3]
 5. ONLINE   3990c6fc37f44f2bbff3e31c23785803 (/dev/exadata_quorum/QD_DATAC3_DB01) [DATAC3]
Located 5 voting disk(s).
If Applicable
Step13>
Relocate the existing voting files from the normal redundancy disk group to the high redundancy disk group.
$Grid_home/bin/crsctl replace votedisk +DATAC1
Step14>Verify the voting disks have been successfully relocated to the high redundancy disk group and that five voting files exist
$crsctl query css votedisk


Comments

Post a Comment

Popular posts from this blog

DataPump Import Of Object Types Fails With Errors ORA-39083 ORA-2304 Or ORA-39117 ORA-39779

<<Back to Oracle DATAPUMP Main Page ORA-39083: Object type TYPE:"TEST_QA01"."LOG_RECORD" failed to create with error: ORA-02304: invalid object identifier literal Import: Release 12.1.0.2.0 - Production on Tue May 29 07:59:12 2018 Copyright (c) 1982, 2014, Oracle and/or its affiliates.  All rights reserved. Connected to: Oracle Database 12c Enterprise Edition Release 12.1.0.2.0 - 64bit Production With the Partitioning, Real Application Clusters, Automatic Storage Management, Oracle Label Security, OLAP, Advanced Analytics and Real Application Testing options Master table "SYSTEM"."SYS_IMPORT_FULL_01" successfully loaded/unloaded Starting "SYSTEM"."SYS_IMPORT_FULL_01":  system/********@TEST_QA parfile=import_TEST.par Processing object type SCHEMA_EXPORT/USER Processing object type SCHEMA_EXPORT/SYSTEM_GRANT Processing object type SCHEMA_EXPORT/ROLE_GRANT Processing object type SCHEMA_EXPORT/DEFAULT_ROLE Pr
Post a Comment
Read more

ORA-28374: typed master key not found in wallet

<<Back to Oracle DB Security Main Page ORA-46665: master keys not activated for all PDBs during REKEY SQL> ADMINISTER KEY MANAGEMENT SET KEY FORCE KEYSTORE IDENTIFIED BY xxxx WITH BACKUP CONTAINER = ALL ; ADMINISTER KEY MANAGEMENT SET KEY FORCE KEYSTORE IDENTIFIED BY xxxx WITH BACKUP CONTAINER = ALL * ERROR at line 1: ORA-46665: master keys not activated for all PDBs during REKEY I found following in the trace file REKEY: Create Key in PDB 3 resulted in error 46658 *** 2019-02-06T15:27:04.667485+01:00 (CDB$ROOT(1)) REKEY: Activation of Key AdnU5OzNP08Qv1mIyXhP/64AAAAAAAAAAAAAAAAAAAAAAAAAAAAA in PDB 3 resulted in error 28374 REKEY: Keystore needs to be restored from the REKEY backup.Aborting REKEY! Cause: All this hassle started because I accidently deleted the wallet and all wallet backup files too and also forgot the keystore password. There was no way to restore the wallet back. Fortunately in my case the PDB which had encrypted data was supposed to be deco
7 comments
Read more

How to Find VIP of an Oracle RAC Cluster

<<Back to Oracle RAC Main Page How to Find Out VIP of an Oracle RAC Cluster Login clusterware owner (oracle) and execute the below command to find out the VIP hostname used in Oracle RAC $ olsnodes -i node1     node1-vip node2     node2-vip OR $ srvctl config nodeapps -viponly Network 1 exists Subnet IPv4: 10.0.0.0/255.255.0.0/bondeth0, static Subnet IPv6: Ping Targets: Network is enabled Network is individually enabled on nodes: Network is individually disabled on nodes: VIP exists: network number 1, hosting node node1 VIP Name: node1-vip VIP IPv4 Address: 10.0.0.1 VIP IPv6 Address: VIP is enabled. VIP is individually enabled on nodes: VIP is individually disabled on nodes: VIP exists: network number 1, hosting node node2 VIP Name: node2-vip VIP IPv4 Address: 10.0.0.2 VIP IPv6 Address: VIP is enabled. VIP is individually enabled on nodes: VIP is individually disabled on nodes:
1 comment
Read more

How to Power On/off Oracle Exadata Machine

<<Back to Exadata Main Page How to Power On/off Oracle Exadata Machine Oracle Exadata machines can be powered on/off either by pressing the power button on front of the server or by logging in to the ILOM interface. Powering on servers using  button on front of the server The power on sequence is as follows. 1. Start Rack, including switches  Note:- Ensure the switches have had power applied for a few minutes to complete power on  configuration before starting Exadata Storage Servers 2.Start Exadata Storage Servers  Note:- Ensure all Exadata Storage Servers complete the boot process before starting the   database servers 3. Start Database Servers Powering On Servers Remotely using ILOM The ILOM can be accessed using the Web console, the command-line interface (CLI), IPMI, or SNMP. For example, to apply power to server dm01cel01 using IPMI, where dm01cel01-ilom is the host name of the ILOM for the server to be powered on, run the
Post a Comment
Read more

How to Attach to a Datapump Job and Check Status of Export or Import

<<Back to Oracle DATAPUMP Main Page How to check the progress of  export or import Jobs You can attach to the export/import  job using ATTACH parameter of oracle datapump utility. Once you are attached to the job you check its status by typing STATUS command. Let us see how Step1>  Find the Export/Import Job Name You can find the datapump job information from  DBA_DATAPUMP_JOBS or  USER_DATAPUMP_JOBS view. SQL> SELECT OWNER_NAME,JOB_NAME,OPERATION,JOB_MODE,STATE from DBA_DATAPUMP_JOBS; OWNER_NAME JOB_NAME                       OPERATION            JOB_MODE   STATE ---------- ------------------------------ -------------------- ---------- ---------- SYSTEM     SYS_EXPORT_FULL_02             EXPORT               FULL       EXECUTING OR You can also find the job name for export/import in logfile in beginning itself. Step2>Attach to the Job and check status One you get the Export/Import Job Name attach the job and check its status. You can attach or det
7 comments
Read more

How to Create Pfile from Spfile and Vice Versa

<<Back to DB Administration Main Page There are instances when a DBA need to start the database using pfile, for example to trouble an instance startup error or to validate init file post parameter changes etc. In such situations you can create a pfile from spfile and once you are done with your changes you can create spfile from updated/modified pfile to start the database. How to Create Pfile from Spfile As sysdba execute following command  SQL> create pfile='/tmp/initOrcl.ora' from spfile; How to Create SPfile from Pfile As sysdba execute following command  SQL> create spfile from  pfile='/tmp/initOrcl.ora'; You can also create the pfile directly from memory How to Create Pfile from Memory As sysdba execute following command  SQL> create  pfile='/tmp/initOrcl.ora' from memory;
3 comments
Read more

How to export only data or only metadata using expdp

<<Back to Oracle DATAPUMP Main Page CONTENT parameter of expdp let you select whether you want to export only data or only metadata or both Default : ALL Syntax and Description CONTENT=[ALL | DATA_ONLY | METADATA_ONLY] DATA_ONLY exports only table row data; no database object definitions are exported. METADATA_ONLY exports only database object definitions; no table row data is exported. Exporting metadata only  $ cat exp_full_pdb01.par directory=dump dumpfile=EXP_PDB01_FULL%U.dmp logfile=EXP_PDB01_FULL.log full=y CONTENT=METADATA_ONLY $ expdp system@PDB01 parfile=exp_full_pdb01.par Exporting data only directory=dump dumpfile=EXP_PDB01_FULL%U.dmp logfile=EXP_PDB01_FULL.log full=y CONTENT=DATA_ONLY $ expdp system@PDB01 parfile=exp_full_pdb01.par
Post a Comment
Read more

Step by Step How to Configure Software Keystore/ Oracle Wallet

<<Back to Oracle DB Security Main Page How to Configure a Software Keystore A software keystore is a container that stores the Transparent Data Encryption master encryption key. To configure a software Keystore follow the steps below. Step 1: Set the Keystore Location in the sqlnet.ora File You can store the software keystore (also known as wallet) in file system or in ASM Diskgroup. Does not matter where you want to store the keystore you have modify the sqlnet.ora and make an entry accordingly Make an entry as shown below in $ORACLE_HOME/network/admin/sqlnet.ora file Example1: If Storing the Wallet in ASM ENCRYPTION_WALLET_LOCATION=  (SOURCE=(METHOD=FILE)    (METHOD_DATA=     (DIRECTORY= +DG_TST_DATA/$ORACLE_SID/wallet )    )  )   Example2: If Storing the Wallet in File System ENCRYPTION_WALLET_LOCATION=  (SOURCE=(METHOD=FILE)    (METHOD_DATA=     (DIRECTORY= /u01/dbatst1/admin/wallet/$ORACLE_SID)    )  ) NOTE: Ensure that the path you entered in  DIREC
7 comments
Read more

ORA-15040: diskgroup is incomplete

<<Back to Oracle ASM Main Page ORA-15040: diskgroup is incomplete SQL> startup ORA-00099: warning: no parameter file specified for ASM instance ASM instance started Total System Global Area 1140850688 bytes Fixed Size                  8629704 bytes Variable Size            1107055160 bytes ASM Cache                  25165824 bytes ORA-15110: no diskgroups mounted Reason: The reason of this error is simply the ASM is not able to find the some or all the disks. Solution: Investigate and make all the disks available to ASM to mount the disk group. Make sure the disks has proper permissions. If you are using AFD check following services are online oracleacfs oracleadvm oracleoks  oracleafd   Source of Problem : Issue started after restart of the server After restarting the server when I tried to start the ASM instance its started throwing error.  ORA-15110: no diskgroups mounted Investigation in my Case Step1> ASM Logfile Scanning  Looked i
2 comments
Read more

ORA-28365: wallet is not open while starting the database

<<Back to DB Administration Main Page ORA-28365: wallet is not open Encountered while Starting the Database $ srvctl start instance -d CDB001 -i CDB0011 PRCR-1013 : Failed to start resource ora.cdb001.db PRCR-1064 : Failed to start resource ora.cdb001.db on node node1.oracle.com CRS-5017: The resource action "ora.cdb001.db start" encountered the following error: ORA-28365: wallet is not open . For details refer to "(:CLSN00107:)" in "/u01/app/oracle/diag/crs/node1.oracle.com/crs/trace/crsd_oraagent_oracle.trc". CRS-2674: Start of 'ora.cdb001.db' on 'node1.oracle.com' failed Solution : Start the instance in mount mode SQL> startup mount; ORACLE instance started. Total System Global Area 2147483648 bytes Fixed Size                  2926472 bytes Variable Size            1392511096 bytes Database Buffers          738197504 bytes Redo Buffers               13848576 bytes Database mounted. Check Wallet status set linesiz
3 comments
Read more