Migrate Exadata X3 to Exadata X8 using Standby

Here are some pre-configuration high-level steps that one should consider before installing an Exadata X8 system:

Verify that the hardware and software meet the minimum requirements for Exadata X8. One can find the hardware and software requirements in the Oracle Exadata Database Machine Installation Guide: https://docs.oracle.com/en/engineered-systems/exadata-database-machine/exadata-database-machine-installation-guide/

Review the Exadata X8 installation documentation and familiarize yourself with the installation process. One can find the installation documentation here: https://docs.oracle.com/en/engineered-systems/exadata-database-machine/exadata-database-machine-installation-guide/

Plan the network configuration for the Exadata X8 system. This includes configuring the Ethernet and InfiniBand networks, assigning IP addresses and hostnames, and setting up DNS and NTP servers. One can find detailed information on network configuration in the Oracle Exadata Database Machine Installation and Configuration Guide: Network Configuration: https://docs.oracle.com/en/engineered-systems/exadata-database-machine/exadata-database-machine-installation-configuration-guide/network-configuration.html

Prepare the existing Oracle databases for migration to Exadata X8. This includes identifying any database features or configurations that may need to be modified, such as database parameters or storage settings. One can find information on migrating databases to Exadata in the Oracle Exadata Database Machine Migration Guide: https://docs.oracle.com/en/engineered-systems/exadata-database-machine/exadata-database-machine-migration-guide/

Ensure that one have the necessary administrative privileges to install and configure the Exadata X8 system. This includes the ability to install software, configure hardware, and create and manage database users and roles.

If one plan to use Oracle Cloud Infrastructure (OCI) with Exadata X8, one should also review the OCI documentation and familiarize yourself with the OCI console and API. One can find the OCI documentation here: https://docs.oracle.com/en-us/iaas/Content/home.htm

These are just a few of the pre-configuration steps that one should consider before installing an Exadata X8 system. I would recommend consulting the Oracle documentation for a more comprehensive list of pre-installation tasks.

~~~~~~~~~~~~~~~~~~~

Install and configure the Exadata X8 hardware components, including the database nodes, storage cells, InfiniBand switches, and PDUs.

Oracle Exadata Database Machine Installation Guide: https://docs.oracle.com/en/engineered-systems/exadata-database-machine/exadata-database-machine-installation-guide/

Configure the network settings, including IP addresses, subnet masks, and hostnames for the database nodes and storage cells.

Oracle Exadata Database Machine Installation and Configuration Guide: Network Configuration: https://docs.oracle.com/en/engineered-systems/exadata-database-machine/exadata-database-machine-installation-configuration-guide/network-configuration.html

Install and configure the Oracle Linux operating system on the database nodes and storage cells.

Oracle Exadata Database Machine Installation and Configuration Guide: Operating System Installation: https://docs.oracle.com/en/engineered-systems/exadata-database-machine/exadata-database-machine-installation-configuration-guide/operating-system-installation.html

Install the Oracle Database software on the database nodes and configure the Oracle Grid Infrastructure.

Oracle Grid Infrastructure Installation Guide: https://docs.oracle.com/en/database/oracle/oracle-database/19/cwlin/oracle-grid-infrastructure-installation-guide-for-linux.html

Configure the Exadata Storage Software on the storage cells, including setting up the Exadata Storage Server Software.

Oracle Exadata Database Machine Installation and Configuration Guide: Storage Configuration: https://docs.oracle.com/en/engineered-systems/exadata-database-machine/exadata-database-machine-installation-configuration-guide/storage-configuration.html

Please note that the above steps are high-level and there may be additional configuration steps or considerations specific to the environment. I would recommend consulting with an Oracle certified professional for assistance with setting up an Exadata X8 system.

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

The commands to create a standby database on Exadata X8 while the primary database is on Exadata X3:

On the primary database on Exadata X3, create a standby control file:

sql

Code

SQL> ALTER DATABASE CREATE STANDBY CONTROLFILE AS '<file path>';

Replace <file path> with the full path and file name for the standby control file.

Copy the standby control file to the Exadata X8 system:

Use a file transfer utility, such as scp or sftp, to copy the standby control file from the Exadata X3 system to the Exadata X8 system.

On the Exadata X8 system, create a standby database using the standby control file:

sql

Code

SQL> CREATE CONTROLFILE REUSE SET DATABASE <database name> RESETLOGS ARCHIVELOG;

SQL> STARTUP NOMOUNT;

SQL> ALTER DATABASE SET STANDBY DATABASE TO '<primary database TNS alias>'';

SQL> ALTER DATABASE MOUNT STANDBY DATABASE;

Replace <database name> with the name of the standby database, and <primary database TNS alias> with the TNS alias for the primary database on the Exadata X3 system.

Configure the standby database to receive redo data from the primary database:

Configure the necessary Oracle Data Guard parameters, such as the LOG_ARCHIVE_DEST_2 parameter, to allow the standby database to receive redo data from the primary database.

Start the managed recovery process:

sql

Code

SQL> ALTER DATABASE RECOVER MANAGED STANDBY DATABASE DISCONNECT;

This will start the managed recovery process, which will apply redo data from the primary database to the standby database.

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

To convert the database on Exadata X3 as the primary and create a standby database on Exadata X8, perform the following steps:

Convert the database on Exadata X3 to a primary database: On the Exadata X3 system, log in to the database as a privileged user and issue the following command to convert the database to a primary database:

sql

Code

SQL> ALTER DATABASE COMMIT TO SWITCHOVER TO PRIMARY;

This will prepare the database for a switchover to a primary role.

Create a standby database on Exadata X8: On the Exadata X8 system, create a new standby database using Oracle Data Guard. This can be done by following the same steps as for creating a standby database on Exadata X3.

Configure the standby database on Exadata X8: Once the standby database is created, configure it to receive redo data from the primary database on Exadata X3. This involves setting up a network connection between the two systems and configuring the necessary Oracle Data Guard parameters.

Start the standby database: Once the standby database is configured, start it up and make sure that it is synchronized with the primary database on Exadata X3.

Switchover to the new primary database: On the Exadata X3 system, issue the following command to perform a switchover to the new primary database on Exadata X8:

sql

Code

SQL> ALTER DATABASE COMMIT TO SWITCHOVER TO STANDBY;

This will switch the roles of the two databases, making the standby database on Exadata X8 the new primary database.

Verify the switchover: Once the switchover is complete, verify that the new primary database is functioning correctly and that all data is being replicated properly to the standby database on Exadata X8.

Delete the old primary database: Once the switchover is complete and verified, the old primary database on Exadata X3 can be deleted or repurposed as needed.

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

CPU, Memory and Storage Ratio for Exadata X3 to Exadata X8 Calculations

 1> Here's a comparison of the hardware specifications of Exadata X3 and X8:

Hardware Component Exadata X3 Exadata X8

CPU Intel Xeon E5-2690 2.9 GHz 8-core Intel Xeon Gold 6230 2.1 GHz 20-core

Memory 512 GB DDR3 1.5 TB DDR4

Storage 22 TB Raw Flash Storage + 168 TB Raw Disk Storage 44 TB Raw Flash Storage + 720 TB Raw Disk Storage

As one can see, the CPU in Exadata X8 has more cores than Exadata X3, which allows it to process more workloads in parallel. The memory capacity is also higher in Exadata X8, which enables it to store and access more data at a faster rate. The storage capacity of Exadata X8 is significantly higher than Exadata X3, which means it can accommodate larger databases.

To convert Exadata X3 to X8, one would need to consider the ratio of each hardware component to ensure a smooth upgrade. For example, if one want to upgrade the CPU, one would need to calculate the ratio of cores and clock speed between the two models to ensure that the new CPU is compatible with the existing motherboard and other components.

2> To calculate the ratio of cores and clock speed between the two models, we can use the following formula:

(Cores X New Clock Speed) / (Old Cores X Old Clock Speed) = Ratio

For example, let's calculate the ratio for the CPUs of Exadata X3 and X8:

Exadata X3 CPU:

Cores: 8

Clock Speed: 2.9 GHz

Exadata X8 CPU:

Cores: 20

Clock Speed: 2.1 GHz

Ratio calculation:

(20 cores X 2.1 GHz) / (8 cores X 2.9 GHz) = 1.81

Therefore, the ratio of cores and clock speed between the Exadata X3 and X8 CPUs is approximately 1.81. This means that the new CPU in Exadata X8 is about 1.81 times faster than the CPU in Exadata X3, which should provide a significant performance improvement.

However, it's important to note that upgrading the CPU alone may not be sufficient for a successful migration, as other hardware components such as memory and storage may also need to be upgraded to ensure compatibility and optimal performance.

3> Here are the memory and storage specifications for Exadata X3 and X8:

Exadata X3:

Memory: 512 GB DDR3

Storage: 22 TB Raw Flash Storage + 168 TB Raw Disk Storage

Exadata X8:

Memory: 1.5 TB DDR4

Storage: 44 TB Raw Flash Storage + 720 TB Raw Disk Storage

To calculate the ratio of memory and storage between the two models, we can use the following formula:

(New Memory or Storage / Old Memory or Storage) = Ratio

a) Memory Ratio Calculation:

(1.5 TB DDR4) / (512 GB DDR3) = 2.93

Therefore, the ratio of memory between the Exadata X3 and X8 is approximately 2.93. This means that the memory in Exadata X8 is almost three times larger than the memory in Exadata X3.

b) Storage Ratio Calculation:

(44 TB Raw Flash Storage + 720 TB Raw Disk Storage) / (22 TB Raw Flash Storage + 168 TB Raw Disk Storage) = 4.49

Therefore, the ratio of storage between the Exadata X3 and X8 is approximately 4.49. This means that the storage in Exadata X8 is almost 4.5 times larger than the storage in Exadata X3.

It's important to note that when upgrading from Exadata X3 to X8, it's not just about the raw size of the memory and storage, but also the technology used. For example, DDR4 memory is faster and more power-efficient than DDR3, and flash storage is faster than disk storage. Upgrading to Exadata X8 will provide both larger memory and storage capacities, as well as access to faster and more efficient technology.

Source: Based on my experience, I referred the hardware specifications for Exadata X3 and X8 are publicly available on the Oracle website and other sources, and the calculations are basic mathematical formulas that are commonly used to compare hardware specifications. However, any confirmed information will be coming from Oracle Support.

ASmLib download

$ rpm -qa | grep asmlib

$ /usr/sbin/oracleasm status
Checking if ASM is loaded: yes
Checking if /dev/oracleasm is mounted: yes

If the ASMLib RPM's are missing in your environment.

Please download the appropriate rpms for your version.

http://www.oracle.com/technetwork/server-storage/linux/asmlib/ol6-1709075.html

Change Hostnames in RAC setup

1> Capture Resource Definitions

Before doing anything, we should capture resource definitions from the current CRS resources. This is an optional step, but it will simplify configuration later.

A single resource definition can be captured with a command $ORA_CRS_HOME/bincrs_stat -p . Here is a small shell script to capture that for every resource and save it into a .cap file. As you will see later these files can be used to easily recreate resources:

for res in `$ORA_CRS_HOME/bin/crs_stat -p | grep "^NAME=" | cut -d = -f 2` ; do
    $ORA_CRS_HOME/bin/crs_stat -p $res >/opt/oracle/resources/$res.cap
done


2> Stop Clusterware

Now you can stop Oracle Clusterware on all nodes using $ORA_CRS_HOME/bin/crsctl stop crs, and then change hostnames. Note that this will stop all databases, listeners, and other resources registered within CRS, so this is the time when outage starts.

3> Rename Hosts
Ask SA to change Hostnames

Please note the following important points with respect to changing hostname.
1> Make sure that aliases in /etc/hosts are amended.
2> Don’t forget to change aliases for VIPs and private IPs. This is not strictly required but you are better off following the standard naming convention (-priv and -vip for interconnect and virtual IP respectively) unless you have really good reason not to. Note that at this stage you should be also able to change IP addresses as well. I didn’t try it, but it should work.
3> Also make sure DNS configuration is also changed by your SA, if your applications use DNS to resolve hostnames.

4> Modify $ORA_CRS_HOME/install/rootconfig

$ORA_CRS_HOME/install/rootconfig is called as part of the root.sh script run after Oracle Clusterware installation. We have to modify it so that it uses different host names.

Generally, you would simply change every appearance of the old hostnames to the new hostnames. If you want to do that in vi, use :%s/old_node/new_node/g. Be careful not to change existing non-relevant parts of the script matching your old hostname. The variables that should be changed are

    CRS_HOST_NAME_LIST
    CRS_NODE_NAME_LIST
    CRS_PRIVATE_NAME_LIST
    CRS_NODELIST
    CRS_NODEVIPS

The latter might need modification if you also change IPs.

At this stage, you can also change your OCR and voting disks locations. The following lines should be changed:

CRS_OCR_LOCATIONS={OCR path},{OCR mirror path}
CRS_VOTING_DISKS={voting disk1 path},{voting disk2 path},{voting disk3 path}

You can also change your cluster name via the variable CRS_CLUSTER_NAME.
5> Cleanup OCR and Voting Disks

You should clear OCR and voting disks, otherwise, the script will refuse to format them. This can be done using dd. In the example below I have mirrored OCR and 3 voting disks:

dd if=/dev/zero of= bs=1024k
dd if=/dev/zero of={OCR2 path} bs=1024k
dd if=/dev/zero of={voting1 path} bs=1024k
dd if=/dev/zero of={voting2 path} bs=1024k
dd if=/dev/zero of={voting3 path} bs=1024k

6> “Break” Clusterware Configuration

rootconfig has some protection  — it checks if Clusterware has been already configured and, if it has, it exits without doing any harm. One way to “break” the configuration and make this script run for a second time is to delete the file /etc/oracle/ocr.loc. (Note that this is a Linux-specific location; other Unix variants might have different path. On HP-UX, for example, it’s something like /var/opt/oracle/ocr.log if I recall correctly.)
Run $ORA_CRS_HOME/install/rootconfig

If everything has gone alright, you should be able to run $ORA_CRS_HOME/install/rootconfig as the root user without any issues. If there are problems, follow the standard CRS troubleshooting procedure — checking /var/log/messages and $ORA_CRS_HOME/log/{nodename} et cetera.

Note that this should be done on every node one by one — sequentially. On the last node of the cluster, the script will try to configure the VIPs, and there is a known bug here if you use a private range IP for VIP. This can be easily fixed by running $ORA_CRS_HOME/bin/vipca manually in graphical mode (i.e. you will need $DISPLAY configured correctly).
Verify Clusterware Configuration and Status

This is a simple check to make sure that all nodes are up and have VIP components configured correctly:

[root@mch10 bin]# $ORA_CRS_HOME/bin/crs_stat -t
Name           Type           Target    State     Host
------------------------------------------------------------
ora.mch10.gsd   application    ONLINE    ONLINE    mch10
ora.mch10.ons   application    ONLINE    ONLINE    mch10
ora.mch10.vip   application    ONLINE    ONLINE    mch10
ora.mch11.gsd   application    ONLINE    ONLINE    mch11
ora.mch11.ons   application    ONLINE    ONLINE    mch11
ora.mch11.vip   application    ONLINE    ONLINE    mch11

7> Adding Listener Resources to CRS

There are two ways to do this — you can either use netca to configure the listener from scratch (you might need to clean it up from listener.ora first), or you can change the configuration manually and register it with CRS from command line. I’ll show how to that manually — obviously, the preferred way when it comes to the real environments. ;-)

First of all, we will need to change the $ORACLE_HOME/network/admin/listener.ora file, and you will probably want to change tnsnames.ora at the same time. You need to replace old node aliases with new ones, and change the IPs if they are used instead of aliases, and if you changed them above during clusterware reconfiguration.

Note that depending on how your LOCAL_LISTENER and REMOTE_LISTENER init.ora parameters are set, you might need to change them: if they reference connections descriptors from tnsname.ora, then only the latter should be changed, but if there are full connection descriptors, they should also be modified).

You should also change listener names to reflect new hostnames. Usually, listeners are named as LISTENER_{hostname}, and you should keep this convention again unless you have a very good reason not to. Do that on both nodes if you don’t have a shared ORACLE_HOME.

Now it’s time to get back to the .cap files with the CRS resource definitions we captured when we began. The files we are interested in are in format ora.{hostname}.LISTENER_{HOSTNAME}.lsnr.cap. In my case, one of them is ora.vs10a.LISTENER_VS10A.lsnr (my old hostname was vs10a). If you changed listener names above, you would need to amend it there as well — NAME=ora.mch10.LISTENER_VS10.lsnr, and rename the file according to the new host name following the same naming convention.

Your VIP name has probably changed, so this line should be modified as well: REQUIRED_RESOURCES=ora.mch10.vip. And finally, the hosting member will change: HOSTING_MEMBERS=mch10. Check the whole file carefully — you should simply modify the old hostname to the new one in both lower and upper case.

Now it’s time to register the resource — the crs_register command does just that. This command specifies the resource name to register and the directory where the .cap file is located. It should be named exactly like resource name plus a “.cap” extension. Each node’s listener can be added from the same node. It’s important that the content of the .cap file is modified appropriately. Assuming I have files ora.mch10.LISTENER_VS10.lsnr and ora.mch11.LISTENER_VS11.lsnr in directory /opt/oracle/A/resources, I run:

$ORA_CRS_HOME/bin/crs_register ora.mch10.LISTENER_VS10.lsnr -dir /opt/oracle/A/resources
$ORA_CRS_HOME/bin/crs_register ora.mch11.LISTENER_VS11.lsnr -dir /opt/oracle/A/resources

Now the output from crs_stat -t should be:

Name           Type           Target    State     Host
------------------------------------------------------------
ora....10.lsnr application    OFFLINE   OFFLINE
ora.mch10.gsd   application    ONLINE    ONLINE    mch10
ora.mch10.ons   application    ONLINE    ONLINE    mch10
ora.mch10.vip   application    ONLINE    ONLINE    mch10
ora....11.lsnr application    OFFLINE   OFFLINE
ora.mch11.gsd   application    ONLINE    ONLINE    mch11
ora.mch11.ons   application    ONLINE    ONLINE    mch11
ora.mch11.vip   application    ONLINE    ONLINE    mch11

It’s now time to start the listeners:

$ORA_CRS_HOME/bin/srvctl start nodeapps -n mch10
$ORA_CRS_HOME/bin/srvctl start nodeapps -n mch11

crs_stat -t should show the listeners online:

Name           Type           Target    State     Host
------------------------------------------------------------
ora....10.lsnr application    ONLINE    ONLINE    mch10
ora.mch10.gsd   application    ONLINE    ONLINE    mch10
ora.mch10.ons   application    ONLINE    ONLINE    mch10
ora.mch10.vip   application    ONLINE    ONLINE    mch10
ora....11.lsnr application    ONLINE    ONLINE    mch11
ora.mch11.gsd   application    ONLINE    ONLINE    mch11
ora.mch11.ons   application    ONLINE    ONLINE    mch11
ora.mch11.vip   application    ONLINE    ONLINE    mch11

8.> Adding ASM Instances to CRS

This step is optional, and it you don’t use ASM, skip it.

Unfortunately, we can’t simply use .cap files to register ASM resources. There are more pieces required and the only way I could find to register ASM instances is to use srvctl which is, actually, a more supported option. This is simple:

$ORACLE_HOME/bin/srvctl add asm -n mch10 -i ASM1 -o $ORACLE_HOME
$ORACLE_HOME/bin/srvctl add asm -n mch11 -i ASM1 -o $ORACLE_HOME
$ORACLE_HOME/bin/srvctl start asm -n mch10
$ORACLE_HOME/bin/srvctl start asm -n mch11

There is a catch — sometimes I had to prefix the name of the ASM instance with a “+” (i.e. making it like -i +ASM1) and sometimes no plus-sign was required.

crs_stat -t should show now:

Name           Type           Target    State     Host
------------------------------------------------------------
ora....SM1.asm application    ONLINE    ONLINE    mch10
ora....10.lsnr application    ONLINE    ONLINE    mch10
ora.mch10.gsd   application    ONLINE    ONLINE    mch10
ora.mch10.ons   application    ONLINE    ONLINE    mch10
ora.mch10.vip   application    ONLINE    ONLINE    mch10
ora....SM2.asm application    ONLINE    ONLINE    mch11
ora....11.lsnr application    ONLINE    ONLINE    mch11
ora.mch11.gsd   application    ONLINE    ONLINE    mch11
ora.mch11.ons   application    ONLINE    ONLINE    mch11
ora.mch11.vip   application    ONLINE    ONLINE    mch11

9.> Register Databases

For each database, you need to register a database resource. Then, for every instance, you need to register an instance resource. So for database A, my two-node cluster, I use:

$ORACLE_HOME/bin/srvctl add database -d A -o $ORACLE_HOME
$ORACLE_HOME/bin/srvctl add instance -d A -i A1 -n mch10
$ORACLE_HOME/bin/srvctl add instance -d A -i A2 -n mch11
$ORACLE_HOME/bin/srvctl start database -d A

10.> Finally, crs_stat -t should show all resources online:

Name           Type           Target    State     Host
------------------------------------------------------------
ora.A.A1.inst  application    ONLINE    ONLINE    mch10
ora.A.A2.inst  application    ONLINE    ONLINE    mch11
ora.A.db       application    ONLINE    ONLINE    mch10
ora....SM1.asm application    ONLINE    ONLINE    mch10
ora....10.lsnr application    ONLINE    ONLINE    mch10
ora.mch10.gsd   application    ONLINE    ONLINE    mch10
ora.mch10.ons   application    ONLINE    ONLINE    mch10
ora.mch10.vip   application    ONLINE    ONLINE    mch10
ora....SM2.asm application    ONLINE    ONLINE    mch11
ora....11.lsnr application    ONLINE    ONLINE    mch11
ora.mch11.gsd   application    ONLINE    ONLINE    mch11
ora.mch11.ons   application    ONLINE    ONLINE    mch11
ora.mch11.vip   application    ONLINE    ONLINE    mch11

11.> Other Resources

If you had other resources like services, user VIPs, or user-defined resources, you will probably be fine using the crs_register command to get them back into CRS. I didn’t try it, but it should work.
Final Check

To make sure that everything is working, you should at least reboot every node and see if everything comes up.

I don’t know if that operation is considered to be supported. The only slippery bit is modifying the $ORA_CRS_HOME/install/rootconfig file, because it’s usually created by the Universal Installer. Another tricky place is the “unusual” registration of listeners. Otherwise, all the commands are pretty much usual stuff, I think. Good luck!

Opatch Permission libraries Issue

I see that all the files which should be owned by root are owned by grid user on node C.

As ROOT user, please run below command on node C - and share " ls -alrt /usr/products/grid/grid_home/12.1.0.2/lib"

/crs/install/rootcrs.sh -prepatch
/rdbms/install/rootadd_rdbms.sh
/crs/install/rootcrs.sh -postpatch

Thanks,

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

unlock and re-lock the GI  to  verify whether permissions remains same or getting changed

As root:

# $GRID_HOME/crs/install/rootcrs.pl -unlock -crshome $GRID_HOME            >>>>>>>stop the crs 
# $GRID_HOME/crs/install/rootcrs.pl -patch>>>>>>>starts the crs

And, then share the " ls -alrt /usr/products/grid/grid_home/12.1.0.2/lib"

Add Node

1> Compare the mentioned file in the new and old environment: "/etc/sysctl.conf" file.
2> Compare the mentioned file in the new and old environment: "/etc/security/limits.conf" file.
3> Compare the mentioned file in the new and old environment "/etc/pam.d/login" file.
4> Create the necessary oracle groups and users.
5> chkconfig ntpd status
6> Create the .profile for setting the environment for grid and oracle user (Compare the mentioned file in the new and old environment)
"/home/oracle/.bash_profile" file and "/home/grid/.bash_profile"
"/home/oracle/grid_env" and "/home/grid/grid_env"
"/home/oracle/db_env" and "/home/grid/db_env"
7> uname -rm
8> All necessary ASM libraries and packages
9> Ping all the IPs of the New_Node
ping -c 3 112-rac1
ping -c 3 112-rac1-priv
ping -c 3 112-rac1-vip

10> ASM must be configured and "#oracleasm listdisks" command on the New_Node should show all the disks.
Few command reference : oracleasm listdisks, oracleasm init, oracleasm scandisks

11> Configure secure shell for oracle user on all nodes

From oracle_home/oui/bin on existing Node -
./runSSHSetup.sh -user oracle -hosts "Existing_Node New_Node" -advanced -exverify

12> Verify New_Node (HWOS)

From grid_home on existing Node
$GRID_HOME/bin/cluvfy stage -post hwos -n New_Node > /u02/hwos.log

13> Verify Peer (REFNODE)

From grid_home on existing Node
$GRID_HOME/bin/cluvfy comp peer -refnode existing Node -n New_Node -orainv oinstall -osdba dba -verbose > /u02/comppeer.log

14> Verify New_Node (New_Node PRE)

From grid_home on existing Node
$GRID_HOME/bin/cluvfy stage -pre nodeadd -n New_Node -fixup -verbose > /u02/fixup.log

14> Extend Clusterware

Run “addNode.sh”

a) [oracle@existing Node bin]$ ./addNode.sh -silent "CLUSTER_NEW_NODES={New_Node}" "CLUSTER_NEW_VIRTUAL_HOSTNAMES={New_Node-vip}"
From root user on New_Node :::

b) [root@New_Node oraInventory]# ./orainstRoot.sh
From root user on New_Node :::

c) [root@New_Node oraInventory]# cd /u01/app/11.2.0/grid/
[root@New_Node grid]# ./root.sh

If successful, clusterware daemons, the listener, the ASM instance, etc. should be started

d) [oracle@New_Node ~]$ crsctl check crs
e) [oracle@New_Node ~]$ crs_stat -t -v
f) [oracle@New_Node ~]$ olsnodes -n
Existing Node  1
Existing_Node 2
New_Node  3

g)[oracle@New_Node ~]$ srvctl status asm -a
ASM is running on Existing_Node,New_Node,existing Node
ASM is enabled.

h) [oracle@New_Node ~]$ ocrcheck
i) [oracle@New_Node ~]$ crsctl query css votedisk

15) Verify New_Node (New_Node POST)

[oracle@existing Node u02]$ $GRID_HOME/bin/cluvfy stage -post nodeadd -n New_Node -verbose > /u02/clusterpost.log

ASM Block Header 0 Corruption

Reference: How To Restore/Repair/Fix An Overwritten (KFBTYP_INVALID) ASM Disk Header (First 4K) 10.2.0.5, 11.1.0.7, 11.2 And Onwards (Doc ID 1088867.1) 

To confirm if just block 0 is corrupted or there are more blocks, please execute this script and upload .out files
kfed.sh
-----
#! /bin/sh
rm /tmp/kfed_DH.out /tmp/kfed_FS.out /tmp/kfed_BK.out /tmp/kfed_FD.out /tmp/kfed_DD.out
for i in `ls /dev/emc*`
do
echo $i >> /tmp/kfed_DH.out
kfed read $i >> /tmp/kfed_DH.out
echo $i >> /tmp/kfed_FS.out
kfed read $i blkn=1 >> /tmp/kfed_FS.out
echo $i >> /tmp/kfed_BK.out
kfed read $i aun=1 blkn=254 >> /tmp/kfed_BK.out
echo $i >> /tmp/kfed_FD.out
kfed read $i aun=2 blkn=1 >> /tmp/kfed_FD.out
echo $i >> /tmp/kfed_DD.out
kfed read $i aun=2 blkn=2 >> /tmp/kfed_DD.out
done
----

That way we can see which au's are affected

Please, upload asmdu output too:
amdu -diskstring '' -dump

and the OS logs

Additionally, please upload:
1) full ASM alertlog

2) Please connect to ASM instance, generate and upload /tmp/ASM_#.html file as result of output from v$asm_disk and v$asm_diskgroup:

---------------------- cut -------------------------
         spool /tmp/ASM_<#>.html
         set markup HTML on
         set pagesize 1000
         set lines 500
         alter session set nls_date_format='DD-MON-YYYY HH24:MI:SS';
         select sysdate "Date and Time" from dual;
         select 'HOSTNAME ASSOCIATED WITH THIS ASM INSTANCE: ==)> ' , MACHINE " " from v$session where program like '%SMON%';

         select * from v$asm_diskgroup order by 1;
         select * from v$asm_disk order by 1, 2, 3;
         select * from gv$asm_operation order by 1;
         select * from v$version where banner like '%Database%' order by 1;
         select * from gv$asm_client order by 1;
         select group_number, name, value FROM v$asm_attribute where NAME like 'disk_repair_time';

         show parameter asm
         show parameter size
         show parameter proc
         show parameter cluster
         show parameter instance_type
         show parameter instance_name

         show parameter pfile

         show sga

         spool off

--------------------- end cut -----------------------