Database Availability Alert Checks

1.1.1.   Instance/Database Down or Hanging or bouncing the database

Alert Format

·         A new connection to instance failed: ORA-01034: ORACLE not available ORA-27101: shared memory realm

·         Database <SID> on is hanging!!!

Action

1. If the host is alive, login to host otherwise raise a ticket with UNIX team to check server availability and escalate to Global Oracle DBA Team.
2. Check if cold backup is going on (ps –eaf | grep dbbackup | grep <SID>). If backup is running, check the backup.log for verification of cold backup.   In this case, keep it pending till the cold backup is completed and escalate to Global Oracle DBA Team to check why alert was raised during cold backup. Ideally it should not raise alert as dbbackup put database in maintenance mode before starting cold backup.

Check alert.log to locate any abnormal lines which caused database down or hanging. Confirm all SID associated file systems are mounted. If any file system is missing, raise a ticket with Unix Team immediately 

1.1.2.  Listener Down

Alert Format

• A new connection to instance failed: ORA-12541: TNS: no listener.

Action

1. Extract listener name and listener password, if any, from $TNS_ADMIN/admin/listener.ora if $TNS_ADMIN exists. Otherwise, extract them from $ORACLE_HOME/network/admin/listener.ora.
2. Check listener status, provide listener password if required.
3. If listener is down, start listener and inspect listener log file to find out any errors.

If listener is up, reload listener. If reload hangs check log/trace files.