Enable Fast Start Failover using DGMGRL Oracle Data Guard Broker

Enable FSFO and starting the Observer

An Observer can be any system with access to the databases in DG configuration and at least Oracle Client installed.

                1. Ensure that SRL's are configured on both the PRIMARY DB and 

                    the STANDBY DB's

               

                2. Ensure that the LogXptMode is set to SYNC

               

                3. Set the FastStartFailoverTarget property of PRIMARY to desired 

                    STANDBY DB.

               

                   edit database orcl1 set property 

                                      'FastStartFailoverTarget'='orcl2';

               

                4. Upgrade protection Mode to MaxAvailability if necessary.

               

                   edit configuration set protection mode as MAxAvailability;

               

                5. Enable Flashback on Primary and Standby DB's

                               

                   Also set DB_FLASHBACK_RETENTION_TARGET to appropriate value.

                   This parameter retains the flashback logs such that the database can be 

                   flashed back to the specified interval of time.

               

                6. Start Observer

                               

                    Connect to standby DB using DGMGRL and issue:

                               

                    start observer;

                               

                7. Enable Fast start Failover

                    enable FAST_START failover;

               

                    This statement creates FSFP process

                               

                   2014-03-12 02:51:34.451      DMON: Creating process FSFP

           2014-03-12 02:51:37.549      FSFP: Process started

           2014-03-12 02:51:38.550      DMON: FSFP successfully started

                         DGMGRL> show configuration

      

              Configuration - dg_config

      

              Protection Mode: MaxAvailability

              Databases:

              orcl1 - Primary database

              orcl2 - (*) Physical standby database

      

              Fast-Start Failover: ENABLED

      

              Configuration Status:

              SUCCESS

Oracle Data Guard Broker Configuration management

Configuration management

1.       Change parameter, database property using DGMGRL

We will change the property 'StandbyFileManagement' for database ORCL2 to AUTO from MANUAL which is default setting for the parameter STANDBY_FILE_MANAGEMENT.

                edit database orcl2 set property 'StandbyFileManagement'='AUTO';

      

      

       DGMGRL> show database verbose orcl2

      

       Database - orcl2

      

         Role:            PHYSICAL STANDBY

         Intended State:  APPLY-ON

         Transport Lag:   0 seconds

         Apply Lag:       0 seconds

         Real Time Query: OFF

         Instance(s):

           orcl2

      

         Properties:

           DGConnectIdentifier             = 'orcl2'

           ObserverConnectIdentifier       = ''

           LogXptMode                      = 'ASYNC'

           DelayMins                       = '0'

           Binding                         = 'OPTIONAL'

           MaxFailure                      = '0'

           MaxConnections                  = '1'

           ReopenSecs                      = '300'

           NetTimeout                      = '30'

           RedoCompression                 = 'DISABLE'

           LogShipping                     = 'ON'

           PreferredApplyInstance          = ''

           ApplyInstanceTimeout            = '0'

           ApplyParallel                   = 'AUTO'

           StandbyFileManagement           = 'AUTO'

           ArchiveLagTarget                = '0'

           LogArchiveMaxProcesses          = '4'

           LogArchiveMinSucceedDest        = '1'

           DbFileNameConvert               = 'orcl1, orcl2'

           LogFileNameConvert              = 'orcl1, orcl2'

           FastStartFailoverTarget         = ''

           StatusReport                    = '(monitor)'

           InconsistentProperties          = '(monitor)'

           InconsistentLogXptProps         = '(monitor)'

           SendQEntries                    = '(monitor)'

           LogXptStatus                    = '(monitor)'

           RecvQEntries                    = '(monitor)'

           HostName                        = 'dg3'

           SidName                         = 'orcl2'

           StaticConnectIdentifier         = '(DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=dg3)(PORT=1521))(CONNECT_DATA=(SERVICE_NAME=orcl2_DGMGRL)(INSTANCE_NAME=orcl2)(SERVER=DEDICATED)))'

           StandbyArchiveLocation          = 'USE_DB_RECOVERY_FILE_DEST'

           AlternateLocation               = ''

           LogArchiveTrace                 = '0'

           LogArchiveFormat                = '%t_%s_%r.dbf'

           TopWaitEvents                   = '(monitor)'

      

       Database Status:

       SUCCESS

We can change the ‘LogXptMode’ property as below. This will update the LOG_ARCHIVE_DEST_n parameter appropriately.

      

       edit database [orcl1|orcl2] set property logxptmode=SYNC;

2.       Change the broker configuration file location

The location of the broker configuration files is determined by the parameters DG_BROKER_CONFIG_FILEn (where n in {1,2}).

Default location is $ORACLE_HOME/dbs

Default names of configuration files are:  dr{1|2}<db_unique_name>             

CHANGING CONFIG FILE LOCATION:

               

The location of the configuration file can only be changed when DMON is not running.

                               

·         set DG_BROKER_START=FALSE

·         alter system set DG_BROKER_CONFIG_FILEn=new loc;

·         manually move files at OS level to new location

                                IF old/new location is ASM, use DBMS_FILE_TRANSFER.COPY_FILE

·         start DMON process

3.       Change DB STATEs

Turn off redo apply,

                edit database orcl2 set STATE='APPLY-OFF';

        Activate active DG aka REAL TIME QUERY,

               

                edit database orcl2 set STATE='APPLY-OFF';

       SQL> alter database open;

       edit database orcl2 set STATE='APPLY-ON';

      

       DGMGRL> show database orcl2

      

       Database - orcl2

      

         Role:            PHYSICAL STANDBY

         Intended State:  APPLY-ON

         Transport Lag:   0 seconds

         Apply Lag:       0 seconds

         Real Time Query: ON

         Instance(s):

           orcl2

      

       Database Status:

       SUCCESS

4.       Change protection mode

a)      Configure the SRL if necessary

b)      Set LogXptMode aptly

                               

edit database orcl1 set property 'LogXptMode'='SYNC';

edit database orcl2 set property 'LogXptMode'='SYNC';

                               

c)       now change the mode for the ENTIRE CONFIGURATION

               

                edit configuration set protection mode as MaxAvailability;

                               

         Effects:

               

                In the STANDBY alert log:

                               

                RFS[6]: Assigned to RFS process 7356

RFS[6]: Identified database type as 'physical standby': Client is LGWR SYNC pid 4753

Primary database is in MAXIMUM AVAILABILITY mode

Changing standby controlfile to MAXIMUM AVAILABILITY mode

Standby controlfile consistent with primary

RFS[6]: Selected log 4 for thread 1 sequence 65 dbid 1224670660 branch 840221191

Recovery of Online Redo Log: Thread 1 Group 4 Seq 65 Reading mem 0

Mem# 0: /u01/app/oracle/oradata/orcl2/srl04.log

Wed Mar 12 01:05:58 2014

RFS[7]: Assigned to RFS process 7366

RFS[7]: Identified database type as 'physical standby': Client is ARCH pid 4817

On the PRIMARY Site

SQL> select dest_id, dest_name, status, affirm, transmit_mode, process from v$archive_dest where status != 'INACTIVE';

DEST_ID DEST_NAME                      STATUS    AFF TRANSMIT_MOD PROCESS

---------- ------------------------------ --------- --- ------------ ----------

1 LOG_ARCHIVE_DEST_1             VALID     NO  SYNCHRONOUS  ARCH

2 LOG_ARCHIVE_DEST_2             VALID     YES PARALLELSYNC LGWR

On STANDBY site :

SQL> select dest_id, dest_name, status, affirm, transmit_mode, process from v$archive_dest where status != 'INACTIVE';

DEST_ID DEST_NAME                      STATUS    AFF TRANSMIT_MOD PROCESS

---------- ------------------------------ --------- --- ------------ ----------

1 LOG_ARCHIVE_DEST_1             VALID     NO  SYNCHRONOUS  ARCH

2 LOG_ARCHIVE_DEST_2             VALID     NO  ASYNCHRONOUS LGWR

32 STANDBY_ARCHIVE_DEST           VALID     YES SYNCHRONOUS  RFS

How to install Data Guard Broker

Installing DG Broker

1.       On both PRIMARY db and STANDBY db's set DG_BROKER_START=TRUE

2.       Add static service DGMGRL entry on both DB's in the LISTENER.ora file

       The static service entry name is of the following format:

    <DB_UNIQUE_NAME>_DGMGRL.<DB_DOMAIN>

               

        SID_LIST_LISTENER =

      (SID_LIST =

        (SID_DESC =

          (GLOBAL_DBNAME = ORCL1_DGMGRL)

          (ORACLE_HOME = /u01/app/oracle/product/11.2.0/dbhome_1)

          (SID_NAME = ORCL1)

        )

      )

        SID_LIST_LISTENER =

      (SID_LIST =

        (SID_DESC =

          (GLOBAL_DBNAME = ORCL2_DGMGRL)

          (ORACLE_HOME = /u01/app/oracle/product/11.2.0/dbhome_1)

          (SID_NAME = ORCL2)

        )

      )

3. CREATE CONFIGURATION

create configuration dg_config as

primary database is 'orcl1'

connect identifier is 'orcl1';

               

This MUST be done from DGMGRL utility while connected to the primary database.

               

If you try to create configuration while connected to standby database, the following error occurs:

               

Error: ORA-16584: operation cannot be performed on a standby database

               

               

               

4.       Add the standby DB to configuration

                ADD database 'orcl2' as

       connect identifier is 'orcl2';         

                adds the standby db.

5.       Show configuration

                DGMGRL> show configuration

      

       Configuration - dg_config

      

         Protection Mode: MaxPerformance

         Databases:

           orcl1 - Primary database

           orcl2 - Physical standby database

      

       Fast-Start Failover: DISABLED

      

       Configuration Status:

       DISABLED

6.       Enable configuration

DGMGRL> show configuration

                                               

From here onwards, the DG BROKER takes over the management of the data guard environment.

This is evident from the following message in the alert log itself.

ALTER SYSTEM SET dg_broker_start=TRUE SCOPE=BOTH;

Tue Mar 11 23:20:31 2014

DMON started with pid=27, OS id=5400

Starting Data Guard Broker (DMON)

Tue Mar 11 23:20:42 2014

INSV started with pid=28, OS id=5409

Tue Mar 11 23:33:23 2014

NSV0 started with pid=30, OS id=5606

Tue Mar 11 23:33:27 2014

RSM0 started with pid=31, OS id=5615

ALTER SYSTEM SET log_archive_trace=0 SCOPE=BOTH SID='orcl2';

ALTER SYSTEM SET log_archive_format='%t_%s_%r.dbf' SCOPE=SPFILE SID='orcl2';

ALTER SYSTEM SET standby_file_management='MANUAL' SCOPE=BOTH SID='*';

ALTER SYSTEM SET archive_lag_target=0 SCOPE=BOTH SID='*';

ALTER SYSTEM SET log_archive_max_processes=4 SCOPE=BOTH SID='*';

ALTER SYSTEM SET log_archive_min_succeed_dest=1 SCOPE=BOTH SID='*';

ALTER SYSTEM SET db_file_name_convert='orcl1','orcl2' SCOPE=SPFILE;

ALTER SYSTEM SET log_file_name_convert='orcl1','orcl2' SCOPE=SPFILE;

ALTER SYSTEM SET fal_server='orcl1' SCOPE=BOTH;

ALTER DATABASE RECOVER MANAGED STANDBY DATABASE  THROUGH ALL SWITCHOVER DISCONNECT  USING CURRENT LOGFILE

Attempt to start background Managed Standby Recovery process (orcl2)

Tue Mar 11 23:33:32 2014

MRP0 started with pid=32, OS id=5620

MRP0: Background Managed Standby Recovery process started (orcl2)

 started logmerger process

Tue Mar 11 23:33:37 2014

Managed Standby Recovery starting Real Time Apply

Parallel Media Recovery started with 2 slaves

Waiting for all non-current ORLs to be archived...

All non-current ORLs have been archived.

Media Recovery Waiting for thread 1 sequence 58 (in transit)

Recovery of Online Redo Log: Thread 1 Group 6 Seq 58 Reading mem 0

  Mem# 0: /u01/app/oracle/oradata/orcl2/srl06.log

Tue Mar 11 23:33:38 2014

Completed: ALTER DATABASE RECOVER MANAGED STANDBY DATABASE  THROUGH ALL SWITCHOVER DISCONNECT  USING CURRENT LOGFILE

Data Guard Broker - Basics

DG BROKER is the management framework for data guard, and allows the DBA’s to automate many of the tasks involved in configuring and monitoring an Oracle Data Guard configuration.

DG Broker provides two interfaces:

1.       DGMGRL utility – this is a command line interface to the DG Broker

2.       OEM  GUI – OEM provides a graphical interface to the DG Broker

Some features such as standby database creation, can be automated and are only available through the GUI interface. If you use the DGMGRL utility and want to create a standby atabase, you must do so manually.

Benefits of DG Broker:

·         Disaster protection

·         High availability and scalability with RAC

·         Automated creation of DG configuration

·         Easy creation of additional standby DB’s (through OEM)

·         Simplified and centralized management

·         Simplified switchover and failover

·         Transparent to applications

DG broker handles the following logical entities:

·         Configuration of databases

A DG Broker configuration consists of following entities:

-          Configuration Object: it is a named collection of database profiles. A database profile in turn is a description of a database object including its current state, status and properties. A configuration object profiles one primary database and its standby databases.

-          Database objects: these correspond to the primary and standby databases. Broker uses the database object’s profile to manage and control the state of a single database on the system.

-          Instance objects: Broker treats databases as a collection of one or more named instances. Broker automatically discovers instances and associates them with their databases.

·         A single database

Data Guard Monitor and DG Configuration files

   The configuration, control and monitoring functions of the Broker are implemented by server-side software and configuration files.

This software is known as Data Guard Monitor.

When the Broker is started, the DMON (data guard monitor) background process runs for every database instance that is managed by the broker. DMON process interacts with the local instance and the DMON processes of all other instances in the broker configuration to perform the requested operation. It is also responsible for monitoring the health of broker configuration and for ensuring that every database has a consistent description of the configuration.

This persistent description of configuration is stored in the binary configuration file. A copy of this file is maintained by the DMON process for each database that belongs to the broker configuration. Configuration file describes the states and properties of the databases in configuration.  

Using SPFile is a must when using DG Broker to ensure that the Broker can update the values of parameters in both the database (parameter file) and the configuration file.

Broker Processes

DMON – Data Guard Monitor process -                This is the main process, responsible for all broker actions, as well as maintaining the configuration files. This process can be enabled or disabled using DG_BROKER_START parameter.

RSM – Broker Resource Manager – This process is responsible for handling SQL commands used by broker that need to be executed on one of the databases in the configuration. These commands need to be run as a result of change in configuration made through the DGMGRL, or the configuration commands run by broker during database startup.

NSVn – Data Guard Net Services – These processes are responsible for making contact with remote databases and sending across any work items to remote databases.

DRCn – Data Guard Network Receiver –  The network receiver processes establish connection from source database NSVn process. This is similar to the LNS and RFS mechanism for redo transport.

Roll Forward Standby Database using Primary Database RMAN Incremental Backup

The Physical Standby can be rolled forward using incremental backups of the Primary database.

This technique can be particularly useful in following cases:

               

-          The Standby is too far behind the Primary and it would take longer for MRP to apply all the redo than it would take the RMAN to create incremental backup of primary and roll forward the standby

-          The Standby is behind the primary and the archivelogs to roll forward the standby have been lost

-          Physical standby has NOLOGGING changes

In any of the above cases, the first step would always be to check the SCN at which the divergence has occurred.

The next steps would be to take an incremental backup of the primary database from that SCN and to recover the standby database using that backup.

Such an incremental backup is not a part of the RMAN retention policy of the primary database.

Following exercise tries to create divergence of Primary and standby by inhibiting the transport of redo from the primary to standby by stopping the listeners. Then a lot of transactions are done on the primary database, thus creating a situation where standby is far behind the primary.

1.       Mount primary and take full backup.

As a best practice, take full backup of the primary.

    RMAN> startup mount;

  RMAN> backup database plus archivelog;

2.       Open primary and create table with huge amount of data

    SQL> alter database open;

  SQL> create table table_dummy as select * from dba_tables;

  SQL> -- repeat  the following insert statement multiple times

  SQL> insert into table_dummy select * from table_dummy; 

3.       Commit the transactions made above  

The Log files will get switched automatically as the table gets large creating a huge backlog trail.

4.        Now mount the standby  and check the  current SCN of the standby

    SQL> select current_scn from v$database;

  1288770     

5.       Take incremental backup on primary for rolling forward the standby database

Connect to Primary database as target and take incremental backup from the SCN at which the divergence occurred.

   

RMAN> backup incremental from scn 1288770 database

      format '/tmp/forStandby_%U' tag 'forStandby';

6.       Send over the backup files from the Primary server to the Standby server

scp /tmp/forStandby_%U oracle@dg2:/tmp/

7.       Catalog the files at standby site using RMAN with standby as target

RMAN> catalog start with '/tmp/forStand';

8.       Recover the standby database with cataloged incremental backup

RMAN> recover database noredo

9.       Backup the Primary DB Controlfile

This step is required in case changes to the database structure have been made.

                Though no structural changes have been done in our case, this step is present for illustration.

With RMAN target as primary, backup current controlfile for standby

 RMAN> backup current controlfile for standby format '/tmp/forStandbyCtrl.bkp'

10.   Send over the controlfile backup to standby

scp /tmp/forStandbyCtrl.bkp oracle@dg2:/tmp/

11.   Shutdown standby and start in nomount mode

RMAN> shutdown immediate;

RMAN> startup nomount;

12.   Restore Standby controlfile at standby site

Connect RMAN with target as standby and restore the controlfile from the standby controlfile just generated

RMAN> restore standby controlfile from '/tmp/forStandbyCtrl.bkp';

13.   Ensure that the db_file_name_convert and the log_file_name_convert parameters are set appropriately

14.   Clear the Standby Redo Log groups on standby

alter database clear logfile group 4;

alter database clear logfile group 5;

alter database clear logfile group 6;

       alter database clear logfile group 7;

15.   Toggle the flashback

alter database flashback off;

alter database flashback on;

16.   Start redo apply at standby site

alter database recover managed standby database using current logfile;

Oracle Dataguard Role Changes - Failover and Switchover

The Dataguard Broker makes it very easy to manage role changes. The following exercise is an attempt to understand what really goes on behind the scenes, by managing the role changes from the SQL prompt.

Disclaimer: One or more statements in this blog might be incorrect, including this one.

Introduction

Environment Description

A. Physical Machines:

B. Oracle Instances

Prerequisite Checks

Switchover steps

Additional Prerequisites

Steps to switchover

1.       Check the switchover status of both DB's.

2.       Initiate switchover at primary Site(DG2)

3.       Mount the former primary (DG2)

4.       check switchover status of the older standby(DG1)

5.        Initiate switchover at the older standby site(DG1)

6.       Open the new primary(DG1)

7.       start redo apply on the new standby

Failover steps

Additional Prerequisites

Steps to Failover

1.       Identify and resolve any gaps

2.       Repeat until all gaps are resolved

3.       Copy any other missing archived logfiles

4.       Flush any unsent redo to the standby if possible

5.       Stop Redo Apply at standby (DG2)

6.       Finish redo apply

7.       check switchover status of the standby(DG2)

8.       Switch physical standby to primary role

9.       Reinstate the failed primary back as a physical standby

i.       Determine SCN at which old standby became new primary

ii.     Flashback failed primary

iii.    Convert primary to failed physical standby

iv.     Check the status of the archive destinations at the primary site (DG2)

v.       Start redo apply at the reinstated database

Introduction

A Data guard environment consists of one Primary and one or more standby databases.

The standby databases could be logical standby or physical standby.

In this article covers the role changes for a data guard environment consisting of one primary and one physical standby database.

Role change can occur in 2 ways:

1.       Switchover: It is reversal of role between a standby database and a primary database. The primary database assumes role of standby database and the standby database assumes role of primary database after the switchover. There is no data loss in a switchover. This is generally a planned activity.

2.       Failover: If the Primary database fails, one of the standby databases can be made to assume the Primary role. This role change of standby database to primary, in case of failure of existing primary, is called failover. There may or may not be data loss. This is not a planned activity and is performed only in catastrophic cases where the Primary database fails.

Environment Description

A. Physical Machines:

               

                Two nodes DG1 and DG2 were configured with following IP addresses:

                DG1: 192.168.4.1

                DG2: 192.168.4.2

B. Oracle Instances

               

                Below are the details of the RDBMS instances

               

                RDBMS instance @DG1: ORCL1

    RDBMS instance @DG2: ORCL2

                At the outset, ORCL2 is the primary database instance and ORCL1 is the standby database instance.

Prerequisite Checks

The following requirements must be checked before any role change:

1.       Verify that each DB is configured properly for the role it is about to assume.

This would include checking the instance parameters for both databases are set appropriately, the network services are configured properly etc.

2.       Verify that there are no redo errors/gaps by querying V$ARCHIVE_DEST_STATUS view at Primary (DG2)

               

SQL> select status, gap_status , db_unique_name from v$archive_dest_status where dest_id=2;

      

       STATUS    GAP_STATUS               DB_UNIQUE_NAME

       --------- ------------------------ ------------------------------

       VALID     NO GAP                   orcl1

               

3.       Ensure that temporary files created at standby site match those at the primary site

4.       Remove any delay if present, configure MRP such that delay is ignored using NODELAY option

5.       Before performing switchover from RAC primary to standby, shutdown all but one instances of the primary

6.       If the standby is in Active data guard mode (real-time query), bring all instances to mounted but not open state

Switchover steps

Additional Prerequisites

·         Verify that the primary DB is open and redo apply active on standby

Steps to switchover

1.       Check the switchover status of both DB's.

The switchover status of the standby would be 'NOT ALLOWED' since there is already an active Primary database.

- PRIMARY (DG2)

       

                 Proceed only if it shows 'TO STANDBY' or 'SESSIONS ACTIVE'

SQL> select instance_name, name, primary_db_unique_name, switchover_status, open_mode, flashback_on, force_logging, database_role from v$database, v$instance;    

      

       INSTANCE_NAME    NAME      PRIMARY_DB_UNIQUE_NAME         SWITCHOVER_STATUS    OPEN_MODE            FLASHBACK_ON       FOR DATABASE_ROLE

       ---------------- --------- ------------------------------ -------------------- -------------------- ------------------ --- ----------------

       orcl2            ORCL1     orcl1                          TO STANDBY           READ WRITE           YES                YES PRIMARY

               

                - STANDBY (DG1)

       

SQL> select instance_name, name, primary_db_unique_name, switchover_status, open_mode, flashback_on, force_logging, database_role from v$database, v$instance;

      

       INSTANCE_NAME    NAME      PRIMARY_DB_UNIQUE_NAME         SWITCHOVER_STATUS    OPEN_MODE            FLASHBACK_ON       FOR DATABASE_ROLE

       ---------------- --------- ------------------------------ -------------------- -------------------- ------------------ --- ----------------

       orcl1            ORCL1     orcl2                          NOT ALLOWED          MOUNTED              YES                YES PHYSICAL STANDBY

               

2.       Initiate switchover at primary Site(DG2)

   

alter database commit to switchover to physical standby with session shutdown;

- At this stage the there is no primary DB, but all physical standbys

- This also leads to change in 'SWITCHOVER_STATUS' of physical standby from 

   'NOT ALLOWED' to 'TO_PRIMARY'

                - This statement takes a backup of the current controlfile to trace

                - closes the database

                - archives the current logfile

                - marks the End-of-redo marker to the end of the last logfile

                - shuts down the MRP process at the standby

               

      A closer look at the logfiles, details what actions are performed by this statement

               

                Primary Logfile (DG2):

                               

   alter database commit to switchover to physical standby with session shutdown

       ALTER DATABASE COMMIT TO SWITCHOVER TO PHYSICAL STANDBY  [Process Id: 4738] (orcl2)

       Sat Mar 29 20:00:13 2014

       LGWR: Standby redo logfile selected to archive thread 1 sequence 165

       LGWR: Standby redo logfile selected for thread 1 sequence 165 for destination LOG_ARCHIVE_DEST_2

       Thread 1 advanced to log sequence 165 (LGWR switch)

         Current log# 3 seq# 165 mem# 0: /u01/app/oracle/oradata/orcl2/redo03.log

       Waiting for all non-current ORLs to be archived...

       Waiting for the ORL for thread 1 sequence 164 to be archived...

       Sat Mar 29 20:00:13 2014

       Archived Log entry 207 added for thread 1 sequence 164 ID 0x4919b14e dest 1:

       Sat Mar 29 20:00:23 2014

       ORL for thread 1 sequence 164 has been archived...

       All non-current ORLs have been archived.

       Waiting for all FAL entries to be archived...

       All FAL entries have been archived.

       Waiting for potential switchover target to become synchronized...

       Active, synchronized Physical Standby  switchover target has been identified

       Sat Mar 29 20:00:24 2014

       ********************************************************************

       LGWR: Resetting 'active' archival for destination LOG_ARCHIVE_DEST_2

       ********************************************************************

       Destination LOG_ARCHIVE_DEST_2 no longer supports SYNCHRONIZATION

       Thread 1 advanced to log sequence 166 (LGWR switch)

         Current log# 1 seq# 166 mem# 0: /u01/app/oracle/oradata/orcl2/redo01.log

       Archived Log entry 209 added for thread 1 sequence 165 ID 0x4919b14e dest 1:

       Sat Mar 29 20:00:25 2014

       Stopping background process CJQ0

       Sat Mar 29 20:00:25 2014

       SMON: disabling tx recovery

       Stopping background process QMNC

       All dispatchers and shared servers shutdown

       CLOSE: killing server sessions.

       CLOSE: all sessions shutdown successfully.

       Stopping background process SMCO

       SMON: disabling cache recovery

       Shutting down archive processes

       Archiving is disabled

       Sat Mar 29 20:00:28 2014

       ARCH shutting downSat Mar 29 20:00:28 2014

      

       ARCH shutting down

       ARC2: Archival stoppedARC3: Archival stopped

      

       Sat Mar 29 20:00:28 2014

       ARCH shutting down

       ARC1: Archival stopped

       Sat Mar 29 20:00:28 2014

       ARCH shutting down

       ARC0: Archival stopped

       Thread 1 closed at log sequence 166

       Successful close of redo thread 1

       ARCH: Noswitch archival of thread 1, sequence 166

       ARCH: End-Of-Redo Branch archival of thread 1 sequence 166

       Expanded controlfile section 11 from 114 to 228 records

       Requested to grow by 114 records; added 4 blocks of records

       Archived Log entry 210 added for thread 1 sequence 166 ID 0x4919b14e dest 1:

       ARCH: Archiving is disabled due to current logfile archival

       Using STANDBY_ARCHIVE_DEST parameter default value as USE_DB_RECOVERY_FILE_DEST

       Primary will check for some target standby to have received all redo

       Final check for a synchronized target standby. Check will be made once.

       LOG_ARCHIVE_DEST_2 is a potential Physical Standby  switchover target

       Active, synchronized target has been identified

       Target has also applied all redo

       Backup controlfile written to trace file /u01/app/oracle/diag/rdbms/orcl2/orcl2/trace/orcl2_ora_4738.trc

       Clearing standby activation ID 1226420558 (0x4919b14e)

       The primary database controlfile was created using the

       'MAXLOGFILES 16' clause.

       There is space for up to 13 standby redo logfiles

       Use the following SQL commands on the standby database to create

       standby redo logfiles that match the primary database:

       ALTER DATABASE ADD STANDBY LOGFILE 'srl1.f' SIZE 52428800;

       ALTER DATABASE ADD STANDBY LOGFILE 'srl2.f' SIZE 52428800;

       ALTER DATABASE ADD STANDBY LOGFILE 'srl3.f' SIZE 52428800;

       ALTER DATABASE ADD STANDBY LOGFILE 'srl4.f' SIZE 52428800;

       Archivelog for thread 1 sequence 166 required for standby recovery

       Switchover: Primary controlfile converted to standby controlfile succesfully.

       Sat Mar 29 20:00:29 2014

       MRP0 started with pid=17, OS id=5176

       MRP0: Background Managed Standby Recovery process started (orcl2)

       Serial Media Recovery started

       Managed Standby Recovery not using Real Time Apply

       Online logfile pre-clearing operation disabled by switchover

       Media Recovery Log /u01/app/oracle/flash_recovery_area/ORCL2/archivelog/2014_03_29/o1_mf_1_166_9mfp85jm_.arc

       Identified End-Of-Redo for thread 1 sequence 166

       Resetting standby activation ID 0 (0x0)

       Media Recovery End-Of-Redo indicator encountered

       Media Recovery Applied until change 1128261

       MRP0: Media Recovery Complete: End-Of-REDO (orcl2)

       MRP0: Background Media Recovery process shutdown (orcl2)

       Sat Mar 29 20:00:35 2014

       idle dispatcher 'D000' terminated, pid = (17, 1)

       Sat Mar 29 20:00:35 2014

       Switchover: Complete - Database shutdown required (orcl2)

       Completed: alter database commit to switchover to physical standby with session shutdown

       Sat Mar 29 20:04:40 2014

       Shutting down instance (immediate)

       .

       .

       ALTER DATABASE CLOSE NORMAL

       ORA-1507 signalled during: ALTER DATABASE CLOSE NORMAL...

       .

       .

       Instance shutdown complete

                 Standby Logfile (DG1):           

   

                Media Recovery Waiting for thread 1 sequence 165 (in transit)

       Recovery of Online Redo Log: Thread 1 Group 5 Seq 165 Reading mem 0

         Mem# 0: /u01/app/oracle/oradata/orcl1/srl05.log

       Sat Mar 29 20:00:23 2014

       Archived Log entry 23 added for thread 1 sequence 165 ID 0x4919b14e dest 1:

       Sat Mar 29 20:00:23 2014

       RFS[5]: Assigned to RFS process 4965

       RFS[5]: Identified database type as 'physical standby': Client is ARCH pid 4754

       Sat Mar 29 20:00:23 2014

       Media Recovery Waiting for thread 1 sequence 166

       Sat Mar 29 20:00:29 2014

       RFS[6]: Assigned to RFS process 4969

       RFS[6]: Identified database type as 'physical standby': Client is Foreground pid 4738

       RFS[6]: Opened log for thread 1 sequence 166 dbid 1224670660 branch 840221191

       Archived Log entry 24 added for thread 1 sequence 166 rlc 840221191 ID 0x4919b14e dest 2:

       Sat Mar 29 20:00:29 2014

       RFS[7]: Assigned to RFS process 4973

       RFS[7]: Identified database type as 'physical standby': Client is Foreground pid 4738

       Media Recovery Log /u01/app/oracle/flash_recovery_area/ORCL1/archivelog/2014_03_29/o1_mf_1_166_9mfp85jg_.arc

       Identified End-Of-Redo for thread 1 sequence 166

       Resetting standby activation ID 1226420558 (0x4919b14e)

       Media Recovery End-Of-Redo indicator encountered

       Media Recovery Applied until change 1128261

       Sat Mar 29 20:00:30 2014

       MRP0: Media Recovery Complete: End-Of-REDO (orcl1)

       MRP0: Background Media Recovery process shutdown (orcl1)

3.       Mount the former primary (DG2)

The former primary needs to be mounted, since it was dismounted by the switchover statement.

                shutdown immediate;

         startup mount;

      

SQL> select instance_name, name, primary_db_unique_name, switchover_status, open_mode, flashback_on, force_logging, database_role from v$database, v$instance;

      

       INSTANCE_NAME    NAME      PRIMARY_DB_UNIQUE_NAME         SWITCHOVER_STATUS    OPEN_MODE            FLASHBACK_ON       FOR DATABASE_ROLE

       ---------------- --------- ------------------------------ -------------------- -------------------- ------------------ --- ----------------

       orcl2            ORCL1     orcl1                          TO PRIMARY           MOUNTED              YES                YES PHYSICAL STANDBY 

4.       check switchover status of the older standby(DG1)

At this point, we have 2 physical standby databases.

SQL> select instance_name, name, primary_db_unique_name, switchover_status, open_mode, flashback_on, force_logging, database_role from v$database, v$instance;

      

       INSTANCE_NAME    NAME      PRIMARY_DB_UNIQUE_NAME         SWITCHOVER_STATUS    OPEN_MODE            FLASHBACK_ON       FOR DATABASE_ROLE

       ---------------- --------- ------------------------------ -------------------- -------------------- ------------------ --- ----------------

       orcl1            ORCL1     orcl2                          TO PRIMARY           MOUNTED              YES                YES PHYSICAL STANDBY

               

5.        Initiate switchover at the older standby site(DG1)

The following statement will switchover the ORCL1 database from ‘Physical Standby’ to ‘Primary’.

SQL> alter database commit to switchover to primary with session shutdown;

      

Database altered.

      

SQL> select instance_name, name, primary_db_unique_name, switchover_status, open_mode, flashback_on, force_logging, database_role from v$database, v$instance;

      

       INSTANCE_NAME    NAME      PRIMARY_DB_UNIQUE_NAME         SWITCHOVER_STATUS    OPEN_MODE            FLASHBACK_ON       FOR DATABASE_ROLE

       ---------------- --------- ------------------------------ -------------------- -------------------- ------------------ --- ----------------

       orcl1            ORCL1     orcl2                          NOT ALLOWED          MOUNTED              YES                YES PRIMARY

               

                New Primary Log (DG1):

               

                ALTER DATABASE SWITCHOVER TO PRIMARY (orcl1)

       Maximum wait for role transition is 15 minutes.

       Backup controlfile written to trace file /u01/app/oracle/diag/rdbms/orcl1/orcl1/trace/orcl1_ora_4733.trc

       SwitchOver after complete recovery through change 1128261

       Online log /u01/app/oracle/oradata/orcl1/redo01.log: Thread 1 Group 1 was previously cleared

       Online log /u01/app/oracle/oradata/orcl1/redo02.log: Thread 1 Group 2 was previously cleared

       Online log /u01/app/oracle/oradata/orcl1/redo03.log: Thread 1 Group 3 was previously cleared

       Standby became primary SCN: 1128259

       Switchover: Complete - Database mounted as primary

       Completed: alter database commit to switchover to primary with session shutdown

       Sat Mar 29 20:25:29 2014

       ARC1: Becoming the 'no SRL' ARCH

6.       Open the new primary(DG1)

      alter database open;

7.       start redo apply on the new standby

      alter database recover managed standby database disconnect;

               

                - The standby database ORL's are cleared when we start redo apply at the

                   just “switched over to Standby” database

   

                New Standby log(DG2):

   

               

                Attempt to start background Managed Standby Recovery process (orcl2)

       Sat Mar 29 20:44:57 2014

       MRP0 started with pid=29, OS id=5425

       MRP0: Background Managed Standby Recovery process started (orcl2)

        started logmerger process

       Sat Mar 29 20:45:02 2014

       Managed Standby Recovery not using Real Time Apply

       Parallel Media Recovery started with 2 slaves

       Waiting for all non-current ORLs to be archived...

       All non-current ORLs have been archived.

       Clearing online redo logfile 1 /u01/app/oracle/oradata/orcl2/redo01.log

       Clearing online log 1 of thread 1 sequence number 166

       Completed: alter database recover managed standby database disconnect

       Sat Mar 29 20:45:07 2014

       Clearing online redo logfile 1 complete

       Clearing online redo logfile 2 /u01/app/oracle/oradata/orcl2/redo02.log

       Clearing online log 2 of thread 1 sequence number 164

       Clearing online redo logfile 2 complete

       Clearing online redo logfile 3 /u01/app/oracle/oradata/orcl2/redo03.log

       Clearing online log 3 of thread 1 sequence number 165

       Clearing online redo logfile 3 complete

       Sat Mar 29 20:45:16 2014

       Media Recovery Waiting for thread 1 sequence 167

  

Failover steps

Additional Prerequisites

·         If standby is running in Max Protection mode, first place it in Max Performance mode

    alter database set standby database to maximize performance;

               

                This is required because one cannot failover to a standby in Max Protection mode.

Steps to Failover

After the switchover operation detailed in previous section, the ORCL1@DG1 is now the Primary DB instance and ORCL2@DG2 is the standby database instance.

We have rebooted the DG1 server, and mounted the ORCL1(db_unq_name) database.

1.       Identify and resolve any gaps

                select * from v$archive_gap;

               

            If any gaps found, manually copy the archived log over to the standby and 

            register it.

  

                alter database register physical logfile '/path/of/archive.log';

2.       Repeat until all gaps are resolved

3.       Copy any other missing archived logfiles

  

            This is basically for the archive log sequences which were not transferred to 

            standby owing to the connection failure, that is the cause of the need to failover.

                Check the V$ARCHIVED_LOG view

               

                select unique thread#, max(sequence#) over (partition by thread#) as last_seq from v$archived_log;

                Register these logfiles at the standby site.

  

                alter database register physical logfile '/path/of/archive.log';

4.       Flush any unsent redo to the standby if possible

  

            If the failed primary can be mounted and the standby can be reached, flush the

            redo available in the non-archived redo log to the standby database.

                alter system flush redo to 'orcl2';

               

                Failed Primary log (DG1):

  

                ALTER SYSTEM FLUSH REDO TO 'orcl2' CONFIRM APPLY

       ALTER SYSTEM FLUSH REDO TO orcl2 CONFIRM APPLY [Process Id: 4829] (orcl1)

       Waiting for all FAL entries to be archived...

       All FAL entries have been archived.

       Waiting for dest_id 2 to become synchronized...

       Active, synchronized Physical Standby  synchronized target has been identified

       Managed  recovery running at physical standby 'LOG_ARCHIVE_DEST_2'

       Flush End-Of-Redo Log thread 1 sequence 178 has been fixed

       Flush Redo: Primary highest seen SCN set to 1162012:0

       ARCH: Noswitch archival of thread 1, sequence 178

       ARCH: End-Of-Redo Branch archival of thread 1 sequence 178

       Identified End-Of-Redo for thread 1 sequence 178

       ARCH: LGWR is scheduled to archive destination LOG_ARCHIVE_DEST_2 after log switch

       Flush End-Of-Redo Log thread 1 sequence 178

       Archived Log entry 45 added for thread 1 sequence 178 ID 0x4930caf9 dest 1:

       ARCH: Archiving is disabled due to current logfile archival

       Primary will wait for orcl2 standby to have applied all redo

       Final check for a target standby that has recovered all redo. Check will be made a few times.

       LOG_ARCHIVE_DEST_2 is a potential  synchronized target

       LOG_ARCHIVE_DEST_2 is a potential  synchronized target

       LOG_ARCHIVE_DEST_2 is a potential  synchronized target

       LOG_ARCHIVE_DEST_2 is a potential  synchronized target

       LOG_ARCHIVE_DEST_2 has also applied all redo from primary

       Active, synchronized target has been identified that has applied all the redo from the primary.

       Flush Redo: Primary redo moved to standby

       Flush Redo: Complete - Database shutdown required (orcl1)

                SQL> archive log list

       Database log mode              Archive Mode

       Automatic archival             Disabled

       Archive destination            USE_DB_RECOVERY_FILE_DEST

       Oldest online log sequence     176

       Next log sequence to archive   0

       Current log sequence           178

                 Standby Log:

  

                Completed: alter database register physical logfile '/u01/app/oracle/flash_recovery_area/ORCL2/archivelog/2014_03_30/o1_mf_1_177_9mhsg19n_.arc'

       Sun Mar 30 15:17:30 2014

       Media Recovery Log /u01/app/oracle/flash_recovery_area/ORCL2/archivelog/2014_03_30/o1_mf_1_177_9mhsg19n_.arc

       Media Recovery Waiting for thread 1 sequence 178

       Sun Mar 30 15:27:08 2014

       RFS[5]: Assigned to RFS process 5716

       RFS[5]: Identified database type as 'physical standby': Client is Foreground pid 4829

       Standby switchover readiness check: Checking whether recoveryapplied all redo..

       Database not available for switchover

         End-Of-REDO archived log file has not been recovered

       Physical Standby did not apply all the redo from the primary.

       Sun Mar 30 15:27:08 2014

       RFS[6]: Assigned to RFS process 5718

       RFS[6]: Identified database type as 'physical standby': Client is ARCH pid 4869

       Sun Mar 30 15:27:08 2014

       RFS[7]: Assigned to RFS process 5724

       RFS[7]: Identified database type as 'physical standby': Client is Foreground pid 4829

       RFS[7]: Opened log for thread 1 sequence 178 dbid 1224670660 branch 840221191

       Archived Log entry 223 added for thread 1 sequence 178 rlc 840221191 ID 0x4930caf9 dest 2:

       Sun Mar 30 15:27:08 2014

       RFS[8]: Assigned to RFS process 5728

       RFS[8]: Identified database type as 'physical standby': Client is Foreground pid 4829

       Standby switchover readiness check: Checking whether recoveryapplied all redo..

       Database not available for switchover

         End-Of-REDO archived log file has been received

         End-Of-REDO archived log file has not been recovered

         Archived log files detected beyond End-Of-REDO

         Incomplete recovery SCN:0:1141681 archive SCN:0:1162012

       Physical Standby did not apply all the redo from the primary.

       Sun Mar 30 15:27:10 2014

       RFS[9]: Assigned to RFS process 5732

       RFS[9]: Identified database type as 'physical standby': Client is Foreground pid 4829

       Standby switchover readiness check: Checking whether recoveryapplied all redo..

       Database not available for switchover

         End-Of-REDO archived log file has been received

         End-Of-REDO archived log file has not been recovered

         Archived log files detected beyond End-Of-REDO

         Incomplete recovery SCN:0:1141681 archive SCN:0:1162012

       Physical Standby did not apply all the redo from the primary.

       Sun Mar 30 15:27:11 2014

       RFS[10]: Assigned to RFS process 5736

       RFS[10]: Identified database type as 'physical standby': Client is Foreground pid 4829

       Standby switchover readiness check: Checking whether recoveryapplied all redo..

       Database not available for switchover

         End-Of-REDO archived log file has been received

         End-Of-REDO archived log file has not been recovered

         Archived log files detected beyond End-Of-REDO

         Incomplete recovery SCN:0:1141681 archive SCN:0:1162012

       Physical Standby did not apply all the redo from the primary.

       Sun Mar 30 15:27:11 2014

       Media Recovery Log /u01/app/oracle/flash_recovery_area/ORCL2/archivelog/2014_03_30/o1_mf_1_178_9mhtmnn2_.arc

       Identified End-Of-Redo for thread 1 sequence 178

       Resetting standby activation ID 1227934457 (0x4930caf9)

       Resetting standby activation ID 1227934457 (0x4930caf9)

       Media Recovery Waiting for thread 1 sequence 179

       Sun Mar 30 15:27:12 2014

       RFS[11]: Assigned to RFS process 5740

       RFS[11]: Identified database type as 'physical standby': Client is Foreground pid 4829

       Standby switchover readiness check: Checking whether recoveryapplied all redo..

       Physical Standby applied all the redo from the primary.

       SQL> archive log list

       Database log mode              Archive Mode

       Automatic archival             Enabled

       Archive destination            USE_DB_RECOVERY_FILE_DEST

       Oldest online log sequence     174

       Next log sequence to archive   0

       Current log sequence           178

5.       Stop Redo Apply at standby (DG2)

                alter database recover managed standby database cancel;

6.       Finish redo apply

                alter database recover managed standby database finish force;

               

                FORCE option stops the RFS process on the standby database so that failover 

                can proceed immediately without waiting for network connections to timeout.

                Standby Log:

  

                Attempt to do a Terminal Recovery (orcl2)

       Media Recovery Start: Managed Standby Recovery (orcl2)

        started logmerger process

       Sun Mar 30 15:46:40 2014

       Managed Standby Recovery not using Real Time Apply

       Parallel Media Recovery started with 2 slaves

       Media Recovery Waiting for thread 1 sequence 179

       RECOVER FINISH applied through switchover EOR logs and stopped.

       Media Recovery Complete: End-Of-REDO (orcl2)

                select recid, first_change#, next_change# , sequence#, blocks, status, applied, END_OF_REDO, END_OF_REDO_TYPE, fal from v$archived_log;

            RECID FIRST_CHANGE# NEXT_CHANGE#  SEQUENCE#     BLOCKS S APPLIED   END END_OF_RED FAL

       ---------- ------------- ------------ ---------- ---------- - --------- --- ---------- ---

              210       1128234      1128261        166         34 A YES       YES SWITCHOVER NO

              211       1128234      1128261        166         34 A NO        YES SWITCHOVER NO

              212       1130130      1130134        169          4 A YES       NO             YES

              213       1128261      1130111        167       2783 A YES       NO             YES

              214       1130111      1130130        168          1 A YES       NO             YES

              215       1131085      1134447        171      17515 A YES       NO             NO

              216       1130134      1131085        170       2365 A YES       NO             YES

              217       1134447      1134454        172          5 A YES       NO             NO

              218       1134454      1134488        173         33 A YES       NO             NO

              219       1134488      1141238        174       8378 A YES       NO             NO

              220       1141238      1141351        175        141 A YES       NO             NO

              221       1141351      1141354        176          1 A YES       NO             NO

              222       1141354      1141681        177        175 A YES       NO             NO

              223       1141681      1162012        178        454 A YES       YES SWITCHOVER NO

                If REDO APPLY FINISH is successful, goto step 7.

                If errors, then  

-          activate standby database | alter database activate physical standby database;

-          goto step 9

               

7.       check switchover status of the standby(DG2)

               

                SQL> select switchover_status from v$database;

      

       SWITCHOVER_STATUS

       --------------------

       TO PRIMARY

8.       Switch physical standby to primary role

                alter database commit to switchover to primary with session shutdown;

              

Standby Logfile:

                ALTER DATABASE SWITCHOVER TO PRIMARY (orcl2)

       Maximum wait for role transition is 15 minutes.

       Backup controlfile written to trace file /u01/app/oracle/diag/rdbms/orcl2/orcl2/trace/orcl2_ora_4676.trc

       SwitchOver after complete recovery through change 1162012

       Online log /u01/app/oracle/oradata/orcl2/redo01.log: Thread 1 Group 1 was previously cleared

       Online log /u01/app/oracle/oradata/orcl2/redo02.log: Thread 1 Group 2 was previously cleared

       Online log /u01/app/oracle/oradata/orcl2/redo03.log: Thread 1 Group 3 was previously cleared

       Standby became primary SCN: 1162010

       Switchover: Complete - Database mounted as primary

       Completed: alter database commit to switchover to primary with session shutdown

               

                SQL> select instance_name, name, primary_db_unique_name, switchover_status, open_mode, flashback_on, force_logging, database_role from v$database, v$instance;

      

       INSTANCE_NAME    NAME      PRIMARY_DB_UNIQUE_NAME         SWITCHOVER_STATUS    OPEN_MODE            FLASHBACK_ON       FOR DATABASE_ROLE

       ---------------- --------- ------------------------------ -------------------- -------------------- ------------------ --- ----------------

       orcl2            ORCL1     orcl1                          FAILED DESTINATION   READ WRITE           YES                YES PRIMARY

9.       Reinstate the failed primary back as a physical standby

The failed primary still assumes the Primary Role, and we will reinstate it as standby.

       SQL> select instance_name, name, primary_db_unique_name, switchover_status, open_mode, flashback_on, force_logging, database_role from v$database, v$instance;

      

       INSTANCE_NAME    NAME    PRIMARY_DB_UNIQUE_NAME         SWITCHOVER_STATUS    OPEN_MODE            FLASHBACK_ON       FOR DATABASE_ROLE

       ---------------- ------- ------------------------------ -------------------- -------------------- ------------------ --- ----------------

       orcl1            ORCL1   orcl2                          NOT ALLOWED          MOUNTED              YES                YES PRIMARY

i.                    Determine SCN at which old standby became new primary

  

                SQL> select standby_became_primary_scn from v$database;

      

       STANDBY_BECAME_PRIMARY_SCN

       --------------------------

                          1162010

ii.                  Flashback failed primary

  

                SQL> flashback database to SCN 1162010;

      

       Flashback complete.

 

iii.                Convert primary to failed physical standby

  

                alter database convert to physical standby;

              ALTER DATABASE CONVERT TO PHYSICAL STANDBY (orcl1)

              Clearing standby activation ID 1227934457 (0x4930caf9)

              The primary database controlfile was created using the

              'MAXLOGFILES 16' clause.

              There is space for up to 13 standby redo logfiles

              Use the following SQL commands on the standby database to create

              standby redo logfiles that match the primary database:

              ALTER DATABASE ADD STANDBY LOGFILE 'srl1.f' SIZE 52428800;

              ALTER DATABASE ADD STANDBY LOGFILE 'srl2.f' SIZE 52428800;

              ALTER DATABASE ADD STANDBY LOGFILE 'srl3.f' SIZE 52428800;

              ALTER DATABASE ADD STANDBY LOGFILE 'srl4.f' SIZE 52428800;

              Archivelog for thread 1 sequence 178 required for standby recovery

              Completed: alter database convert to physical standby

                      This statement dismounts the DB, so we need to mount it again.

                               

                                shutdown immediate;

                  startup mount;

               

                                SQL> select instance_name, name, primary_db_unique_name, switchover_status, open_mode, flashback_on, force_logging, database_role from v$database, v$instance;

             

              INSTANCE_NAME    NAME    PRIMARY_DB_UNIQUE_NAME         SWITCHOVER_STATUS    OPEN_MODE            FLASHBACK_ON       FOR DATABASE_ROLE

              ---------------- ------- ------------------------------ -------------------- -------------------- ------------------ --- ----------------

              orcl1            ORCL1   orcl2                          TO PRIMARY           MOUNTED              YES                YES PHYSICAL STANDBY

iv.                 Check the status of the archive destinations at the primary site (DG2)

  

                SQL> select dest_id, dest_name, status, protection_mode, destination, error,srl from v$archive_dest_status where dest_id=2;

          DEST_ID DEST_NAME            STATUS    PROTECTION_MODE      DESTINATION                    ERROR                          SRL

       ---------- -------------------- --------- -------------------- ------------------------------ ------------------------------ ---

               2 LOG_ARCHIVE_DEST_2   VALID     MAXIMUM AVAILABILITY orcl1                                                         YES

v.                   Start redo apply at the reinstated database

alter database recover managed standby database using current logfile disconnect;