FAST_START FAILOVER (FSFO) & Reinstate Failed Primary database

Start Failover using Dataguard Broker

Fast-Start Failover: Concepts
Oracle introduced the Fast-Start Failover (FSF) feature set in Release 10gR2, but it’s been enhanced significantly in Oracle 11g to permit much finer-grained control over the conditions under which a FSF would be initiated. Simply put, FSF ensures that under the appropriate circumstances – some of which are mandatory, and some of which are optionally-configured - a failover to the chosen standby target database will occur without DBA intervention. The following occurrences will initiate FSF:

Fast-Start Failover detects one of these failover situations through the Fast-Start Failover Observer (FSFO). The FSFO leverages the Oracle Call Interface (OCI) architecture to decide when a failover is necessary, which physical standby database should be the target of the failover, and how long to wait until it declares a failover is absolutely necessary. The good news is that I have quite a bit of control over these three directives, all of which are completely configurable using the Data Guard Broker Manager utility (DGMGRL).

Topics to be discussed:
1) Basic checks / requirements for FAST_START FAILOVER
2) Enable fast start failover.
3) Verify the fast-start failover configuration.
4) Fast_start failover gracefull - seamless application access
5) Managing observer
6) Reinstating a Failed Primary Database


1) Basic checks / requirements for FAST_START FAILOVER

Here are the following basic check for fast_start failover in active dataguad environment using dataguad broker:
1.1) Ensure standby redo logs are configured on the primary and target standby databases.
1.2) Ensure the LogXptMode Property is set to SYNC/ASYNC for both primary and standby database
1.3) Set the FastStartFailoverTarget configuration property.
1.4) Upgrade the protection mode to MAXAVAILABILITY ( if necessary)
1.5) Enable Flashback Database on the primary and target standby databases.
1.6) Start the observer

Note: Active dataguard setup and broker setup should be completed carefully. ( See my next publication )

1.1) Ensure standby redo logs are configured on the primary and target standby databases:

Standby redo logs must be configured on the primary and standby databases. Configure n+1 standby redo logs. i.e., if in primary database, there are 3 redolog groups then set 4 standby redo logs.

1.2) Ensure the LogXptMode Property is set to SYNC/ASYNC for both primary and standby database:

The LogXptMode configurable database property must be set to SYNC on the primary and target standby databases. To set the redo transport service that corresponds to the protection mode you plan to set, use the EDIT DATABASE (property) command on the primary and target standby databases. For example, if the protection mode to be set is MAXAVAILABILITY, you must set the LogXptMode property to SYNC on the primary database and on the target standby database, as shown in the following examples:
Primary:

DGMGRL> show database 'PRIMARY';

Database - PRIMARY
  Role:            PRIMARY
  Intended State:  TRANSPORT-ON
  Instance(s):     PRIMARY

Database Status: SUCCESS

DGMGRL> show configuration;

Configuration - DR_Config
  Protection Mode: MaxAvailability
  Databases:
    PRIMARY - Primary database
    STANDBY - Physical standby database
Fast-Start Failover: DISABLED

Configuration Status: SUCCESS

Standby]

DGMGRL> show database 'STANDBY';

Database - STANDBY
  Role:            PHYSICAL STANDBY
  Intended State:  APPLY-ON
  Transport Lag:   0 seconds
  Apply Lag:       0 seconds
  Real Time Query: OFF
  Instance(s):     STANDBY

Database Status: SUCCESS

DGMGRL> show configuration;

Configuration - DR_Config
  Protection Mode: MaxAvailability
  Databases:
    PRIMARY - Primary database
    STANDBY - Physical standby database

Fast-Start Failover: DISABLED

Configuration Status: SUCCESS
DGMGRL>


1.3) Set the FastStartFailoverTarget configuration property.:

If you have two or more standby databases, set up the FastStartFailoverTarget configuration property on the primary database to indicate the desired target standby database. For example:

PRIMARY]
DGMGRL> connect sys@PRIMARY
Password:
Connected.

DGMGRL> EDIT DATABASE 'PRIMARY' SET PROPERTY FastStartFailoverTarget='STANDBY';
Property "faststartfailovertarget" updated

DGMGRL>

STANDBY]
DGMGRL> connect sys@STANDBY
Password:
Connected.

DGMGRL> EDIT DATABASE 'STANDBY' SET PROPERTY FastStartFailoverTarget='PRIMARY';

Property "faststartfailovertarget" updated

1.4 : Upgrade the protection mode to MAXAVAILABILITY, if necessary.

If it is necessary to upgrade the protection mode, use the following DGMGRL EDIT CONFIGURATION command. For example:

DGMGRL> EDIT CONFIGURATION SET PROTECTION MODE AS MAXAVAILABILITY;

1.5 : Enable Flashback Database on the primary and target standby databases.

If it is not already enabled on the primary and standby databases, enable Flashback Database by issuing the following statements on each database:
ALTER SYSTEM SET UNDO_RETENTION=3600 SCOPE=SPFILE;
ALTER SYSTEM SET UNDO_MANAGEMENT='AUTO' SCOPE=SPFILE;
SHUTDOWN IMMEDIATE;
STARTUP MOUNT;
SHOW PARAMETER UNDO;
ALTER SYSTEM SET DB_FLASHBACK_RETENTION_TARGET=4320 SCOPE=BOTH;
ALTER DATABASE ARCHIVELOG;
ALTER SYSTEM SET db_recovery_file_dest_size=<size>;
ALTER SYSTEM SET db_recovery_file_dest=<directory-specification>;
ALTER DATABASE FLASHBACK ON;
ALTER DATABASE OPEN;

Note : Ensure the UNDO_RETENTION and DB_FLASHBACK_RETENTION_TARGET initialization parameters are set to sufficiently large values so that reinstatement is still possible after a prolonged outage.

1.6) Start the observer

Start the observer by logging into the observer computer and running DGMGRL. Connect to the configuration as SYS and then issue the START OBSERVER command. Note that the command does not return; that is you will not get DGMGRL prompt after issuing the command.

DGMGRL> CONNECT sys@PRIMARY;
Password: password
Connected.

DGMGRL> START OBSERVER;
Observer started

Note : The terminal session will appear to hang at this point.

When starting the observer interactively, Oracle recommends that connection credentials be supplied as a command parameter to the DGMGRL CONNECT command, as shown in the example, rather than as a command line parameter to the DGMGRL command. This practice prevents other users on the system from using a utility (for example, the UNIX ps utility) to display the connection credentials.
When starting the observer from a script, Oracle recommends that you use a method that supports 'connect /', so that database connection credentials do not have to be embedded within the script. If you choose to use a client-side Oracle Wallet as a secure external password store (see Oracle Database Advanced Security Administrator's Guide), be sure to add credentials for both the primary and fast-start failover target standby databases. The database connect string that you specify when adding the credentials for each database must match the ObserverConnectIdentifer or DGConnectIdentifier configurable database property.
To maximize the benefits of FSFO, the observer should run on a different host than the primary and standby databases. Ideally the primary, standby, and observer will be in geographically separate areas. The observer is very lightweight, requiring few system resources.  Unlike the primary / standby interconnect, where bandwidth and latency are determining performance factors, the observer requires very little network bandwidth and is not overly latency sensitive, allowing the it to be placed practically anywhere a reliable connection is available.

Since the observer is a specialized instance of a DGMGRL session, the observer host should be installed with either the Oracle Client Administrator software or the full Oracle Database software stack.
Note that Now My Oracle Client Version is 11.2.0.1.0. We will proceed to start the observer.

Running a StatusReport on the primary should verify that the error is due to a missing observer.

DGMGRL> show database 'PRIMARY' statusreport;
STATUS REPORT
       INSTANCE_NAME   SEVERITY ERROR_TEXT

Note: No error came means, you can proceed. Be sure flashback enabled, undo retain value will be greater as discussed earlier. Don't initiate failover unless at least 30 minutes of history is available


2) Enable fast start failover:

You can enable fast-start failover while connected to any database system in the broker configuration. For example:

DGMGRL> ENABLE FAST_START FAILOVER;
Enabled.

3) Verify the Fast-Start Failover configuration.

Now Cross-check and start "FAST_START FAILOVER":

Use the SHOW FAST_START FAILOVER command to display the fast-start failover settings:

DGMGRL> show FAST_START FAILOVER;

Fast-Start Failover: ENABLED
  Threshold:        30 seconds
  Target:           STANDBY
  Observer:         edc-noc-01d
  Lag Limit:        30 seconds (not in use)
  Shutdown Primary: TRUE
  Auto-reinstate:   TRUE
Configurable Failover Conditions
  Health Conditions:
    Corrupted Controlfile          YES
    Corrupted Dictionary           YES
    Inaccessible Logfile            NO
    Stuck Archiver                  NO
    Datafile Offline               YES
  Oracle Error Conditions:     (none)
DGMGRL>

Note : The FastStartFailoverPmyShutdown and FastStartFailoverAutoReinstate configuration properties do not affect whether the former primary database is shut down and automatically reinstated, respectively, when a fast-start failover occurs if either a user configurable fast-start failover condition is detected or if an application initiated a fast-start failover by calling the DBMS_DG.INITIATE_FS_FAILOVER function. In these cases, the former primary database is always shut down and never automatically reinstated.


4) Fast_start fail-over graceful - seamless application access:

4.1) Create and Start Services
4.2) 4.2) Configure client tnsnames.ora entry
4.3) Verify created service status
4.4) Demo table ( records more than 10 Lakh) to test access when failover happen
4.5) Test the Failover and test graceful access
4.6) Verify the target standby database is ready for failover. in PRIMARY]

-- create service
SQL>
exec dbms_service.create_service(service_name => 'REPORT',network_name => 'REPORT',aq_ha_notifications =>
TRUE,failover_method => 'BASIC',failover_type => 'SELECT',failover_retries => 30,failover_delay => 5);

-- Start Service
sql> exec dbms_service.START_SERVICE('REPORT');

-- Create procedure to trigger for starting/ stopping service in case of shut down of Primary database.
SQL>
create or replace procedure p_cmc_taf_service is
  v_role VARCHAR(30);
begin
  select DATABASE_ROLE into v_role from V$DATABASE;
  if v_role = 'PRIMARY' then
    DBMS_SERVICE.START_SERVICE('REPORT');
  else
    DBMS_SERVICE.STOP_SERVICE('REPORT');
  end if;
end;
/

-- Create triggers to call above procedure when start the database
SQL> 
create or replace TRIGGER trg_cmc_taf_service_startup
  after startup on database
begin
  p_cmc_taf_service;
end;
/
-- Create triggers to call above procedure when database role will be changed in the database
SQL>
create or replace TRIGGER trg_cmc_taf_manage_rolechange
  after db_role_change on database
begin
  p_cmc_taf_service;
end;
/

4.2) Configure client tnsnames.ora entry

in Client]
-- Add tns entry on client side : First do this entry and confirm

-- tnsnames.ora entry:
-- For single IP
REPORT =
  (DESCRIPTION =
    (ADDRESS = (PROTOCOL = TCP)(HOST = 172.18.1.18)(PORT = 1522))
    (CONNECT_DATA =
      (SERVER = DEDICATED)
      (SERVICE_NAME = REPORT)
    )
  )

-- For Application to access seamlessly

REPORT =
(DESCRIPTION =
   (ADDRESS_LIST=
     (ADDRESS = (PROTOCOL = TCP)(HOST = 172.18.1.18)(PORT = 1522))
     (ADDRESS = (PROTOCOL = TCP)(HOST = 172.18.1.17)(PORT = 1521))
   )
     (CONNECT_DATA =
    (SERVER = DEDICATED)
    (SERVICE_NAME = REPORT)
    (FAILOVER_MODE=(TYPE=SELECT)(METHOD=BASIC)(RETRIES=30)(DELAY=5))
     )
)


4.3) Veryfy created service status
-- Test the listener status

$ lsnrctl status

LSNRCTL for Linux: Version 11.2.0.3.0 - Production on 16-AUG-2014 17:39:38
Copyright (c) 1991, 2011, Oracle.  All rights reserved.
Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=EH-UAT-DB)(PORT=1522)))
STATUS of the LISTENER
------------------------
Alias                     LISTENER
Version                   TNSLSNR for Linux: Version 11.2.0.3.0 - Production
Start Date                04-AUG-2014 17:27:58
Uptime                    12 days 0 hr. 11 min. 39 sec
Trace Level               off
Security                  ON: Local OS Authentication
SNMP                      OFF
Listener Parameter File   /rmandisk/app/oracle/product/11.2.0/db_1/network/admin/listener.ora
Listener Log File         /rmandisk/app/oracle/diag/tnslsnr/EH-UAT-DB/listener/alert/log.xml
Listening Endpoints Summary...
  (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=EH-UAT-DB)(PORT=1522)))
  (DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=EXTPROC1522)))
Services Summary...
Service "PLSExtProc" has 1 instance(s).
  Instance "PLSExtProc", status UNKNOWN, has 1 handler(s) for this service...
Service "PRIMARY" has 2 instance(s).
  Instance "PRIMARY", status UNKNOWN, has 1 handler(s) for this service...
  Instance "PRIMARY", status READY, has 1 handler(s) for this service...
Service "PRIMARYXDB" has 1 instance(s).
  Instance "PRIMARY", status READY, has 1 handler(s) for this service...
Service "PRIMARY_DGB" has 1 instance(s).
  Instance "PRIMARY", status READY, has 1 handler(s) for this service...
Service "PRIMARY_DGMRGL.EH-UAT-DB" has 1 instance(s).
  Instance "PRIMARY", status UNKNOWN, has 1 handler(s) for this service...
Service "REPORT" has 1 instance(s).
  Instance "PRIMARY", status READY, has 1 handler(s) for this service...
The command completed successfully

Conclusion:
1) able to connect through service 'REPORT'.
2) Listener status showing the created service 'REPORT'

i.e., OK about service. Next to test below tnsnames entry for seamless application connection.

-- To confirm:
1) Run tnsping REPORT --  succeeded
2) Connect sqlplus window using service name 'REPORT' -- succeeded

Now it is ready for failover testing.....

4.4) Demo table ( records more than 10 Lakh) to test access when failover happen

Create Big table to test a “select” query furring failover operation.

-- Create a big table to test during failover in client machine
-- Create structure
create table big_table as
select rownum id,
               OWNER, OBJECT_NAME, SUBOBJECT_NAME,
               OBJECT_ID, DATA_OBJECT_ID,
               OBJECT_TYPE, CREATED, LAST_DDL_TIME,
               TIMESTAMP, STATUS, TEMPORARY,
               GENERATED, SECONDARY
  from all_objects a  where 1=0;

-- Insert data ( input 100000 when execute the pl-sql block)
declare
    l_cnt number;
    l_rows number := &1;
begin
    insert /*+ append */
    into big_table
    select rownum,OWNER, OBJECT_NAME, SUBOBJECT_NAME,
          OBJECT_ID, DATA_OBJECT_ID, OBJECT_TYPE,
        CREATED, LAST_DDL_TIME, TIMESTAMP, STATUS,
        TEMPORARY, GENERATED, SECONDARY
      from all_objects a where rownum <= &1;
    l_cnt := sql%rowcount;
    commit;
    while (l_cnt < l_rows)
    loop
        insert /*+ APPEND */ into big_table
        select rownum+l_cnt,
               OWNER, OBJECT_NAME, SUBOBJECT_NAME,
               OBJECT_ID, DATA_OBJECT_ID,
               OBJECT_TYPE, CREATED, LAST_DDL_TIME,
               TIMESTAMP, STATUS, TEMPORARY,
               GENERATED, SECONDARY
          from big_table
         where rownum <= l_rows-l_cnt;
        l_cnt := l_cnt + sql%rowcount;
        commit;
    end loop;
end;
/

-- add a constraint
SQL> alter table big_table add constraint big_table_pk primary key(id);
-- Gather the statistics
SQL> exec dbms_stats.gather_table_stats(ownname => 'SYS',tabname => 'BIG_TABLE',cascade => TRUE);

4.5) Test the Failover and test graceful access

Now our preparation is over and ready to test fire the missile “failover”.

PRIMARY]
DGMGRL> connect sys@PRIMARY;
Password:
Connected.

DGMGRL> show database 'PRIMARY';
Database - PRIMARY
  Role:            PRIMARY
  Intended State:  TRANSPORT-ON
  Instance(s):     PRIMARY
Database Status: SUCCESS
DGMGRL>
DGMGRL> show configuration;
Configuration - DR_Config
  Protection Mode: MaxAvailability
  Databases:
    PRIMARY - Primary database
    STANDBY - (*) Physical standby database
Fast-Start Failover: ENABLED
Configuration Status:SUCCESS
DGMGRL>

DGMGRL> SHOW FAST_START FAILOVER;
Fast-Start Failover: ENABLED
  Threshold:        30 seconds
  Target:           STANDBY
  Observer:         edc-noc-01d
  Lag Limit:        30 seconds (not in use)
  Shutdown Primary: TRUE
  Auto-reinstate:   TRUE
Configurable Failover Conditions
  Health Conditions:
    Corrupted Controlfile          YES
    Corrupted Dictionary           YES
    Inaccessible Logfile            NO
    Stuck Archiver                  NO
    Datafile Offline               YES
  Oracle Error Conditions:     (none)
DGMGRL>

4.6) Verify the target standby database is ready for failover.

If fast-start failover is initiated, the observer verifies the target standby database is ready to fail over to the primary database role.

• Fast-start failover cannot occur if:
• Fast-start failover is no longer enabled
• The observer cannot connect to the target standby database
• The observer and the target standby database are inconsistent with regard to the current state of the broker configuration
• The observer is not running
• If the protection mode is maximum availability and the target standby database was not synchronized with the primary database at the time the primary database failed
• If the protection mode is maximum performance and the apply point of the target standby database lags the redo generation point of the primary database by more than the amount specified by the FastStartFailoverLagLimit configuration property at the time the primary database failed
• The target standby database has contact with the primary database
• The FS_FAILOVER_STATUS column in the V$DATABASE view for the target standby database displays a reason why fast-start failover cannot occur
• A manual failover is already in progress.
• The primary database was shut down without using the ABORT option

Follow the steps test Fast_start Failover :

Step 1   Connect to the target standby database.

DGMGRL> connect sys@STANDBY;

Step 2   Issue the failover command.

    Now you can issue the failover command to make the target standby database the new primary database for the configuration. Note that after the failover completes, the original primary database cannot be used as a standby database of the new primary database unless it is re-enabled.

DGMGRL> FAILOVER TO database-name [IMMEDIATE];
e.g.,
DGMGRL> FAILOVER TO 'STANDBY';
Performing failover NOW, please wait...
Error: ORA-16600: not connected to target standby database for failover

Failed.
Unable to failover
DGMGRL>

oops !!! Failed, Oh my God... Ok No. issues, We will troubleshoot

Solution: You may be in other than Standby database. Be sure you logged into Standby database though broker.

DGMGRL> connect sys@STANDBY;
Password:
Connected.

DGMGRL> show database 'STANDBY';
Database - STANDBY
  Role:            PHYSICAL STANDBY
  Intended State:  APPLY-ON
  Transport Lag:   0 seconds
  Apply Lag:       0 seconds
  Real Time Query: OFF
  Instance(s):
    STANDBY
Database Status: SUCCESS

DGMGRL> show configuration;
Configuration - DR_Config
  Protection Mode: MaxAvailability
  Databases:
    PRIMARY - Primary database
    STANDBY - (*) Physical standby database
Fast-Start Failover: ENABLED

Configuration Status: SUCCESS

DGMGRL>
DGMGRL> FAILOVER TO 'STANDBY';
Performing failover NOW, please wait...
Failover succeeded, new primary is "STANDBY"
DGMGRL>
                      
Ohhh.. Yes , Working fine.


Test Case:

I have connected to created service 'REPORT' and started executing a long running query. In the mean time I fired command manual failover. I observed, my “select” query continued and not terminated the session. This situation tells the story that application is continued seamlessly. Application server can't disturb. Only Transactions will be failed what are in primary. Next connection onwards it will continue. Now its time to re-instate the old primary as standby. Because existing standby is new primary database.


    If the target standby database is an Oracle RAC physical or snapshot standby database, the broker will direct CRS to shut down all instances except the apply instance before it continues the failover. If the other instances cannot be shut down, the failover fails. In this case, you must manually shut down all instances except the apply instance and issue the FAILOVER command again. It is also important that you do not start any new instances during the failover. The broker will direct CRS to restart instances that were shut down prior to the failover. No instances will be shut down if failing over to a logical standby database.


5) Managing observer

The observer is integrated in the DGMGRL client-side component of the broker and typically runs on a different computer from the primary or standby databases and from the computer where you manage the broker configuration. The observer continuously monitors the fast-start failover environment to ensure the primary database is available The observer's main purpose is to enhance high availability and lights out computing by reducing the human intervention required by the manual failover process that can add minutes or hours to downtime.
You can manage the observer through either the Data Guard Overview pages in Oracle Enterprise Manager or using DGMGRL commands

1.1) Stopping the Observer:
5.2) How the Observer Maintains Fast-Start Failover Configuration Information
5.3) What Happens if the Observer Fails?

1.1) Stopping the Observer:

You may want to stop the observer when you no longer want to use fast-start failover or if you want to move the observer to a different host machine.

To stop the observer when fast-start failover is enabled, the primary database and target standby database must be connected and communicating with each other. Stopping the observer does not disable the fast-start failover. However, fast-start failover cannot occur when the target standby database is in the unobserved state.

To stop the observer when fast-start failover is not enabled, the primary database must be running.You can stop the observer while connected to any database in the broker configuration that has network connectivity to the primary database, as follows:

--Using Enterprise Manager

Choose the Stop Observer option on the first page of the fast-start failover wizard and click Continue at the bottom of the page. See the Enterprise Manager online help system for more information.

--Using DGMGRL

Issue the following command:

DGMGRL> STOP OBSERVER;

Note:
The observer does not stop immediately when you issue STOP OBSERVER command. When the broker receives the STOP OBSERVER request, it informs the observer the next time the observer contacts the broker.

5.2) How the Observer Maintains Fast-Start Failover Configuration Information:

The observer persistently maintains information about the fast-start failover configuration in a binary file created in the working directory where you started the observer. By default, the observer creates this file in the current working directory when it is started and names the file fsfo.dat. This file contains connect identifiers to both the primary and the target standby databases.
Ensure this file cannot be read by unauthorized users.
Once the observer is started, you cannot change the file's name and location. However, you can change the name or the location of the file if you start the observer using the DGMGRL START OBSERVER command and include the FILE qualifier. See the START OBSERVER command for more information.
Note:
If the observer is stopped abnormally (for example, by typing CTRL/C), restart it and reference the existing fsfo.dat file with the FILE qualifier.

5.3) What Happens if the Observer Fails?

If the primary and target standby databases stay connected but the connection to the observer is lost, then the broker reports that the configuration is not observed. The configuration and database status report that the observer is not running and return one of the following
status messages:
ORA-16658: unobserved fast-start failover configuration
ORA-16820: fast-start failover observer is no longer observing this database
While the configuration is in the unobserved state, fast-start failover cannot happen. Therefore, the primary database can continue processing transactions, even if the target standby database fails. The configuration status returns the SUCCESS status after the observer reestablishes its connection to the primary database, which then notifies the target standby database.

5.4) How the Broker Performs an Immediate Failover Operation

Once you start an immediate failover, the broker:
1.    Verifies that the target standby database is enabled. If the standby database is not enabled for management by the broker, then the failover cannot occur.
2.    Stops Redo Apply or SQL Apply on the standby database immediately, without waiting until all available redo data has been applied. This may result in data loss.
3.    Transitions the target standby database into the primary role, opens the new primary database in read/write mode, and starts redo transport services.
After an immediate failover completes, all the standby databases in the configuration, regardless of their type, are disabled. They must be reenabled before they can serve as standby database to the new primary database.The broker allows the failover to proceed as long as there are no errors for the standby database that you selected to participate in the failover.

6) Reinstating a Failed Primary Database:

You can use the broker's REINSTATE command to re-enable the failed primary database after performing a complete failover to either a physical, snapshot, or logical standby database. You can also use the broker's REINSTATE command to reenable any physical standby databases that were not the target of the failover operation but were disabled during a complete failover to a physical standby database.

6.1) Whan failed primary can be re-instated
6.2) Basic steps to reinstate a database
6.3) Reinstatement Using Enterprise Manager
6.4) How to Re-create and Reenable a Disabled Database
6.5) Reinstating a Failed Primary Database


6.1) Whan failed primary can be re-instated
Databases that can be reinstated will have the following status value:
ORA-16661: the standby database needs to be reinstated
For the REINSTATE command to succeed, Flashback Database must have been enabled on the database prior to the failover and there must be sufficient flashback logs on that database. In addition, the database to be reinstated and the new primary database must have network connectivity.

6.2) Basic steps to reinstate a database:
1.    Restart the database to the mounted state
2.    Connect to the new primary database
3.    Use Enterprise Manager or DGMGRL to reinstate the database
When reinstating a failed primary database, the broker reenables it as a standby database of the same type (physical or logical standby database) as the old standby database. When reinstating physical standby databases that were disabled during a failover, the broker reenables them as physical standby databases to the new primary database.

6.3) Reinstatement Using Enterprise Manager

On the Data Guard Overview page, click the Database must be reinstated link. This brings up the General Properties page that provides a Reinstate button. After you click the Reinstate button, Enterprise Manager begins reinstating the database.
When the process is complete, the database will be enabled as a standby database to the new primary database, and Enterprise Manager displays the Data Guard Overview page.
Reinstatement Using DGMGRL
Issue the following command while connected to any database in the broker configuration, except the database that is to be reinstated:

DGMGRL> REINSTATE DATABASE db_unique_name;
The newly reinstated standby database will begin serving as standby database to the new primary database. If the database is not reinstated successfully, then you must reenable it from a copy of the new primary database.

6.4) How to Re-create and Reenable a Disabled Database :

If you performed a failover or switchover that requires you to re-create the failed primary database or standby databases that were disabled during the role transition, follow the procedures in Oracle Data Guard Concepts and Administration.
Note that if you are re-creating the old primary database, it must be created as the standby type of the old standby database. For example, if the old standby was a physical standby, then the old primary must be re-created as a physical standby. It can then be properly enabled.
After the database has been re-created, enable broker management of the re-created standby database by using the DGMGRL ENABLE DATABASE command.


6.5) Reinstating a Failed Primary Database

After failover, now primary database is 'STANDBY'.  So old primay database to be re-instead as physical stand by database. See below status from current primary database 'STANDBY'.

DGMGRL> FAILOVER TO 'STANDBY';
Performing failover NOW, please wait...
Failover succeeded, new primary is "STANDBY"
DGMGRL>

DGMGRL> show database 'STANDBY';

Database - STANDBY

  Role:            PRIMARY
  Intended State:  TRANSPORT-ON
  Instance(s):
    STANDBY

  Database Warning(s):
    ORA-16817: unsynchronized fast-start failover configuration

Database Status: WARNING
DGMGRL>

DGMGRL> show configuration;

Configuration - DR_Config

  Protection Mode: MaxAvailability
  Databases:
    STANDBY - Primary database
      Warning: ORA-16817: unsynchronized fast-start failover configuration

    PRIMARY - (*) Physical standby database (disabled)
      ORA-16661: the standby database needs to be reinstated

Fast-Start Failover: ENABLED

Configuration Status: WARNING

DGMGRL>

If your primary database had been configured with Flashback Database, you can easily reinstate the failed primary database as a standby database of the new primary database. The failed primary database will be reinstated as a standby type that matches the old standby database. For example, if you failed over to a physical standby database, the old primary will be reinstated as a physical standby database.

To reinstate the failed primary database, start it to the mounted state. Then run DGMGRL, connect to the new primary database and reinstate the old primary database.

Step 1   Restart the Old Primary Database

SQL> startup mount;
ORACLE instance started.

Database mounted.
SQL>
SQL> select name,open_mode from v$database;

NAME          OPEN_MODE
---------         --------------------
PRIMARY       MOUNTED

SQL>


Step 2   Connect to new primary database do the following:

DGMGRL> connect sys@STANDBY;
Password:
Connected.
DGMGRL>

DGMGRL> REINSTATE DATABASE 'PRIMARY';
Reinstating database "PRIMARY", please wait...
Error: ORA-16653: failed to reinstate database

Failed.
Reinstatement of database "PRIMARY" failed

Note: Your primary database may not in mout state.

Step 3  Now REINSTATE the old primary database to make it physical standby

DGMGRL> REINSTATE DATABASE 'PRIMARY';
Reinstating database "PRIMARY", please wait...
Reinstatement of database "PRIMARY" succeeded
DGMGRL>
Now, old primary database ( PRIMARY) is re-instated as new physical standby database.


Step 4   Verify / Cross-check status:
-- See configuration
DGMGRL> SHOW CONFIGURATION;

Configuration - DR_Config
  Protection Mode: MaxAvailability
  Databases:
    STANDBY - Primary database
    PRIMARY - (*) Physical standby database
Fast-Start Failover: ENABLED

Configuration Status: SUCCESS

DGMGRL>

Connect to new physical standby ( old primary):
DGMGRL> connect sys@PRIMARY;
Password:
Connected.
DGMGRL> 

DGMGRL> show configuration;

Configuration - DR_Config

  Protection Mode: MaxAvailability
  Databases:
    STANDBY - Primary database
    PRIMARY - (*) Physical standby database

Fast-Start Failover: ENABLED

Configuration Status: SUCCESS

DGMGRL>
-- See the standby database
DGMGRL> show database 'PRIMARY';

Database - PRIMARY

  Role:            PHYSICAL STANDBY
  Intended State:  APPLY-ON
  Transport Lag:   0 seconds
  Apply Lag:       0 seconds
  Real Time Query: OFF
  Instance(s):     PRIMARY
Database Status: SUCCESS
DGMGRL>

-- See the status of new standby database
DGMGRL> SHOW DATABASE 'PRIMARY' 'StatusReport';
STATUS REPORT
       INSTANCE_NAME   SEVERITY ERROR_TEXT

DGMGRL>

-- See the current status of fast_stat failover

DGMGRL> SHOW FAST_START FAILOVER;

Fast-Start Failover: ENABLED

  Threshold:        30 seconds
  Target:           PRIMARY
  Observer:         edc-noc-01d
  Lag Limit:        30 seconds (not in use)
  Shutdown Primary: TRUE
  Auto-reinstate:   TRUE

Configurable Failover Conditions
  Health Conditions:
    Corrupted Controlfile          YES
    Corrupted Dictionary           YES
    Inaccessible Logfile            NO
    Stuck Archiver                  NO
    Datafile Offline               YES

  Oracle Error Conditions:
    (none)

DGMGRL> 

Thanks .. Cheers !!!!