Start Failover using Dataguard Broker
Fast-Start Failover: Concepts
Oracle
introduced the
Fast-Start Failover
(FSF) feature set in Release 10
gR2,
but it’s been enhanced significantly in Oracle 11
g to permit much finer-grained control over the conditions
under which a FSF would be initiated. Simply put, FSF ensures that under the
appropriate circumstances – some of which are
mandatory,
and some of which are
optionally-configured
- a failover to the chosen standby target database will occur
without DBA
intervention. The following occurrences will initiate FSF:
Fast-Start
Failover detects one of these failover situations through the
Fast-Start Failover Observer (FSFO). The
FSFO leverages the
Oracle Call Interface
(OCI) architecture to decide
when
a failover is necessary,
which
physical standby database should be the target of the failover, and
how long to wait until it declares a
failover is absolutely necessary. The good news is that I have quite a bit of
control over these three directives, all of which are completely configurable
using the
Data Guard Broker Manager
utility (
DGMGRL).
Topics to be discussed:
1) Basic checks / requirements for FAST_START FAILOVER
2) Enable fast start failover.
3) Verify the fast-start failover configuration.
4) Fast_start failover gracefull - seamless application access
5) Managing observer
6) Reinstating a Failed Primary Database
1) Basic checks / requirements for FAST_START FAILOVER
Here are the following basic check for fast_start failover in active dataguad environment using dataguad broker:
1.1) Ensure standby redo logs are configured on the primary and target standby databases.
1.2) Ensure the LogXptMode Property is set to SYNC/ASYNC for both primary and standby database
1.3) Set the FastStartFailoverTarget configuration property.
1.4) Upgrade the protection mode to MAXAVAILABILITY ( if necessary)
1.5) Enable Flashback Database on the primary and target standby databases.
1.6) Start the observer
Note: Active dataguard setup and broker setup should be completed carefully. ( See my next publication )
1.1) Ensure standby redo logs are configured on the primary and target standby databases:
Standby redo logs must be configured on the primary and standby databases. Configure n+1 standby redo logs. i.e., if in primary database, there are 3 redolog groups then set 4 standby redo logs.
1.2) Ensure the LogXptMode Property is set to SYNC/ASYNC for both primary and standby database:
The LogXptMode configurable database property must be set to SYNC on the primary and target standby databases. To set the redo transport service that corresponds to the protection mode you plan to set, use the EDIT DATABASE (property) command on the primary and target standby databases. For example, if the protection mode to be set is MAXAVAILABILITY, you must set the LogXptMode property to SYNC on the primary database and on the target standby database, as shown in the following examples:
Primary:
DGMGRL> show database 'PRIMARY';
Database - PRIMARY
Role: PRIMARY
Intended State: TRANSPORT-ON
Instance(s): PRIMARY
Database Status: SUCCESS
DGMGRL> show configuration;
Configuration - DR_Config
Protection Mode: MaxAvailability
Databases:
PRIMARY - Primary database
STANDBY - Physical standby database
Fast-Start Failover: DISABLED
Configuration Status: SUCCESS
Standby]
DGMGRL> show database 'STANDBY';
Database - STANDBY
Role: PHYSICAL STANDBY
Intended State: APPLY-ON
Transport Lag: 0 seconds
Apply Lag: 0 seconds
Real Time Query: OFF
Instance(s): STANDBY
Database Status: SUCCESS
DGMGRL> show configuration;
Configuration - DR_Config
Protection Mode: MaxAvailability
Databases:
PRIMARY - Primary database
STANDBY - Physical standby database
Fast-Start Failover: DISABLED
Configuration Status: SUCCESS
DGMGRL>
1.3) Set the FastStartFailoverTarget configuration property.:
If you have two or more standby databases, set up the FastStartFailoverTarget configuration property on the primary database to indicate the desired target standby database. For example:
PRIMARY]
DGMGRL> connect sys@PRIMARY
Password:
Connected.
DGMGRL> EDIT DATABASE 'PRIMARY' SET PROPERTY FastStartFailoverTarget='STANDBY';
Property "faststartfailovertarget" updated
DGMGRL>
STANDBY]
DGMGRL> connect sys@STANDBY
Password:
Connected.
DGMGRL> EDIT DATABASE 'STANDBY' SET PROPERTY FastStartFailoverTarget='PRIMARY';
Property "faststartfailovertarget" updated
1.4 : Upgrade the protection mode to MAXAVAILABILITY, if necessary.
If it is necessary to upgrade the protection mode, use the following DGMGRL EDIT CONFIGURATION command. For example:
DGMGRL> EDIT CONFIGURATION SET PROTECTION MODE AS MAXAVAILABILITY;
1.5 : Enable Flashback Database on the primary and target standby databases.
If it is not already enabled on the primary and standby databases, enable Flashback Database by issuing the following statements on each database:
ALTER SYSTEM SET UNDO_RETENTION=3600 SCOPE=SPFILE;
ALTER SYSTEM SET UNDO_MANAGEMENT='AUTO' SCOPE=SPFILE;
SHUTDOWN IMMEDIATE;
STARTUP MOUNT;
SHOW PARAMETER UNDO;
ALTER SYSTEM SET DB_FLASHBACK_RETENTION_TARGET=4320 SCOPE=BOTH;
ALTER DATABASE ARCHIVELOG;
ALTER SYSTEM SET db_recovery_file_dest_size=<size>;
ALTER SYSTEM SET db_recovery_file_dest=<directory-specification>;
ALTER DATABASE FLASHBACK ON;
ALTER DATABASE OPEN;
Note : Ensure the UNDO_RETENTION and DB_FLASHBACK_RETENTION_TARGET initialization parameters are set to sufficiently large values so that reinstatement is still possible after a prolonged outage.
1.6) Start the observer
Start the observer by logging into the observer computer and running DGMGRL. Connect to the configuration as SYS and then issue the START OBSERVER command. Note that the command does not return; that is you will not get DGMGRL prompt after issuing the command.
DGMGRL> CONNECT sys@PRIMARY;
Password: password
Connected.
DGMGRL> START OBSERVER;
Observer started
Note : The terminal session will appear to hang at this point.
When starting the observer interactively, Oracle recommends that connection credentials be supplied as a command parameter to the DGMGRL CONNECT command, as shown in the example, rather than as a command line parameter to the DGMGRL command. This practice prevents other users on the system from using a utility (for example, the UNIX ps utility) to display the connection credentials.
When starting the observer from a script, Oracle recommends that you use a method that supports 'connect /', so that database connection credentials do not have to be embedded within the script. If you choose to use a client-side Oracle Wallet as a secure external password store (see Oracle Database Advanced Security Administrator's Guide), be sure to add credentials for both the primary and fast-start failover target standby databases. The database connect string that you specify when adding the credentials for each database must match the ObserverConnectIdentifer or DGConnectIdentifier configurable database property.
To maximize the benefits of FSFO, the observer should run on a different host than the primary and standby databases. Ideally the primary, standby, and observer will be in geographically separate areas. The observer is very lightweight, requiring few system resources. Unlike the primary / standby interconnect, where bandwidth and latency are determining performance factors, the observer requires very little network bandwidth and is not overly latency sensitive, allowing the it to be placed practically anywhere a reliable connection is available.
Since the observer is a specialized instance of a DGMGRL session, the observer host should be installed with either the Oracle Client Administrator software or the full Oracle Database software stack.
Note that Now My Oracle Client Version is 11.2.0.1.0. We will proceed to start the observer.
Running a StatusReport on the primary should verify that the error is due to a missing observer.
DGMGRL> show database 'PRIMARY' statusreport;
STATUS REPORT
INSTANCE_NAME SEVERITY ERROR_TEXT
Note: No error came means, you can proceed. Be sure flashback enabled, undo retain value will be greater as discussed earlier. Don't initiate failover unless at least 30 minutes of history is available
2) Enable fast start failover:
You can enable fast-start failover while connected to any database system in the broker configuration. For example:
DGMGRL> ENABLE FAST_START FAILOVER;
Enabled.
3) Verify the Fast-Start Failover configuration.
Now Cross-check and start "FAST_START FAILOVER":
Use the SHOW FAST_START FAILOVER command to display the fast-start failover settings:
DGMGRL> show FAST_START FAILOVER;
Fast-Start Failover: ENABLED
Threshold: 30 seconds
Target: STANDBY
Observer: edc-noc-01d
Lag Limit: 30 seconds (not in use)
Shutdown Primary: TRUE
Auto-reinstate: TRUE
Configurable Failover Conditions
Health Conditions:
Corrupted Controlfile YES
Corrupted Dictionary YES
Inaccessible Logfile NO
Stuck Archiver NO
Datafile Offline YES
Oracle Error Conditions: (none)
DGMGRL>
Note : The FastStartFailoverPmyShutdown and FastStartFailoverAutoReinstate configuration properties do not affect whether the former primary database is shut down and automatically reinstated, respectively, when a fast-start failover occurs if either a user configurable fast-start failover condition is detected or if an application initiated a fast-start failover by calling the DBMS_DG.INITIATE_FS_FAILOVER function. In these cases, the former primary database is always shut down and never automatically reinstated.
4) Fast_start fail-over graceful - seamless application access:
4.1) Create and Start Services
4.2) 4.2) Configure client tnsnames.ora entry
4.3) Verify created service status
4.4) Demo table ( records more than 10 Lakh) to test access when failover happen
4.5) Test the Failover and test graceful access
4.6) Verify the target standby database is ready for failover.
in PRIMARY]
-- create service
SQL>
exec dbms_service.create_service(service_name => 'REPORT',network_name => 'REPORT',aq_ha_notifications =>
TRUE,failover_method => 'BASIC',failover_type => 'SELECT',failover_retries => 30,failover_delay => 5);
-- Start Service
sql> exec dbms_service.START_SERVICE('REPORT');
-- Create procedure to trigger for starting/ stopping service in case of shut down of Primary database.
SQL>
create or replace procedure p_cmc_taf_service is
v_role VARCHAR(30);
begin
select DATABASE_ROLE into v_role from V$DATABASE;
if v_role = 'PRIMARY' then
DBMS_SERVICE.START_SERVICE('REPORT');
else
DBMS_SERVICE.STOP_SERVICE('REPORT');
end if;
end;
/
-- Create triggers to call above procedure when start the database
SQL>
create or replace TRIGGER trg_cmc_taf_service_startup
after startup on database
begin
p_cmc_taf_service;
end;
/
-- Create triggers to call above procedure when database role will be changed in the database
SQL>
create or replace TRIGGER trg_cmc_taf_manage_rolechange
after db_role_change on database
begin
p_cmc_taf_service;
end;
/
4.2) Configure client tnsnames.ora entry
in Client]
-- Add tns entry on client side : First do this entry and confirm
-- tnsnames.ora entry:
-- For single IP
REPORT =
(DESCRIPTION =
(ADDRESS = (PROTOCOL = TCP)(HOST = 172.18.1.18)(PORT = 1522))
(CONNECT_DATA =
(SERVER = DEDICATED)
(SERVICE_NAME = REPORT)
)
)
-- For Application to access seamlessly
REPORT =
(DESCRIPTION =
(ADDRESS_LIST=
(ADDRESS = (PROTOCOL = TCP)(HOST = 172.18.1.18)(PORT = 1522))
(ADDRESS = (PROTOCOL = TCP)(HOST = 172.18.1.17)(PORT = 1521))
)
(CONNECT_DATA =
(SERVER = DEDICATED)
(SERVICE_NAME = REPORT)
(FAILOVER_MODE=(TYPE=SELECT)(METHOD=BASIC)(RETRIES=30)(DELAY=5))
)
)
4.3) Veryfy created service status
-- Test the listener status
$ lsnrctl status
LSNRCTL for Linux: Version 11.2.0.3.0 - Production on 16-AUG-2014 17:39:38
Copyright (c) 1991, 2011, Oracle. All rights reserved.
Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=EH-UAT-DB)(PORT=1522)))
STATUS of the LISTENER
------------------------
Alias LISTENER
Version TNSLSNR for Linux: Version 11.2.0.3.0 - Production
Start Date 04-AUG-2014 17:27:58
Uptime 12 days 0 hr. 11 min. 39 sec
Trace Level off
Security ON: Local OS Authentication
SNMP OFF
Listener Parameter File /rmandisk/app/oracle/product/11.2.0/db_1/network/admin/listener.ora
Listener Log File /rmandisk/app/oracle/diag/tnslsnr/EH-UAT-DB/listener/alert/log.xml
Listening Endpoints Summary...
(DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=EH-UAT-DB)(PORT=1522)))
(DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=EXTPROC1522)))
Services Summary...
Service "PLSExtProc" has 1 instance(s).
Instance "PLSExtProc", status UNKNOWN, has 1 handler(s) for this service...
Service "PRIMARY" has 2 instance(s).
Instance "PRIMARY", status UNKNOWN, has 1 handler(s) for this service...
Instance "PRIMARY", status READY, has 1 handler(s) for this service...
Service "PRIMARYXDB" has 1 instance(s).
Instance "PRIMARY", status READY, has 1 handler(s) for this service...
Service "PRIMARY_DGB" has 1 instance(s).
Instance "PRIMARY", status READY, has 1 handler(s) for this service...
Service "PRIMARY_DGMRGL.EH-UAT-DB" has 1 instance(s).
Instance "PRIMARY", status UNKNOWN, has 1 handler(s) for this service...
Service "REPORT" has 1 instance(s).
Instance "PRIMARY", status READY, has 1 handler(s) for this service...
The command completed successfully
Conclusion:
1) able to connect through service 'REPORT'.
2) Listener status showing the created service 'REPORT'
i.e., OK about service. Next to test below tnsnames entry for seamless application connection.
-- To confirm:
1) Run tnsping REPORT -- succeeded
2) Connect sqlplus window using service name 'REPORT' -- succeeded
Now it is ready for failover testing.....
4.4) Demo table ( records more than 10 Lakh) to test access when failover happen
Create Big table to test a “select” query furring failover operation.
-- Create a big table to test during failover in client machine
-- Create structure
create table big_table as
select rownum id,
OWNER, OBJECT_NAME, SUBOBJECT_NAME,
OBJECT_ID, DATA_OBJECT_ID,
OBJECT_TYPE, CREATED, LAST_DDL_TIME,
TIMESTAMP, STATUS, TEMPORARY,
GENERATED, SECONDARY
from all_objects a where 1=0;
-- Insert data ( input 100000 when execute the pl-sql block)
declare
l_cnt number;
l_rows number := &1;
begin
insert /*+ append */
into big_table
select rownum,OWNER, OBJECT_NAME, SUBOBJECT_NAME,
OBJECT_ID, DATA_OBJECT_ID, OBJECT_TYPE,
CREATED, LAST_DDL_TIME, TIMESTAMP, STATUS,
TEMPORARY, GENERATED, SECONDARY
from all_objects a where rownum <= &1;
l_cnt := sql%rowcount;
commit;
while (l_cnt < l_rows)
loop
insert /*+ APPEND */ into big_table
select rownum+l_cnt,
OWNER, OBJECT_NAME, SUBOBJECT_NAME,
OBJECT_ID, DATA_OBJECT_ID,
OBJECT_TYPE, CREATED, LAST_DDL_TIME,
TIMESTAMP, STATUS, TEMPORARY,
GENERATED, SECONDARY
from big_table
where rownum <= l_rows-l_cnt;
l_cnt := l_cnt + sql%rowcount;
commit;
end loop;
end;
/
-- add a constraint
SQL> alter table big_table add constraint big_table_pk primary key(id);
-- Gather the statistics
SQL> exec dbms_stats.gather_table_stats(ownname => 'SYS',tabname => 'BIG_TABLE',cascade => TRUE);
4.5) Test the Failover and test graceful access
Now our preparation is over and ready to test fire the missile “failover”.
PRIMARY]
DGMGRL> connect sys@PRIMARY;
Password:
Connected.
DGMGRL> show database 'PRIMARY';
Database - PRIMARY
Role: PRIMARY
Intended State: TRANSPORT-ON
Instance(s): PRIMARY
Database Status: SUCCESS
DGMGRL>
DGMGRL> show configuration;
Configuration - DR_Config
Protection Mode: MaxAvailability
Databases:
PRIMARY - Primary database
STANDBY - (*) Physical standby database
Fast-Start Failover: ENABLED
Configuration Status:SUCCESS
DGMGRL>
DGMGRL> SHOW FAST_START FAILOVER;
Fast-Start Failover: ENABLED
Threshold: 30 seconds
Target: STANDBY
Observer: edc-noc-01d
Lag Limit: 30 seconds (not in use)
Shutdown Primary: TRUE
Auto-reinstate: TRUE
Configurable Failover Conditions
Health Conditions:
Corrupted Controlfile YES
Corrupted Dictionary YES
Inaccessible Logfile NO
Stuck Archiver NO
Datafile Offline YES
Oracle Error Conditions: (none)
DGMGRL>
4.6) Verify the target standby database is ready for failover.
If fast-start failover is initiated, the observer verifies the target standby database is ready to fail over to the primary database role.
- Fast-start failover cannot occur if:
- Fast-start failover is no longer enabled
- The observer cannot connect to the target standby database
- The observer and the target standby database are inconsistent with regard to the current state of the broker configuration
- The observer is not running
- If the protection mode is maximum availability and the target standby database was not synchronized with the primary database at the time the primary database failed
- If the protection mode is maximum performance and the apply point of the target standby database lags the redo generation point of the primary database by more than the amount specified by the FastStartFailoverLagLimit configuration property at the time the primary database failed
- The target standby database has contact with the primary database
- The FS_FAILOVER_STATUS column in the V$DATABASE view for the target standby database displays a reason why fast-start failover cannot occur
- A manual failover is already in progress.
- The primary database was shut down without using the ABORT option
Follow the steps test Fast_start Failover :
Step 1 Connect to the target standby database.
DGMGRL> connect sys@STANDBY;
Step 2 Issue the failover command.
Now you can issue the failover command to make the target standby database the new primary database for the configuration. Note that after the failover completes, the original primary database cannot be used as a standby database of the new primary database unless it is re-enabled.
DGMGRL> FAILOVER TO database-name [IMMEDIATE];
e.g.,
DGMGRL> FAILOVER TO 'STANDBY';
Performing failover NOW, please wait...
Error: ORA-16600: not connected to target standby database for failover
Failed.
Unable to failover
DGMGRL>
oops !!! Failed, Oh my God... Ok No. issues, We will troubleshoot
Solution: You may be in other than Standby database. Be sure you logged into Standby database though broker.
DGMGRL> connect sys@STANDBY;
Password:
Connected.
DGMGRL> show database 'STANDBY';
Database - STANDBY
Role: PHYSICAL STANDBY
Intended State: APPLY-ON
Transport Lag: 0 seconds
Apply Lag: 0 seconds
Real Time Query: OFF
Instance(s):
STANDBY
Database Status: SUCCESS
DGMGRL> show configuration;
Configuration - DR_Config
Protection Mode: MaxAvailability
Databases:
PRIMARY - Primary database
STANDBY - (*) Physical standby database
Fast-Start Failover: ENABLED
Configuration Status: SUCCESS
DGMGRL>
DGMGRL> FAILOVER TO 'STANDBY';
Performing failover NOW, please wait...
Failover succeeded, new primary is "STANDBY"
DGMGRL>
Ohhh.. Yes , Working fine.
Test Case:
I have connected to created service 'REPORT' and started executing a long running query. In the mean time I fired command manual failover. I observed, my “select” query continued and not terminated the session. This situation tells the story that application is continued seamlessly. Application server can't disturb. Only Transactions will be failed what are in primary. Next connection onwards it will continue. Now its time to re-instate the old primary as standby. Because existing standby is new primary database.
If the target standby database is an Oracle RAC physical or snapshot standby database, the broker will direct CRS to shut down all instances except the apply instance before it continues the failover. If the other instances cannot be shut down, the failover fails. In this case, you must manually shut down all instances except the apply instance and issue the FAILOVER command again. It is also important that you do not start any new instances during the failover. The broker will direct CRS to restart instances that were shut down prior to the failover. No instances will be shut down if failing over to a logical standby database.
5) Managing observer
The observer is integrated in the DGMGRL client-side component of the broker and typically runs on a different computer from the primary or standby databases and from the computer where you manage the broker configuration. The observer continuously monitors the fast-start failover environment to ensure the primary database is available The observer's main purpose is to enhance high availability and lights out computing by reducing the human intervention required by the manual failover process that can add minutes or hours to downtime.
You can manage the observer through either the Data Guard Overview pages in Oracle Enterprise Manager or using DGMGRL commands
1.1) Stopping the Observer:
5.2) How the Observer Maintains Fast-Start Failover Configuration Information
5.3) What Happens if the Observer Fails?
1.1) Stopping the Observer:
You may want to stop the observer when you no longer want to use fast-start failover or if you want to move the observer to a different host machine.
To stop the observer when fast-start failover is enabled, the primary database and target standby database must be connected and communicating with each other. Stopping the observer does not disable the fast-start failover. However, fast-start failover cannot occur when the target standby database is in the unobserved state.
To stop the observer when fast-start failover is not enabled, the primary database must be running.You can stop the observer while connected to any database in the broker configuration that has network connectivity to the primary database, as follows:
--Using Enterprise Manager
Choose the Stop Observer option on the first page of the fast-start failover wizard and click Continue at the bottom of the page. See the Enterprise Manager online help system for more information.
--Using DGMGRL
Issue the following command:
DGMGRL> STOP OBSERVER;
Note:
The observer does not stop immediately when you issue STOP OBSERVER command. When the broker receives the STOP OBSERVER request, it informs the observer the next time the observer contacts the broker.
5.2) How the Observer Maintains Fast-Start Failover Configuration Information:
The observer persistently maintains information about the fast-start failover configuration in a binary file created in the working directory where you started the observer. By default, the observer creates this file in the current working directory when it is started and names the file fsfo.dat. This file contains connect identifiers to both the primary and the target standby databases.
Ensure this file cannot be read by unauthorized users.
Once the observer is started, you cannot change the file's name and location. However, you can change the name or the location of the file if you start the observer using the DGMGRL START OBSERVER command and include the FILE qualifier. See the START OBSERVER command for more information.
Note:
If the observer is stopped abnormally (for example, by typing CTRL/C), restart it and reference the existing fsfo.dat file with the FILE qualifier.
5.3) What Happens if the Observer Fails?
If the primary and target standby databases stay connected but the connection to the observer is lost, then the broker reports that the configuration is not observed. The configuration and database status report that the observer is not running and return one of the following
status messages:
ORA-16658: unobserved fast-start failover configuration
ORA-16820: fast-start failover observer is no longer observing this database
While the configuration is in the unobserved state, fast-start failover cannot happen. Therefore, the primary database can continue processing transactions, even if the target standby database fails. The configuration status returns the SUCCESS status after the observer reestablishes its connection to the primary database, which then notifies the target standby database.
5.4) How the Broker Performs an Immediate Failover Operation
Once you start an immediate failover, the broker:
1. Verifies that the target standby database is enabled. If the standby database is not enabled for management by the broker, then the failover cannot occur.
2. Stops Redo Apply or SQL Apply on the standby database immediately, without waiting until all available redo data has been applied. This may result in data loss.
3. Transitions the target standby database into the primary role, opens the new primary database in read/write mode, and starts redo transport services.
After an immediate failover completes, all the standby databases in the configuration, regardless of their type, are disabled. They must be reenabled before they can serve as standby database to the new primary database.The broker allows the failover to proceed as long as there are no errors for the standby database that you selected to participate in the failover.
6) Reinstating a Failed Primary Database:
You can use the broker's REINSTATE command to re-enable the failed primary database after performing a complete failover to either a physical, snapshot, or logical standby database. You can also use the broker's REINSTATE command to reenable any physical standby databases that were not the target of the failover operation but were disabled during a complete failover to a physical standby database.
6.1) Whan failed primary can be re-instated
6.2) Basic steps to reinstate a database
6.3) Reinstatement Using Enterprise Manager
6.4) How to Re-create and Reenable a Disabled Database
6.5) Reinstating a Failed Primary Database
6.1) Whan failed primary can be re-instated
Databases that can be reinstated will have the following status value:
ORA-16661: the standby database needs to be reinstated
For the REINSTATE command to succeed, Flashback Database must have been enabled on the database prior to the failover and there must be sufficient flashback logs on that database. In addition, the database to be reinstated and the new primary database must have network connectivity.
6.2) Basic steps to reinstate a database:
1. Restart the database to the mounted state
2. Connect to the new primary database
3. Use Enterprise Manager or DGMGRL to reinstate the database
When reinstating a failed primary database, the broker reenables it as a standby database of the same type (physical or logical standby database) as the old standby database. When reinstating physical standby databases that were disabled during a failover, the broker reenables them as physical standby databases to the new primary database.
6.3) Reinstatement Using Enterprise Manager
On the Data Guard Overview page, click the Database must be reinstated link. This brings up the General Properties page that provides a Reinstate button. After you click the Reinstate button, Enterprise Manager begins reinstating the database.
When the process is complete, the database will be enabled as a standby database to the new primary database, and Enterprise Manager displays the Data Guard Overview page.
Reinstatement Using DGMGRL
Issue the following command while connected to any database in the broker configuration, except the database that is to be reinstated:
DGMGRL> REINSTATE DATABASE db_unique_name;
The newly reinstated standby database will begin serving as standby database to the new primary database. If the database is not reinstated successfully, then you must reenable it from a copy of the new primary database.
6.4) How to Re-create and Reenable a Disabled Database :
If you performed a failover or switchover that requires you to re-create the failed primary database or standby databases that were disabled during the role transition, follow the procedures in Oracle Data Guard Concepts and Administration.
Note that if you are re-creating the old primary database, it must be created as the standby type of the old standby database. For example, if the old standby was a physical standby, then the old primary must be re-created as a physical standby. It can then be properly enabled.
After the database has been re-created, enable broker management of the re-created standby database by using the DGMGRL ENABLE DATABASE command.
6.5) Reinstating a Failed Primary Database
After failover, now primary database is 'STANDBY'. So old primay database to be re-instead as physical stand by database. See below status from current primary database 'STANDBY'.
DGMGRL> FAILOVER TO 'STANDBY';
Performing failover NOW, please wait...
Failover succeeded, new primary is "STANDBY"
DGMGRL>
DGMGRL> show database 'STANDBY';
Database - STANDBY
Role: PRIMARY
Intended State: TRANSPORT-ON
Instance(s):
STANDBY
Database Warning(s):
ORA-16817: unsynchronized fast-start failover configuration
Database Status: WARNING
DGMGRL>
DGMGRL> show configuration;
Configuration - DR_Config
Protection Mode: MaxAvailability
Databases:
STANDBY - Primary database
Warning: ORA-16817: unsynchronized fast-start failover configuration
PRIMARY - (*) Physical standby database (disabled)
ORA-16661: the standby database needs to be reinstated
Fast-Start Failover: ENABLED
Configuration Status: WARNING
DGMGRL>
If your primary database had been configured with Flashback Database, you can easily reinstate the failed primary database as a standby database of the new primary database. The failed primary database will be reinstated as a standby type that matches the old standby database. For example, if you failed over to a physical standby database, the old primary will be reinstated as a physical standby database.
To reinstate the failed primary database, start it to the mounted state. Then run DGMGRL, connect to the new primary database and reinstate the old primary database.
Step 1 Restart the Old Primary Database
SQL> startup mount;
ORACLE instance started.
Database mounted.
SQL>
SQL> select name,open_mode from v$database;
NAME OPEN_MODE
--------- --------------------
PRIMARY MOUNTED
SQL>
Step 2 Connect to new primary database do the following:
DGMGRL> connect sys@STANDBY;
Password:
Connected.
DGMGRL>
DGMGRL> REINSTATE DATABASE 'PRIMARY';
Reinstating database "PRIMARY", please wait...
Error: ORA-16653: failed to reinstate database
Failed.
Reinstatement of database "PRIMARY" failed
Note: Your primary database may not in mout state.
Step 3 Now REINSTATE the old primary database to make it physical standby
DGMGRL> REINSTATE DATABASE 'PRIMARY';
Reinstating database "PRIMARY", please wait...
Reinstatement of database "PRIMARY" succeeded
DGMGRL>
Now, old primary database ( PRIMARY) is re-instated as new physical standby database.
Step 4 Verify / Cross-check status:
-- See configuration
DGMGRL> SHOW CONFIGURATION;
Configuration - DR_Config
Protection Mode: MaxAvailability
Databases:
STANDBY - Primary database
PRIMARY - (*) Physical standby database
Fast-Start Failover: ENABLED
Configuration Status: SUCCESS
DGMGRL>
Connect to new physical standby ( old primary):
DGMGRL> connect sys@PRIMARY;
Password:
Connected.
DGMGRL>
DGMGRL> show configuration;
Configuration - DR_Config
Protection Mode: MaxAvailability
Databases:
STANDBY - Primary database
PRIMARY - (*) Physical standby database
Fast-Start Failover: ENABLED
Configuration Status: SUCCESS
DGMGRL>
-- See the standby database
DGMGRL> show database 'PRIMARY';
Database - PRIMARY
Role: PHYSICAL STANDBY
Intended State: APPLY-ON
Transport Lag: 0 seconds
Apply Lag: 0 seconds
Real Time Query: OFF
Instance(s): PRIMARY
Database Status: SUCCESS
DGMGRL>
-- See the status of new standby database
DGMGRL> SHOW DATABASE 'PRIMARY' 'StatusReport';
STATUS REPORT
INSTANCE_NAME SEVERITY ERROR_TEXT
DGMGRL>
-- See the current status of fast_stat failover
DGMGRL> SHOW FAST_START FAILOVER;
Fast-Start Failover: ENABLED
Threshold: 30 seconds
Target: PRIMARY
Observer: edc-noc-01d
Lag Limit: 30 seconds (not in use)
Shutdown Primary: TRUE
Auto-reinstate: TRUE
Configurable Failover Conditions
Health Conditions:
Corrupted Controlfile YES
Corrupted Dictionary YES
Inaccessible Logfile NO
Stuck Archiver NO
Datafile Offline YES
Oracle Error Conditions:
(none)
DGMGRL>
Thanks .. Cheers !!!!
