ORA-19909: datafile 1 belongs to an orphan incarnation

When I tried to Flashback on in standby database then suddenly below message found in alert log and standby database not able to apply archive-logs also. So, here are messages from standby database alert log:

-- Message from alert log ( in standby database)

Mon Dec 12 12:43:24 2016
alter database recover managed standby database disconnect from session
Attempt to start background Managed Standby Recovery process (PROD)
Mon Dec 12 12:43:24 2016
MRP0 started with pid=26, OS id=22750
MRP0: Background Managed Standby Recovery process started (PROD)
 started logmerger process
Mon Dec 12 12:43:29 2016
Managed Standby Recovery not using Real Time Apply
Warning: Recovery target destination is in a sibling branch
of the controlfile checkpoint. Recovery will only recover
changes to datafiles.
Datafile 1 (ckpscn 5983588137777) is orphaned on incarnation#=2
MRP0: Detected orphaned datafiles!
Recovery will possibly be retried after flashback...
Errors in file /oracle/diag/rdbms/PROD/PROD/trace/PROD_pr00_22755.trc:
ORA-19909: datafile 1 belongs to an orphan incarnation
ORA-01110: data file 1: '/u03/oradata/PROD/PROD/system01.dbf'
Recovery Slave PR00 previously exited with exception 19909
Completed: alter database recover managed standby database disconnect from session
Mon Dec 12 12:43:51 2016
MRP0: Background Media Recovery process shutdown (PROD)

RCA:

Primary and standby have different incarnations:

-- in standby db

RMAN> list incarnation of database;

using target database control file instead of recovery catalog

List of Database Incarnations
DB Key  Inc Key DB Name  DB ID            STATUS  Reset SCN  Reset Time
------- ------- -------- ---------------- --- ---------- ----------
1       1       PROD  3680942556       PARENT  5982315579513 30-NOV-2016 00:31:04
2       2       PROD  3680942556       PARENT  5982448719535 30-NOV-2016 22:31:15
4       4       PROD  3680942556       ORPHAN  5983354187673 09-DEC-2016 04:01:27
3       3       PROD  3680942556       CURRENT 5983354274164 10-DEC-2016 04:02:53

RMAN>


-- in primary db

RMAN> list incarnation of database;

using target database control file instead of recovery catalog

List of Database Incarnations
DB Key  Inc Key DB Name  DB ID            STATUS  Reset SCN  Reset Time
------- ------- -------- ---------------- --- ---------- ----------
1       1       PROD  3680942556       PARENT  5982315579513 30-NOV-2016 00:31:04
2       2       PROD  3680942556       CURRENT 5982448719535 30-NOV-2016 22:31:15

RMAN>


-- Solution:

You need to reset the standby database's incarnation to match the primary's:

-- in standby

RMAN> list incarnation of database;

using target database control file instead of recovery catalog

List of Database Incarnations
DB Key  Inc Key DB Name  DB ID            STATUS  Reset SCN  Reset Time
------- ------- -------- ---------------- --- ---------- ----------
1       1       PROD  3680942556       PARENT  5982315579513 30-NOV-2016 00:31:04
2       2       PROD  3680942556       PARENT  5982448719535 30-NOV-2016 22:31:15
4       4       PROD  3680942556       ORPHAN  5983354187673 09-DEC-2016 04:01:27
3       3       PROD  3680942556       CURRENT 5983354274164 10-DEC-2016 04:02:53

RMAN> reset database  to incarnation 2;

database reset to incarnation 2

RMAN> list incarnation of database;


List of Database Incarnations
DB Key  Inc Key DB Name  DB ID            STATUS  Reset SCN  Reset Time
------- ------- -------- ---------------- --- ---------- ----------
1       1       PROD  3680942556       PARENT  5982315579513 30-NOV-2016 00:31:04
2       2       PROD  3680942556       CURRENT 5982448719535 30-NOV-2016 22:31:15
4       4       PROD  3680942556       ORPHAN  5983354187673 09-DEC-2016 04:01:27
3       3       PROD  3680942556       ORPHAN  5983354274164 10-DEC-2016 04:02:53

RMAN>

-- Verify MRP is active or not

$ ps -ef|grep mrp
oracle   23358 21699  0 12:56 pts/1    00:00:00 grep mrp
$

-- Verify standby database status and role

SQL> select name,open_mode ,database_role from v$database;

NAME      OPEN_MODE            DATABASE_ROLE
--------- -------------------- ----------------
PROD      MOUNTED              PHYSICAL STANDBY

SQL>

-- now start MRP in standby database
$ !sql
SQL> alter database recover managed standby database disconnect from session;

Database altered.

SQL>

-- Verify the MRP process

$ ps -ef|grep mrp
oracle   23455     1  0 12:58 ?        00:00:00 ora_mrp0_PROD
oracle   24125 21699  0 13:11 pts/1    00:00:00 grep mrp

$

Now archive-logs are applying.

Reference
Metalink Note 554358.1 - MRP Recovery Failed With ORA-19909