ORA-19909: datafile 1 belongs to an orphan incarnation

When I tried to Flashback on in standby database then suddenly below message found in alert log and standby database not able to apply archive-logs also. So, here are messages from standby database alert log:

-- Message from alert log ( in standby database)

Mon Dec 12 12:43:24 2016
alter database recover managed standby database disconnect from session
Attempt to start background Managed Standby Recovery process (PROD)
Mon Dec 12 12:43:24 2016
MRP0 started with pid=26, OS id=22750
MRP0: Background Managed Standby Recovery process started (PROD)
 started logmerger process
Mon Dec 12 12:43:29 2016
Managed Standby Recovery not using Real Time Apply
Warning: Recovery target destination is in a sibling branch
of the controlfile checkpoint. Recovery will only recover
changes to datafiles.
Datafile 1 (ckpscn 5983588137777) is orphaned on incarnation#=2
MRP0: Detected orphaned datafiles!
Recovery will possibly be retried after flashback...
Errors in file /oracle/diag/rdbms/PROD/PROD/trace/PROD_pr00_22755.trc:
ORA-19909: datafile 1 belongs to an orphan incarnation
ORA-01110: data file 1: '/u03/oradata/PROD/PROD/system01.dbf'
Recovery Slave PR00 previously exited with exception 19909
Completed: alter database recover managed standby database disconnect from session
Mon Dec 12 12:43:51 2016
MRP0: Background Media Recovery process shutdown (PROD)

RCA:

Primary and standby have different incarnations:

-- in standby db

RMAN> list incarnation of database;

using target database control file instead of recovery catalog

List of Database Incarnations
DB Key  Inc Key DB Name  DB ID            STATUS  Reset SCN  Reset Time
------- ------- -------- ---------------- --- ---------- ----------
1       1       PROD  3680942556       PARENT  5982315579513 30-NOV-2016 00:31:04
2       2       PROD  3680942556       PARENT  5982448719535 30-NOV-2016 22:31:15
4       4       PROD  3680942556       ORPHAN  5983354187673 09-DEC-2016 04:01:27
3       3       PROD  3680942556       CURRENT 5983354274164 10-DEC-2016 04:02:53

RMAN>


-- in primary db

RMAN> list incarnation of database;

using target database control file instead of recovery catalog

List of Database Incarnations
DB Key  Inc Key DB Name  DB ID            STATUS  Reset SCN  Reset Time
------- ------- -------- ---------------- --- ---------- ----------
1       1       PROD  3680942556       PARENT  5982315579513 30-NOV-2016 00:31:04
2       2       PROD  3680942556       CURRENT 5982448719535 30-NOV-2016 22:31:15

RMAN>


-- Solution:

You need to reset the standby database's incarnation to match the primary's:

-- in standby

RMAN> list incarnation of database;

using target database control file instead of recovery catalog

List of Database Incarnations
DB Key  Inc Key DB Name  DB ID            STATUS  Reset SCN  Reset Time
------- ------- -------- ---------------- --- ---------- ----------
1       1       PROD  3680942556       PARENT  5982315579513 30-NOV-2016 00:31:04
2       2       PROD  3680942556       PARENT  5982448719535 30-NOV-2016 22:31:15
4       4       PROD  3680942556       ORPHAN  5983354187673 09-DEC-2016 04:01:27
3       3       PROD  3680942556       CURRENT 5983354274164 10-DEC-2016 04:02:53

RMAN> reset database  to incarnation 2;

database reset to incarnation 2

RMAN> list incarnation of database;


List of Database Incarnations
DB Key  Inc Key DB Name  DB ID            STATUS  Reset SCN  Reset Time
------- ------- -------- ---------------- --- ---------- ----------
1       1       PROD  3680942556       PARENT  5982315579513 30-NOV-2016 00:31:04
2       2       PROD  3680942556       CURRENT 5982448719535 30-NOV-2016 22:31:15
4       4       PROD  3680942556       ORPHAN  5983354187673 09-DEC-2016 04:01:27
3       3       PROD  3680942556       ORPHAN  5983354274164 10-DEC-2016 04:02:53

RMAN>

-- Verify MRP is active or not

$ ps -ef|grep mrp
oracle   23358 21699  0 12:56 pts/1    00:00:00 grep mrp
$

-- Verify standby database status and role

SQL> select name,open_mode ,database_role from v$database;

NAME      OPEN_MODE            DATABASE_ROLE
--------- -------------------- ----------------
PROD      MOUNTED              PHYSICAL STANDBY

SQL>

-- now start MRP in standby database
$ !sql
SQL> alter database recover managed standby database disconnect from session;

Database altered.

SQL>

-- Verify the MRP process

$ ps -ef|grep mrp
oracle   23455     1  0 12:58 ?        00:00:00 ora_mrp0_PROD
oracle   24125 21699  0 13:11 pts/1    00:00:00 grep mrp

$

Now archive-logs are applying.

Reference
Metalink Note 554358.1 - MRP Recovery Failed With ORA-19909

Resolve Gaps in Standby with incremental backup

Sometimes due to some activity or archive log header corruption, standby database will be in out of synch and gap will come. This can be verified from queries or alert logs of standby database.

Note: Primary instance is in RAC ( 2-node) and standby is in Stand-alone server.

Real-time steps:

Example: ( from alert log)

…………………………

…………………………

Mon Oct 10 02:20:17 2016

Media Recovery Log /u02/prod_archivelogs/2016_10_10/thread_1_seq_142321.362.924828783

Media Recovery Waiting for thread 2 sequence 51848

Fetching gap sequence in thread 2, gap sequence 51848-51849

FAL[client]: Error fetching gap sequence, no FAL server specified

…………………………

…………………………

Here one archive log is missing, i.e., thread_2_seq_51848.xxx

So, Now you verify whether you have that archive log in production or not. If available then copy and catlog/ register it to recover again. Otherwise you can take an incremental backup and recover it. Here are the following steps:

Steps:

1.   At standby check the current scn.

sql> select to_char(current_scn) from v$database;

   5967451101100

2. Create a standby control file from Primary Database.

sql> ALTER DATABASE CREATE STANDBY CONTROLFILE AS '/u02/incremental_bkp/standby_control.ctl';

3. Copy the standby controlfile to standby server.

$ scp -r -p '/u02/incremental_bkp/standby_control.ctl' oracle@160.192.1.2:/u03/incremental_bkp

4. Take incremental backup on primary starting from scn# of standby database

rman> Backup incremental from SCN 5969890030000 database tag='FOR_STANDBY' format '/u02/incremental_bkp/%d_%t_%s_%p';

Note: Take scn some less value than standby.

5. Move the backup pieces to the standby host

6. Shut down the standby database and replace the current controlfile with new one.

            cp control01.ctl control01.ctl_bkp

            cp control02.ctl control01.ctl_bkp

            cp standby_control.ctl control01.ctl

            cp standby_control.ctl control02.ctl

7. Now mount the standby database.

            sql> startup nomount;

            sql> alter database mount standby database;

8. If Primary and standby datafiles location are different then go with below steps.

            rman> catalog start with '/u02/flash_recovery_area/PROD/ORADATA';

            rman> switch database to copy;

            Some time we can get below error while above command try to execute.

Err:

RMAN> switch database to copy;

using target database control file instead of recovery catalog

RMAN-00571: ===========================================================

RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============

RMAN-00571: ===========================================================

RMAN-03002: failure of switch to copy command at 05/30/2016 23:35:44

RMAN-06571: datafile 1 does not have recoverable copy

It happened if standby database is in different incarnation with primary.

In Standby:

RMAN> list incarnation;

List of Database Incarnations

DB Key  Inc Key DB Name  DB ID            STATUS  Reset SCN  Reset Time

------- ------- -------- ---------------- --- ---------- ----------

1       1       PROD  2886743190       PARENT  1          25-MAR-16

2       2       PROD  2886743190       PARENT  5965541899890 27-MAR-16

4       4       PROD  2886743190       ORPHAN  5969888275728 30-MAY-16

3       3       PROD  2886743190       CURRENT 5969888309840 30-MAY-16

In primary:

RMAN> list incarnation;

List of Database Incarnations

DB Key  Inc Key DB Name  DB ID            STATUS  Reset SCN  Reset Time

------- ------- -------- ---------------- --- ---------- ----------

1       1       PROD  2886743190       PARENT  1          25-MAR-16

2       2       PROD  2886743190       CURRENT  5965541899890 27-MAR-16

NOW change the standby database incarnation which is there in primary.

In standby:

            RMAN> reset database to incarnation 2;

                        database reset to incarnation 2

            rman> switch database to copy;

            Now shutdown and mount the database again.

9.  Recover the standby database with incremental backup pieces.

            rman> crosscheck backup;

            rman> delete expired backup;

            rman> catalog start with ' /u03/incremental_bkp';

            rman> recover database noredo;

10. Now verify change in scn status.

            SQL> select to_char(current_scn) from v$database;

             CURRENT_SCN

                        ---------------------------

                        5970008638143

11. On the standby database, start the Managed Recovery Process.

            sql>alter database recover managed standby database disconnect from session;

Now recovery is started. MRP process also started. You can catalog the archivelogs. It will apply archivelogs automatically. 

EXPDP backup failed with error ORA-04031

In one of my production environment I found my EXPDP backup failed with below error. It is not always. Same I re-created in one of my test instance. Here are the reported errors.

Error while running EXPDP backup: 
============================== 
UDE-31623: operation generated ORACLE error 31623
ORA-31623: a job is not attached to this session via the specified handle
ORA-06512: at "SYS.DBMS_DATAPUMP", line 3326
ORA-06512: at "SYS.DBMS_DATAPUMP", line 4551
ORA-06512: at line 1


Error in alert log at that time: 
======================== 
Sun Oct 23 00:02:48 2016
Errors in file /u01/app/oracle/diag/rdbms/prod/PROD/trace/PROD_ora_787060.trc (incident=132844):
ORA-04031: unable to allocate 376 bytes of shared memory ("streams pool","unknown object","streams pool","kwqbsinfy:cco")
Incident details in: /u01/app/oracle/diag/rdbms/prod/PROD/incident/incdir_132844/PROD_ora_787060_i132844.trc
Sun Oct 23 00:02:53 2016
Dumping diagnostic data in directory=[cdmp_20161023000253], requested by (instance=1, osid=787060), summary=[incident=132844].


Solution / Fix :
============

By setting minimum value to streams_pool_size of minimum 200M we can avoid the error.

OR

In Oracle 11g, it is highly recommended to use AMM i.e memory_target where SGA and PGA are automatically managed and set minimum values to SGA_TARGET, PGA and other pools.

This configuration will give better performance and reduces fragmentation and avoids ORA-04031 errors.

AND 

set the parameter _shared_pool_reserved_pct instead of the parameter shared_pool_reserved_size. 

Set the shared pool reserved percent with: 

alter system set "_shared_pool_reserved_pct"=10 scope=spfile; 

NOTE: Restart the database and listener once the changes are done. 

The above settings will not only fix ORA-04031 but also helps for optimal memory utilization. 

Thanks .

Error: Failed to open the wallet - Fix

During cloning activity in one of our EBS R12, the whole cloning process completed successfully without reporting any issues. But when we are trying to start application services all “opmn” services are started except HTTP_Server.

e.g.,

[applprod@app scripts]$ ./adopmnctl.sh status

You are running adopmnctl.sh version 120.6.12010000.5

Checking status of OPMN managed processes...

Processes in Instance: PRODHCM_app.app.testdomain.com
---------------------------------+--------------------+---------+---------
ias-component                    | process-type       |     pid | status
---------------------------------+--------------------+---------+---------
OC4JGroup:default_group          | OC4J:oafm          |    2418 | Alive  
OC4JGroup:default_group          | OC4J:forms         |    2219 | Alive  
OC4JGroup:default_group          | OC4J:forms         |    2218 | Alive  
OC4JGroup:default_group          | OC4J:forms         |    2216 | Alive  
OC4JGroup:default_group          | OC4J:forms         |    2215 | Alive  
OC4JGroup:default_group          | OC4J:oacore        |    1889 | Alive  
OC4JGroup:default_group          | OC4J:oacore        |    1888 | Alive  
OC4JGroup:default_group          | OC4J:oacore        |    1887 | Alive  
OC4JGroup:default_group          | OC4J:oacore        |    1886 | Alive  
HTTP_Server                      | HTTP_Server        |     N/A | Down  


adopmnctl.sh: exiting with status 0

adopmnctl.sh: check the logfile /u01/applprod/inst/apps/PRODHCM_app/logs/appl/admin/log/adopmnctl.txt for more information ...

[applprod@app scripts]$

In the opmn logfile below error messages are reported:

$ cat /u01/applprod/inst/apps/PRODHCM_app/logs/ora/10.1.3/opmn/HTTP_Server~1.log

--------
16/10/17 10:34:10 Start process
--------
/u01/applprod/inst/apps/PRODHCM_app/ora/10.1.3/Apache/Apache/bin/apachectl startssl: execing httpd
Error: Failed to open the wallet [Hint: incorrect password, bad wallet file, ...] (Server app.apollohospitals.com:8030, wallet file:/u01/applprod/inst/apps/PRODHCM_app/certs/Apache)

Then I tried to start the "apache"server only but issue not resolved. Restart of application serer also not resolved the issue. From the above error it seems there is a issue with Apache server certificate. So, Finally below workaround worked fine:

So, connected to App server and stopped the all application services.

$ cd $ADMIN_SCRIPT_HOME
$ ./adstpall.sh

Then, go to Apache sertificate directory.

$ cd /u01/applprod/inst/apps/PRODHCM_app/certs/Apache

Just rename exiting file in the folder:

$ mv cwallet.sso cwallet.sso_BAK

And, copy the same file from source ( prod) to this cloned environment and start the application. It worked for me.

Restore Pluggable database using RMAN in Oracle 12c

Here you can see how to take a pluggable database backup with compress mode and some restore and recover scenarios.

-- To Take pluggable database backup:

$ rman target /

RMAN>

RMAN> backup as compressed backupset pluggable database salespdb format '/u10/EXAM/fast_recovery_area/salespdb_full_12c_db_%U';

Starting backup at 08-OCT-2016 17:10:27

using target database control file instead of recovery catalog

allocated channel: ORA_DISK_1

channel ORA_DISK_1: SID=212 device type=DISK

channel ORA_DISK_1: starting compressed full datafile backup set

channel ORA_DISK_1: specifying datafile(s) in backup set

input datafile file number=00012 name=/u10/EXAM/ORADATA/CDB/salespdb/sysaux01.dbf

input datafile file number=00011 name=/u10/EXAM/ORADATA/CDB/salespdb/system01.dbf

input datafile file number=00013 name=/u10/EXAM/ORADATA/CDB/salespdb/sales01.dbf

channel ORA_DISK_1: starting piece 1 at 08-OCT-2016 17:10:29

channel ORA_DISK_1: finished piece 1 at 08-OCT-2016 17:11:44

piece handle=/u10/EXAM/fast_recovery_area/salespdb_full_12c_db_0krhs1k5_1_1 tag=TAG20161008T171028 comment=NONE

channel ORA_DISK_1: backup set complete, elapsed time: 00:01:15

Finished backup at 08-OCT-2016 17:11:44

Starting Control File and SPFILE Autobackup at 08-OCT-2016 17:11:44

piece handle=/u10/EXAM/fast_recovery_area/CDB/autobackup/2016_10_08/o1_mf_s_924714704_czkpvw5x_.bkp comment=NONE

Finished Control File and SPFILE Autobackup at 08-OCT-2016 17:11:51

RMAN>

Verify schemas:

RMAN> report schema;

using target database control file instead of recovery catalog

Report of database schema for database with db_unique_name CDB

List of Permanent Datafiles

===========================

File Size(MB) Tablespace           RB segs Datafile Name

---- -------- -------------------- ------- ------------------------

1    800      SYSTEM               YES     /u10/EXAM/ORADATA/CDB/system01.dbf

3    830      SYSAUX               NO      /u10/EXAM/ORADATA/CDB/sysaux01.dbf

4    225      UNDOTBS1             YES     /u10/EXAM/ORADATA/CDB/undotbs01.dbf

5    250      PDB$SEED:SYSTEM      NO      /u10/EXAM/ORADATA/CDB/pdbseed/system01.dbf

6    5        USERS                NO      /u10/EXAM/ORADATA/CDB/users01.dbf

7    570      PDB$SEED:SYSAUX      NO      /u10/EXAM/ORADATA/CDB/pdbseed/sysaux01.dbf

8    260      PDB1:SYSTEM          NO      /u10/EXAM/ORADATA/CDB/pdb1/system01.dbf

9    590      PDB1:SYSAUX          NO      /u10/EXAM/ORADATA/CDB/pdb1/sysaux01.dbf

10   5        PDB1:USERS           NO      /u10/EXAM/ORADATA/CDB/pdb1/pdb1_users01.dbf

11   260      SALESPDB:SYSTEM      NO      /u10/EXAM/ORADATA/CDB/salespdb/system01.dbf

12   580      SALESPDB:SYSAUX      NO      /u10/EXAM/ORADATA/CDB/salespdb/sysaux01.dbf

13   250      SALESPDB:SALES       NO      /u10/EXAM/ORADATA/CDB/salespdb/sales01.dbf

14   260      PDB2:SYSTEM          NO      /u10/EXAM/ORADATA/CDB/pdb2/system01.dbf

15   580      PDB2:SYSAUX          NO      /u10/EXAM/ORADATA/CDB/pdb2/sysaux01.dbf

16   5        PDB2:USERS           NO      /u10/EXAM/ORADATA/CDB/pdb2/pdb1_users01.dbf

17   260      HRPDB:SYSTEM         NO      /u10/EXAM/ORADATA/CDB/hrpdb/system01.dbf

18   580      HRPDB:SYSAUX         NO      /u10/EXAM/ORADATA/CDB/hrpdb/sysaux01.dbf

19   250      HRPDB:SALES          NO      /u10/EXAM/ORADATA/CDB/hrpdb/hr01.dbf

List of Temporary Files

=======================

File Size(MB) Tablespace           Maxsize(MB) Tempfile Name

---- -------- -------------------- ----------- --------------------

1    197      TEMP                 32767       /u10/EXAM/ORADATA/CDB/temp01.dbf

2    100      PDB$SEED:TEMP        32767       /u10/EXAM/ORADATA/CDB/pdbseed/pdbseed_temp012016-10-04_03-18-32-PM.dbf

3    20       PDB1:TEMP            32767       /u10/EXAM/ORADATA/CDB/pdb1/temp012016-10-04_03-18-32-PM.dbf

4    20       SALESPDB:TEMP        32767       /u10/EXAM/ORADATA/CDB/salespdb/pdbseed_temp012016-10-04_03-18-32-PM.dbf

5    20       PDB2:TEMP            32767       /u10/EXAM/ORADATA/CDB/pdb2/temp012016-10-04_03-18-32-PM.dbf

6    20       HRPDB:TEMP           32767       /u10/EXAM/ORADATA/CDB/hrpdb/pdbseed_temp012016-10-04_03-18-32-PM.dbf

RMAN>

Note: You can take multiple pluggable databases using below format script also:

rman>

run { 
    setlimit channel ch1 maxopenfiles 8; 
    backup 
    incremental level = 0
    filesperset = 32
    pluggable database pdb1, ..pdbn;
} 
exit;

rman>

Now do following activity for test:

SQL> alter pluggable database SALESPDB close immediate;

Pluggable database altered.

SQL>

$ cd /u10/EXAM/ORADATA/CDB/salespdb

]$ pwd

/u10/EXAM/ORADATA/CDB/salespdb

$ mv sales01.dbf sales01.dbf_bkp

SQL> alter pluggable database salespdb open;

alter pluggable database salespdb open

*

ERROR at line 1:

ORA-01157: cannot identify/lock data file 13 - see DBWR trace file

ORA-01110: data file 13: '/u10/EXAM/ORADATA/CDB/salespdb/sales01.dbf'

SQL>

Here I am trying to open my pdb. It is clearly showing “sales01.dbf” datafile is missing. So you  need to restore and recover from last valid backup.

Now start restore and Recover:

RMAN> restore pluggable database salespdb;

Starting restore at 10-OCT-2016 15:25:51

using channel ORA_DISK_1

channel ORA_DISK_1: starting datafile backup set restore

channel ORA_DISK_1: specifying datafile(s) to restore from backup set

channel ORA_DISK_1: restoring datafile 00011 to /u10/EXAM/ORADATA/CDB/salespdb/system01.dbf

channel ORA_DISK_1: restoring datafile 00012 to /u10/EXAM/ORADATA/CDB/salespdb/sysaux01.dbf

channel ORA_DISK_1: restoring datafile 00013 to /u10/EXAM/ORADATA/CDB/salespdb/sales01.dbf

channel ORA_DISK_1: reading from backup piece /u10/EXAM/fast_recovery_area/salespdb_full_12c_db_0krhs1k5_1_1

channel ORA_DISK_1: piece handle=/u10/EXAM/fast_recovery_area/salespdb_full_12c_db_0krhs1k5_1_1 tag=TAG20161008T171028

channel ORA_DISK_1: restored backup piece 1

channel ORA_DISK_1: restore complete, elapsed time: 00:00:55

Finished restore at 10-OCT-2016 15:26:46

Now Recover the pluggable database:

RMAN> recover pluggable database salespdb;

Starting recover at 10-OCT-2016 15:26:58

using channel ORA_DISK_1

starting media recovery

archived log for thread 1 with sequence 54 is already on disk as file /u10/EXAM/fast_recovery_area/CDB/archivelog/2016_10_08/o1_mf_1_54_czkvhl14_.arc

archived log for thread 1 with sequence 55 is already on disk as file /u10/EXAM/fast_recovery_area/CDB/archivelog/2016_10_08/o1_mf_1_55_czl60bfv_.arc

archived log for thread 1 with sequence 56 is already on disk as file /u10/EXAM/fast_recovery_area/CDB/archivelog/2016_10_08/o1_mf_1_56_czlf1tko_.arc

archived log for thread 1 with sequence 57 is already on disk as file /u10/EXAM/fast_recovery_area/CDB/archivelog/2016_10_09/o1_mf_1_57_czm3ywck_.arc

archived log for thread 1 with sequence 58 is already on disk as file /u10/EXAM/fast_recovery_area/CDB/archivelog/2016_10_09/o1_mf_1_58_czm95ftg_.arc

archived log for thread 1 with sequence 59 is already on disk as file /u10/EXAM/fast_recovery_area/CDB/archivelog/2016_10_09/o1_mf_1_59_czmlktv5_.arc

archived log for thread 1 with sequence 60 is already on disk as file /u10/EXAM/fast_recovery_area/CDB/archivelog/2016_10_09/o1_mf_1_60_czn1s62o_.arc

archived log for thread 1 with sequence 61 is already on disk as file /u10/EXAM/fast_recovery_area/CDB/archivelog/2016_10_09/o1_mf_1_61_cznh65n2_.arc

archived log for thread 1 with sequence 62 is already on disk as file /u10/EXAM/fast_recovery_area/CDB/archivelog/2016_10_09/o1_mf_1_62_cznpw5wk_.arc

archived log for thread 1 with sequence 63 is already on disk as file /u10/EXAM/fast_recovery_area/CDB/archivelog/2016_10_09/o1_mf_1_63_czny6jb4_.arc

archived log for thread 1 with sequence 64 is already on disk as file /u10/EXAM/fast_recovery_area/CDB/archivelog/2016_10_10/o1_mf_1_64_czom11oo_.arc

archived log for thread 1 with sequence 65 is already on disk as file /u10/EXAM/fast_recovery_area/CDB/archivelog/2016_10_10/o1_mf_1_65_czpkzhmk_.arc

archived log file name=/u10/EXAM/fast_recovery_area/CDB/archivelog/2016_10_08/o1_mf_1_54_czkvhl14_.arc thread=1 sequence=54

archived log file name=/u10/EXAM/fast_recovery_area/CDB/archivelog/2016_10_08/o1_mf_1_55_czl60bfv_.arc thread=1 sequence=55

archived log file name=/u10/EXAM/fast_recovery_area/CDB/archivelog/2016_10_08/o1_mf_1_56_czlf1tko_.arc thread=1 sequence=56

archived log file name=/u10/EXAM/fast_recovery_area/CDB/archivelog/2016_10_09/o1_mf_1_57_czm3ywck_.arc thread=1 sequence=57

archived log file name=/u10/EXAM/fast_recovery_area/CDB/archivelog/2016_10_09/o1_mf_1_58_czm95ftg_.arc thread=1 sequence=58

archived log file name=/u10/EXAM/fast_recovery_area/CDB/archivelog/2016_10_09/o1_mf_1_59_czmlktv5_.arc thread=1 sequence=59

archived log file name=/u10/EXAM/fast_recovery_area/CDB/archivelog/2016_10_09/o1_mf_1_60_czn1s62o_.arc thread=1 sequence=60

archived log file name=/u10/EXAM/fast_recovery_area/CDB/archivelog/2016_10_09/o1_mf_1_61_cznh65n2_.arc thread=1 sequence=61

archived log file name=/u10/EXAM/fast_recovery_area/CDB/archivelog/2016_10_09/o1_mf_1_62_cznpw5wk_.arc thread=1 sequence=62

archived log file name=/u10/EXAM/fast_recovery_area/CDB/archivelog/2016_10_09/o1_mf_1_63_czny6jb4_.arc thread=1 sequence=63

media recovery complete, elapsed time: 00:00:07

Finished recover at 10-OCT-2016 15:27:06

RMAN> quit

Recovery Manager complete.

Now verify your pluggbale database datafiles. Sure it is recovered.

[oracle@eh-uatdb1 salespdb]$ ls

pdbseed_temp012016-10-04_03-18-32-PM.dbf  sales01.dbf  sales01.dbf_bkp  sysaux01.dbf  system01.dbf

$ ls sales01.dbf

sales01.dbf

$

Now You can open the "salespdb" pluggable database.

SQL> alter pluggable database salespdb open;

Pluggable database altered.

SQL> 
SQL> show pdbs;

    CON_ID CON_NAME                       OPEN MODE  RESTRICTED
---------- ------------------------------ ---------- ----------
         2 PDB$SEED                       READ ONLY  NO
         3 PDB1                           READ WRITE NO
         4 SALESPDB                       READ WRITE NO
         5 PDB2                           READ WRITE NO
         6 HRPDB                          READ WRITE NO

SQL> 

Fix: ORA-65086: cannot open/close the pluggable database

During cloning activity I unplugged a pdb and then try to open it but I faced "ORA-65086: cannot open/close the pluggable database  " error. I view Oracle document and found below concept to do the following steps:

-- Unplug a pluggable database

SQL> ALTER PLUGGABLE DATABASE hrpdb close immediate;

Pluggable database altered.

SQL> ALTER PLUGGABLE DATABASE hrpdb UNPLUG INTO '/u10/EXAM/bkp_pdb/hrpdb.xml';

Pluggable database altered.

Now you can see the status of pdbs.

SQL> show pdbs;

    CON_ID CON_NAME                       OPEN MODE  RESTRICTED

---------- ------------------------------ ---------- ----------

         2 PDB$SEED                       READ ONLY  NO

         3 PDB1                           READ WRITE NO

         4 SALESPDB                       READ WRITE NO

         5 PDB2                           READ WRITE NO

         6 HRPDB                          MOUNTED

SQL>

Here, I am trying to open the same database. But it is not opening due to unplugged.

SQL> alter pluggable database hrpdb open;

alter pluggable database hrpdb open

*

ERROR at line 1:

ORA-65086: cannot open/close the pluggable database

SQL>

It is not possible to open again the pluggable database… The official documentation claims:

After a PDB is unplugged, it remains in the CDB with an open mode of MOUNTED and a status of UNPLUGGED. The only operation you can perform on an unplugged PDB is DROP PLUGGABLE DATABASE, which will remove it from the CDB. You must drop the PDB before you can plug it into the same CDB or another CDB.

So, following official documentation recommendations:

SQL> drop pluggable database hrpdb;

Pluggable database dropped.

SQL> CREATE pluggable DATABASE hrpdb USING '/u10/EXAM/bkp_pdb/hrpdb.xml' NOCOPY;

Pluggable database created.

SQL> show pdbs;

    CON_ID CON_NAME                       OPEN MODE  RESTRICTED

---------- ------------------------------ ---------- ----------

         2 PDB$SEED                       READ ONLY  NO

         3 PDB1                           READ WRITE NO

         4 SALESPDB                       READ WRITE NO

         5 PDB2                           READ WRITE NO

         6 HRPDB                          MOUNTED

SQL> alter pluggable database hrpdb open;

Pluggable database altered.

SQL> show pdbs;

    CON_ID CON_NAME                       OPEN MODE  RESTRICTED

---------- ------------------------------ ---------- ----------

         2 PDB$SEED                       READ ONLY  NO

         3 PDB1                           READ WRITE NO

         4 SALESPDB                       READ WRITE NO

         5 PDB2                           READ WRITE NO

         6 HRPDB                          READ WRITE NO

SQL> 

Hope, It may helped you.