Verification of Compressed piece HARD block error in RMAN
Validating backups in RMAN
Troubleshoot ORA-19870
Problem Description :
When I tried in one of my standby database to restore database, I received the below error.
Error from RMAN log:
$ rman target /
RMAN> RUN
{
allocate channel ch1 type disk;
RESTORE DATABASE;
SWITCH DATAFILE ALL;
}2> 3> 4> 5> 6>
using target database control file instead of recovery catalog
allocated channel: ch1
channel ch1: SID=273 device type=DISK
Starting restore at 23-JAN-15
channel ch1: starting datafile backup set restore
channel ch1: specifying datafile(s) to restore from backup set
channel ch1: restoring datafile 00003 to /u01/ORADATA/PROD/undotbs01.dbf
channel ch1: restoring datafile 00004 to /u01/ORADATA/PROD/users01.dbf
...
...
channel ch1: reading from backup piece
/u02/bkp/prodbkp/backupset/backupset/2015_01_22/o1_mf_nnndf_TAG20150122T050216_bd0fr199_.bkp
channel ch1: ORA-19870: error while restoring backup piece
/u02/bkp/prodbkp/backupset/backupset/2015_01_22/o1_mf_nnndf_TAG20150122T050216_bd0fr199_.bkp
ORA-19599: block number 167207 is corrupt in backup piece
/u02/bkp/prodbkp/backupset/backupset/2015_01_22/o1_mf_nnndf_TAG20150122T050216_bd0fr199_.bkp
channel ch1: starting datafile backup set restore
channel ch1: specifying datafile(s) to restore from backup set
channel ch1: restoring datafile 00001 to /u01/ORADATA/PROD/system01.dbf
channel ch1: restoring datafile 00002 to /u01/ORADATA/PROD/sysaux01.dbf
...
...
channel ch1: reading from backup piece
/u02/bkp/prodbkp/backupset/backupset/2015_01_22/o1_mf_nnndf_TAG20150122T050216_bd0gpk9j_.bkp
channel ch1: ORA-19870: error while restoring backup piece
/u02/bkp/prodbkp/backupset/backupset/2015_01_22/o1_mf_nnndf_TAG20150122T050216_bd0gpk9j_.bkp
ORA-19599: block number 204403 is corrupt in backup piece
/u02/bkp/prodbkp/backupset/backupset/2015_01_22/o1_mf_nnndf_TAG20150122T050216_bd0gpk9j_.bkp
failover to previous backup
released channel: ch1
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03002: failure of restore command at 01/23/2015 14:30:27
RMAN-06026: some targets not found - aborting restore
RMAN-06023: no backup or copy of datafile 62 found to restore
RMAN-06023: no backup or copy of datafile 58 found to restore
RMAN-06023: no backup or copy of datafile 33 found to restore
RMAN-06023: no backup or copy of datafile 14 found to restore
RMAN-06023: no backup or copy of datafile 13 found to restore
RMAN>
Error From Alert log:
Fri Jan 23 14:22:55 2015
Full restore complete of datafile 35 /u01/ORADATA/PROD/HRM_INDEX01.dbf. Elapsed time: 0:04:44
checkpoint is 101068670522
last deallocation scn is 98368972366
Fri Jan 23 14:23:11 2015
Corrupt block 167207 found during reading backup piece,
file=/u02/bkp/prodbkp/backupset/backupset/2015_01_22/o1_mf_nnndf_TAG20150122T050216_bd0fr199_.bkp,
corr_type=3
Reread of blocknum=167207,
.....
.....
file=/u02/bkp/prodbkp/backupset/backupset/2015_01_22/o1_mf_nnndf_TAG20150122T050216_bd0fr199_.bkp,
found same corrupt data
Reread of blocknum=167207,
file=/u02/bkp/prodbkp/backupset/backupset/2015_01_22/o1_mf_nnndf_TAG20150122T050216_bd0fr199_.bkp,
found same corrupt data
Compressed piece HARD block error, bit 8 off in block number 00000001 for file 0
Fri Jan 23 14:25:12 2015
....
....
Investigation:
A) Primary database:
Step: 1 - Is there any block corruption in the database?
Action:
SQL> select * from v$database_block_corruption;
no rows selected
Step:2 - Is there any failure backups?
RMAN> list failure;
using target database control file instead of recovery catalog
no failures found that match specification
RMAN>
Step:3 - Was it created with the compression option set?
If No, no issues, If yes, then we will see the backup transfer process.
Step:4 - Are all required backup pices validated?
RMAN> list backuppiece
'/u02/bkp/prodbkp/backupset/backupset/2015_01_22/o1_mf_nnndf_TAG20150122T050216_bd0fr199_.bkp';
using target database control file instead of recovery catalog
List of Backup Pieces
BP Key BS Key Pc# Cp# Status Device Type Piece Name
------- ------- --- --- ----------- ----------- ----------
1227 1227 1 1 AVAILABLE DISK
/u02/bkp/prodbkp/backupset/backupset/2015_01_22/o1_mf_nnndf_TAG20150122T050216_bd0fr199_.bkp
RMAN>
-- Take the BP Key and run the below command.
RMAN> validate backupset 1308; -- Just verified with taking a BP key
Starting validate at 23-JAN-15
allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=807 device type=DISK
channel ORA_DISK_1: starting validation of archived log backup set
channel ORA_DISK_1: reading from backup piece
/u01/app/oracle/fast_recovery_area/PROD/backupset/2015_01_27/o1_mf_annnn_TAG20150127T053329_bdfog
l4f_.bkp
channel ORA_DISK_1: piece
handle=/u01/app/oracle/fast_recovery_area/PROD/backupset/2015_01_27/o1_mf_annnn_TAG20150127T05332
9_bdfogl4f_.bkp tag=TAG20150127T053329
channel ORA_DISK_1: restored backup piece 1
channel ORA_DISK_1: validation complete, elapsed time: 00:00:01
Finished validate at 23-JAN-15
RMAN>
-- It is successfully validated. i.e., there is no issue with backup piece with production.
Note :
You can use the VALIDATE option of the BACKUP command to verify that database files exist and are in the correct locations, and have no physical or logical corruptions that would prevent RMAN from creating backups of them. When performing a BACKUP... VALIDATE, RMAN reads the files to be backed up in their entirety, as it would during a real backup. It does not, however, actually produce any backup sets or image copies.
If the backup validation discovers corrupt blocks, then RMAN updates the $DATABASE_BLOCK_CORRUPTION view with rows describing the corruptions. You can repair corruptions using block media recovery, documented in Oracle Database Backup and Recovery Advanced User's Guide. After a corrupt block is repaired, the row identifying this block is deleted from the view.
For example, you can validate that all database files and archived logs can be backed up by running a command as follows:
RMAN> BACKUP VALIDATE DATABASE ARCHIVELOG ALL;
The RMAN client displays the same output that it would if it were really backing up the files. If RMAN cannot validate the backup of one or more of the files, then it issues an error message. For example, RMAN may show output similar to the following:
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03002: failure of backup command at 08/29/2002 14:33:47
ORA-19625: error identifying file /oracle/oradata/trgt/arch/archive1_6.dbf
ORA-27037: unable to obtain file status
SVR4 Error: 2: No such file or directory
Additional information: 3
Now verification in standby database:
1) How the backup pieces are transferred?
If SCP, then may be a block corruption in backup piece. Here the rman log and alert log is showing this type of error. So to irradicate, tranfer the backups and archivelogs using ftp binary mode.
2) Have you validated backups in standby after catalog all backup pieces?
If no. Do it. If yes, then see the status. If it is showing like below, then there is corrupt block in backup piece.
RMAN> list backuppiece '/u02/bkp/prodbkp/backupset/backupset/2015_01_22/o1_mf_nnndf_TAG20150122T050216_bd0fr199_.bkp';
using target database control file instead of recovery catalog
List of Backup Pieces
BP Key BS Key Pc# Cp# Status Device Type Piece Name
------- ------- --- --- ----------- ----------- ----------
1227 1227 1 1 AVAILABLE DISK /u02/bkp/prodbkp/backupset/backupset/2015_01_22/o1_mf_nnndf_TAG20150122T050216_bd0fr199_.bkp
RMAN> validate backupset 1227;
Starting validate at 23-JAN-15
allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=273 device type=DISK
channel ORA_DISK_1: starting validation of datafile backup set
channel ORA_DISK_1: reading from backup piece /u02/bkp/prodbkp/backupset/backupset/2015_01_22/o1_mf_nnndf_TAG20150122T050216_bd0fr199_.bkp
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03009: failure of validate command on ORA_DISK_1 channel at 01/23/2015 20:42:34
ORA-19870: error while restoring backup piece /u02/bkp/prodbkp/backupset/backupset/2015_01_22/o1_mf_nnndf_TAG20150122T050216_bd0fr199_.bkp
ORA-19599: block number 167207 is corrupt in backup piece /u02/bkp/prodbkp/backupset/backupset/2015_01_22/o1_mf_nnndf_TAG20150122T050216_bd0fr199_.bkp
RMAN>
-- It seems block corruption in backup piece.
Others:
Verification of FTP services:
-- Verify service status , All services
# service --status-all
-- Specific services
# service vsftpd restart
# service vsftpd status
# service vsftpd stop
Example:
# service vsftpd status
vsftpd is stopped
# service vsftpd start
Starting vsftpd for vsftpd: [ OK ]
# service vsftpd status
vsftpd (pid 32674) is running...
Validating backups in RMAN
Troubleshoot ORA-19870
Problem Description :
When I tried in one of my standby database to restore database, I received the below error.
Error from RMAN log:
$ rman target /
RMAN> RUN
{
allocate channel ch1 type disk;
RESTORE DATABASE;
SWITCH DATAFILE ALL;
}2> 3> 4> 5> 6>
using target database control file instead of recovery catalog
allocated channel: ch1
channel ch1: SID=273 device type=DISK
Starting restore at 23-JAN-15
channel ch1: starting datafile backup set restore
channel ch1: specifying datafile(s) to restore from backup set
channel ch1: restoring datafile 00003 to /u01/ORADATA/PROD/undotbs01.dbf
channel ch1: restoring datafile 00004 to /u01/ORADATA/PROD/users01.dbf
...
...
channel ch1: reading from backup piece
/u02/bkp/prodbkp/backupset/backupset/2015_01_22/o1_mf_nnndf_TAG20150122T050216_bd0fr199_.bkp
channel ch1: ORA-19870: error while restoring backup piece
/u02/bkp/prodbkp/backupset/backupset/2015_01_22/o1_mf_nnndf_TAG20150122T050216_bd0fr199_.bkp
ORA-19599: block number 167207 is corrupt in backup piece
/u02/bkp/prodbkp/backupset/backupset/2015_01_22/o1_mf_nnndf_TAG20150122T050216_bd0fr199_.bkp
channel ch1: starting datafile backup set restore
channel ch1: specifying datafile(s) to restore from backup set
channel ch1: restoring datafile 00001 to /u01/ORADATA/PROD/system01.dbf
channel ch1: restoring datafile 00002 to /u01/ORADATA/PROD/sysaux01.dbf
...
...
channel ch1: reading from backup piece
/u02/bkp/prodbkp/backupset/backupset/2015_01_22/o1_mf_nnndf_TAG20150122T050216_bd0gpk9j_.bkp
channel ch1: ORA-19870: error while restoring backup piece
/u02/bkp/prodbkp/backupset/backupset/2015_01_22/o1_mf_nnndf_TAG20150122T050216_bd0gpk9j_.bkp
ORA-19599: block number 204403 is corrupt in backup piece
/u02/bkp/prodbkp/backupset/backupset/2015_01_22/o1_mf_nnndf_TAG20150122T050216_bd0gpk9j_.bkp
failover to previous backup
released channel: ch1
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03002: failure of restore command at 01/23/2015 14:30:27
RMAN-06026: some targets not found - aborting restore
RMAN-06023: no backup or copy of datafile 62 found to restore
RMAN-06023: no backup or copy of datafile 58 found to restore
RMAN-06023: no backup or copy of datafile 33 found to restore
RMAN-06023: no backup or copy of datafile 14 found to restore
RMAN-06023: no backup or copy of datafile 13 found to restore
RMAN>
Error From Alert log:
Fri Jan 23 14:22:55 2015
Full restore complete of datafile 35 /u01/ORADATA/PROD/HRM_INDEX01.dbf. Elapsed time: 0:04:44
checkpoint is 101068670522
last deallocation scn is 98368972366
Fri Jan 23 14:23:11 2015
Corrupt block 167207 found during reading backup piece,
file=/u02/bkp/prodbkp/backupset/backupset/2015_01_22/o1_mf_nnndf_TAG20150122T050216_bd0fr199_.bkp,
corr_type=3
Reread of blocknum=167207,
.....
.....
file=/u02/bkp/prodbkp/backupset/backupset/2015_01_22/o1_mf_nnndf_TAG20150122T050216_bd0fr199_.bkp,
found same corrupt data
Reread of blocknum=167207,
file=/u02/bkp/prodbkp/backupset/backupset/2015_01_22/o1_mf_nnndf_TAG20150122T050216_bd0fr199_.bkp,
found same corrupt data
Compressed piece HARD block error, bit 8 off in block number 00000001 for file 0
Fri Jan 23 14:25:12 2015
....
....
Investigation:
A) Primary database:
Step: 1 - Is there any block corruption in the database?
Action:
SQL> select * from v$database_block_corruption;
no rows selected
Step:2 - Is there any failure backups?
RMAN> list failure;
using target database control file instead of recovery catalog
no failures found that match specification
RMAN>
Step:3 - Was it created with the compression option set?
If No, no issues, If yes, then we will see the backup transfer process.
Step:4 - Are all required backup pices validated?
RMAN> list backuppiece
'/u02/bkp/prodbkp/backupset/backupset/2015_01_22/o1_mf_nnndf_TAG20150122T050216_bd0fr199_.bkp';
using target database control file instead of recovery catalog
List of Backup Pieces
BP Key BS Key Pc# Cp# Status Device Type Piece Name
------- ------- --- --- ----------- ----------- ----------
1227 1227 1 1 AVAILABLE DISK
/u02/bkp/prodbkp/backupset/backupset/2015_01_22/o1_mf_nnndf_TAG20150122T050216_bd0fr199_.bkp
RMAN>
-- Take the BP Key and run the below command.
RMAN> validate backupset 1308; -- Just verified with taking a BP key
Starting validate at 23-JAN-15
allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=807 device type=DISK
channel ORA_DISK_1: starting validation of archived log backup set
channel ORA_DISK_1: reading from backup piece
/u01/app/oracle/fast_recovery_area/PROD/backupset/2015_01_27/o1_mf_annnn_TAG20150127T053329_bdfog
l4f_.bkp
channel ORA_DISK_1: piece
handle=/u01/app/oracle/fast_recovery_area/PROD/backupset/2015_01_27/o1_mf_annnn_TAG20150127T05332
9_bdfogl4f_.bkp tag=TAG20150127T053329
channel ORA_DISK_1: restored backup piece 1
channel ORA_DISK_1: validation complete, elapsed time: 00:00:01
Finished validate at 23-JAN-15
RMAN>
-- It is successfully validated. i.e., there is no issue with backup piece with production.
Note :
You can use the VALIDATE option of the BACKUP command to verify that database files exist and are in the correct locations, and have no physical or logical corruptions that would prevent RMAN from creating backups of them. When performing a BACKUP... VALIDATE, RMAN reads the files to be backed up in their entirety, as it would during a real backup. It does not, however, actually produce any backup sets or image copies.
If the backup validation discovers corrupt blocks, then RMAN updates the $DATABASE_BLOCK_CORRUPTION view with rows describing the corruptions. You can repair corruptions using block media recovery, documented in Oracle Database Backup and Recovery Advanced User's Guide. After a corrupt block is repaired, the row identifying this block is deleted from the view.
For example, you can validate that all database files and archived logs can be backed up by running a command as follows:
RMAN> BACKUP VALIDATE DATABASE ARCHIVELOG ALL;
The RMAN client displays the same output that it would if it were really backing up the files. If RMAN cannot validate the backup of one or more of the files, then it issues an error message. For example, RMAN may show output similar to the following:
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03002: failure of backup command at 08/29/2002 14:33:47
ORA-19625: error identifying file /oracle/oradata/trgt/arch/archive1_6.dbf
ORA-27037: unable to obtain file status
SVR4 Error: 2: No such file or directory
Additional information: 3
Now verification in standby database:
1) How the backup pieces are transferred?
If SCP, then may be a block corruption in backup piece. Here the rman log and alert log is showing this type of error. So to irradicate, tranfer the backups and archivelogs using ftp binary mode.
2) Have you validated backups in standby after catalog all backup pieces?
If no. Do it. If yes, then see the status. If it is showing like below, then there is corrupt block in backup piece.
RMAN> list backuppiece '/u02/bkp/prodbkp/backupset/backupset/2015_01_22/o1_mf_nnndf_TAG20150122T050216_bd0fr199_.bkp';
using target database control file instead of recovery catalog
List of Backup Pieces
BP Key BS Key Pc# Cp# Status Device Type Piece Name
------- ------- --- --- ----------- ----------- ----------
1227 1227 1 1 AVAILABLE DISK /u02/bkp/prodbkp/backupset/backupset/2015_01_22/o1_mf_nnndf_TAG20150122T050216_bd0fr199_.bkp
RMAN> validate backupset 1227;
Starting validate at 23-JAN-15
allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=273 device type=DISK
channel ORA_DISK_1: starting validation of datafile backup set
channel ORA_DISK_1: reading from backup piece /u02/bkp/prodbkp/backupset/backupset/2015_01_22/o1_mf_nnndf_TAG20150122T050216_bd0fr199_.bkp
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03009: failure of validate command on ORA_DISK_1 channel at 01/23/2015 20:42:34
ORA-19870: error while restoring backup piece /u02/bkp/prodbkp/backupset/backupset/2015_01_22/o1_mf_nnndf_TAG20150122T050216_bd0fr199_.bkp
ORA-19599: block number 167207 is corrupt in backup piece /u02/bkp/prodbkp/backupset/backupset/2015_01_22/o1_mf_nnndf_TAG20150122T050216_bd0fr199_.bkp
RMAN>
-- It seems block corruption in backup piece.
Others:
Verification of FTP services:
-- Verify service status , All services
# service --status-all
-- Specific services
# service vsftpd restart
# service vsftpd status
# service vsftpd stop
Example:
# service vsftpd status
vsftpd is stopped
# service vsftpd start
Starting vsftpd for vsftpd: [ OK ]
# service vsftpd status
vsftpd (pid 32674) is running...
No comments:
Post a Comment