PRCR-1013 : Failed to start resource - while start service in RAC database

Suddenly I found one of the service which is created for load-balancing TAF purpose, is down. When I tried to start it, then below error came.

$ srvctl start service -d PROD -s HRPROD -n rac2 -o open
PRCR-1013 : Failed to start resource ora.PROD.HRPROD.svc
PRCR-1064 : Failed to start resource ora.PROD.HRPROD.svc on node rac2
CRS-5017: The resource action "ora.PROD.HRPROD.svc start" encountered the following error:
ORA-06550: line 1, column 34:
PLS-00553: character set name is not recognized
ORA-06550: line 0, column 0:
PL/SQL: Compilation unit analysis terminated
. For details refer to "(:CLSN00107:)" in "/u01/app/grid4/log/rac2/agent/crsd/oraagent_oracle/oraagent_oracle.log".

CRS-2674: Start of 'ora.PROD.HRPROD.svc' on 'rac2' failed
CRS-5017: The resource action "ora.PROD.HRPROD.svc clean" encountered the following error:
ORA-06550: line 1, column 33:
PLS-00553: character set name is not recognized
ORA-06550: line 0, column 0:
PL/SQL: Compilation unit analysis terminated
. For details refer to "(:CLSN00106:)" in "/u01/app/grid4/log/rac2/agent/crsd/oraagent_oracle/oraagent_oracle.log".

Note: I has a service with name "HRPROD" in my PROD database.

Now read the error written in oraagent_oracle.log file. Best way is raise SR and follow it immedialtly.

I did below activities and resolved the issues:

Before any activity, output of "crsctl stat res -t":

--------------------------------------------------------------------------------
NAME           TARGET  STATE        SERVER              
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.DATA.dg
               ONLINE  ONLINE       rac1                                       
               ONLINE  ONLINE       rac2                                       
ora.FRA.dg
               ONLINE  ONLINE       rac1                                       
               ONLINE  ONLINE       rac2                                       
ora.LISTENER.lsnr
               ONLINE  ONLINE       rac1                                       
               ONLINE  ONLINE       rac2                                       
ora.asm
               ONLINE  ONLINE       rac1                    Started            
               ONLINE  ONLINE       rac2                    Started            
ora.gsd
               OFFLINE OFFLINE      rac1                                       
               OFFLINE OFFLINE      rac2                                       
ora.net1.network
               ONLINE  ONLINE       rac1                                       
               ONLINE  ONLINE       rac2                                       
ora.ons
               ONLINE  ONLINE       rac1                                       
               ONLINE  ONLINE       rac2                                       
ora.registry.acfs
               ONLINE  ONLINE       rac1                                       
               ONLINE  ONLINE       rac2                                       
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
      1        ONLINE  ONLINE       rac2                                       
ora.LISTENER_SCAN2.lsnr
      1        ONLINE  ONLINE       rac1                                       
ora.LISTENER_SCAN3.lsnr
      1        ONLINE  ONLINE       rac1                                       
ora.cvu
      1        ONLINE  ONLINE       rac1                                       
ora.rac1.vip
      1        ONLINE  ONLINE       rac1                                       
ora.rac2.vip
      1        ONLINE  ONLINE       rac2                                       
ora.PROD.db
      1        ONLINE  ONLINE       rac1                    Open               
      2        ONLINE  ONLINE       rac2                    Open               
ora.PROD.HRPROD.svc
      1        ONLINE  ONLINE       rac1                                       
      2        ONLINE  OFFLINE                                                  
ora.oc4j
      1        ONLINE  ONLINE       rac1                                       
ora.scan1.vip
      1        ONLINE  ONLINE       rac2                                       
ora.scan2.vip
      1        ONLINE  ONLINE       rac1                                       
ora.scan3.vip
      1        ONLINE  ONLINE       rac1  

Here service is not running in node2. In very first step stop entire service in all nodes and start it again. If same issue is coming you can follow below practice.

1. Ensure that the database is properly started up, that is, mounted and opened.

2. Go to the oraagent_oracle.pid and very as like below and kill the process

$cd $GRID_HOME/log/nodename/agent/crsd/oraagent_oracle
$cat oraagent_oracle.pid 

You should see a pid number in the file.

$ ps -ef |grep <pid> -----> verify if this pid is the oraagent.bin process.
if it is the oraagent.bin process kill it.
$ kill -9 <pid>

3. If not ps -ef|grep d.bin. You should find two oraagent.bin processes kill these.

Killing the process should not have any effect and they should respawn again automatically.
Confirm that the processes have started.

4. Then restart the service using srvctl.

$ srvctl start service -d <database_name> -s <service_name> -i <instance_name>

Re-created same issue and following activities done in one of my test env.:

$ cd /u01/app/grid4/log/rac2/agent/crsd/oraagent_oracle
$ cat oraagent_oracle.pid
7798818

$ps -ef|grep 7798818

$ ps -ef|grep 7798818
  oracle  7798818        1   2   Jul 22      - 43:22 /u01/app/grid4/bin/oraagent.bin
  oracle 16974300 36307328   0 17:35:08  pts/4  0:00 grep 7798818

$ kill -9 7798818

$  ps -ef|grep d.bin
  oracle  5767238        1   0   Jul 22      -  7:16 /u01/app/grid4/bin/evmd.bin
  oracle  5832844        1   0   Jul 22      -  1:45 /u01/app/grid4/bin/gpnpd.bin
    root  5963902        1   0   Jul 22      - 15:07 /u01/app/grid4/bin/ohasd.bin reboot
  oracle  8192028        1   0   Jul 22      -  0:05 /u01/app/grid4/bin/mdnsd.bin
  oracle  6291842  8585614   3   Jul 22      - 70:27 /u01/app/grid4/bin/ocssd.bin
    root  7668170        1   0   Jul 22      -  7:40 /u01/app/grid4/bin/octssd.bin reboot
    root 26411340        1   3   Jul 23      - 41:36 /u01/app/grid4/bin/osysmond.bin
  oracle 28049762 36307328   0 17:36:52  pts/4  0:00 grep d.bin
    root 45875706        1   0   Jul 22      - 18:22 /u01/app/grid4/bin/crsd.bin reboot
  oracle 51052946        1   0   Jul 22      - 10:53 /u01/app/grid4/bin/gipcd.bin

Note : There is no pid value as we killed. Now start the service again:

$ srvctl start service -d PROD -s HRPROD -n rac2 -o open

Yah. It is started. Now see the "crsctl stat res -t" output:

$ crsctl stat res -t
--------------------------------------------------------------------------------
NAME           TARGET  STATE        SERVER                   STATE_DETAILS      
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.DATA.dg
               ONLINE  ONLINE       rac1                                       
               ONLINE  ONLINE       rac2                                       
ora.FRA.dg
               ONLINE  ONLINE       rac1                                       
               ONLINE  ONLINE       rac2                                       
ora.LISTENER.lsnr
               ONLINE  ONLINE       rac1                                       
               ONLINE  ONLINE       rac2                                       
ora.asm
               ONLINE  ONLINE       rac1                    Started            
               ONLINE  ONLINE       rac2                    Started            
ora.gsd
               OFFLINE OFFLINE      rac1                                       
               OFFLINE OFFLINE      rac2                                       
ora.net1.network
               ONLINE  ONLINE       rac1                                       
               ONLINE  ONLINE       rac2                                       
ora.ons
               ONLINE  ONLINE       rac1                                       
               ONLINE  ONLINE       rac2                                       
ora.registry.acfs
               ONLINE  ONLINE       rac1                                       
               ONLINE  ONLINE       rac2                                       
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
      1        ONLINE  ONLINE       rac2                                       
ora.LISTENER_SCAN2.lsnr
      1        ONLINE  ONLINE       rac1                                       
ora.LISTENER_SCAN3.lsnr
      1        ONLINE  ONLINE       rac1                                       
ora.cvu
      1        ONLINE  ONLINE       rac1                                       
ora.rac1.vip
      1        ONLINE  ONLINE       rac1                                       
ora.rac2.vip
      1        ONLINE  ONLINE       rac2                                       
ora.PROD.db
      1        ONLINE  ONLINE       rac1                    Open               
      2        ONLINE  ONLINE       rac2                    Open               
ora.PROD.HRPROD.svc
      1        ONLINE  ONLINE       rac1                                       
      2        ONLINE  ONLINE       rac2                                       
ora.oc4j
      1        ONLINE  ONLINE       rac1                                       
ora.scan1.vip
      1        ONLINE  ONLINE       rac2                                       
ora.scan2.vip
      1        ONLINE  ONLINE       rac1                                       
ora.scan3.vip
      1        ONLINE  ONLINE       rac1

Hope, it helped....