Cluster related process in 11g RAC R2 Environment

CLUSTERWARE PROCESSES in 11g RAC R2 Environment 

In any RAC environment Cluster daemons are the main agent to communicate between instances. At one glance below commands can be used what cluster daemons are running:

$ ps -ef|grep d.bin
    root  4456622        1   0   Oct 23      - 15:24 /u01/app/11.2.0/grid/bin/ohasd.bin reboot
    root  4849726        1   3   Oct 23      - 158:25 /u01/app/11.2.0/grid/bin/orarootagent.bin
    root  4915400        1   0   Oct 23      -  3:51 /u01/app/11.2.0/grid/bin/cssdagent
    root  5898304        1   0   Oct 23      - 18:57 /u01/app/11.2.0/grid/bin/orarootagent.bin
    grid  9634040        1   0   Oct 23      -  0:10 /u01/app/11.2.0/grid/bin/tnslsnr LISTENER_SCAN2 -inherit
    grid 11337848        1   0   Oct 23      -  0:07 /u01/app/11.2.0/grid/bin/mdnsd.bin
    grid 11862222        1   0   Oct 23      -  0:10 /u01/app/11.2.0/grid/bin/tnslsnr LISTENER_SCAN3 -inherit
    grid  2294256        1   0   Oct 23      -  6:45 /u01/app/11.2.0/grid/bin/evmd.bin
    grid  2621900        1   0   Oct 23      -  1:21 /u01/app/11.2.0/grid/bin/scriptagent.bin
    grid  2818414  2949552   0   Oct 23      - 16:59 /u01/app/11.2.0/grid/bin/ocssd.bin
    root  2949552        1   0   Oct 23      -  0:00 /bin/sh /u01/app/11.2.0/grid/bin/ocssd
    root  3867104        1   0   Oct 23      - 19:40 /u01/app/11.2.0/grid/bin/osysmond.bin
    root  3997956        1   0   Oct 23      -  3:35 /u01/app/11.2.0/grid/bin/cssdmonitor
    grid  4456732  2294256   0   Oct 23      -  0:07 /u01/app/11.2.0/grid/bin/evmlogger.bin -o /u01/app/11.2.0/grid/evm/log/evmlogger.info -l /u01/app/11.2.0/grid/evm/log/evmlogger.log
    grid  4719074        1   0   Oct 23      - 17:19 /u01/app/11.2.0/grid/bin/oraagent.bin
    grid  4784526        1   0   Oct 23      - 12:56 /u01/app/11.2.0/grid/bin/gipcd.bin
    root  5046556        1   1   Oct 23      - 53:09 /u01/app/11.2.0/grid/bin/ologgerd -m ehdb02 -r -d /u01/app/11.2.0/grid/crf/db/ehdb01
    root  5112254        1   0   Oct 23      - 19:11 /u01/app/11.2.0/grid/bin/crsd.bin reboot
    root  5439972        1   0   Oct 23      - 11:01 /u01/app/11.2.0/grid/bin/octssd.bin reboot
    grid  5505490        1   0   Oct 23      -  2:06 /u01/app/11.2.0/grid/bin/gpnpd.bin
    grid  7930344        1   0   Oct 23      -  0:08 /u01/app/11.2.0/grid/bin/tnslsnr LISTENER -inherit
  oracle  9371946        1   0   Oct 23      - 25:28 /u01/app/11.2.0/grid/bin/oraagent.bin
  oracle 11141592 11599954   0 17:00:11  pts/0  0:00 grep d.bin
    grid 11796808        1   0   Oct 23      - 16:56 /u01/app/11.2.0/grid/bin/oraagent.bin

i).Cluster Ready Services (CRS)

$ ps -ef | grep crs | grep -v grep
    root  5112254        1   0   Oct 23      - 19:09 /u01/app/11.2.0/grid/bin/crsd.bin reboot

crsd.bin => The above process is responsible for start, stop, monitor and failover of resource. It maintains OCR and also restarts the resources when the failure occurs.

This is applicable for RAC systems. For Oracle Restart and ASM ohasd is used.

ii).Cluster Synchronization Service (CSS)

$ ps -ef | grep -v grep | grep css
    root  4915400        1   0   Oct 23      -  3:50 /u01/app/11.2.0/grid/bin/cssdagent
    grid  2818414  2949552   0   Oct 23      - 16:57 /u01/app/11.2.0/grid/bin/ocssd.bin
    root  2949552        1   0   Oct 23      -  0:00 /bin/sh /u01/app/11.2.0/grid/bin/ocssd
    root  3997956        1   0   Oct 23      -  3:33 /u01/app/11.2.0/grid/bin/cssdmonitor

cssdmonitor => Monitors node hangs(via oprocd functionality) and monitors OCCSD process hangs (via oclsomon functionality) and monitors vendor clusterware(via vmon functionality).This is the multi threaded process that runs with elavated priority.

Startup sequence: INIT --> init.ohasd --> ohasd --> ohasd.bin --> cssdmonitor

cssdagent => Spawned by OHASD process.Previously(10g) oprocd, responsible for I/O fencing.Killing this process would cause node reboot.Stops,start checks the status of occsd.bin daemon

Startup sequence: INIT --> init.ohasd --> ohasd --> ohasd.bin --> cssdagent

occsd.bin => Manages cluster node membership runs as oragrid user.Failure of this process results in node restart.

Startup sequence: INIT --> init.ohasd --> ohasd --> ohasd.bin --> cssdagent --> ocssd --> ocssd.bin

iii) Event Management (EVM)

$ ps -ef | grep evm | grep -v grep
    grid  2294256        1   1   Oct 23      -  6:45 /u01/app/11.2.0/grid/bin/evmd.bin
    grid  4456732  2294256   0   Oct 23      -  0:07 /u01/app/11.2.0/grid/bin/evmlogger.bin -o /u01/app/11.2.0/grid/evm/log/evmlogger.info -l /u01/app/11.2.0/grid/evm/log/evmlogger.log

evmd.bin => Distributes and communicates some cluster events to all of the cluster members so that they are aware of the cluster changes.

evmlogger.bin => Started by EVMD.bin reads the configuration files and determines what events to subscribe to from EVMD and it runs user defined actions for those events.

iv).Oracle Root Agent

$ ps -ef | grep -v grep | grep orarootagent
    root  4849726        1   0   Oct 23      - 158:11 /u01/app/11.2.0/grid/bin/orarootagent.bin
    root  5898304        1   0   Oct 23      - 18:54 /u01/app/11.2.0/grid/bin/orarootagent.bin

orarootagent.bin => A specialized oraagent process that helps crsd manages resources owned by root, such as the network, and the Grid virtual IP address.

The above 2 process are actually threads which looks like processes. This is a Linux specific

v).Cluster Time Synchronization Service (CTSS)

ps -ef | grep ctss | grep -v grep
    root  5439972        1   0   Oct 23      - 10:59 /u01/app/11.2.0/grid/bin/octssd.bin reboot

octssd.bin => Provides Time Management in a cluster for Oracle Clusterware

vi) Oracle Agent

$ ps -ef | grep -v grep | grep oraagent
    grid  4719074        1   0   Oct 23      - 17:18 /u01/app/11.2.0/grid/bin/oraagent.bin
  oracle  9371946        1   0   Oct 23      - 25:26 /u01/app/11.2.0/grid/bin/oraagent.bin
    grid 11796808        1   0   Oct 23      - 16:55 /u01/app/11.2.0/grid/bin/oraagent.bin

oraagent.bin => Extends clusterware to support Oracle-specific requirements and complex resources. This process runs server callout scripts when FAN events occur. This process was known as RACG in Oracle Clusterware 11g Release 1 (11.1).


ORACLE HIGH AVAILABILITY SERVICES STACK

i) Cluster Logger Service

$ ps -ef | grep -v grep | grep ologgerd
    root  5046556        1   0   Oct 23      - 53:05 /u01/app/11.2.0/grid/bin/ologgerd -m ehdb02 -r -d /u01/app/11.2.0/grid/crf/db/ehdb01

ologgerd => Receives information from all the nodes in the cluster and persists in a CHM repository-based database. This service runs on only two nodes in a cluster

ii).System Monitor Service (osysmond)

$ ps -ef | grep -v grep | grep osysmond
    root  3867104        1   0   Oct 23      - 19:38 /u01/app/11.2.0/grid/bin/osysmond.bin

osysmond => The monitoring and operating system metric collection service that sends the data to the cluster logger service. This service runs on every node in a cluster

iii). Grid Plug and Play (GPNPD):

$ ps -ef | grep gpn
  oracle  9306330 11599954   0 16:49:41  pts/0  0:00 grep gpn
    grid  5505490        1   0   Oct 23      -  2:05 /u01/app/11.2.0/grid/bin/gpnpd.bin

gpnpd.bin => Provides access to the Grid Plug and Play profile, and coordinates updates to the profile among the nodes of the cluster to ensure that all of the nodes have the most recent profile.

iv).Grid Interprocess Communication (GIPC):

$ ps -ef | grep -v grep | grep gipc
    grid  4784526        1   0   Oct 23      - 12:55 /u01/app/11.2.0/grid/bin/gipcd.bin

gipcd.bin => A support daemon that enables Redundant Interconnect Usage.

v). Multicast Domain Name Service (mDNS):

$ ps -ef | grep -v grep | grep dns
    grid 11337848        1   0   Oct 23      -  0:07 /u01/app/11.2.0/grid/bin/mdnsd.bin

mdnsd.bin => Used by Grid Plug and Play to locate profiles in the cluster, as well as by GNS to perform name resolution. The mDNS process is a background process on Linux and UNIX and on Windows.

vi).Oracle Grid Naming Service (GNS)

$ ps -ef | grep -v grep | grep gns

gnsd.bin => Handles requests sent by external DNS servers, performing name resolution for names defined by the cluster.

Note: No output will come if gns is not configured.

I hope this doc will help you.