Common OEM 13c agent issues and fixes with Log directory structure

You may face following issues in OEM 13c like :
a) Agent is unable to communicate with the OMS. (REASON = Agent is Unreachable (REASON : Agent to OMS Communication is broken. ).
b) Huge volume Alerts

So, In this case, first try to connect the OMS like below:

$ telnet oms01.example.com 3872 
Trying 10.20.30.40... 
Connected to oms01.example.com. 
$

If there is no connectivity issue, then you can follow below steps and monitor the agent :

Step 1: Increase the agent memory 

1. Stop the agent. 
<AGENT_INST_HOME>/bin>./emctl stop agent

2. Take a backup of AGENT_INST_HOME/sysman/config/emd.properties and edit it 
From: 
agentJavaDefines=-Xmx270M -XX:MaxPermSize=96M 

To: 
agentJavaDefines=-Xmx512M -XX:MaxPermSize=96M 

3.Backup the file <AGENT_INST>/sysman/config/s_jvm_options.opt and add following parameter at EOF 

-XX:+UseMembar 

4.Start the agent 
./emctl start agent 

Then verify the log "emagent.nohup" file for rest of the agents and follow the above steps to avoid the same issue.

5. Verify the status of the agent

./emctl status agent


Note: 
To get a report on this issue, you can connect to the OMS and run the below query to find out the history:

Connected to the Repository database as the user SYSMAN, execute the following query:

SQL> Select b.target_name agent , c.target_name buddy 
from em_agent_buddy_map a, em_targets b, em_targets c 
where a.agent_target_guid = b.target_guid 

and a.buddy_target_guid = c.target_guid; 



DIRECTORY STRUCTURE AND LOCATIONS OF IMPORTANT TRACE AND LOG FILES OF ENTERPRISE MANAGER CLOUD CONTROL 13C:

The Trace and Log Files of OMS:

The Oracle Management Service has two applications: OMS Console (emgc) and OMS Platform (empbs). The trace and log files of these applications are located in “sysman/log” directory in the EM Instance Home (for example: /u01/oracle/gc_inst/em/EMGC_OMS1/sysman/log ). These logs are very useful to diagnose OMS related problems.

emoms.log : The main log file for the OMS Console (emgc) application.
emoms.trc : The main trace file for the OMS Console (emgc) application.
emoms_pbs.log : The main log file for the OMS pbs (empbs) application.
emoms_pbs.trc : The main trace file for the OMS pbs (empbs) application.
secure.log ; This log provides additional logs for “emctl secure oms” command.
emctl.log : This log contains OMS startup/shutdown details.
emctl.msg : This log is created by the OMS HealthMonitor thread, when it re-starts the OMS due to a critical error.

The Trace and Log Files of EM Managed Server:

EM Managed Server is the weblogic server hosting OMS applications. The logs are located in “logs” directory in EM Manager Server Home folder (for example: /u01/oracle/gc_inst/user_projects/domains/GCDomain/servers/EMGC_OMS1/logs):

EMGC_OMS1.log : The EMGC_OMSn instance writes all messages from its subsystems and applications to this logfile.
EMGC_OMS1.out : The messages written to sysout and syserr are saved in this file by the nodemanager. This log file is rotated by the nodemanager only at the time of Managed server start, and older files need to be manually purged.
access.log : This log file contains details of HTTP requests processed by EM Managed Server.
EMGC_OMS1-diagnostic.log: This log file contains application-related security errors.

The Trace and Log Files of Oracle HTTP Server (OHS):

Oracle HTTP Server (OHS) is the web server component of EM13c. It works as a reverse proxy for Oracle WebLogic Server and the framework for hosting static pages. It’s located in “logs” folder in the “OHS Home” (/u01/oracle/gc_inst/user_projects/domains/GCDomain/servers/ohs1/logs):

access_log : Access logs stores all the client requests. The HTTP Response code is useful in identifying whether the client request was successful or not.
ohs1.log : This is the error log for the OHS.
admin_log : This log file keeps Oracle Dynamic Monitoring Service (DMS) requests.
em_upload_http_access_log : The log file contains entries for Agent uploads in http mode.
em_upload_https_access_log : The log file contains entries for Agent uploads in https mode.

The Trace and Log Files of EM Agents:

The agent log files are located in the “sysman/log” folder in the “Agent Instance Home” (for example: /u01/oracle/emagent/agent_inst/sysman/log).

gcagent.log : This log file contains trace, debug, information, error, or warning messages from the Agent. It can be used for debugging Agent framework issues.
gcagent_errors.log : It’s similar to the gcagent.log but it contains only the log messages of ERROR and FATAL levels. The log has no size limit.
emagent.nohup : Nohup logs of the agent. This log contains startup and shutdown messages including JVM flags. It can be used to determine when the agent is down, and the errors generated on startup of shutdown can be examined with this log.
gcagent_mdu.log : This log tracks the metadata updates (such as save or remove targets, blackouts, deployments of metric extensions) to the Agent.
gcagent_sdk.trc : The gcagent_sdk.trc file contains log messages from each of the plugins and can be used for debugging plugin related issues.
emctl.log : emctl.log provides logging for emctl commands.
emdctlj.log : emdctlj.log contains logging for the entry point and the exit codes of the emctl commands.
secure.log : This log provides additional logs for “emctl secure agent” command, and can be useful to diagnose failures during securing of the Agent.
emagent_perl.trc : Trace file for the PERL scripts. EM uses perl scripts to gather some metrics and target discovery. The log level can be changed by modifying EMAGENT_PERL_TRACE_LEVEL variable in “sysman/config/emd.properties” file.