May 29, 2015

Monitor load on database servers - using iostat & vmstat

The iostat command is used to monitor the load on server input/output (I/O) devices by observing the time the devices are active compared to the average transfer rate of the device. iostat generates several report lines that can be used to monitor and subsequently change the system configuration to better balance the I/O workload between physical disk devices.

The initial report detail lines generated by iostat provide statistics encompassing the time since the system was last booted. Subsequent sets of detail lines cover the time since the previous report interval.
Each set of report lines starts with a header row with CPU statistics which represents the CPU usage across all processors.  Following the CPU information, a device header row is displayed with subsequent detail lines of statistics for each device in the system.

The following example shows the invocation of iostat specifying a three second interval or delay with a total of two samplings or counts:





























For the average CPU report, %user, %nice, %iowait, and %idle are defined the same as they were in the mpstat command output.  One remaining piece of information is defined as:
* %sys: The percentage of processor utilization occurring at the system kernel level.

For the device utilization report:

* device: The device name as listed in the /dev directory is displayed.  These device names are mapped to mount points in the file /etc/fstab and are also listed in the output of the df command.
* tps: The number of transfers (I/O requests) per second issued to the device.
* blk_read/s: The number of blocks per second read from the device.
* blk_wrtn/s: The number of blocks per second written to the device.
* blk_read: The total number of blocks read.
* blk_wrtn: The total number of blocks written.
This information can assist in the determination of which devices are more heavily used than others and perhaps help with the determination of how to better distribute data to balance the workload.

More Examples on iostat:
$ iostat 
Display a single history since boot report for all CPU and Devices.

$ iostat -d 2 
Display a continuous device report at two second intervals.

$ iostat -d 2 6 
Display six reports at two second intervals for all devices.

$ iostat -x hda hdb 2 6 
Display six reports of extended statistics at two second intervals for devices hda and hdb.

$ iostat -p sda 2 6 
Display six reports at two second intervals for device sda and all its partitions (sda1, etc.)

$ iostat -ct 2
Display a continues two seconds interval report of CPU utimization. Use iostat -tT 2 in AIX environments.

Displaying Virtual Memory Statistics:

The vmstat command displays information about processes, memory, paging, block IO, and different levels of CPU activity. As with iostat, the first detail lines produce report averages since the last reboot. Subsequent detail lines report information using the interval specified on the command line.
As with the other commands in this section, the vmstat command is driven by delay and count options that determine the time interval between report lines and the totals number of intervals to be reported.













The Linux man page for vmstat defines the fields displayed as follows:
* procs
* r: The number of processes waiting for run time
* b: The number of processes in uninterruptible sleep, which means they are waiting on a resource
* memory
* swpd: Virtual memory used
* free: Idle memory
* buff: Amount of memory used as buffers
* cache: Current memory used as cache
* swap
* si: Memory swapped in per second from disk
* so: Memory swapped out per second to disk
* io
* bi: Blocks per second received from a block device
* bo: Blocks per second sent to a block device
* system
* in:.Number of interrupts per second, including the clock
* cs: Number of context switches per second
* cpu: These statistics are percentages of total CPU time:
* us: User time spent running non-kernel code, includes nice time
* sy: System time spent running kernel code
* id: Idle time
* wa: Wait time spent waiting for I/O

The vmstat information can be invaluable when studying resource utilization trends.  Here are a few examples of how vmstat output can be interpreted:

If over time the run queue value, procs-r, remains consistently higher than the number of processors in the server and CPU idle time is low, the system is CPU bound and can benefit from the addition of more and/or faster processors. Alternatively a high number displayed in the procs-b column also indicates a bottleneck, but one where processes are waiting on other resources.

If the virtual memory used (memory-swpd) remains high and the free memory (memory-free) remains low, then the system is memory constrained and will benefit from additional RAM.

Consistently high I/O rates paired with consistently low CPU utilization (cpu-us) indicates an I/O bound system that could benefit from a highly buffered disk array or possibly solid-state disk.

Click here to see more about vmstat, iostat, mpstat and nmon reports from my earlier post.

Understanding %CPU while running top command

Most of Oracle DBAs or Linux administrators have following common questions on "TOP" command.

What does %CPU means when "top" is running?
If %CPU for my application as 400 or 500 most of the times, then how I will interpret? What number is a high number?

To answer this, you need to collect some statistics like Total CPU and cores available etc.

e.g.,

$ lscpu 

gives me below output:

Architecture:          x86_64
CPU op-mode(s):        32-bit, 64-bit
Byte Order:            Little Endian
CPU(s):                32
On-line CPU(s) list:   0-31
Thread(s) per core:    2
Core(s) per socket:    8
Socket(s):             2
NUMA node(s):          2
Vendor ID:             GenuineIntel
CPU family:            6
Model:                 45
Stepping:              7
CPU MHz:               2599.928
BogoMIPS:              5199.94
Virtualization:        VT-x
L1d cache:             32K
L1i cache:             32K
L2 cache:              256K
L3 cache:              20480K
NUMA node0 CPU(s):     0,2,4,6,8,10,12,14,16,18,20,22,24,26,28,30
NUMA node1 CPU(s):     1,3,5,7,9,11,13,15,17,19,21,23,25,27,29,31


so, we can understad like belwo:

%CPU -- CPU Usage : The percentage of your CPU that is being used by the process. By default, top displays this as a percentage of a single CPU. On multi-core systems, you can have percentages that are greater than 100%. For example, if 3 cores are at 60% use,  top will show a CPU use of 180%. See here for more information. You can  toggle this behavior by hitting "Shift+i" while top is running to show the overall percentage of available CPUs in use.

According to your lscpu output:

You have 32 cores (CPU(s)) in total.
You have 2 physical sockets (Socket(s)), each contains 1 physical processor.
Each processor of yours has 8 physical cores (Core(s) per socket) inside, which means you have 8 * 2 = 16 real cores.
Each real core can have 2 threads (Thread(s) per core), which means you have real cores * threads = 16 * 2 = 32 cores in total.
So you have 16 virtual cores and 16 real cores.

It means you have a lot of cores. One core at max is 100%. So the highest it can be is number_of_cores × 100%.

If you want to see core-wise CPU statistics, then below command can be used:

$ mpstat -P ALL 1

It shows how much each core is busy and it updates automatically each second. The output would be something like this
(on a quad-core processor):

10:54:41 PM  CPU    %usr   %nice    %sys %iowait    %irq   %soft  %steal  %guest   %idle
10:54:42 PM  all    8.20    0.12    0.75    0.00    0.00    0.00    0.00    0.00   90.93
10:54:42 PM    0   24.00    0.00    2.00    0.00    0.00    0.00    0.00    0.00   74.00
10:54:42 PM    1   22.00    0.00    2.00    0.00    0.00    0.00    0.00    0.00   76.00
10:54:42 PM    2    2.02    1.01    0.00    0.00    0.00    0.00    0.00    0.00   96.97
10:54:42 PM    3    2.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00   98.00
10:54:42 PM    4   14.15    0.00    1.89    0.00    0.00    0.00    0.00    0.00   83.96
10:54:42 PM    5    1.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00   99.00
10:54:42 PM    6    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
10:54:42 PM    7    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00





May 26, 2015

Instance terminated by PMON / Serial Transaction recovery caught exception 30319 : an workaround

Recently I found below  error from alert log during an workaround.

Tue May 26 15:52:35 2015
Exception [type: SIGSEGV, Address not mapped to object] [ADDR:0x10979BB8] [PC:0x101ECBAF4, opiexe()+28884] [flags: 0x0, count: 1]
Errors in file /u01/app/oracle/diag/rdbms/PROD/PROD/trace/PROD_smon_9306262.trc  (incident=112111):
ORA-07445: exception encountered: core dump [opiexe()+28884] [SIGSEGV] [ADDR:0x10979BB8] [PC:0x101ECBAF4] [Address not mapped to object] []
Incident details in: /u01/app/oracle/diag/rdbms/PROD/PROD/incident/incdir_112111/PROD_smon_9306262_i112111.trc
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Tue May 26 15:52:38 2015
Dumping diagnostic data in directory=[cdmp_20150526155238], requested by (instance=1, osid=9306262 (SMON)), summary=[incident=112111].
Tue May 26 15:52:40 2015
Sweep [inc][112111]: completed
Sweep [inc2][112111]: completed
Tue May 26 15:52:41 2015
PMON (ospid: 16449862): terminating the instance due to error 474
System state dump requested by (instance=1, osid=16449862 (PMON)), summary=[abnormal instance termination].
System State dumped to trace file /u01/app/oracle/diag/rdbms/PROD/PROD/trace/PROD_diag_10223776.trc
Dumping diagnostic data in directory=[cdmp_20150526155242], requested by (instance=1, osid=16449862 (PMON)), summary=[abnormal instance termination].
Instance terminated by PMON, pid = 16449862


From the related .trc file I found below issues:

......
......
ktsmgtur(): TUR was not tuned for 3761 secs
ktsmg_advance_slot(): MMNL advances slot after 3761 seconds
Serial Transaction recovery caught exception 30319

*** 2015-05-26 12:51:32.986
Serial Transaction recovery caught exception 30319
Serial Transaction recovery caught exception 30319

*** 2015-05-26 13:03:05.122
Serial Transaction recovery caught exception 30319



From the above investigations and  I changed the below described parameter. Now no issues are coming.

SQL> show parameter fast_start_parallel_rollback;

NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
fast_start_parallel_rollback         string      LOW


SQL> ALTER SYSTEM SET fast_start_parallel_rollback='FALSE';

system altered.

Reason: From alert log I found more parallel recoveries. As it is a report server POC, I changed "fast_start_parallel_rollback='FALSE' ".

If any issue / cause I will find, I will update the same topic.

Note : Instead of running this as “TRUE” or “LOW” for parallel rollback, setting it to “FALSE” is the best choice for a large, complicated transaction, (as there are other transactional processes updating the tables involved in this rollback!)

If SMON is doing some crash recovery, then use the below query and calculate your time:

sql> alter session set NLS_DATE_FORMAT=’DD-MON-YYYY HH24:MI:SS';

sql> select usn,
       state,
       undoblockstotal "Total",
       undoblocksdone "Done",
       undoblockstotal - undoblocksdone "ToDo",
       decode(cputime,
              0,
              'unknown',
              sysdate + (((undoblockstotal - undoblocksdone) /
              (undoblocksdone / cputime)) / 86400)) "Estimated time to complete"
              from v$fast_start_transactions;


When a large transaction got killed, terminated we all know that smon will do the rollback of the transaction, “smon: enable tx recovery”  which can evident in alert log.



May 22, 2015

Create user with admin option in Oracle

creating super admin user in Oracle
OR
Create user with admin option


-- To create user

create user applsys
identified by xxxxx
  default tablespace USERS
  temporary tablespace TEMP
  profile default;
grant CONNECT to applsys;
GRANT dba TO applsys WITH ADMIN OPTION;


Some theory :

WITH GRANT OPTION :

Specify WITH GRANT OPTION to enable the grantee to grant the object privileges to other users and roles.

Restriction on Granting WITH GRANT OPTION :

You can specify WITH GRANT OPTION only when granting to a user or to PUBLIC, not when granting to a role.

WITH HIERARCHY OPTION :

Specify WITH HIERARCHY OPTION to grant the specified object privilege on all subobjects of object, such as subviews created under a view, including subobjects created subsequent to this statement.

This clause is meaningful only in combination with the SELECT object privilege.

May 18, 2015

Fix : unable to spawn jobq slave process, slot 0, error 1089

-- Found from alert log. Message continusly writing to alert log

Mon May 18 14:15:58 2015
kkjcre1p: unable to spawn jobq slave process, slot 0, error 1089


RCA:

1) If you have fired "shut immediate" and presses "^C" to cancel the command.
OR
2) If you have some memory issue and you unable to down / start the database again.


Solution:

1) start it force fully if you don't have memory issues.

$ sqlplus / as sysdba
SQL> startup force;

This will cause your database will be force fully down and start it again.

Note: Shut abort will not work in this situation.


May 9, 2015

Resolve Adding redolog issues in RAC db: ORA-01156, ORA-01275

Issue Not allowing to add new redolog

SQL> ALTER DATABASE ADD LOGFILE THREAD 3 GROUP 3 ('+FRA/PROD/ONLINELOG/redo3a.log') size 200M reuse;
ALTER DATABASE ADD LOGFILE THREAD 3 GROUP 3 ('+FRA/PROD/ONLINELOG/redo3a.log') size 200M reuse
*
ERROR at line 1:
ORA-01156: recovery or flashback in progress may need access to files


Solution:

Cancel the recovery mode if you are recovering your standby database.

SQL> alter database recover managed standby database cancel;

Database altered.

Then you may face the below issue...

SQL> ALTER DATABASE ADD LOGFILE THREAD 3 GROUP 3 ('+FRA/PROD/ONLINELOG') size 200M;
ALTER DATABASE ADD LOGFILE THREAD 3 GROUP 3 ('+FRA/PROD/ONLINELOG') size 200M
*
ERROR at line 1:
ORA-01275: Operation ADD LOGFILE is not allowed if standby file management is automatic.


Change standby_file_management value "AUTO" to MANUAL.

SQL> alter system set standby_file_management=MANUAL;

System altered.


When I did the above, then my below commands worked.

SQL> ALTER DATABASE ADD LOGFILE THREAD 3 GROUP 3 ('+FRA/PROD/ONLINELOG/redo3a.log') size 200M;

Database altered.

SQL> ALTER DATABASE ADD LOGFILE THREAD 3 GROUP 4 ('+FRA/PROD/ONLINELOG/redo4b.log') size 200M;

Database altered.

SQL>


Translate >>