Oracle ASM Disks and multi path issue- Solution

So today I needed to create some new ASM disks on Hitachi enterprise storage with HDLM Hitachi Dynamic Link Manager multipathing setup with Oracle Linux 6.6 environment on a two node Oracle 12c RAC cluster. Unfortunately when I tried to create new ASM disks using the ASMLib and the Oracle ASM create disk command it failed to create the disks! I thought this to be odd because I had used the fdisk command on Linux to add a new partition on each of the LUNs presented to the OS for ASM. Permissions were correct and fdisk showed that the partitions were created for the newly provisioned LUNs to Linux.

I found a blog post that mentions a similar issue and MOS note that references the issue and workaround solution:

ASMLib: oracleasm createdisk command fails: Device '/dev/emcpowera1 is not a partition [Failed] (Doc ID 469163.1)

Now while this references EMC storage, the same issue and solution applies to other multipathing software and storage arrays such as Hitachi storage with HDLM in my case.

Instead of using the 'oracleasm createdisk' command I had to use an internal tool called asmtool that usually is called behind the scenes by Oracle tools.

# /usr/sbin/asmtool -C -l /dev/oracleasm -n VOL1 -s /dev/emcpowera1 -a force=yes

This saved me hours of frustration this week when I had to build new ASM disks and do a migration for a customer solution in a very short time period. Hopefully it will also save you hours of grief .

Getting used to Hitachi storage is a bit new for me but since I have years of EMC storage administration experience it is not too steep a learning curve and quite a bit of fun learning new things!

On Solving the Right Problem

Dear Readers,


It has been quite a while since I wrote on the blog due to some past things that occupied my personal time that had to be addressed. Recently I began to solve new performance related problems for a customer. Here is the scenario:


Oracle 12c RAC
Large enterprise SAN
Converged infrastructure


The target goal is to achieve a minimal performance level (SLA) for storage and system performance.


The customer is only receiving a low number of storage performance in terms for overall IOPS with their two node Oracle 12c RAC configuration.


So with little information to go by from past experiences, I asked how they measure such performance values.


Enter the Benchmark Tools!


The customer is using a third party performance tool to measure performance for CPU, system and storage figures. So what is this tool you might ask? Well hold on for a minute we will get to that question in a short while. My first DBA spidey sense was to get the data from the horse's mouth so to speak that is from Oracle! I logged into the Oracle 12c RAC cluster and pulled the recent AWR reports from the cluster and noticed that overall there were no serious performance issues!


It turns out that the performance tool is reporting different numbers than the Oracle database!


Hmm well that sure is very odd! I review the infrastructure between customer sites and lo and behold find that configurations are different! So that calls to mind that Oracle per se is not guilty but a more fundamental issue of apples to oranges comparisons. Which leads to my next thought process


FIND AND SOLVE THE RIGHT PROBLEM


ASK THE RIGHT QUESTION


I know this seems obvious to most of us Oracle DBA types right? Well you'd be surprised at how many customers spin their wheels attempting to solve a performance issue by mistaking the forest for the weeds. Instead of immediate jumping to conclusions, take a deep breath and step back to look at the big picture. The following comes to mind:


Examine
1. Storage configuration- disks, HBA, HBA, multi path configurations
2. Firmware and patch levels for infrastructure- servers, SAN, networks
3. Review OS configurations and releases
4. Run basic tests to collect data points- Oracle AWR, sar, vmstat, et al.




Stay tuned on my next series of blog posts on how to exactly solve these types of problems. Oh yeah and get ready, set and go for Oracle Open World this coming week!!


Cheers,
-B