Exachk to the rescue!

 So do you have a large and complex Oracle Exadata environment and need to perform a health check or resolve a tricky issue? Well, one thing to keep in mind is that Oracle support may ask you to run some diagnostics to provide logs and so forth. This is where the utility exachk comes to the resue!

What is exachk? Well according to the Oracle Exadata documentation, this nifty utility "provide a lightweight and non-intrusive health check framework for the Oracle stack of software and hardware components."

As an extra bonus, the ExaChk utility works not only for Oracle Exadata engineered systems, but can diagnose issues with other Oracle database appliances such as the Exalogic and Oracle Database Appliance (ODA) systems. Recently, we had a power issue at work that required me to troubleshoot why the Oracle Clusterware (CRS) and ASM failed to come back on line after the power reboot. I tried to start the CRS using standard Oracle 19c RAC commands like crsctl and srvctl with no success as it complained about storage related issues on the Exadata Cloud @ Customer environment since we DBAs unfortunately are not provided with access to the grid disks and cell storage servers due to the setup. So how do you use Exachk? It is really simple actually. You can run it by default as the grid user and by default it runs all of the options to perform a full collection of the Exadata ecosystem. If you use the exachk -a that will run all options for diagnostic collection tasks. It then collects the details for the health check of the Exadata system into a series of tar files to upload to Oracle support.

So the next time you face a challenging Exadata or Oracle engineered system issue, be sure to use the powerful ExaChk utility!

References and Further Reading

Oracle Exadata Database Machine EXAchk (Doc ID 1070954.1)

Oracle Exachk Quick Start Guide

https://docs.oracle.com/en/engineered-systems/health-diagnostics/exachk/oexug/quick-start-guide.html#GUID-7C8BEBDC-5125-4493-A8FC-799023FAFC06

Happy 2021 A New Age

 Hello my dear readers,

It has been a long time since I posted but ya know life sometimes gets in the way. Lots of moving, work, travel and projects and the past few years have been life changing on so many levels.  This will be a non tech post but next ones will be some new database content!

I learned Exadata and Oracle Cloud! After many years of hearing about these amazing platforms and attending sessions at past Oracle conferences, I was able last year to get hands on experience working on several large Exadata X4-8 systems. It has been a lot of fun learning and also using Oracle 19c new features like multi-tenancy pluggable databases, Oracle Cloud (OCI) and setting up Oracle 13c Cloud control as well. So fun! 

More to come my fellow data professionals and colleagues!

-Ben

ASM Diskgroup Analysis Tip

Recently I have been performing multiple Oracle 12c RAC setups and digging deep into ASM internals. Besides query the ASM environment via the V$ASM_DISK and V$ASM_DISKGROUP dynamic tables, these lesser known command line tools provide detailed troubleshooting assistance for solving difficult ASM issues with your Oracle environments.


As a reference point for today's discussion, I found a very useful support note from Oracle that explains all three utilities to analyze Oracle ASM environments:


ASM tools used by Support : KFOD, KFED, AMDU
 (Doc ID 1485597.1)


So let's take a look at each one in terms of how and why you would use these ASM diagnostic tools.


kfod


Kfod is a quick and dirty tool to examine the current ASM configuration for ownership, sizes and device mappings. Ensure that your kfod is in the local path for $GRID_HOME/bin/kfod to allow execution of the tool.


$ which kfod
/u01/app/grid/12.1.0/grid/bin/kfod


To get help with kfod use the help=y option:


$ kfod help=y
_asm_a/llow_only_raw_disks              KFOD allow only raw devices [_asm_allow_only_raw_disks=(TRUE)/FALSE]
_asm_l/ibraries         ASM Libraries[_asm_libraries=lib1,lib2,...]
_asms/id                ASM Instance[_asmsid=sid]
_b/oot          Running in pre-install env (boot=TRUE/FALSE)
_f/lexinfo              Provide flexinfo      (_flexinfo=TRUE/FALSE)
_p/atch_lib             Patchlib [_patch_lib=]
_u/ser          OS Username
asm_/diskstring         ASM Diskstring [asm_diskstring=discoverystring, discoverystring ...]
asmc/ompatibility               Include diskgroup ASM compatibility [asmcompatibility=TRUE/(FALSE)]
cli/ent_cluster         client cluster name
clus_/version           cluster version
clust/er                KFOD cluster [cluster=TRUE/(FALSE)]
db_/unique_name         db_unique_name for ASM instance[db_unique_name=dbname]
dbc/ompatibility                Include diskgroup DB compatibility [dbcompatibility=TRUE/(FALSE)]
disk_/access            Disk access method [disk_access=DIRECT/(INDIRECT)]
disks           Disks to discover [disks=raw,asm,badsize,all]
ds/cvgroup              Include group name [dscvgroup=TRUE/(FALSE)]
f/orce          Force option to delete files (force=TRUE/FALSE)
g/roup          Disks in diskgroup [group=diskgroup]
h/ostlist               hostlist[hostlist=host1,host2,...]
metadata_a/usize                AU Size for Metadata Size Calculation
metadata_c/lients               Client Count for Metadata Size Calculation
metadata_d/isks         Disk Count for Metadata Size Calculation
metadata_n/odes         Node Count for Metadata Size Calculation
metadata_r/edundancy            Redundancy for Metadata Size Calculation
na/me           Include disk name [name=TRUE/(FALSE)]
no/hdr          KFOD header suppression [nohdr=TRUE/(FALSE)]
ol/r            Import credentials to OLR [olr=TRUE/(FALSE)]
op              KFOD options type [OP=DISKS/CANDIDATES/MISSING/GROUPS/INSTS/VERSION/PATCHES/PATCHLVL/CLIENTS/RM/RMVERS/DFLTDSTR/GPNPDSTR/METADATA/CREDCRECLUS/GETCLSTYPE/CREDEXPORT/GETASMGUID/CREDDELCLUS/CREDVERIFY/UPGRADEVERIFY/ALL]
p/file          ASM parameter file [pfile=parameterfile]
r/im_disk_access                Rim disk access method [rim_disk_access=DIRECT/(INDIRECT)]
s/tatus         Include disk header status [status=TRUE/(FALSE)]
v/erbose                KFOD verbose errors [verbose=TRUE/(FALSE)]
w/rap           wrap file for credentials
KFOD-01000: USAGE: kfod op= asm_diskstring=... | pfile=...


kfod status=TRUE asm_diskstring='' disk=ALL


$ kfod verbose=true, disks=all status=true op=disks \
> asm_diskstring='/dev/oracleasm/disks/*'
WARNING: Using brute force method to determine the size of /dev/oracleasm/disks/sddlmfdrv0.
 There will be performance issues. Please check configuration to determine the cause for the failure of ioctl
--------------------------------------------------------------------------------
 Disk          Size Header    Path                                     User     Group
================================================================================
   1:     409600 Mb MEMBER    /dev/oracleasm/disks/sddlma             grid     oinstall
   2:     409600 Mb MEMBER    /dev/oracleasm/disks/sddlmb             grid     oinstall
   3:     409600 Mb MEMBER    /dev/oracleasm/disks/sddlmc             grid     oinstall
   4:     409600 Mb MEMBER    /dev/oracleasm/disks/sddlmd             grid     oinstall
   5:     409600 Mb MEMBER    /dev/oracleasm/disks/sddlme             grid     oinstall
   6:     409600 Mb MEMBER    /dev/oracleasm/disks/sddlmf             grid     oinstall
   7:     409600 Mb MEMBER    /dev/oracleasm/disks/sddlmg             grid     oinstall




kfed


Kfed differs from the other tools in that it shows lower level metadata mappings for block size and endian counts for ASM disks and diskgroups. The help is provided by entering kfed with no options:


$ kfed
as/mlib         ASM Library [asmlib='lib']
aun/um          AU number to examine or update [AUNUM=number]
aus/z           Allocation Unit size in bytes [AUSZ=number]
blkn/um         Block number to examine or update [BLKNUM=number]
blks/z          Metadata block size in bytes [BLKSZ=number]
ch/ksum         Update checksum before each write [CHKSUM=YES/NO]
cn/t            Count of AUs to process [CNT=number]
de/v            ASM device to examine or update [DEV=string]
dm/pall         Don't suppress repeated lines when dumping corrupt blocks [DMPALL=YES/NO]
o/p             KFED operation type [OP=READ/WRITE/MERGE/REPAIR/NEW/FORM/FIND/STRUCT]
p/rovnm         Name for provisioning purposes [PROVNM=string]
s/eek           AU number to seek to [SEEK=number]
te/xt           File name for translated block text [TEXT=string]
ty/pe           ASM metadata block type number [TYPE=number]
v/erb           Verbose execution [verbose=YES|NO]




To run kfed, you need to specify the ASM disk mapping including the fully qualified device path as shown in the below example:


$ kfed read /dev/oracleasm/disks/sddlmbd
kfbh.endian:                          1 ; 0x000: 0x01
kfbh.hard:                          130 ; 0x001: 0x81
kfbh.type:                            1 ; 0x002: KFBTYP_DISKHEAD
kfbh.datfmt:                          1 ; 0x003: 0x01
kfbh.block.blk:                       0 ; 0x004: blk=0
kfbh.block.obj:              2147483675 ; 0x008: disk=22
kfbh.check:                  2341676564 ; 0x00c: 0x8b932613
kfbh.fcn.base:                        0 ; 0x010: 0x00000000
kfbh.fcn.wrap:                        0 ; 0x014: 0x00000000
kfbh.spare1:                          0 ; 0x018: 0x00000000





amdu


Last but not least is probably one of my favorite ASM command line tools called amdu which provides a nicely formatted report for a specific ASM diskgroup that consolidates most of the information generated by kfod and kfed into a simple file called report.txt


For the many options with amdu to get help make sure to specify the help=y parameter as shown below:


$ amdu help=y
al/lides                Dump indirect blks unconditionally
-allides: AMDU ordinarily skips over empty indirect blocks. Specifying
    this option tells AMDU to dump those blocks unconditionally. Be
    warned that this can make the resulting AMDU dump quite large.
au/size         AU size for corrupt disks
-ausize : This option must be set when -baddisks is set. It
    must be a power of 2. This size is required to scan a disk looking
    for metadata, and it is normally read from the disk header. The
    value applies to all disks that do not have a valid header. The
    value from the disk header will be used if a valid header is
    found.
ba/ddisks               Include disks with bad headers
-baddisks :  Normally disks with bad disk headers, or that
    look like they were never part of a disk group, will not be
    scanned. This option forces them to be scanned anyway and to be
    considered part of the given diskgroup. This is most useful when
    a disk header has been damaged. The disk will still need to have
    a valid allocation table to drive the scan unless -fullscan is
    used. In any case at least one block in the first two AUs must be
    valid so that the disk number can be determined. The options
    -ausize and -blksize are required since these values are normally
    fetched from the disk header. If the diskgroup uses external
    redundancy then -external should be specified. These values will
    be compared against any valid disks found in the diskgroup and
    they must be the same.
bl/ksize                ASM block size for corrupt disks
-blksize : This option must be set when -baddisks is set. It
    must be a power of 2. This size is required to scan a disk looking
    for metadata, and it is normally read from the disk header. The
    value applies to all disks that do not have a valid header. The
    value from the disk header will be used if a valid header is
    found.
c/ompare                Compare file mirrors
-compare: This option only applies to file extraction from a normal or
    high redundancy disk group. Every extent that is mirrored on more
    than one discovered disk will have all sides of its mirror
    compared. If they are not identical a message will be reported
    on standard error and the report file. The message will indicate
    which copy was extracted. A count of the blocks that are not
    identical will be in the report file.
dir/ectory              Directory from previous dump
-directory : This option completely eliminates the discovery
    phase of operation. It specifies the name of a dump directory from
    a previous run of AMDU. The report file and map files are read
    instead of doing a discovery and scan. The parsing of these ASCII
    files is very dependent on them being exactly as written by AMDU.
    AMDU is unlikely to work properly if they have been modified by
    a text editor, or if some of the files are missing or truncated.
    Note that the directory may be a copy FTPed from another
    machine. The other machine may even be a different platform
    with a different endianess.
dis/kstring             Diskstring for discovery
 -diskstring : By default the null string is used for
    discovery. The null string should discover all disks the user has
    access to. Many installations specify an asm_diskstring parameter
    for their ASM instance. If so that parameter value should be given
    here. Multiple discovery strings can be specified by multiple
    occurrences of -diskstring . Beware of shell syntax
    conflicts with discovery strings. Diskstrings are usually the same
    syntax the shell uses for expanding path names on command lines so
    they will most likely need to be enclosed in single quotes.
du/mp           Diskgroups to dump
-dump : This option specifies the name of a diskgroup to
    have its metadata dumped. This option may be specified multiple
    times to dump multiple diskgroups. If the diskgroup name is ALL
    then all diskgroups encountered will be dumped. The diskgroup name
    is not case sensitive, but will be converted to uppercase for all
    reports. If this option is not specified then no map or image
    files will be created, but -extract and -print may still work.
exc/lude                Disks to exclude
-exclude : Multiple exclude options may be specified. These
    strings are used for discovery just like the values for diskstring.
    Only shallow discovery is done on these diskstrings. Any disks
    found in the exclude discovery will not be accessed. If they are
    also discovered using the -diskstring strings, then the report will
    include the information from shallow discovery along with a message
    indicating the disk was excluded.
exte/rnal               Assume external redundancy
-external: Normally AMDU determines the diskgroup redundancy from the
    disk headers. However this is not possible with the -baddisks
    option. It is assumed that the redundancy of the -baddisks
    diskgroup is normal or high unless this option is given to specify
    external redundancy.
extr/act                Files to extract
-extract .: This extracts the numbered file
    from the named diskgroup, case insensitive. This option may be
    specified multiple times to extract multiple files. The extracted
    file is placed in the dump directory under the name
    _.f  where is the diskgroup name
    in uppercase, and is the file number. The -output option
    may be used to write the file to any location. The extracted file
    will appear to have the same contents it would have if accessed
    through the database. If some portion of the file is unavailable
    then that portion of the output file will be filled with
    0xBADFDA7A, and a message will appear on stderr.
fi/ledump               Dump files rather than extract
-filedump: This option causes the file objects in the command line to
    have their blocks dumped to the image files rather than extracted.
    This can be combined with the -novirtual option to selectively
    dump only some of the metadata files. It may also be used to dump
    user files (number >= 256) so that all mirrored copies can be
    examined.
fo/rmer         Include dropped disks
-former: Normally disks marked as former are not scanned, but this
    option will scan them and include their contents in the output.
    This is useful when it is necessary to look at the contents of a
    disk that was dropped. Note that dropped normal disks will not have
    any entries in their allocation tables and thus only the physically
    addressed extents will be dumped. Force dropped disks will not have
    status former in their disk headers and are not affected by this
    option. However if DROP DISKGROUP is used, the disks will have the
    contents as of the time of the drop, and will be in status former.
    Thus this option is useful for extracting files from a dropped
    diskgroup.
fu/llscan               Scan entire disk
-fullscan: This option reads every AU on the disk and looks at the
    contents of the AU rather than limiting the AU's read based on the
    allocation table. This is useful when the allocation table is
    corrupt or needs recovery. An AU will be written to the image file
    if it starts with a block that contains a valid ASM block header.
    The file and extent information for the map will be extracted from
    the block header. Physically addressed metadata will be dumped
    regardless of its contents. This option is incompatible with
    extracting a file. It is an error to specify -extract with this
    option. Note that this option is likely to find old garbage
    metadata in unallocated AU's since there is no means of
    determining what is allocated. Thus there may be many different
    copies of the same block, possibly of different versions.
h/ex            Always print block contents in hex
-hex: This prints the block contents in hex without attempting to print
    them as ASM metadata. This is useful when the block is known to not
    be ASM metadata. It avoids the ASM block header dump and ensures
    the block is not accidentally interpreted as ASM metadata. This
    option requires at least one -print option.
noa/cd          Do not dump ACD
-noacd: This option limits the dumping of the Active Change Directory
    to just the control blocks that contain the checkpoint. There is
    126 MB of ACD per ASM instance (42 MB for external redundancy). It
    is normally of no interest if there has been a clean shutdown or
    no updates for a while. This option avoids dumping a lot of
    unimportant data. The blocks will still be read and checked for
    corruption. The map file will still contain entries for the ACD
    extents, but the block counts will be zero.
nod/ir          Do not create a dump directory
-nodir: No dump directory is created, and no files are created in it.
    The directory name is not written to standard out. The report file
    is written to standard out before any block printouts from any
    -print options.  This option conflicts with -filedump. It is an
    error to specify this and extract a file to the dump directory.
noe/xtract              Do not create extracted file
-noextract: This prevents files from being extracted to an output
    file, but the file will be read and any errors in selecting the
    correct output will be reported. This is most useful in
    combination with the -compare option.
noh/eart                Do not check for heartbeat
-noheart: Normally the heartbeat block will be saved at discovery time
    and checked when the disk is scanned. A sleep is added between
    discovery and scanning to ensure there is time for the heartbeat
    to be written. If the heartbeat block changes then it is most
    likely that the diskgroup containing this disk is mounted by an
    active ASM instance. An error and warning is generated but
    operation proceeds normally. This option suppresses this check
    and avoids the sleep.
noi/mage                Do not create image files
-noimage: No image files will be created n the dump directory. All
    the reads specified by the read options will still be done. The
    map files may be used to find blocks on the disks themselves. In
    the map file, the count of blocks dumped, the image file sequence
    number, and the byte offset in the image file will all always be
    zero (C00000 S0000 B0000000000)
nom/ap          Do not create map or image files
-nomap: No map file is created and no image file is created. The only
    output is the report file. The -noimage option is assumed if this
    is set since an image file without a map is useless. The options
    -noscan and -noread also result in no map or image files, but
    -nomap still reads the metadata to check for I/O errors and corrupt
    blocks.
nop/rint                Do no print block contents
-noprint: This suppresses the printout of the block contents for
    blocks printed with the -print option. It is useful for getting
    just the block reports without a lot of data. This option requires
    at least one -print option.
norea/d         Shallow discovery only
-noread: This eliminates any reading of any disks at all. Only shallow
    discovery will be done. The report will end after the discovery
    section. It is an error to specify this option and specify a file
    to extract or blocks to print. It is an error to specify this
    and -fullscan.
norep/ort               Do not generate a report
-noreport: This suppresses the generation of the report file. It is
    most useful in combination with -nodir and -print to get block
    printouts without a lot of clutter. It is unnecessary to include
    this with -directory since no report is generated then anyway.
nosc/an         Deep discovery only
-noscan: This eliminates any reading of any disks after deep
    discovery. This results in just doing a deep discovery using the
    disksting parameter. The report will end after the discovery
    section. It is an error to specify this option and specify a file
    to extract. It is an error to specify this and -fullscan.
nosu/bdir               Do not create a dump directory
-nosubdir: No dump directory is created, but files are still created.
    The directory name is not written to standard out. The report file
    and any other dump or extract  files are written to the current
    directory or to the directory indicated by -parentdir. This means
    that if multiple AMDU dumps are requested using this option, the
    report file will always correspond to the last dump requested.
nov/irtual              Do not dump virtual metadata
-novirtual: This option eliminates reading of any virtual metadata.
    Only the physically addressed metadata will be read. This
    implicitly eliminates the ACD and extent maps so -noacd and
    -noxmap will be assumed.
nox/map         Do not dump extent maps
-noxmap: This option eliminates reading of the indirect extents
    containing the file extent maps. This is the bulk of the metadata
    in most diskgroups. Even the entries in the map file will be
    eliminated.
o/utput         Files to create for extract
-output : This option specifies a different file for
    writing an extracted file. The file will be overwritten if it
    already exists. This option requires that exactly one file is
    extracted via the -extract option. Required with -directory
pa/rent         Parent for dump directory
-parent : By default the dump directory is created in the
    current directory, but another directory can be specified using
    this option. The parent directory for the dump directory must
    already exist.
pr/int          Block to print
-print : This option prints one or more blocks to standard
    out. This option may be specified multiple times to print multiple
    s. The printout contains information about how each
    block was read as well as a formatted printout. Multiple blocks
    matching the same may be found when scanning the
    disks. For example there may be multiple disks that have headers
    for the same diskgroup and disk number. If the block is from a
    mirrored file then multiple copies should exist on different disks.
    If multiple copies of the same block have identical contents then
    only one formatted printout of the contents will be generated, but
    a header will be printed for each copy. A may include
    a count of sequential blocks to print. A may specify
    a block either by disk or file.
   ::= | .C
   ::= | |
        | |
   ::=
        .N.A.B
   ::=
         .D.A.B
   ::=
         .F.X.B
   ::=
         .F.V
   ::=
         .F.M
r/egistry               Dump registry files
-registry: The ASM registries will be read and dumped to the image
    file. There will be no block consistency checks since these files
    do not have ASM cache headers. To dump one specific registry
    specify -filedump and include the file object for the registry
    (e.g. DATA.255).
s/pfile         Extract usable spfile
-spfile: This causes extract to render the resulting file in a form
    that is directly usable by startup. Without this option, AMDU
    will extract the file as a regular ASM file including all ASM
    specific headers and such


Now let's take a look at the overall syntax for using AMDU is below:
.
amdu -diskstring ''
 
AMDU creates a new subdirectory with the prefix amdu_ plus date and time as naming convention after execution that houses the reports for ASM metadata.


$ amdu -diskstring '/dev/oracleasm/disks/*' -dump DG_DATA


amdu_2015_07_16_16_27_02/
WARNING: Using brute force method to determine the size of /dev/oracleasm/disks/sddlmfdrv0.
 There will be performance issues. Please check configuration to determine the cause for the failure of ioctl
The report.txt file contains the details on the ASM disk metadata for review.


******************************* AMDU Settings ********************************
ORACLE_HOME = /u01/app/grid/12.1.0/grid
System name:    Linux
Node name:      ora-ben01
Release:        2.6.32-504.el6.x86_64
Version:        #1 SMP Tue Sep 16 01:56:35 EDT 2014
Machine:        x86_64
amdu run:       19-JUL-15 16:27:02
Endianess:      1
--------------------------------- Operations ---------------------------------
       -dump DG_DATA
------------------------------- Disk Selection -------------------------------
 -diskstring '/dev/oracleasm/disks/*'
------------------------------ Reading Control -------------------------------
------------------------------- Output Control -------------------------------


********************************* DISCOVERY **********************************
----------------------------- DISK REPORT N0001 ------------------------------
                Disk Path: /dev/oracleasm/disks/sddlmk
           Unique Disk ID:
               Disk Label:
     Physical Sector Size: 512 bytes
                Disk Size: 409600 megabytes
               Group Name: DG_DATA
                Disk Name: DG_DATA_0000
       Failure Group Name: DG_DATA_0000
              Disk Number: 0
            Header Status: 3
       Disk Creation Time: 2015/07/16 19:55:33.792000
          Last Mount Time: 2015/07/18 10:24:19.387000
    Compatibility Version: 0x0c100000(12010000)
         Disk Sector Size: 512 bytes


So when you are faced with a challenging ASM issue or for general verification after a complex large scale Oracle RAC and ASM build, be sure to take advantage of these excellent ASM tools!

Oracle ASM Disks and multi path issue- Solution

So today I needed to create some new ASM disks on Hitachi enterprise storage with HDLM Hitachi Dynamic Link Manager multipathing setup with Oracle Linux 6.6 environment on a two node Oracle 12c RAC cluster. Unfortunately when I tried to create new ASM disks using the ASMLib and the Oracle ASM create disk command it failed to create the disks! I thought this to be odd because I had used the fdisk command on Linux to add a new partition on each of the LUNs presented to the OS for ASM. Permissions were correct and fdisk showed that the partitions were created for the newly provisioned LUNs to Linux.

I found a blog post that mentions a similar issue and MOS note that references the issue and workaround solution:

ASMLib: oracleasm createdisk command fails: Device '/dev/emcpowera1 is not a partition [Failed] (Doc ID 469163.1)

Now while this references EMC storage, the same issue and solution applies to other multipathing software and storage arrays such as Hitachi storage with HDLM in my case.

Instead of using the 'oracleasm createdisk' command I had to use an internal tool called asmtool that usually is called behind the scenes by Oracle tools.

# /usr/sbin/asmtool -C -l /dev/oracleasm -n VOL1 -s /dev/emcpowera1 -a force=yes

This saved me hours of frustration this week when I had to build new ASM disks and do a migration for a customer solution in a very short time period. Hopefully it will also save you hours of grief .

Getting used to Hitachi storage is a bit new for me but since I have years of EMC storage administration experience it is not too steep a learning curve and quite a bit of fun learning new things!

On Solving the Right Problem

Dear Readers,


It has been quite a while since I wrote on the blog due to some past things that occupied my personal time that had to be addressed. Recently I began to solve new performance related problems for a customer. Here is the scenario:


Oracle 12c RAC
Large enterprise SAN
Converged infrastructure


The target goal is to achieve a minimal performance level (SLA) for storage and system performance.


The customer is only receiving a low number of storage performance in terms for overall IOPS with their two node Oracle 12c RAC configuration.


So with little information to go by from past experiences, I asked how they measure such performance values.


Enter the Benchmark Tools!


The customer is using a third party performance tool to measure performance for CPU, system and storage figures. So what is this tool you might ask? Well hold on for a minute we will get to that question in a short while. My first DBA spidey sense was to get the data from the horse's mouth so to speak that is from Oracle! I logged into the Oracle 12c RAC cluster and pulled the recent AWR reports from the cluster and noticed that overall there were no serious performance issues!


It turns out that the performance tool is reporting different numbers than the Oracle database!


Hmm well that sure is very odd! I review the infrastructure between customer sites and lo and behold find that configurations are different! So that calls to mind that Oracle per se is not guilty but a more fundamental issue of apples to oranges comparisons. Which leads to my next thought process


FIND AND SOLVE THE RIGHT PROBLEM


ASK THE RIGHT QUESTION


I know this seems obvious to most of us Oracle DBA types right? Well you'd be surprised at how many customers spin their wheels attempting to solve a performance issue by mistaking the forest for the weeds. Instead of immediate jumping to conclusions, take a deep breath and step back to look at the big picture. The following comes to mind:


Examine
1. Storage configuration- disks, HBA, HBA, multi path configurations
2. Firmware and patch levels for infrastructure- servers, SAN, networks
3. Review OS configurations and releases
4. Run basic tests to collect data points- Oracle AWR, sar, vmstat, et al.




Stay tuned on my next series of blog posts on how to exactly solve these types of problems. Oh yeah and get ready, set and go for Oracle Open World this coming week!!


Cheers,
-B

Oracle OpenWorld 2014

Dear Readers,

This year's Oracle Open World 2014 was a lot of learning and meeting old and new friends.
What a fun and great conference!

The big highlights for me were:

Oracle 12c In Memory Database- in sight and everywhere this was a key mantra at the event.

Big Data SQL- another new thing that intrigues me is using SQL to manage Big Data applications with Hadoop, NoSQL and other applications.

Engineered systems- Oracle demo grounds featured Exadata, Exalogic, Big Data Appliance (BDA), etc front and center.

Last but not least, Oracle Public Cloud was a centerpiece in the keynotes and sessions. Oracle is now a serious contender to not only IBM, HP, Cisco, and EMC but to Amazon Web Services (AWS) with public cloud offerings.

Last year, Oracle introduced hands on labs (HOL) to allow participants to learn first hand new technologies aside from the standard product marketing and general conference sessions.

This year the hands on labs (HOL) were greatly expanded and I spent the majority of my time attending a dozen or so of these excellent sessions for learning first hand how to work with Oracle Public Cloud, virtualization, big data and NoSQL.

Prior to OOW, I attended the 2 day Oracle ACE Director summit and had a great experience learning first hand from Oracle's Thomas Kurian, EVP of Product Development as well as many product and engineering folks from Oracle.

Last but certainly not least, I presented at the Delphix booth on performing Oracle E-Business Suite Upgrades with Delphix and had a great attendance. Next month I am presenting at the BGOUG conference in Sofia, Bulgaria which will be an exciting event.

OOW 2014 events

Dear readers,

This year promises to be an exciting Oracle OpenWorld.

I will be presenting in the Delphix booth on using Delphix to perform upgrades with Oracle E-Business Suite. Hope to see you all there and we also will have a Clone Attack with demo software available to learn about Delphix. In addition, DBVisit will be presenting REPAttack to show demos how to replicate seamless Oracle environments with the DBVisit software.

I am headed to the Oracle ACE Director briefing so stay tuned and hope to see everyone there.

Oracle 12c RAC and Flex ASM Overview

Dear Readers,

In preparation for upcoming Oracle Open World 2014 and BGOUG 2014 (Bulgarian User Oracle Group) conferences, I will write a series of tips and tricks for new Oracle 12c RAC and ASM features. Today I would like to review the new Oracle 12c RAC Flexible ASM feature. For our discussion as a reference point you can source details in the useful My Oracle Support (http://support.oracle.com) note FAQ: Oracle Flex ASM 12c / 12.1 (Doc ID 1573137.1)

Which contains a wealth of detailed information for the setup and configuration of an Oracle 12c Flex ASM environment. So, what is Oracle 12c Flex ASM you might be asking? Well it is a new method for managing clustered storage within Oracle 12c RAC environments. In the past releases of Oracle RAC and ASM prior to 12c, one had to build individual ASM instances on each cluster node for an Oracle RAC configuration. If any single node failed, the surviving RAC nodes would take control in a quorum event managed by Oracle cluster ware. However, there were some performance and availability issues with that architecture. To remediate the weak points of the Oracle RAC and ASM environment, Oracle development re-architected the model for ASM. By allowing shared control of ASM within a cluster via a single ASM instance, flex ASM was born. Oracle provides a good architecture diagram below for the design.

Source: Oracle 12c ASM Documentation, Oracle Corporation 2014 Users, applications and database clients can now connect from a single ASM instance in the cluster with makes for better availability and additional failover capabilities. Stay tuned for future Oracle 12c RAC and ASM installments.

I am now a pilot!

Dear readers,

Last week, I passed my private pilot oral exam and flight test called a checkride and now have my private pilot license!

The checkride was challenging and intense but I had a great experience and learned a lot about myself and from the awesome examiner in Santa Rosa.

Here is a picture of me after passing my checkride with my awesome instructor:

I was able to fly from Palo Alto to Sacramento and then on to Oroville to visit family and friends for this year's Thanksgiving dinner and to avoid the nasty holiday traffic on the freeway! As I soared to over 5000 feet, I could see the massive traffic on the ground and smiled that I was able to bypass this. Flying into Oroville was fun and a very nice large airport for a very small town of less than 50,000 people.

Big Data Interview with OTN from Oracle Open World 2013

Below is my interview with the Oracle Technology Network on Big Data from this year's Oracle OpenWorld conference:

Big Data Interview at Oracle Open World

OOW 2013 Review

Dear readers,

This year was quite an exciting yet different OOW 2013 conference. Before the conference, I attended the Oracle ACE Director briefing with Oracle product management to learn the inside scoop before the show for new products. We then had a nice dinner thanks to the Oracle ACE program at the Walt Disney Museum.

At the museum, I had the opportunity to view many cool things from Walt Disney's lifetime of achievement as the father of animation.

At the start of the Oracle Openworld 2013 conference, the big announcements were that for the new 12c in memory database as well as the Oracle Public cloud.

To me the in memory database functions of Oracle 12c are the most exciting and even the bus at SF had them too!

Oracle 12c New Feature: How to backup pluggable databases

Oracle 12c introduced the new multi-tenant feature called Pluggable Databases (PDB). We will show how to take a backup of the pluggable database components in this post.

Setup for RMAN with Oracle 12c

In order to use the Oracle 12c Recovery Manager (RMAN) utility for pluggable database backups, you need to first enable archivelog mode.

Once archivelog mode is enabled, we can take a backup of the pluggable database

Now we can verify that the backup image is available from RMAN for our pluggable database

Backup for root component of Oracle 12c Pluggable Databases

In a nutshell, an Oracle 12c PDB consists of two parts: a root component and a seed component that includes the data. Earlier we performed a full database backup of the entire pluggable database but let us say that we just want to backup the root itself. We can do so with the RMAN command BACKUP DATABASE ROOT as shown in the following example:

Now let us verify the root backup for our PDB with Oracle 12c:

Stay tuned when we visit how to restore pluggable databases with RMAN and Oracle 12c!

Oracle 12c Installation on Oracle Linux 6

Dear readers, I decided to finally get around to installing Oracle 12c (12.1.0) on Oracle Linux 6 today. One key difference is the increased disk space requirements from 11g to 12c for the basic database configuration. I recall that 11gR2 requires almost 4GB of disk storage space. Now, for a basic enterprise installation with Oracle 12c, you need at least 6.4Gb of disk space!

Pre-requisites for Oracle Linux 6 with Oracle 12c

Oracle Linux 6 requirements are available online from Oracle documentation. To simplify package and OS dependency requirements, you can deploy the Oracle Linux 6 RPM available from Oracle called oracle-rdbms-server-12cR1-preinstall

Wim Coekerts has a good write up on this:

https://blogs.oracle.com/wim/entry/easily_install_oracle_rdbms_12cr1

Here is the basic screen shot of the storage requirements for 12c:

Like the previous setup screens for 11gR2 we have choice of desktop versus server class database system with Oracle so I will not bore you with these details.

Now we have the new choice to configure the database and also the new multi-tenant feature called pluggable database

One difference in the configuration lies instead of using the Oracle Enterprise Manager (OEM) console, Oracle 12c installer now asks you if you wish to use the new Oracle 12c Cloud Control. Since for sake of first setup with 12c, we are not using this option, we ignore it.

Now we are off to the races for the actual installation for 12c:

At the end, run the usual root.sh scripts that are called out for under root user.

We verify the status of the newly created Oracle 12c database:

The new Oracle 12c management console has a host of new features as shown below that we will explore in future posts:

Stay tuned!

Software Defined Networks (SDN)

One hot topic that I would like to discuss today is that of software defined networks (SDN) in respect to Oracle and other database platforms. Key vendors that provide core technologies around this area are provided by VMware, Redhat, and Oracle. First let us cover the basics of SDN.

VMware Software Defined Network

http://www.vmware.com/solutions/datacenter/software-defined-datacenter/networking.html

From VMware approach, the new vCloud Director suite provides the tools to implement an end to end software defined network and virtualized ecosystem along with robust technologies such as VMware vCenter and VMware vSphere. According to VMware the Software Data Center:

The Software-Defined Data Center: Operating Principles

Rather than mask the inherent rigidity of specialized hardware under a tangle of scripts, the Software-Defined Data center leapfrogs these constraints to change the way all data center services are delivered.

As implemented in VMware’s vCloud Suite, the virtualization principles of abstraction, pooling and automation are applied to compute, storage, networking, security, and availability. This creates the Virtual Data Center, a new construct that aggregates these software-defined services and enables intelligent, policy-based provisioning, automation, and monitoring. APIs and other connectors provide seamless extensibility to third-party platforms and public cloud services.

Below is the architecture for vCloud environment for SDN:

Taken together, these capabilities comprise VMware Cloud Infrastructure, the virtualized infrastructure at the heart of the Software-Defined Data Center. As highly capable as it is, this “engine” is not sufficient on its own. It must be fully managed, which encompasses everything from delivering access to the right services with the right approvals to ensuring the performance, compliance and efficiency of your private cloud.

These management capabilities—delivered via VMware Cloud Management—are specifically designed to provide deep insight into cloud infrastructure performance, while enabling services to be provisioned and accessed in any available cloud.

For many companies, the Software-Defined Data Center will coexist in a heterogeneous environment with multiple hypervisors, hardware from different vendors, and various public cloud services. You can abstract away this complexity by extending VMware Cloud Management beyond your vSphere-enabled Software-Defined Data Center—to impose uniform governance, control, access and self-service over your entire heterogeneous, hybrid cloud environment.

Cloud Service Provisioning: Automate the provisioning of infrastructure and applications within the Software-Defined Data Center, and beyond it across multiple clouds and platforms. Cloud Operations Management: Manage the health, risk, efficiency and compliance of your infrastructure and applications. Cloud Business Management: Govern and manage cloud services as a critical element of running IT like a business.

Oracle Software Defined Network

Oracle has a different approach to software defined networking and virtualized data centers that is more application focused along the core suite of applications that run Oracle database and Oracle systems.

http://www.oracle.com/us/products/networking/virtual-networking/sdn/overview/index.html

Below is an overview of the approach for SDN taken by Oracle:

Last but not least, completely integrated systems that merge SDN with hardware, storage, compute and network converged infrastructure are now on the market and provide turn key solutions to large customers who require a quick time to market data center such as that provided by VCE (http://www.vce.com) via the vBlock platform. A good overview of the vBlock is available below:

http://www.cisco.com/web/GR/connect2013/pdfs/014_emc_vmware_thanos.pdf

Until next time

Oracle 12c New Features and Release

Oracle 12c has been released this month and this is the biggest news since Oracle 11g came out some time back.

Below are some new features:

Oracle cloning

Oracle PDB

Oracle RAC and ASM enhancements

Oracle security enhancements

Oracle 12c New Features

As an Oracle RAC expert, the deployment features for Oracle 12c RAC and ASM are quite exciting and should make highly available scalable clusters with the new Flex cluster more fun and powerful for large enterprise data centers. Per the 12c new features for Flex RAC clusters:

"Oracle Flex Cluster is a new Oracle Clusterware based topology utilizing two types of cluster nodes: Hub Nodes and Leaf Nodes. Hub Nodes represent traditional nodes, tightly coupled using network and storage. Leaf Nodes are a new type of node that runs a lighter weight stack and does not require direct shared storage connectivity."

I also like the new 12c policy based approach to database and RAC deployments:

"Oracle Grid Infrastructure allows running multiple applications in one cluster. Using a policy-based approach, the workload introduced by these applications can be allocated across the cluster using a policy set. In addition, a policy set enables different policies to be applied to the cluster over time as required. Policy sets can be defined using a web-based interface or a command-line interface.

Hosting various workloads in the same cluster helps to consolidate the workloads into a shared infrastructure that provides high availability and scalability. Using a centralized policy-based approach allows for dynamic resource reallocation and prioritization as the demand changes."

Stay tuned for exciting new developments as I test out the many new 12c features from Oracle. Oracle has truly raised the bar once again for enterprise database and application technology! Bravo!

Oracle RAC Timezone woes and solution

Recently had a timezone issue with an Oracle 11gR2 RAC cluster. Even though I had set the host time and date, for some reason it did not get propagated to the Oracle RAC configuration.

There are couple of ways to set timezone for an Oracle 11.2 RAC cluster. First is the easy way by using srvctl commands:

srvctl setenv database -d -T =

srvctl setenv database -d racdb -T TZ=US/Eastern

In the first case, if this did not work, we need to edit a configuration file called s_crsconfig_{racnodename}_env.txt located under the $GRID_HOME/crs/install directory.

Here is my example for the edits performed of the above file on both RAC nodes, using first node as sample:

[root@racnode1$ vi /oracle/11.2.0/grid/crs/install/s_crsconfig_racnode1_env.txt

### This file can be used to modify the NLS_LANG environment variable, which determines the charset to be used for messages.

### For example, a new charset can be configured by setting NLS_LANG=JAPANESE_JAPAN.UTF8

### Do not modify this file except to change NLS_LANG, or under the direction of Oracle Support Services

TZ=US/Pacific

NLS_LANG=AMERICAN_AMERICA.AL32UTF8

TNS_ADMIN=

ORACLE_BASE=

Restart the Oracle 11.2 Clusterware services and database and run srvctl to confirm setting changes.

Migrating single instance Oracle 11g database to Oracle 11gR2 RAC and ASM

Recently over the past few years, I've been involved with customer engagements to migrate legacy single instance environmnents to Oracle 11gR2 RAC and ASM. In this post, I will detail out the high level steps on how to perform this for ERP environments with SAP and Oracle.

I will spend a few blog posts on this topic due to the complex nature of migrating from single instance to Oracle RAC.

1. Migrate single instance Oracle non-ASM to Oracle 11gR2 RAC and ASM

2. Migrate single instance Oracle ASM to Oracle 11gR2 RAC.

SAP ECC 7.x now allows you to directly install Oracle ASM in the new Oracle 11gR2 Grid infrastructure environment. This means that you can build and configure your Oracle 11gR2 ASM disks and provision them without having to first migrate with RMAN from single instance Oracle to ASM.

Case 1: Migrate single instance Oracle non-ASM to Oracle 11gR2 RAC and ASM

The steps to migrate Oracle single instance to Oracle 11gR2 RAC are as follows:

1. Install Oracle 11gR2 Grid infrastructure

2. Provision disks for ASM

3. Configure and create ASM instances

4. Use RMAN to transfer non-ASM Oracle database to ASM configuration

5. Install Oracle 11gR2 RAC database software in new ORACLE_HOME

6. Perform cluster infrastructure configuration for public, private, VIP, and SCAN requirements per Oracle 11gR2 documentation

7. Install Oracle 11gR2 RAC grid infrastructure for a cluster

8. Use rconfig to complete the migration to Oracle 11gR2 RAC

Note: for SAP with Oracle RAC, you MUST use Oracle 11.2.0.2 or later release to obtain support from SAP. ASM must use a ACFS fileystem as well.

Case 2: Migrate single instance Oracle ASM to Oracle 11gR2 RAC

The steps for case 2 are as follows for an environment with Oracle single instance and ASM:

1. Install Oracle 11gR2 RAC database software in new ORACLE_HOME

2. Run the rootcrs.pl -deconfig -force script for single instance GRID_HOME with Oracle ASM

3. Install Oracle 11gR2 RAC Grid infrastructure for new GRID_HOME on cluster node hosts

4. Add ASM disks from single instance to cluster services with ASMCA utility

5. Migrate single instance with rconfig scripts

I will add details in future posts as the tasks are very complex and involved. For now, I wanted to give a high level summary since this is not well documented.

First solo flight and my quest to become a private pilot for fun!

This year I started my journey to accomplish a childhood dream of earning my private pilot license to have the freedom and skill to fly aircraft in my free time. This past week I completed my first solo flight successfully at KPAO airport in Palo Alto! It was exciting when my instructor, Gordon Reade at Sundance Flying Club stepped out after signing my logbook to solo and wished me luck before I taxi and took off from Runway 31. After the tower granted me clearance to take off, I lined up and used full throttle to rotate (Vr) at about 55kts in the Cessna 172SP aircraft (837SP) to complete my first solo flight of three takeoffs and landings in the traffic pattern over the bay at KPAO airport. It was fun and best of all I was relaxed and confident in my newly acquired skills. All three landings were smooth on touchdown and my instructor congratulated me on a solo flight well done. Now I look forward to completing my cross country flights and passing my checkride to receive my private pilot license!

NoCOUG Events for April and May 2013

As the new conference director for the Northern California Oracle User Group (NoCOUG), we have many exciting events planned for the Oracle user community. Next week, on April 16th we have a meetup for cloud computing with Oracle technology at the Hacker Dojo in Mountain View, CA: http://www.meetup.com/Northern-California-Oracle-Users-Group/events/112651522/?a=me1_grp&rv=me1&_af_eid=112651522&_af=event In May, we have the NoCOUG spring/summer conference planned with events for Big Data and Oracle technology. http://www.technicalconferencesolutions.com/pls/caat/caat_abstract_reports.schedule?conference_id=124 To attend this exciting conference you may sign up below and RSVP: http://www.nocoug.org/rsvp.html So if you are in northern California please come learn the latest greatest things about all things Oracle.

End of 2012

Year in review As we end 2012, in closing I have not posted much due to work and many activities wrapping up the year. My new Oracle R12 EBS book is available and ready for folks to learn the tips and tricks for Oracle R12 Financials from Oracle Press. Each year I pursue a new goal and try to master or improve a skill or ability. This keeps me sharp and motivated and happy. Last year, I learned to scuba dive. Right now, I am learning to fly! My goal by 2013 is to pass my checkride and obtain my private pilot license. I have always had a passion for aviation and aeronautics. I am doing this purely as a hobby and do not want to be a professional pilot. For one, the pay is quite low until one has a decade of experience and two, I like working in technology as a job better than to fly planes for a living. I also will be focused on bring new events to the NoCOUG Oracle User Group in the bay area and northern California for Oracle users who want to learn new developments in database technology from Oracle. Happy Holidays and best wishes to all for a safe, peaceful and prosperous 2013! Cheers, Ben