ASM Diskgroup Analysis Tip

Recently I have been performing multiple Oracle 12c RAC setups and digging deep into ASM internals. Besides query the ASM environment via the V$ASM_DISK and V$ASM_DISKGROUP dynamic tables, these lesser known command line tools provide detailed troubleshooting assistance for solving difficult ASM issues with your Oracle environments.


As a reference point for today's discussion, I found a very useful support note from Oracle that explains all three utilities to analyze Oracle ASM environments:


ASM tools used by Support : KFOD, KFED, AMDU
 (Doc ID 1485597.1)


So let's take a look at each one in terms of how and why you would use these ASM diagnostic tools.


kfod


Kfod is a quick and dirty tool to examine the current ASM configuration for ownership, sizes and device mappings. Ensure that your kfod is in the local path for $GRID_HOME/bin/kfod to allow execution of the tool.


$ which kfod
/u01/app/grid/12.1.0/grid/bin/kfod


To get help with kfod use the help=y option:


$ kfod help=y
_asm_a/llow_only_raw_disks              KFOD allow only raw devices [_asm_allow_only_raw_disks=(TRUE)/FALSE]
_asm_l/ibraries         ASM Libraries[_asm_libraries=lib1,lib2,...]
_asms/id                ASM Instance[_asmsid=sid]
_b/oot          Running in pre-install env (boot=TRUE/FALSE)
_f/lexinfo              Provide flexinfo      (_flexinfo=TRUE/FALSE)
_p/atch_lib             Patchlib [_patch_lib=]
_u/ser          OS Username
asm_/diskstring         ASM Diskstring [asm_diskstring=discoverystring, discoverystring ...]
asmc/ompatibility               Include diskgroup ASM compatibility [asmcompatibility=TRUE/(FALSE)]
cli/ent_cluster         client cluster name
clus_/version           cluster version
clust/er                KFOD cluster [cluster=TRUE/(FALSE)]
db_/unique_name         db_unique_name for ASM instance[db_unique_name=dbname]
dbc/ompatibility                Include diskgroup DB compatibility [dbcompatibility=TRUE/(FALSE)]
disk_/access            Disk access method [disk_access=DIRECT/(INDIRECT)]
disks           Disks to discover [disks=raw,asm,badsize,all]
ds/cvgroup              Include group name [dscvgroup=TRUE/(FALSE)]
f/orce          Force option to delete files (force=TRUE/FALSE)
g/roup          Disks in diskgroup [group=diskgroup]
h/ostlist               hostlist[hostlist=host1,host2,...]
metadata_a/usize                AU Size for Metadata Size Calculation
metadata_c/lients               Client Count for Metadata Size Calculation
metadata_d/isks         Disk Count for Metadata Size Calculation
metadata_n/odes         Node Count for Metadata Size Calculation
metadata_r/edundancy            Redundancy for Metadata Size Calculation
na/me           Include disk name [name=TRUE/(FALSE)]
no/hdr          KFOD header suppression [nohdr=TRUE/(FALSE)]
ol/r            Import credentials to OLR [olr=TRUE/(FALSE)]
op              KFOD options type [OP=DISKS/CANDIDATES/MISSING/GROUPS/INSTS/VERSION/PATCHES/PATCHLVL/CLIENTS/RM/RMVERS/DFLTDSTR/GPNPDSTR/METADATA/CREDCRECLUS/GETCLSTYPE/CREDEXPORT/GETASMGUID/CREDDELCLUS/CREDVERIFY/UPGRADEVERIFY/ALL]
p/file          ASM parameter file [pfile=parameterfile]
r/im_disk_access                Rim disk access method [rim_disk_access=DIRECT/(INDIRECT)]
s/tatus         Include disk header status [status=TRUE/(FALSE)]
v/erbose                KFOD verbose errors [verbose=TRUE/(FALSE)]
w/rap           wrap file for credentials
KFOD-01000: USAGE: kfod op= asm_diskstring=... | pfile=...


kfod status=TRUE asm_diskstring='' disk=ALL


$ kfod verbose=true, disks=all status=true op=disks \
> asm_diskstring='/dev/oracleasm/disks/*'
WARNING: Using brute force method to determine the size of /dev/oracleasm/disks/sddlmfdrv0.
 There will be performance issues. Please check configuration to determine the cause for the failure of ioctl
--------------------------------------------------------------------------------
 Disk          Size Header    Path                                     User     Group
================================================================================
   1:     409600 Mb MEMBER    /dev/oracleasm/disks/sddlma             grid     oinstall
   2:     409600 Mb MEMBER    /dev/oracleasm/disks/sddlmb             grid     oinstall
   3:     409600 Mb MEMBER    /dev/oracleasm/disks/sddlmc             grid     oinstall
   4:     409600 Mb MEMBER    /dev/oracleasm/disks/sddlmd             grid     oinstall
   5:     409600 Mb MEMBER    /dev/oracleasm/disks/sddlme             grid     oinstall
   6:     409600 Mb MEMBER    /dev/oracleasm/disks/sddlmf             grid     oinstall
   7:     409600 Mb MEMBER    /dev/oracleasm/disks/sddlmg             grid     oinstall




kfed


Kfed differs from the other tools in that it shows lower level metadata mappings for block size and endian counts for ASM disks and diskgroups. The help is provided by entering kfed with no options:


$ kfed
as/mlib         ASM Library [asmlib='lib']
aun/um          AU number to examine or update [AUNUM=number]
aus/z           Allocation Unit size in bytes [AUSZ=number]
blkn/um         Block number to examine or update [BLKNUM=number]
blks/z          Metadata block size in bytes [BLKSZ=number]
ch/ksum         Update checksum before each write [CHKSUM=YES/NO]
cn/t            Count of AUs to process [CNT=number]
de/v            ASM device to examine or update [DEV=string]
dm/pall         Don't suppress repeated lines when dumping corrupt blocks [DMPALL=YES/NO]
o/p             KFED operation type [OP=READ/WRITE/MERGE/REPAIR/NEW/FORM/FIND/STRUCT]
p/rovnm         Name for provisioning purposes [PROVNM=string]
s/eek           AU number to seek to [SEEK=number]
te/xt           File name for translated block text [TEXT=string]
ty/pe           ASM metadata block type number [TYPE=number]
v/erb           Verbose execution [verbose=YES|NO]




To run kfed, you need to specify the ASM disk mapping including the fully qualified device path as shown in the below example:


$ kfed read /dev/oracleasm/disks/sddlmbd
kfbh.endian:                          1 ; 0x000: 0x01
kfbh.hard:                          130 ; 0x001: 0x81
kfbh.type:                            1 ; 0x002: KFBTYP_DISKHEAD
kfbh.datfmt:                          1 ; 0x003: 0x01
kfbh.block.blk:                       0 ; 0x004: blk=0
kfbh.block.obj:              2147483675 ; 0x008: disk=22
kfbh.check:                  2341676564 ; 0x00c: 0x8b932613
kfbh.fcn.base:                        0 ; 0x010: 0x00000000
kfbh.fcn.wrap:                        0 ; 0x014: 0x00000000
kfbh.spare1:                          0 ; 0x018: 0x00000000





amdu


Last but not least is probably one of my favorite ASM command line tools called amdu which provides a nicely formatted report for a specific ASM diskgroup that consolidates most of the information generated by kfod and kfed into a simple file called report.txt


For the many options with amdu to get help make sure to specify the help=y parameter as shown below:


$ amdu help=y
al/lides                Dump indirect blks unconditionally
-allides: AMDU ordinarily skips over empty indirect blocks. Specifying
    this option tells AMDU to dump those blocks unconditionally. Be
    warned that this can make the resulting AMDU dump quite large.
au/size         AU size for corrupt disks
-ausize : This option must be set when -baddisks is set. It
    must be a power of 2. This size is required to scan a disk looking
    for metadata, and it is normally read from the disk header. The
    value applies to all disks that do not have a valid header. The
    value from the disk header will be used if a valid header is
    found.
ba/ddisks               Include disks with bad headers
-baddisks :  Normally disks with bad disk headers, or that
    look like they were never part of a disk group, will not be
    scanned. This option forces them to be scanned anyway and to be
    considered part of the given diskgroup. This is most useful when
    a disk header has been damaged. The disk will still need to have
    a valid allocation table to drive the scan unless -fullscan is
    used. In any case at least one block in the first two AUs must be
    valid so that the disk number can be determined. The options
    -ausize and -blksize are required since these values are normally
    fetched from the disk header. If the diskgroup uses external
    redundancy then -external should be specified. These values will
    be compared against any valid disks found in the diskgroup and
    they must be the same.
bl/ksize                ASM block size for corrupt disks
-blksize : This option must be set when -baddisks is set. It
    must be a power of 2. This size is required to scan a disk looking
    for metadata, and it is normally read from the disk header. The
    value applies to all disks that do not have a valid header. The
    value from the disk header will be used if a valid header is
    found.
c/ompare                Compare file mirrors
-compare: This option only applies to file extraction from a normal or
    high redundancy disk group. Every extent that is mirrored on more
    than one discovered disk will have all sides of its mirror
    compared. If they are not identical a message will be reported
    on standard error and the report file. The message will indicate
    which copy was extracted. A count of the blocks that are not
    identical will be in the report file.
dir/ectory              Directory from previous dump
-directory : This option completely eliminates the discovery
    phase of operation. It specifies the name of a dump directory from
    a previous run of AMDU. The report file and map files are read
    instead of doing a discovery and scan. The parsing of these ASCII
    files is very dependent on them being exactly as written by AMDU.
    AMDU is unlikely to work properly if they have been modified by
    a text editor, or if some of the files are missing or truncated.
    Note that the directory may be a copy FTPed from another
    machine. The other machine may even be a different platform
    with a different endianess.
dis/kstring             Diskstring for discovery
 -diskstring : By default the null string is used for
    discovery. The null string should discover all disks the user has
    access to. Many installations specify an asm_diskstring parameter
    for their ASM instance. If so that parameter value should be given
    here. Multiple discovery strings can be specified by multiple
    occurrences of -diskstring . Beware of shell syntax
    conflicts with discovery strings. Diskstrings are usually the same
    syntax the shell uses for expanding path names on command lines so
    they will most likely need to be enclosed in single quotes.
du/mp           Diskgroups to dump
-dump : This option specifies the name of a diskgroup to
    have its metadata dumped. This option may be specified multiple
    times to dump multiple diskgroups. If the diskgroup name is ALL
    then all diskgroups encountered will be dumped. The diskgroup name
    is not case sensitive, but will be converted to uppercase for all
    reports. If this option is not specified then no map or image
    files will be created, but -extract and -print may still work.
exc/lude                Disks to exclude
-exclude : Multiple exclude options may be specified. These
    strings are used for discovery just like the values for diskstring.
    Only shallow discovery is done on these diskstrings. Any disks
    found in the exclude discovery will not be accessed. If they are
    also discovered using the -diskstring strings, then the report will
    include the information from shallow discovery along with a message
    indicating the disk was excluded.
exte/rnal               Assume external redundancy
-external: Normally AMDU determines the diskgroup redundancy from the
    disk headers. However this is not possible with the -baddisks
    option. It is assumed that the redundancy of the -baddisks
    diskgroup is normal or high unless this option is given to specify
    external redundancy.
extr/act                Files to extract
-extract .: This extracts the numbered file
    from the named diskgroup, case insensitive. This option may be
    specified multiple times to extract multiple files. The extracted
    file is placed in the dump directory under the name
    _.f  where is the diskgroup name
    in uppercase, and is the file number. The -output option
    may be used to write the file to any location. The extracted file
    will appear to have the same contents it would have if accessed
    through the database. If some portion of the file is unavailable
    then that portion of the output file will be filled with
    0xBADFDA7A, and a message will appear on stderr.
fi/ledump               Dump files rather than extract
-filedump: This option causes the file objects in the command line to
    have their blocks dumped to the image files rather than extracted.
    This can be combined with the -novirtual option to selectively
    dump only some of the metadata files. It may also be used to dump
    user files (number >= 256) so that all mirrored copies can be
    examined.
fo/rmer         Include dropped disks
-former: Normally disks marked as former are not scanned, but this
    option will scan them and include their contents in the output.
    This is useful when it is necessary to look at the contents of a
    disk that was dropped. Note that dropped normal disks will not have
    any entries in their allocation tables and thus only the physically
    addressed extents will be dumped. Force dropped disks will not have
    status former in their disk headers and are not affected by this
    option. However if DROP DISKGROUP is used, the disks will have the
    contents as of the time of the drop, and will be in status former.
    Thus this option is useful for extracting files from a dropped
    diskgroup.
fu/llscan               Scan entire disk
-fullscan: This option reads every AU on the disk and looks at the
    contents of the AU rather than limiting the AU's read based on the
    allocation table. This is useful when the allocation table is
    corrupt or needs recovery. An AU will be written to the image file
    if it starts with a block that contains a valid ASM block header.
    The file and extent information for the map will be extracted from
    the block header. Physically addressed metadata will be dumped
    regardless of its contents. This option is incompatible with
    extracting a file. It is an error to specify -extract with this
    option. Note that this option is likely to find old garbage
    metadata in unallocated AU's since there is no means of
    determining what is allocated. Thus there may be many different
    copies of the same block, possibly of different versions.
h/ex            Always print block contents in hex
-hex: This prints the block contents in hex without attempting to print
    them as ASM metadata. This is useful when the block is known to not
    be ASM metadata. It avoids the ASM block header dump and ensures
    the block is not accidentally interpreted as ASM metadata. This
    option requires at least one -print option.
noa/cd          Do not dump ACD
-noacd: This option limits the dumping of the Active Change Directory
    to just the control blocks that contain the checkpoint. There is
    126 MB of ACD per ASM instance (42 MB for external redundancy). It
    is normally of no interest if there has been a clean shutdown or
    no updates for a while. This option avoids dumping a lot of
    unimportant data. The blocks will still be read and checked for
    corruption. The map file will still contain entries for the ACD
    extents, but the block counts will be zero.
nod/ir          Do not create a dump directory
-nodir: No dump directory is created, and no files are created in it.
    The directory name is not written to standard out. The report file
    is written to standard out before any block printouts from any
    -print options.  This option conflicts with -filedump. It is an
    error to specify this and extract a file to the dump directory.
noe/xtract              Do not create extracted file
-noextract: This prevents files from being extracted to an output
    file, but the file will be read and any errors in selecting the
    correct output will be reported. This is most useful in
    combination with the -compare option.
noh/eart                Do not check for heartbeat
-noheart: Normally the heartbeat block will be saved at discovery time
    and checked when the disk is scanned. A sleep is added between
    discovery and scanning to ensure there is time for the heartbeat
    to be written. If the heartbeat block changes then it is most
    likely that the diskgroup containing this disk is mounted by an
    active ASM instance. An error and warning is generated but
    operation proceeds normally. This option suppresses this check
    and avoids the sleep.
noi/mage                Do not create image files
-noimage: No image files will be created n the dump directory. All
    the reads specified by the read options will still be done. The
    map files may be used to find blocks on the disks themselves. In
    the map file, the count of blocks dumped, the image file sequence
    number, and the byte offset in the image file will all always be
    zero (C00000 S0000 B0000000000)
nom/ap          Do not create map or image files
-nomap: No map file is created and no image file is created. The only
    output is the report file. The -noimage option is assumed if this
    is set since an image file without a map is useless. The options
    -noscan and -noread also result in no map or image files, but
    -nomap still reads the metadata to check for I/O errors and corrupt
    blocks.
nop/rint                Do no print block contents
-noprint: This suppresses the printout of the block contents for
    blocks printed with the -print option. It is useful for getting
    just the block reports without a lot of data. This option requires
    at least one -print option.
norea/d         Shallow discovery only
-noread: This eliminates any reading of any disks at all. Only shallow
    discovery will be done. The report will end after the discovery
    section. It is an error to specify this option and specify a file
    to extract or blocks to print. It is an error to specify this
    and -fullscan.
norep/ort               Do not generate a report
-noreport: This suppresses the generation of the report file. It is
    most useful in combination with -nodir and -print to get block
    printouts without a lot of clutter. It is unnecessary to include
    this with -directory since no report is generated then anyway.
nosc/an         Deep discovery only
-noscan: This eliminates any reading of any disks after deep
    discovery. This results in just doing a deep discovery using the
    disksting parameter. The report will end after the discovery
    section. It is an error to specify this option and specify a file
    to extract. It is an error to specify this and -fullscan.
nosu/bdir               Do not create a dump directory
-nosubdir: No dump directory is created, but files are still created.
    The directory name is not written to standard out. The report file
    and any other dump or extract  files are written to the current
    directory or to the directory indicated by -parentdir. This means
    that if multiple AMDU dumps are requested using this option, the
    report file will always correspond to the last dump requested.
nov/irtual              Do not dump virtual metadata
-novirtual: This option eliminates reading of any virtual metadata.
    Only the physically addressed metadata will be read. This
    implicitly eliminates the ACD and extent maps so -noacd and
    -noxmap will be assumed.
nox/map         Do not dump extent maps
-noxmap: This option eliminates reading of the indirect extents
    containing the file extent maps. This is the bulk of the metadata
    in most diskgroups. Even the entries in the map file will be
    eliminated.
o/utput         Files to create for extract
-output : This option specifies a different file for
    writing an extracted file. The file will be overwritten if it
    already exists. This option requires that exactly one file is
    extracted via the -extract option. Required with -directory
pa/rent         Parent for dump directory
-parent : By default the dump directory is created in the
    current directory, but another directory can be specified using
    this option. The parent directory for the dump directory must
    already exist.
pr/int          Block to print
-print : This option prints one or more blocks to standard
    out. This option may be specified multiple times to print multiple
    s. The printout contains information about how each
    block was read as well as a formatted printout. Multiple blocks
    matching the same may be found when scanning the
    disks. For example there may be multiple disks that have headers
    for the same diskgroup and disk number. If the block is from a
    mirrored file then multiple copies should exist on different disks.
    If multiple copies of the same block have identical contents then
    only one formatted printout of the contents will be generated, but
    a header will be printed for each copy. A may include
    a count of sequential blocks to print. A may specify
    a block either by disk or file.
   ::= | .C
   ::= | |
        | |
   ::=
        .N.A.B
   ::=
         .D.A.B
   ::=
         .F.X.B
   ::=
         .F.V
   ::=
         .F.M
r/egistry               Dump registry files
-registry: The ASM registries will be read and dumped to the image
    file. There will be no block consistency checks since these files
    do not have ASM cache headers. To dump one specific registry
    specify -filedump and include the file object for the registry
    (e.g. DATA.255).
s/pfile         Extract usable spfile
-spfile: This causes extract to render the resulting file in a form
    that is directly usable by startup. Without this option, AMDU
    will extract the file as a regular ASM file including all ASM
    specific headers and such


Now let's take a look at the overall syntax for using AMDU is below:
.
amdu -diskstring ''
 
AMDU creates a new subdirectory with the prefix amdu_ plus date and time as naming convention after execution that houses the reports for ASM metadata.


$ amdu -diskstring '/dev/oracleasm/disks/*' -dump DG_DATA


amdu_2015_07_16_16_27_02/
WARNING: Using brute force method to determine the size of /dev/oracleasm/disks/sddlmfdrv0.
 There will be performance issues. Please check configuration to determine the cause for the failure of ioctl
The report.txt file contains the details on the ASM disk metadata for review.


******************************* AMDU Settings ********************************
ORACLE_HOME = /u01/app/grid/12.1.0/grid
System name:    Linux
Node name:      ora-ben01
Release:        2.6.32-504.el6.x86_64
Version:        #1 SMP Tue Sep 16 01:56:35 EDT 2014
Machine:        x86_64
amdu run:       19-JUL-15 16:27:02
Endianess:      1
--------------------------------- Operations ---------------------------------
       -dump DG_DATA
------------------------------- Disk Selection -------------------------------
 -diskstring '/dev/oracleasm/disks/*'
------------------------------ Reading Control -------------------------------
------------------------------- Output Control -------------------------------


********************************* DISCOVERY **********************************
----------------------------- DISK REPORT N0001 ------------------------------
                Disk Path: /dev/oracleasm/disks/sddlmk
           Unique Disk ID:
               Disk Label:
     Physical Sector Size: 512 bytes
                Disk Size: 409600 megabytes
               Group Name: DG_DATA
                Disk Name: DG_DATA_0000
       Failure Group Name: DG_DATA_0000
              Disk Number: 0
            Header Status: 3
       Disk Creation Time: 2015/07/16 19:55:33.792000
          Last Mount Time: 2015/07/18 10:24:19.387000
    Compatibility Version: 0x0c100000(12010000)
         Disk Sector Size: 512 bytes


So when you are faced with a challenging ASM issue or for general verification after a complex large scale Oracle RAC and ASM build, be sure to take advantage of these excellent ASM tools!