Finishing the Oracle 11gR2 Grid Infrastructure setup

In the previous post, we discussed how to resolve installation issues for time synchronization (NTP) between our cluster nodes as well as how to configure shared disks for 11gR2 RAC by using iSCSI. This post concludes a summary of the steps with our 11gR2 Grid setup. We need to run two scripts, oraInstRoot.sh and root.sh on both cluster nodes.

Open a new terminal shell window on the first 11gR2 cluster node and as root user,
execute the script called orainstRoot.sh located under the /u01/app/oraInventory directory.




This script will set the correct permissions for the configuration.

Next as root user, execute the script called root.sh located under the
/u02/app/11.2.0/grid directory as shown below.

This will take some time to run and will startup the clusterware processes including
CRS and OHASD (Oracle HA service).






My advice is to allocate at least 2.5Gb of RAM memory per VM (Virtual Machine) cluster node if you are using virtualization to setup 11gR2 RAC. The big change with 11gR2 RAC is that consumes at least 10x more memory and is a memory pig so you need to account for this major change. This is big difference from 11gR1 which does consume as much memory.

Shared disks not accessible during 11gR2 RAC Grid Infrastructure Installation

During the initial setup of the shared disks on my iSCSI SAN for the 11gR2 RAC cluster nodes,
the Oracle 11gR2 Grid Infrastructure complained that it could not access the LUNS for the iSCSI storage. To verify and resolve this issue, I had to run the discovery commands for iSCSI as shown below.

First, on the iSCSI SAN we need to check the status for the LUNS:

To do so, we issue the Linux iSCSI command tgtadm --lld iscsi --op show --mode target

root@san ~]# tgtadm --lld iscsi --op show --mode target
Target 1: iqm.mgmt.volumes-san
System information:
Driver: iscsi
State: ready
I_T nexus information:
I_T nexus: 2
Initiator: iqn.1994-05.com.redhat:7d8594f48e2a
Connection: 0
IP Address: 1.99.1.1
I_T nexus: 3
Initiator: iqn.1994-05.com.redhat:6cc883d783b6
Connection: 0
IP Address: 1.99.1.2
LUN information:
LUN: 0
Type: controller
SCSI ID: IET 00010000
SCSI SN: beaf10
Size: 0 MB
Online: Yes
Removable media: No
Backing store type: rdwr
Backing store path: None
LUN: 1
Type: disk
SCSI ID: IET 00010001
SCSI SN: beaf11
Size: 2147 MB
Online: Yes
Removable media: No
Backing store type: rdwr
Backing store path: /disks/disk1.dat
LUN: 2
Type: disk
SCSI ID: IET 00010002
SCSI SN: beaf12
Size: 2147 MB
Online: Yes
Removable media: No
Backing store type: rdwr
Backing store path: /disks/disk2.dat
LUN: 3
Type: disk
SCSI ID: IET 00010003
SCSI SN: beaf13
Size: 2147 MB
Online: Yes
Removable media: No
Backing store type: rdwr
Backing store path: /disks/disk3.dat
LUN: 4
Type: disk
SCSI ID: IET 00010004
SCSI SN: beaf14
Size: 2147 MB
Online: Yes
Removable media: No
Backing store type: rdwr
Backing store path: /disks/disk4.dat
LUN: 5
Type: disk
SCSI ID: IET 00010005
SCSI SN: beaf15
Size: 2147 MB
Online: Yes
Removable media: No
Backing store type: rdwr
Backing store path: /disks/disk5.dat
LUN: 6
Type: disk
SCSI ID: IET 00010006
SCSI SN: beaf16
Size: 2147 MB
Online: Yes
Removable media: No
Backing store type: rdwr
Backing store path: /disks/disk6.dat
LUN: 7
Type: disk
SCSI ID: IET 00010007
SCSI SN: beaf17
Size: 2147 MB
Online: Yes
Removable media: No
Backing store type: rdwr
Backing store path: /disks/disk7.dat
LUN: 8
Type: disk
SCSI ID: IET 00010008
SCSI SN: beaf18
Size: 2147 MB
Online: Yes
Removable media: No
Backing store type: rdwr
Backing store path: /disks/disk8.dat
LUN: 9
Type: disk
SCSI ID: IET 00010009
SCSI SN: beaf19
Size: 2147 MB
Online: Yes
Removable media: No
Backing store type: rdwr
Backing store path: /disks/disk9.dat
LUN: 10
Type: disk
SCSI ID: IET 0001000a
SCSI SN: beaf110
Size: 2147 MB
Online: Yes
Removable media: No
Backing store type: rdwr
Backing store path: /disks/disk10.dat
LUN: 11
Type: disk
SCSI ID: IET 0001000b
SCSI SN: beaf111
Size: 2147 MB
Online: Yes
Removable media: No
Backing store type: rdwr
Backing store path: /disks/disk11.dat
LUN: 12
Type: disk
SCSI ID: IET 0001000c
SCSI SN: beaf112
Size: 2147 MB
Online: Yes
Removable media: No
Backing store type: rdwr
Backing store path: /disks/disk12.dat
Account information:
ACL information:
ALL
[root@san ~]#

Good, so on my iSCSI SAN I can see the LUNs. Next step is to initiate discovery on the second RAC node that cannot see the shared disks.

[root@vxnode2 ~]# iscsiadm -m discovery -t sendtargets -p san.mgmt.example.com
1.99.1.254:3260,1 iqm.mgmt.volumes-san

You should see the status as shown above with details for the SAN. If not you need to run the rediscovery process and check to ensure that the iscsi services are running on the cluster node. Next verify that the disks are accessible and have the correct ownership and privileges granted:

[root@vxnode2 ~]# ls -l /dev/sd*
brw-r----- 1 root disk 8, 0 May 16 03:28 /dev/sda
brw-r----- 1 root disk 8, 1 May 16 03:29 /dev/sda1
brw-r----- 1 root disk 8, 2 May 16 03:28 /dev/sda2
brw-r----- 1 oracle oinstall 8, 16 May 16 03:30 /dev/sdb
brw-r----- 1 oracle oinstall 8, 32 May 16 03:30 /dev/sdc
brw-r----- 1 oracle oinstall 8, 48 May 16 03:30 /dev/sdd
brw-r----- 1 oracle oinstall 8, 64 May 16 03:30 /dev/sde
brw-r----- 1 oracle oinstall 8, 80 May 16 03:30 /dev/sdf
brw-r----- 1 oracle oinstall 8, 96 May 16 03:30 /dev/sdg
brw-r----- 1 oracle oinstall 8, 112 May 16 03:30 /dev/sdh
brw-r----- 1 oracle oinstall 8, 128 May 16 03:30 /dev/sdi
brw-r----- 1 oracle oinstall 8, 144 May 16 03:30 /dev/sdj
brw-r----- 1 oracle oinstall 8, 160 May 16 03:30 /dev/sdk
brw-r----- 1 oracle oinstall 8, 176 May 16 03:30 /dev/sdl
brw-r----- 1 oracle oinstall 8, 192 May 16 03:30 /dev/sdm

These should match for both cluster nodes.

Now we can kick off the installation for 11gR2 RAC as shown below.


Are you ready for a coffee break? I sure am! Ok so this will take a LONG time to run since I am using VMWare Fusion on my 4Gb Macbook Pro and it will run for half hour or so.


I will cover how to setup an iSCSI SAN for Oracle 11gR2 RAC in a future blog post.
My Oracle Support has a nice walk through guide on how to configure Open Filer iSCSI for Oracle RAC environments.

Using Openfiler iSCSI with an Oracle RAC database on Linux [ID 371434.1]

NTP configuration issues with Oracle 11gR2 RAC setup

During the Grid infrastructure setup for Oracle 11gR2 RAC, I ran into a weird error with ntp not working with the Grid
infrastructure installer.





The solution thanks to he Grumpy DBA, better known as Jay Caviness

http://grumpy-dba.com/2009/09/04/slewing-the-11gr2-dragon/

is to edit ntp configuration on both cluster nodes and restart ntp services.

To do so, logon to each node as root and edit the /etc/sysconfig/ntpd configuration file to make change for options settings as shown below:

[root@vxnode2 ntp]# cat /etc/sysconfig/ntpd
# Drop root to id 'ntp:ntp' by default.
OPTIONS="-u ntp:ntp -p /var/run/ntpd.pid"

# Set to 'yes' to sync hw clock after successful ntpdate
SYNC_HWCLOCK=no

# Additional options for ntpdate
NTPDATE_OPTIONS=""
OPTIONS="-x -u ntp:ntp -p /var/run/ntpd.pid"


You will need to restart ntp services for this to take affect:

root@vxnode2 ntp]# service ntpd stop
Shutting down ntpd: [ OK ]
[root@vxnode2 ntp]# service ntpd start
ntpd: Synchronizing with time server: [ OK ]
Starting ntpd: [ OK ]
[root@vxnode2 ntp]# ps -ef|grep ntp
root 7363 1 0 03:51 ? 00:00:00 ntpd -x -u ntp:ntp -p /var/run/ntpd.pid

Now we can move forward in the 11gR2 RAC Grid Infrastructure setup.

Be aware that since we are using virtualization for 11gR2 RAC with VMware, you may need to restart the ntp services each time that you power on the VMs. Another option around this issue with ntp is to use the Oracle 11gR2 RAC cluster synchronization services (CSS) instead of ntp to let Oracle handle the network synchronization services for the cluster.

Configure Cluster Time Synchronization Service - (CTSS)

If you want to use Cluster Time Synchronization Service to do this, you need to remove your ntp configuration and shutdown the ntp services on both cluster nodes.

To deactivate the NTP service, you need to shutdown and disable the existing ntpd service, and remove the ntp.conf file. To complete these steps on CentOS Linux, run the following commands as the root user on both cluster nodes:

1) /sbin/service ntpd stop
2) chkconfig ntpd off
3) mv /etc/ntp.conf /etc/ntp.conf.original
4) Delete the following file:

rm /var/run/ntpd.pid
This file maintains the pid for the NTP daemon.

When the installer finds that the NTP protocol is not active, the Cluster Time Synchronization Service is automatically installed in active mode and synchronizes the time across the nodes. If NTP is found configured, then the Cluster Time Synchronization Service is started in observer mode, and no active time synchronization is performed by Oracle Clusterware within the cluster.

To confirm that ctssd is active after installation, enter the following command as the Grid installation owner (grid):

1) crsctl check ctss
CRS-4701: The Cluster Time Synchronization Service is in Active mode.
CRS-4702: Offset (in msec): 0

Thanks to Jeffrey Hunter for these tips. He has an excellent writeup for 11gR2 RAC with iSCSI:

http://www.oracle.com/technology/pub/articles/hunter-rac11gr2-iscsi.html#10


Next we will take a look at how to resolve issues with iSCSI disks not accessed by the 11gR2 RAC Grid infrastructure setup program.

Oracle 11g Release 2 RAC Setup on VMWare on laptop

This weekend, I decided to setup Oracle 11g Release 2 RAC and ASM with the new Grid infrastructure on Macbook laptop with 4Gb of RAM. There is a way to setup a basic two node sandbox cluster with 4Gb of RAM. Some good references listed below discuss the detailed steps involved to configure the cluster with Oracle 11gR2 so I will not bore you with details.

Marly's Blog discusses the setup steps in great detail:

http://wiki.oracle.com/page/11gR2+RAC+on+a+Mac+-+Part+1
http://wiki.oracle.com/page/11gR2+RAC+on+a+Mac+-+Part+2
http://wiki.oracle.com/page/11gR2+RAC+on+a+Mac+-+Part+3
http://wiki.oracle.com/page/11gR2+RAC+on+a+Mac+-+Part+4

Masterschema has an excellent video series on how to do this as well:

http://www.masterschema.com/2009/09/install-oracle-rac-11g-r2-in-vmware-3-nodes/

The biggest decision you need to make is that with respect to shared storage. With 11g Release 2 RAC, you now have the option to store both the clusterware, OCR and ASM in ASM. Also, since VMWare Fusion and Oracle Sun VirtualBox do not have the ability to share disks which is required for RAC, you need to either setup NFS mounts between the cluster node VMs or use
an iSCSI server such as OpenFiler. However, on a laptop with 4Gb of memory, you need to conserve resources and Open Filer is a pig so I recommend using a third VM with Linux to host the iSCSI shared disks. There are also some excellent white papers on how to setup 11gR2 RAC with iSCSI and OpenFiler if you decide to buy a few low cost servers listed below:

http://www.oracle.com/technology/pub/articles/hunter-rac11gr2-iscsi.html#10

OpenFiler is a free iSCSI appliance that offers many similar features to that with NetApp filer iSCSI servers for free!
Since its easier to setup iSCSI server on a bare bones Linux VM, I decided to bypass using OpenFiler since it takes up too much memory that I cannot afford to waste on my 4Gb Macbook Pro laptop computer and the bare bones Linux VM uses only 350Mb of RAM compared to the 1-2Gb of RAM memory required for a decent OpenFiler server. Maybe if I get a laptop with 16Gb then I will test 11gR2 RAC with OpenFiler but for now, I just did a poor man's iSCSI virtual machine with CentOS Linux.

In the past, I have used Red Hat Linux and Oracle Enterprise Linux. However, this time I downloaded the ISO image for CentOS from http://centos.org/ and I must admit I am impressed! Its friendly, free and powerful and easier to use than Oracle Enterprise Linux.

Ok, So far, I am half way through the 11gR2 RAC install and finishing the Grid Infrastructure setup. There are a couple of cool new features that I like:

1) SSH setup- now you can have Oracle set this up and verify that ssh equivalency is working before the install
2) Validation of IP addresses- during the Grid Infrastructure installation process, Oracle verifies that the IP addresses for the cluster nodes are correct as well as that of the VIP (Virtual IP addresses).

Another note: Oracle 11gR2 RAC is a serious memory PIG compared to even 11gR1 and earlier releases. Ideally you want a server with at least 8GB or more RAM to setup the VMs. There is a workaround to use less memory resources and to prevent the OHASD and clusterware processes from using as much memory.

Alex Gorbachev of Pythian has a good writeup on this:

http://www.pythian.com/news/9179/oracle-11gr2-grid-infrastructure-memory-footprint/


There is a workaround to decrease the memory usage.

After installation update the ohasd in each of the GRID homes. Add: ulimit -Ss like this:

[root@server1 ~]# vi /u01/app/11.2.0/grid/bin/ohasd

# forcibly eliminate LD_ASSUME_KERNEL to ensure NPTL where available
ulimit -Ss
LD_ASSUME_KERNEL=
export LD_ASSUME_KERNEL

The result, it will same around 500MB of memory.

You still need more memory in compare to 11gR1, but at least it will same 1G for 2 nodes.

Thanks to Bernhard de Cock Buning for this tip. I have not tested it yet but will see if it helps me when I finish the Grid Infrastructure setup with both VMs.

Stay tuned for a lessons learned discussion of what 11gR2 RAC means for the enterprise. I also plan to discuss how to RAC enable Oracle Release 12 E-Business Suite in a future blog post.

I do prefer the Mac OS X platform compared to setup of RAC virtualization on Windows because the Mac OS is based on BSD UNIX so I can use most of the same UNIX commands that I am happy with as a DBA.

Cheers,
Ben

Collaborate 10 Conference Experiences in Vegas

Collaborate was a wonderful conference. Plus, it was in Vegas and since I had never spent time before in Las Vegas, a new experience for me as well! Here is where I stayed during the conference.



The conference was at the Mandalay Bay casino/resort center and also where I stayed for the week to make it convenient to get to and from the sessions and events. Besides, the Mandalay has probably one of the best pools/beaches in Vegas which made it pleasant for those scorching hot days while I was there.



The cool thing about the main pool was the concert stage above it with live shows in the evenings with bands playing good music and the waves to simulate surfing!

Of course the coolest looking hotel would be the Luxor fashioned after the pyramids of Egypt along with the Disneyland like Castles of the Excalibur.






Next door to the Luxor is the medieval themed Excalibur.




I had the chance to walk the strip a few times which was fun in spite of the heat and need to drink massive quantities of water and the pain of the blisters that I received- next time need to find better walking shoes! So I liked the lights on the strip as well as the cool view from the top of the Mandalay as well.








I am not much of a gambler but I did manage to break even and win $50 on blackjack, $30 on craps and lose $20 on slot machines. Met an older gentleman who won $11,000 that day on Craps! And saw a guy lose several thousand dollars in one hand of blackjack as well. The Oracle ACE dinner on Sunday night was fantastic and I give special thanks to folks at Oracle ACE program who planned a fun evening and dinner- Vikki and others, thanks! Rick Moonen's RM Seafood has excellent seafood and a hip decor as well. For the conference a nice snapshot of the event center.



Oracle Magazine had a party at Red Square in Vegas which was fun chance to meet new people and try some speciality vodkas. Ran into Steven Feuerstein of Quest and PL/SQL fame and he is a very nice guy.

I feel blessed to have met so many new friends and many of the top Oracle experts in the world there as well. I especially enjoyed meeting Ahmed Alomari, Mark Farnham, Rich Niemec and Brad Brown from TUSC, Ian Abramson, John Stoufer, Craig Shallahamer, Iggy Fernandez and many others at the conference. I also has a chance to meet founder and chairman of Pythian, Paul Vallee which was nice since Pythian is one of the most well known Oracle database consulting providers with many bright Oracle DBA folks working there.
Had chance to connect with Arup Nanda several times at the conference as well as meet John Garmany from Burleson Consulting. So how did my session go? It was ok, I did not have a lot of people in my session probably because it was on Thursday the last day of the conference and most of the attendees left on Wednesday evening.

So what was the highlight of the conference for me? I have to say that what made my day was when one of the senior Oracle Applications DBAs who attended my session on Thursday praised the quality and let me know she really enjoyed my session! Thank you! When a DBA tells you that he/she learned new tips and liked the presentation better than most at the conference that gives you a fantastic feeling that it was all worth it for the hard work you put into the paper and preparation!

Hope to see more folks at next conference. With the US and world economy in the worst shape since the Great Depression, my prayer is that things come back so I can have more folks attend to learn new things on Oracle technology.


Cheers,
Ben