R12 EBS Installation Tips and Tricks: Failed Installation Solutions for RW-50015 Errors

After countless hours of frustrations with attempts to get a test Oracle EBS R12 installed on my Macbook Pro laptop running VMWare Fusion and Linux, I found a solution recently to a problem with installing a test Oracle R12 EBS environment on Oracle Enterprise Linux 5 (OEL 5- 32 bit) with VMWare Fusion. There is a bug in R12 EBS for Linux platforms that will cause the installation to fail on the Apache HTTP server process.

What happens is that you will receive an RW-50015 Error message during the Rapidwiz install process.


RW-50015: Error: - HTTP Listener is not responding. The service might not have started on the port yet. Please check the service and use the retry button.


Also at same time, the Rapid Install window will show many errors for HTTP.


I then used adapcctl.sh status to check on these services and sure enough the http server would not start:

[oracle@ebs scripts]$ ./adapcctl.sh status

You are running adapcctl.sh version 120.6.12000000.2

Checking status of OPMN managed Oracle HTTP Server (OHS) instance ...

Processes in Instance: PROD_ebs.ebs.sandiego.com
-------------------+--------------------+---------+---------
ias-component | process-type | pid | status
-------------------+--------------------+---------+---------
OC4J | oafm | 21168 | Alive
OC4J | forms | 21078 | Alive
OC4J | oacore | 20943 | Alive
HTTP_Server | HTTP_Server | N/A | Down


adapcctl.sh: exiting with status 0

adapcctl.sh: check the logfile /d01/app/oracle/inst/apps/PROD_ebs/logs/appl/admin/log/adapcctl.txt for more information ...



According to the release notes from Oracle Metalink Support (Note #402310.1)
Oracle Applications Installation and Upgrade Notes Release 12 (12.0.4) for Linux (32-bit)
Oracle HTTP Server Patch (on OEL 5 and RHEL 5 only)

There is an issue with the Oracle HTTP Server that comes with the R12 EBS Suite that causes the linking process to fail for Apache. The solution is to exit the Rapid Install program, shutdown the OC4J and web services and apply the patch. Actually, it is really a library called libdb.so.2 which is missing from the default OEL 5 and RHEL 5 operating system. Even after I ran the up2date oracle-validated command to install all of the fixes for Oracle on Linux, it was still missing. So the patch really is to download the patch which contains this essential missing library file and copy it over to the /usr/lib directory structure.


Download and apply the patch 6078836 from OracleMetaLink to fix an issue with the Oracle HTTP Server bundled with the E-Business Suite technology stack. We unzip the patch:

[oracle@ebs oracle]$ unzip p6078836_101330_LINUX.zip
Archive: p6078836_101330_LINUX.zip
creating: 6078836/
inflating: 6078836/libdb.so.2
inflating: 6078836/README.txt
[oracle@ebs oracle]$ ls
6078836 apps db inst p6078836_101330_LINUX.zip StageR12
[oracle@ebs oracle]$ cd 6078836/

[oracle@ebs 6078836]$ ls
libdb.so.2 README.txt


Then we copy to /usr/lib.

[root@ebs 6078836] $ cp libdb.so.2 /usr/lib

[root@ebs lib]# cd /usr/lib

[root@ebs lib]# pwd
/usr/lib


[root@ebs lib]# ls libdb.so*
libdb.so libdb.so.2



After we apply the patch, we need to restart the web services using the adapcctl.sh utility. Actually, just re-run the Rapid Installer program which will do this step for us. After it is successful and Apache comes up, we can verify status for R12 EBS by running the adapcctl.sh command again as listed in the below example:

[oracle@ebs scripts]$ ./adapcctl.sh status

You are running adapcctl.sh version 120.6.12000000.2

Checking status of OPMN managed Oracle HTTP Server (OHS) instance ...

Processes in Instance: PROD_ebs.ebs.sandiego.com
-------------------+--------------------+---------+---------
ias-component | process-type | pid | status
-------------------+--------------------+---------+---------
OC4J | oafm | 21168 | Alive
OC4J | forms | 21078 | Alive
OC4J | oacore | 20943 | Alive
HTTP_Server | HTTP_Server | 28519 | Alive

We then re-start rapidwiz and voila! It works!




Now, we can access the R12 EBS Webportal main logon page as shown below:





We can now login to R12 EBS as SYSADMIN user using default sysadmin password as shown below to verify that all is working for our new R12 EBS installation:





Cheers,
Ben

Fun with Oracle EBS R12 Installation

I am setting up a new test environment with Oracle EBS R12 release on OEL 5 with VMWare Fusion on a new Macbook Pro.

To install Oracle E-Business, we have two main options to stage the more than 20 DVDs that are part of a normal 11i or R12 EBS implementation. The first is to manually feed each DVD into the drive during the Rapid Install program for EBS installation. As you can imagine, this is very tedious and not convenient. The second option is far more convenient, use the perl script called adautostg.pl that comes with the Rapid Install DVD. The adautostg.pl script is on the Oracle R12 EBS Applications Rapid Install Start Here DVD and located under the /startCD/Disk1/rapidwiz directory on the DVD. However, there is a problem that may cause the autostage feature to fail with your R12 EBS installation. I ran into this problem today when I was trying to stage my R12 EBS test environment by running adautostg.pl. It appears that sometimes adautostg.pl cannot find the correct disk label and fails to access the automounted DVD. Since my new firm just became an Oracle partner, it still is a week or two before I can open a new iTar to resolve
this issue. However, suffice it to say, this issue with adautostg.pl has been around since 11i EBS release. My solution? Either load the DVDs one at a time during the Rapid Install process or copy over the 20 DVDs to a manually created staging area.

Stay tuned as we will cover how to implement RAC with single instance R12 EBS in an upcoming post!

Cheers,
Ben

A new chapter

Yesterday afternoon, I was let go from my job. Fortunately, it gives me desire now to build my own successful consulting firm. I have launched a new business and technical site to complement blogs on database technology called Ben Prusinski and Associates.

Please visit us for the best onsite and remote Oracle services possible for your budget as well as future white papers and tips on database technology.

Ben Prusinski and Associates
We are an Oracle practice with a focus on customer quality service and affordable solutions compared to larger firms.

I am also adding a second blog to host additional tips and tricks as the business site as I will have more space and room to add more detailed posts than in the past. Next month I will be presenting on more black magic for Oracle 11g at the Oracle conference in Santiago, Chile in April for CLOUG

They say that in every cloud is a silver lining. As a new Oracle ACE, I look forward to making even bigger contributions to the Oracle community this year and into the future!

New Developments for 2009: I am now an Oracle ACE!

Greetings fellow Oracle professionals!

In a departure from my technical tips, I decided to make a few key announcements today.

I just found out today that I have received the prestigious Oracle ACE award!
I am excited to be among the elite group of Oracle technologists and look forward to sharing new tips and tricks in the future. One thing that I am interested in is to do a presentation on the Exadata. If anyone has access to a test Exadata system and wants a top DBA to do some stress testing, I am available! Another fellow Oracle professional, Chen Shapira, whom I met last year at OpenWorld also is a new Oracle ACE. So wish us double congratulations. Chen is a wonderful person and brilliant Oracle DBA. Her blog is great as well and I found myself intrigued by the discussions on concurrency and Dining Philosophers. We had a nice discussion on the pros and cons of hidden parameters for Oracle at Open World last year :-)

The second exciting new development in my professional life is that I have been chosen to participate in the Oracle 11gR2 beta program. Beta testing is underway currently and it is a lot of fun and great challenge thus far.

Third, I passed all of my certification exams for the Oracle DBA track to become 10g OCP and OCA certified Oracle professional. Now, I have to do the upgrade exam to 11g and then study and train for the 11g OCM.
I have mixed views on the values of certification. Suffice it to say, I feel it definitely helps! I was able to learn new things and brush up on basics. I can say that I have a better understanding of 10g features and backup and recovery now.

Add to that I am in the middle of writing a book on Oracle technology and have one piece of advice to aspiring Oracle authors: you must be willing to dedicate your free time to the book and beware of tech changes before your book is published so that it is not out of date when release party is underway! The benefit of writing a book on Oracle is you do tons of research and testing and really learn the area of focus inside and out. As a new Oracle author, I have deep appreciation for folks like Robert Freeman, Tom Kyte, Jonathan Lewis, Don Burleson, Mike Ault, Arup Nanda, and Kevin Loney who are prolific writers in the Oracle technology field and yet were busy raising families, children and working at challenging Oracle positions.

With the difficult economy, every bit that you contribute to others will have a major karmic affect on the world and enrich your own life as well.

Oracle 10g/11g RAC Clusterware Installation Tips

Part II: Oracle RAC High Availability Series


Since I have been building clusters with Oracle 10g and 11g RAC environments for many
new clients, I have decided to present a series on Oracle RAC tips and tricks. The previous installment for Oracle RAC technology included design and implementation
guidelines for best practices. This week, we will cover the post installation tips
with the Oracle 10g/11g Cluster Verification Utility called cluvfy. This wonderful tool provides insight into whether or not your Oracle RAC environment has been installed correctly.

The Oracle Clusterware utility for Oracle RAC comes in two forms: the cluvfy utility which is available after installation of the clusterware software and the runcluvfy.sh shell script which can be used before installation. In the following example, we will use the post install verification feature available with cluvfy to verify our Oracle 10g RAC clusterware installation.


Verify Oracle RAC Clusterware Installation using the Clusterware Verification Utility (CLUVFY)


Cluvfy uses stages to check the status of the installation for before and after each
phase of the Oracle RAC installation process.

Below is the basic syntax for cluvfy:

oracle@racnode1 $ cluvfy


USAGE:
cluvfy [ -help ]
cluvfy stage { -list | -help }
cluvfy stage {-pre|-post} [-verbose]
cluvfy comp { -list | -help }
cluvfy comp [-verbose]

USAGE:
cluvfy stage -post crsinst -n [-verbose]

is the comma separated list of non-domain qualified nodenames, on which the test should be conducted. If "all" is specified, then all the nodes in the cluster will be used for verification.

DESCRIPTION:
performs the appropriate checks on all the nodes in the nodelist after setting up Cluster Ready Services(CRS).




Now we run the post-CRS verification with cluvfy to ensure that our Oracle RAC clusterware has been correctly installed. This is useful since the Oracle 10g/11g RAC Clusterware (CRS) is the most challenging aspect of a RAC implementation.


oracle@racnode1> cluvfy stage -post crsinst -n all -verbose

Performing post-checks for cluster services setup

Checking node reachability...

Check: Node reachability from node "racnode1"
Destination Node Reachable?
------------------------------------ ------------------------
racnode1 yes
racnode2 yes

Result: Node reachability check passed from node "racnode1".


Checking user equivalence...

Check: User equivalence for user "oracle"
Node Name Comment
------------------------------------ ------------------------
racnode1 passed
racnode2 passed
Result: User equivalence check passed for user "oracle".

Checking Cluster manager integrity...


Checking CSS daemon...
Node Name Status
------------------------------------ ------------------------
racnode2 running
racnode1 running
Result: Daemon status check passed for "CSS daemon".

Cluster manager integrity check passed.

Checking cluster integrity...

Node Name
------------------------------------
racnode1
racnode2

Cluster integrity check passed


Checking OCR integrity...

Checking the absence of a non-clustered configuration...
All nodes free of non-clustered, local-only configurations.

Uniqueness check for OCR device passed.

Checking the version of OCR...
OCR of correct Version "2" exists.

Checking data integrity of OCR...
Data integrity check for OCR passed.

OCR integrity check passed.

Checking CRS integrity...

Checking daemon liveness...

Check: Liveness for "CRS daemon"
Node Name Running
------------------------------------ ------------------------
racnode2 yes
racnode1 yes
Result: Liveness check passed for "CRS daemon".

Checking daemon liveness...

Check: Liveness for "CSS daemon"
Node Name Running
------------------------------------ ------------------------
racnode2 yes
racnode1 yes
Result: Liveness check passed for "CSS daemon".

Checking daemon liveness...

Check: Liveness for "EVM daemon"
Node Name Running
------------------------------------ ------------------------
racnode2 yes
racnode1 yes
Result: Liveness check passed for "EVM daemon".

Liveness of all the daemons
Node Name CRS daemon CSS daemon EVM daemon
------------ ------------------------ ------------------------ ----------
racnode2 yes yes yes
racnode1 yes yes yes

Checking CRS health...

Check: Health of CRS
Node Name CRS OK?
------------------------------------ ------------------------
racnode2 yes
racnode1 yes
Result: CRS health check passed.

CRS integrity check passed.

Checking node application existence...


Checking existence of VIP node application
Node Name Required Status Comment
------------ ------------------------ ------------------------ ----------
racnode2 yes exists passed
racnode1 yes exists passed
Result: Check passed.

Checking existence of ONS node application
Node Name Required Status Comment
------------ ------------------------ ------------------------ ----------
racnode2 no exists passed
racnode1 no exists passed
Result: Check passed.

Checking existence of GSD node application
Node Name Required Status Comment
------------ ------------------------ ------------------------ ----------
racnode2 no exists passed
racnode1 no exists passed
Result: Check passed.


Post-check for cluster services setup was successful.


The beauty of the Oracle RAC cluster verification tool is that it can pinpoint trouble spots before, during and after the installation process. It is an essential tool in the Oracle RAC expert's workbench.

Some additional details on the use of the cluvfy utility are available in the following Oracle Support online Metalink notes:

ML # 339939.1- Running Cluster Verification Utility to Diagnose Install Problems
ML # 316817.1- CLUSTER VERIFICATION UTILITY FAQ


Regards,
Ben

Design considerations for Oracle RAC High Availability Design

I have recently been working with clients to implement and review Oracle 10g RAC environments. One item that has
been of importance is how to design a truly redundant and highly available (HA) infrastructure.

The goal is to avoid any single point of failure (SPOF) while maximizing performance and scalable factors

Here are my notes:

Hardware Considerations
Implement 3-4 nodes for the RAC design in the initial phase. By using at least 3-4 nodes for a new RAC cluster, you
help to protect yourself against the dreaded "Split Brain" condition. If you lose a single node, you still will have 2-3 nodes
to use for failover and servicing current mission critical applications.

Network Considerations for RAC
1. Implement multiple switches for both the interconnect to avoid loss in communication between nodes in the RAC cluster.
The danger of using only a single switch is that if this switch has a failure, the entire RAC cluster will crash and result will be
downtime. I see a lot of clients skimp on this item. By using 2 switches at both the interconnect (private network) and storage level (ie: the SAN fabric or iSCSI layer) you protect yourself against a network failure.

2. Use a fat pipe for the interconnect. Go with at least 4Gb+ Ethernet or even better, fiber channel for best throughput.
Even better, the Infiniband has robust performance for heavy duty applications.

3. Implement multiple dual homed NIC cards to avoid loss of a network adapter in the server. Other good network interfaces
have this built into the network card such as Sun's IPMP (IP Multipathing - Sun Traffic manager).

Storage Considerations for Oracle 10g/11g RAC
Invest in the best SAN possible with Fiber Channel (FC/FC-AL) for best performance and support.

Fiber is the best overall performance and HA solution for enterprise storage. Another suitable option is iSCSI which provides many similar benefits.

Other tips for Oracle 10g/11g RAC Design

1. Mirror and protect multiple copies of the Oracle 10g/11g Clusterware: have several copies of the OCR (Oracle Cluster Registry) and Voting Disks. These are small footprints in size and if you only have a single copy, guess what will happen to your entire RAC cluster if you lose your only copy of these critical files? The entire RAC cluster will fail because it will not be able to communicate. I have seen clients that have 1 copy of the OCR and vote disk and they put their RAC clustered environments at great risk. Even if the Unix or storage administrator says they mirror copies of them on storage, one cannot be too cautious to have multiple copies.

2. Use multiple ASM Disk Groups.

At least 4-5 ASM disk groups are recommended to split up the various Oracle 10g/11g database files for performance and availabilty reasons. For example, we can have the following sample ASM configuration:

+FLASHDG for flash recovery area within ASM to store backups and archivelogs
+DATADG for Oracle 10g/11g data files
+INDEXDG for Oracle 10g/11g indexes
+DATADG2 for additional Oracle 10g/11g application database files

Implement Oracle Data Guard for RAC
While Oracle RAC provides performance, scalability, and data protection against a single node failure for a RAC instance in the cluster, it does not protect against data loss in the event that the RAC database has a media failure and data loss. This is because the RAC cluster nodes all share the same database. Many folks incorrectly assume that RAC is a total HA solution. It is not. I recommend that a standby physical database be implemented with RAC environments to provide for protection against data loss and the single point of failure (SPOF) which is the Achilles Heel with RAC. By using an Oracle Data Guard with RAC, you gain failover and switchover features to protect against data loss. Downtime is bad enough for an already stressed DBA to worry about, but data loss will get a DBA fired and potentially cause a company to go out of business. Thus, Data Guard is the perfect solution to complement RAC for a comprehensive HA solution as part of the Oracle Maximum Availability (MAA) architecture.


Implement and Test RMAN Backup and Recovery with Oracle 10g/11g RAC
Oracle provides the ultimate backup and recovery tool called the Recovery Manager (RMAN) for free out of the box to provide essential backup and recovery for complex RAC environments. User managed hot backups were fine years ago before the RMAN age but sorry folks, they really do not cut the mustard for modern times. RMAN provides a ton of features such as block level media recovery and point in time recovery that are not available in the old user backups. Plus RMAN can be used to clone RAC databases and implement standby Data Guard environments as well as backup and recovery ASM disk groups with Oracle 10g and 11g for RAC.





Hope these tips and tricks help you with building a reliable and stable RAC environment!

Cheers,
Ben

My Delayed Review for Open World

Well dear readers, sorry for the delay. I caught a sinus infection on the return flight home to San Diego and took me a week to recover. That and things have been hectic for me.

This year's OpenWorld conference was amazing. Since it was my very first time at OpenWorld as well as speaking at the conference, I had a great time. I want to give thanks to the hundreds of folks who attended my session. Below is a link to my session if you wish to download it:


But first, some photos from the OpenWorld conference.

And Oracle definitely knows how to throw a party with this year's Appreciation event on Treasure Island in San Francisco. I did enjoy the sushi and later on the music from live performances by Seal, Psychedelic Furs (which I remember from high school!), UB40 and Alan Jackson.





The massive Oracle Ferris wheel at the event entrance for the fun Appreciation event was incredible.

Even more amazing and since I love bright neon light shows was the gigantic stage and light show. Reminds me of the city in the sci fi classic movie Blade Runner:



The big news of the event was that Larry announced that Oracle was entering the hardware and storage business called the Exadata database server with HP. I have mixed views on this offering but it should prove interesting! Some details on it:

Here is a photo of the new beast from Oracle and HP:




Of course, no keynote from Larry would be complete without an introduction to his new super trimaran yacht as Larry has a passion for sailing and racing! Alex Gorbachev whom I had the pleasure to meet at the conference posted a nice little video of Larry's new racing boat

All that I can say is wow: I would love to have the chance to be on the racing boat and try to see if I can sail that ship.

Now the best part for me anyways was meeting new colleagues, business contacts and new friends!!!! After all, its really all about the people and sharing tips, tricks and camaraderie. I was fortunate to meet Tim Hall, Arup Nanda, Tom Kyte, Cary Millsap, Lewis Cunningham, Mark Williams, Greg Rahn and many other top Oracle experts at the conference. Poor Tim Hall got ill the last day of the conference and I hope that he is feeling better now. I have enjoyed his Oracle-Base website chock full of Oracle tips and scripts. And not to mention, it was fantastic meeting colleagues from ITC especially my collegues from Buenos Aires, Argentina office.