Thursday, March 18, 2010

Cloning Release 12: Oracle Applications Part II: Rapid Clone techniques

In the previous blog post, I walked you through the pre-clone steps for Release 12 with Oracle E-Business Suite. Now that our pre-clone is complete, its time to do the actual cloning operation. With Release 12 for Oracle Applications, Oracle provides
a useful perl script called as part of Rapid Clone that performs most of the heavy lifting for cloning the E-Business Suite. In addition to years of experience an an Oracle Apps DBA, I referenced the following MyOracleSupport (formerly Metalink) notes listed below to guide me in the cloning for the E-Business Suite:

Note: 406982.1 - Cloning Oracle Applications Release 12 with Rapid Clone

Note: 603104.1 - Troubleshooting RapidClone issues with Oracle Applications R12

In a nutshell the cloning process is as follows:

I. Clone database tier from source to target environment
II. Clone applications tier from source to target

Cloning Release 12 Oracle E-Business Suite with Rapid Clone

Ok so we ran the scripts from last time and finished the prep work for the cloning. So how do we do the actual cloning or refresh for Release 12?
Actually the answer is that it depends on your environment. You can do a hot clone using the Oracle Applications Management Pack (AMP) with Oracle Enterprise Manager (OEM) Grid Control to cut down on downtime window. However that requires additional licensing and installation time. Or you could use RMAN or an Oracle hot backup to take the copy of your source database tier and clone to the target environment. However, for our example, we will use a simple cold backup to copy over the apps tier and database tier from our Source EBS environment to our target system.

As I mentioned above, for cloning an Oracle Applications environment, you would use your backup as well as copy commands manually or via a series of UNIX or Linux shell scripts to copy over the apps and db tier as part of the cloning.

However, in my test environment, I do not have an extra 300Gb of disk space
so we will use the Unix/Linux mv (move) commands to simulate the clone procedures.

If you decide to use the Oracle Recovery Manager (RMAN) utility to clone the database the procedure will be different.

Reference the Oracle documentation on usage for RMAN for details.

Some details on how to use RMAN to do this are listed below:

Note: 362473.1 - Cloning E-Business Suite Using Hot Backup for Minimal Downtime of Source Environment

Oracle® Database Backup and Recovery Basics 10g Release 2 (10.2)

Oracle® Database Backup and Recovery Advanced User's Guide 10g Release 2 (10.2)

In my opinion, you should use the RMAN Duplicate database command to take a hot backup of the database tier.

This works well and details are provided in

Note: 228257.1 RMAN 'Duplicate Database' Feature in Oracle9i / 10G and 11G

Note: If your database tier server is on 11gR1 or 11gR2 the new features for RMAN such as encrypted backups, Data Recovery Advisor (DRA), et al may also be of use to your backup and recovery strategy at future date. I have a blog
post on 11g RMAN new features in a previous blog post if you have questions on how this works with RMAN and 11g.

Ok so let's get started and cloning! First, we start with cloning the database tier.
On the database tier, we login as the oracle OS user and shutdown both the listener and database:

I. Clone Source database tier to target database tier server

Since we are simulating a clone procedure we create the new clone directory for database and applications tiers as follows:

mkdir -p /dbtier12/CLONE
mkdir -p /appstier12/CLONE

Before we use Rapid Clone script ( we need to copy over the database and application tier files.

First we copy the application tier file system
Log on to the source system application tier nodes as the APPLMGR user and shut down the application tier server processes. Copy the following application tier directories from the source node to the target application tier node:

Applications Technology Stack:
Oracle 10gAS 10.1.2 Tools ORACLE_HOME
Oracle 10gAS 10.1.3 Web ($IAS_ORACLE_HOME)

So this means that for the Apps tier you need to copy over the $INST_TOP, $COMMON_TOP, $APPL_TOP, 10.1.2 ORACLE_HOME, and 10.1.3 ORACLE_HOME directories and files

Copy the database node file system

Log on to the source system database node as the ORACLE user:

1. Perform a normal shutdown of the source system database
2. Copy the database (.dbf) files from the source system to the target system
3. Copy the source database ORACLE_HOME to the target system
4. Start the source Applications system database and application tier processes

For the database tier this includes the datafiles, parameter file (init.ora, spfile) and tnsnames.ora files

In this test environment, I do not have 300+ Gb of disk space so we simulated the copy with the UNIX/Linux mv command. However in a real world environment you would use the cp or scp command to transfer files from source to target systems. You would need to shutdown the database and take a cold backup of the database and apps tiers unless you do hot cloning and hot backup.

So now on the target we have the following clone directories:

1) Apps tier for CLONE

/apptier12/CLONE base directory for apps tier

-> 10.1.2 for CLONE ORACLE_HOME 10.1.2
-> 10.1.3 for CLONE ORACLE_HOME 10.1.3
-> appl for CLONE APPL_TOP
-> comn for CLONE COMMON_TOP

2) db tier for CLONE

CLONE 10g ORACLE_HOME -> /dbtier12/CLONE/10.2.0
CLONE datafiles /dbtier12/CLONE/db/apps_st/data


We will need to keep track of this information for the upcoming step when we run the clone scripts!

Do not set the environment files for the database tier! We need a clean slate for
the database tier clone procedure.

Note: if you decide to do a copy of the db and apps tier files from your source to target system, you will need to start the database tier and apps tier services on the source system. In this example we did a Unix/Linux move (mv) to simulate the copy
so we do not start the apps tier and db tier since both source and target are on
the same virtual machine.

Run the clone script for db tier this is called we can take a sneak peak at it with vi or emacs or whatever your favorite editor

perl dbTier

How the clone script flows calls the script which then calls forth the script and

Note: if you do not have PD ksh installed the work around to get working is to edit the script to comment out the reference in the script to look for the PD ksh shell.

Clone source database tier to target database tier

cd /dbtier12/CLONE/10.2.0/appsutil/clone/bin

perl dbTier

enter the password for APPS then hit the key

enter the details for target system at the prompts

In our case we take the defaults since we are cloning from source to target on same system as shown in the example below

Once we enter the choices for the prompt settings the clone will begin for the db tier. You will need to enter settings for your target clone datafile location (DATA_TOP) as well as database tier ORACLE_HOME location, and other settings such as port pool.

Time to go for a coffee break! It will take some time and depends on the size of your database. I have scripts for cloning that I usually fire off at night time and babysit to make sure all runs smoothly.

You can tail the log file to keep track of your cloning operation. I like to dump the clone output to a logfile as part of a script to verify things.

When it finishes you should see a successful screen like below:

Rapid Clone starts the listener and database back up once the clone is created. We can run a few simple commands to verify that our new database clone is online and available:

My clone took about 90 minutes for the database tier since I have a very slow server running in a virtual machine. Your mileage may vary based on hardware, OS and available resources. Now lets clone the apps tier and finish our cloning!

Clone Source applications tier to target applications tier

Now that our database tier has been cloned its time to clone the apps tier to the new target! We run the script on the source apps tier as shown below:

For our example, The cloning script for the apps tier lives under the
following directory:

$ cd $INST_TOP/clone/bin

Login as the applmgr OS user and we run it with the following syntax:

perl appsTier

Cloning the Applications Tier

Since I do not have the PD ksh installed it errors and then I need to edit the script which is called by like last time so that it does not try to search for the PD ksh.

Rapid Clone uses a questionaire to prompt us for the details for the source to target cloning for the applications tier similar to the questionaire in the cloning script ( that we ran earlier to clone the database tier.

We enter the following for the target apps tier system prompts:

target system base directory: /appstier12/CLONE
target system Forms ORACLE_HOME /appstier12/CLONE/10.1.2
target system Web ORACLE_HOME /appstier12/CLONE/10.1.3

choose 0 for port pool like we did earlier for db tier clone

A complete list is given below:

Now we want to keep a close watch on the logs to make sure no errors occur during the cloning for the apps tier in which we tail the logfile:

Are you ready for another coffee break? I sure hope so because I am! So lets grab a mocha and maybe a donut while this clone runs for the next few hours. I usually run a clone script at night time and check it every half hour until its done.

In between the coffee and donut break, I check the status and its at 50% and before I can drink the java or eat my donut a frantic Oracle project manager and functional analyst are waiting for me in my DBA cube!

I calm the functional team down and fix their performance issue. Was resource problem with JVM for the apps tier for another environment. I will blog on a solution to this in future post. OK so we check our clone and enter additional prompts after we had to restart it after it failed due to port issue that I fixed and re-started the clone.

Our clone finishes and all of the app tier services are started automatically by Rapid Clone ( script.

And we have verification without errors.

Troubleshooting Errors during Rapid Clone - Appstier

Recall that earlier our clone failed at 74% during the cloning for the applications tier when we ran appsTier so I checked
the logfile under

/appstier12/CLONE/inst/apps/CLONE_tusebs/admin/log to find the
specific failure since it failed during the ApplyAppsTier phase

After looking at the logfile ApplyAppsTier_02251717.log I found the root cause
why my clone failed was due to unable to connect to the listener and database.

The port pool needs to be same on database and applications tier
so if you set 0 as port pool during dbTier for db tier you MUST USE same port pool for clone with apps tier or the clone will fail during Appstier apply phase!!!

For Linux, you can use lsof command to find ports in use and use kill command to free up these ports.

After the cloning scripts have run successfully, the apps and db tiers will be started automatically by the Rapid Clone scripts.

I then reviewed the logs under


for both the source database and applications tiers and verified that all application and database services are available.

Next we need to verify that, users can login to OAM as cloned environment.

Sure enough, I hit the dreaded 500 Internal Error when I tried to login to the OAM website.

Here are some good troubleshooting notes if you have 500 Internal errors when attempt to access the OAM login page

Note: 761869.1 500 Internal Server Error
Note: 813523.1 500 Internal Server
Note: 443353.1 Guest Password mismatch
Note: 564536.1 ADSTRTAL.SH returns error TIMEOUT OUT Interrupted Exception
Note: 331322.1

Note: 391406.1 How to get a clean Autoconfig
Note: 387859.1 Autoconfig R12

I used these to resolve the issue which involved changing settings for the context file to allow JVM to function correctly without consuming all server resources as well as bouncing the app and database tiers on the target clone server.

After all was said and done, I verified that things worked for my new clone.

As a final sanity check, I checked the database and application tier servers for the new cloned environment.

Good the cloned database is functional and available. Lastly, I checked that all of the application tier services were online.

Finishing Tasks for Cloning Operations

1. Workflow settings for new clone
2. Verify login and accounts work correctly
3. Check status for SSO and OID

While we did not check on workflow, this is one area that you must address in your cloned environments. Also, run tests to check that the authentication is working correctly with accounts that use SSO and OID. If you do not perform these checks then users will have problems logging into the cloned systems after a refresh.

Congratulations! You now can perform the cloning and refresh tasks that are critical skills to have as a hardcore Oracle Apps DBA!

Tuesday, March 16, 2010

Cloning Release 12 Oracle Applications E-Business Suite: Part I pre-clone steps

Dear readers,

Today we begin a new series on cloning the E-Business Suite for Release 12 with Oracle Applications. In this post, we cover the pre-clone steps required to perform a successful clone for Oracle Applications R12. As an Apps DBA, the ability to quickly produce clones is mandatory to be a successful EBS DBA. The following post will show details on how to clone. First lets get started with the prep work.

Part I: Pre-Clone Steps

Unlike a regular Oracle database environment, cloning the E-Business Suite is a complex affair. You need to prepare the source and target environments before a clone operation can be performed.

Part IA: Prepare clone environments

As a precaution, I advise you to follow any pre-reqs on My Oracle Support (formerly Metalink) and to download and apply the latest AD, Autoconfig template, and Rapid Clone patches before we begin. This will save you the potential headache and risk of failed clone operations due to a patch that should have been applied before doing the cloning steps.

1. First make sure to source the environments for oracle and applmgr environments for the database and applications tiers.

2. Shutdown apps tier and run autoconfig on apps tier

$ADMIN_SCRIPTS_HOME/ or $INST_TOP/admin/scripts/
$ADMIN_SCRIPTS_HOME/ or $INST_TOP/admin/scripts/

Note: you will need to enter your password for APPS for these scripts

As a precaution, I advise you to first backup context file before cloning.

3. as applmgr user on apps tier, create the appsutil file for the pre-clone steps
Oracle supplies a script called to generate the new appsutil file which you will need to do the pre-clone. On the applications tier as the applmgr OS user run:

perl $AD_TOP/bin/

Review the log files from the script to ensure that no errors have occurred before taking the next step.

Scroll down and verify it worked:

Open another window or shell prompt and
as the oracle user, copy the file via ftp from the $INST_TOP/admin/out
directory to the database tier server to the $ORACLE_HOME directory

cd $ORACLE_HOME; unzip -o

as oracle user, run autoconfig on the database tier

cd $ORACLE_HOME/appsutil/scripts/ where indicates _ where SID is
the database SID and hostname the database tier hostname

in our case, this SBOX_tusebs would be as follows:

Now that we have run auto config, we are ready for the pre-clone steps!

Part IB: run pre-clone scripts for database tier

Now that we have completed the required prep work for our cloning operations, we need to run the pre-clone scripts for the database and applications tiers before we can clone the environment.

Oracle Release 12 actually starting with release 11i for E-Business Suite has a suite of perl scripts that perform the heavy lifting for us in terms of these cloning scripts. First, we need to pre-clone the database tier
and then we run the pre-clone script for the apps tier. Both the Sequence and syntax are critical here or the clone will fail.

The pre-clone script for database and apps tier are called It lives under the
$ORACLE_HOME/appsutil/scripts/ directory on the database tier.

If you are a curious DBA like me,
you can look at the script with your favorite editor such as VI or EMACS.
Here is a sneak peak at the script:

Warning (Achtung!): Do not play around and make changes
to any of these scripts without the advice of Oracle Support!

Of course, one test that I may try at a future date on my
sandbox play environment is to tweak the scripts to see if I can do something funny like run my preclone
with silly names like
perl mickeymouseDBtier
perl plutoAppsTier

Then I could call it Disney EBS Rapid Cloning!

Warning: Kids do not try changing these scripts on a production system! Tricks are for kids not silly rabbits to paraphrase my favorite cereal commercial.

OK Let's run it!

you will need to enter the APPS password for the script to execute

it runs for a few minutes and then completes without errors

we verify the status in the logfile

scroll down and check to see that all ran fine without errors

Ok so now we need to run the pre-clone script on the apps tier as the applmgr OS user. Since we are running this on Linux platform the script is a shell script.

Part IC: Run the pre-clone scripts for apps tier

The pre-clone script for apps tier is located under the $INST_TOP/admin/scripts or $ADMIN_SCRIPTS_HOME directory and is called To execute it we logon as applmgr user and run

perl $INST_TOP/admin/scripts/ appsTier as shown below and you need to enter the APPS password
at the prompt.

Be sure to use the syntax EXACTLY as shown above or the preclone will FAIL!

This will run for a short time maybe 5-10 minutes and then complete. Verify that your preclone ran without errors. Check the log file generated by the script for the apps tier to ensure no errors have occurred as shown in the examples below.

Scroll down the remainder of the log file:

Now we are finally ready to clone! In the next blog post, I will show you the actual cloning steps to clone R12 EBS. Stay tuned!

Ben Prusinski

Saturday, March 6, 2010

Fusion Middleware Integration with Oracle E-Business Suite: Part II: SOA Suite and Release 12

In the last post, we discussed how to install SOA Suite for Oracle Fusion Middleware. This post will provide some quick tips on how to configure Release 12 of the Oracle Applications E-Business Suite with SOA Fusion Middleware. As a caveat, I want to advise you all that all versions of Oracle EBS prior to 12.1 are not supported for SOA Fusion Middleware based on discussions with Oracle support. With that said, I will summarize the details. Since my test environment is 12.0.4 and SOA requires 12.1, I do not have actual screen shots yet. What I will do is to revisit the specifics in a future post once I have upgraded my test 12.0.4 EBS environment to 12.1.2 ( a lengthy process!). OK so lets get started.

The reference details are further contained and hidden away in a few key notes from MyOracleSupport (formerly Metalink):

Oracle Metalink Note: 755067.1 - Using Oracle BPEL 10g with E-Business Suite Release 12.1.1

Oracle Metalink Note 556540.1 Installing Oracle E-Business Suite Integrated SOA Gateway, Release 12.1

MetaLink Note 755069.1 -EBS R12 Packager and Adapters for External Application Servers

Step A: Perform the following steps for enabling Oracle E-Business Suite Integrated SOA Gateway Release 12.1:

1. Apply the following Oracle E-Business Suite patch:
Patch 9060361:R12.TXK.B: FSG: Log Location and Level Properties Not Accessible from OAFM

As with all EBS patches, you will need to verify that all pre-requisite and co-requisite patches have been applied successfully to the 12.1.x E-Business Suite via Auto Patch without errors. Be sure to review the readme instructions!

2. Set applications environment and stop all application tier processes.
From the applications instance $APPL_TOP, set the environment by running the APPS.env script

3. Shutdown the application tier processes for your EBS instance by running $ADMIN_SCRIPTS_HOME/adstpall

4. Run TXK development script to install Oracle Application Server Adapter for Oracle Applications as shown below:
$FND_TOP/bin/ -script=CfgOC4JApp -applicationname=pcapps -oracleinternal=Yes -oc4jpass=welcome -runautoconfig=No

5. Run TXK deployment script to install forms-c4ws.ear and configure container Forms-c4ws J2EE group correctly, as shown below:

$FND_TOP/bin/ -script=DeployForms-c4ws

6. Enable the new container forms-c4ws. (By default, the new OC4J container forms-c4ws will be disabled.)
To enable the new container forms-c4ws, modify the following values of context variables in the $CONTEXT_FILE:

Set 's_forms-c4wsstatus' to "enabled".
Set 's_forms-c4ws_nprocs' to "1".

7. Run AutoConfig. For information on how to run AutoConfig, see "Using AutoConfig to Manage System Configurations in Oracle E-Business Suite Release 12", My Oracle Support Knowledge Document 387859.1 for details.

8. Start all application tier processes for the instance by running the script $ADMIN_SCRIPTS_HOME/adstrtal

9. Enable ASADMIN user per details in the Oracle Metalink suport notes

10. Log on to Oracle E-Business Suite using sysadmin

11. Bounce the middle tier for R12 EBS environment

Part B: Configure SOA Suite with Release 12.1.x for Oracle Applications E-Business Suite

I will provide a summary of steps for the required tasks to configure the Oracle SOA Suite with Release 12 for Oracle EBS:

1. Register your installation of SOA Suite with R12 EBS instance per details from Oracle MetaLink Note 755069.1
This involves generating a new DBC file to use with SOA Suite and your EBS instance.

2. Create BPEL domains in SOA environment

3. Create JDBC connection pools in SOA instance for OC4J to access each EBS 12.1 instance

4. Define data sources between SOA instance and EBS 12.1 instances

5. Create Oracle adapter connection between SOA and 12.1 EBS instances

Hope this helps you to get started on your quest to configure Oracle Release 12.1 with SOA Suite Fusion Middleware!