Annoying interruption problem solved for Oracle R12 E-Business Suite

I run most of my test environments using virtual servers with VMWare and Oracle Sun Virtualbox. As such the network connections use DHCP to access wireless internet access points. Recently a new issue came up that gave me hours of grief. After making some changes to my Oracle R12 context file, I was unable to get the middle tier application services started successfully. I received the following errors:

******************************************************
ERROR: Timed out (100000): Interrupted Exception
Your are running adoafmctl.sh version 120.6.12000000.2
Starting OPMN managed OAFM OC4J instance
*****************************************************

A quick search on My Oracle Support found similar issue in Note: 564536.1.
The note recommends changing the timeout parameters for oacore parameter
under the $ORA_CONFIG_HOME/10.1.3/opmn.xml file for the middle application tier server as well as changing the context file parameters s_oacoretimeout, s_formstimeout, and s_oafmtimeout for the middle tier application context file and running autoconfig on the middle tier. I tried this and unfortunately, it did not fix my problem.

So, upon further investigation, I found the solution given by My Oracle Support Note: 412980.1 Problem: oafm, forms, and oacore not starting after IP Address Change.

Aha! I thought since I use DHCP and virtual networks, this plays havoc on my Oracle R12 EBS virtual environments. The root cause of my problem and why the middle tier services failed to start is that the OC4J lock files store the old IP address which are incorrect after an IP address is changed by the DHCP and virtual network adapters whenever I travel to use different networks. To solve my problem, I had to perform the following tasks.

1. Take backup of current middle tier files
2. shutdown all current running middle app tier services with the adopmnctl.sh script
under the $INST_TOP/admin/scripts/home directory

3. delete the lock files under the $INST_TOP/ora/10.1.3/j2ee directories

[applmgr@tusebs scripts]$ echo $INST_TOP
/appstier12/CLONE/inst/apps/CLONE_tusebs
[applmgr@tusebs scripts]$ rm -fr /appstier12/CLONE/inst/apps/CLONE_tusebs/ora/10.1.3/j2ee/oacore/persistence/*
[applmgr@tusebs scripts]$ rm -fr /appstier12/CLONE/inst/apps/CLONE_tusebs/ora/10.1.3/j2ee/oafm/persistence/*
[applmgr@tusebs scripts]$ rm -fr /appstier12/CLONE/inst/apps/CLONE_tusebs/ora/10.1.3/j2ee/forms/persistence/*

4. Run Autoconfig on the apps middle tier:
[applmgr@tusebs scripts]$ ./adautocfg.sh
Enter the APPS user password:

The log file for this session is located at: /appstier12/CLONE/inst/apps/CLONE_tusebs/admin/log/02101314/adconfig.log

AutoConfig is configuring the Applications environment...

AutoConfig will consider the custom templates if present.
Using CONFIG_HOME location : /appstier12/CLONE/inst/apps/CLONE_tusebs
Classpath : /appstier12/CLONE/comn/java/lib/appsborg2.zip:/appstier12/CLONE/comn/java/classes

Using Context file : /appstier12/CLONE/inst/apps/CLONE_tusebs/appl/admin/CLONE_tusebs.xml

Context Value Management will now update the Context file

Updating Context file...COMPLETED

Attempting upload of Context file and templates to database...COMPLETED

Configuring templates from all of the product tops...
Configuring AD_TOP........COMPLETED
Configuring FND_TOP.......COMPLETED
Configuring ICX_TOP.......COMPLETED
Configuring IEO_TOP.......COMPLETED
Configuring BIS_TOP.......COMPLETED
Configuring AMS_TOP.......COMPLETED
Configuring CCT_TOP.......COMPLETED
Configuring WSH_TOP.......COMPLETED
Configuring CLN_TOP.......COMPLETED
Configuring OKE_TOP.......COMPLETED
Configuring OKL_TOP.......COMPLETED
Configuring OKS_TOP.......COMPLETED
Configuring CSF_TOP.......COMPLETED
Configuring IGS_TOP.......COMPLETED
Configuring IBY_TOP.......COMPLETED
Configuring JTF_TOP.......COMPLETED
Configuring MWA_TOP.......COMPLETED
Configuring CN_TOP........COMPLETED
Configuring CSI_TOP.......COMPLETED
Configuring WIP_TOP.......COMPLETED
Configuring CSE_TOP.......COMPLETED
Configuring EAM_TOP.......COMPLETED
Configuring FTE_TOP.......COMPLETED
Configuring ONT_TOP.......COMPLETED
Configuring AR_TOP........COMPLETED
Configuring AHL_TOP.......COMPLETED
Configuring OZF_TOP.......COMPLETED
Configuring IES_TOP.......COMPLETED
Configuring CSD_TOP.......COMPLETED
Configuring IGC_TOP.......COMPLETED

AutoConfig completed successfully.

5. Start application middle tier services and verify all is working now

[applmgr@tusebs scripts]$ $ADMIN_SCRIPTS_HOME/adstrtal.sh apps/apps

You are running adstrtal.sh version 120.13.12000000.3

Executing service control script:
/appstier12/CLONE/inst/apps/CLONE_tusebs/admin/scripts/adopmnctl.sh start
script returned:
****************************************************

You are running adopmnctl.sh version 120.4.12000000.3

Starting Oracle Process Manager (OPMN) ...

adopmnctl.sh: exiting with status 0

adopmnctl.sh: check the logfile /appstier12/CLONE/inst/apps/CLONE_tusebs/logs/appl/admin/log/adopmnctl.txt for more information ...


.end std out.

.end err out.

****************************************************


Executing service control script:
/appstier12/CLONE/inst/apps/CLONE_tusebs/admin/scripts/adapcctl.sh start
script returned:
****************************************************

You are running adapcctl.sh version 120.6.12000000.2

Starting OPMN managed Oracle HTTP Server (OHS) instance ...

adapcctl.sh: exiting with status 0

adapcctl.sh: check the logfile /appstier12/CLONE/inst/apps/CLONE_tusebs/logs/appl/admin/log/adapcctl.txt for more information ...


.end std out.

.end err out.

****************************************************


Executing service control script:
/appstier12/CLONE/inst/apps/CLONE_tusebs/admin/scripts/adoacorectl.sh start
script returned:
****************************************************

You are running adoacorectl.sh version 120.11.12000000.2

Starting OPMN managed OACORE OC4J instance ...

adoacorectl.sh: exiting with status 0

adoacorectl.sh: check the logfile /appstier12/CLONE/inst/apps/CLONE_tusebs/logs/appl/admin/log/adoacorectl.txt for more information ...


.end std out.

.end err out.

****************************************************


Executing service control script:
/appstier12/CLONE/inst/apps/CLONE_tusebs/admin/scripts/adformsctl.sh start
script returned:
****************************************************

You are running adformsctl.sh version 120.12.12000000.3

Starting OPMN managed FORMS OC4J instance ...

adformsctl.sh: exiting with status 0

adformsctl.sh: check the logfile /appstier12/CLONE/inst/apps/CLONE_tusebs/logs/appl/admin/log/adformsctl.txt for more information ...


.end std out.

.end err out.

****************************************************


Executing service control script:
/appstier12/CLONE/inst/apps/CLONE_tusebs/admin/scripts/adoafmctl.sh start
script returned:
****************************************************

You are running adoafmctl.sh version 120.6.12000000.2

Starting OPMN managed OAFM OC4J instance ...

adoafmctl.sh: exiting with status 0

adoafmctl.sh: check the logfile /appstier12/CLONE/inst/apps/CLONE_tusebs/logs/appl/admin/log/adoafmctl.txt for more information ...


.end std out.

.end err out.

****************************************************


Executing service control script:
/appstier12/CLONE/inst/apps/CLONE_tusebs/admin/scripts/adalnctl.sh start
script returned:
****************************************************

adalnctl.sh version 120.3

Checking for FNDFS executable.
Listener APPS_CLONE has already been started.

adalnctl.sh: exiting with status 2


adalnctl.sh: check the logfile /appstier12/CLONE/inst/apps/CLONE_tusebs/logs/appl/admin/log/adalnctl.txt for more information ...


.end std out.

.end err out.

****************************************************


Executing service control script:
/appstier12/CLONE/inst/apps/CLONE_tusebs/admin/scripts/adcmctl.sh start
script returned:
****************************************************

You are running adcmctl.sh version 120.14

Starting concurrent manager for CLONE ...
Starting CLONE_0210@CLONE Internal Concurrent Manager
Default printer is noprint

adcmctl.sh: exiting with status 0


adcmctl.sh: check the logfile /appstier12/CLONE/inst/apps/CLONE_tusebs/logs/appl/admin/log/adcmctl.txt for more information ...


.end std out.

.end err out.

****************************************************


Executing service control script:
/appstier12/CLONE/inst/apps/CLONE_tusebs/admin/scripts/jtffmctl.sh start
script returned:
****************************************************

You are running jtffmctl.sh version 120.3

Validating Fulfillment patch level via /appstier12/CLONE/comn/java/classes
Fulfillment patch level validated.
Starting Fulfillment Server for CLONE on port 9300 ...

jtffmctl.sh: exiting with status 0


.end std out.

.end err out.

****************************************************


All enabled services for this node are started.

adstrtal.sh: Exiting with status 0

adstrtal.sh: check the logfile /appstier12/CLONE/inst/apps/CLONE_tusebs/logs/appl/admin/log/adstrtal.log for more information ...

[applmgr@tusebs scripts]$ $ADMIN_SCRIPTS_HOME/adapcctl.sh status

You are running adapcctl.sh version 120.6.12000000.2

Checking status of OPMN managed Oracle HTTP Server (OHS) instance ...

Processes in Instance: CLONE_tusebs.tusebs.localdomain
-------------------+--------------------+---------+---------
ias-component | process-type | pid | status
-------------------+--------------------+---------+---------
OC4J | oafm | 25028 | Alive
OC4J | forms | 24953 | Alive
OC4J | oacore | 24866 | Alive
HTTP_Server | HTTP_Server | 24812 | Alive


adapcctl.sh: exiting with status 0

And Voila! Issue has been resolved and no more annoying interrupted exception!