Monday, November 28, 2011
How to fix archive stuck error with Oracle 11gR2
ORA-00257: archiver error. Connect internal only, until freed.
Sure enough, upon examination of the ADRCI alert.log entries, I found this:
* 2011-10-27 16:24:51.990 4265 krsh.c
ARC1: Error 19809 Creating archive log file to '+FRA'
*** 2011-10-27 16:24:51.990 2864 krsi.c
krsi_dst_fail: dest:1 err:19809 force:0 blast:1
DDE rules only execution for: ORA 312
----- START Event Driven Actions Dump ----
---- END Event Driven Actions Dump ----
----- START DDE Actions Dump -----
Executing SYNC actions
----- START DDE Action: 'DB_STRUCTURE_INTEGRITY_CHECK' (Async) -----
Successfully dispatched
----- END DDE Action: 'DB_STRUCTURE_INTEGRITY_CHECK' (SUCCESS, 0 csec) -----
Executing ASYNC actions
----- END DDE Actions Dump (total 0 csec) -----
DDE: Problem Key 'ORA 312' was flood controlled (0x5) (no incident)
ORA-00312: online log 2 thread 3: '+DATA/vis/onlinelog/group_2.259.743840813'
ORA-00312: online log 2 thread 3: '+FRA/vis/onlinelog/group_2.259.743840815'
Solution:
The Fast Recovery Area (FRA) is full with old archive log files so we need to clean this up by removing old archive logs.
If you have production environment, back these up to tape or storage for recovery and archival purposes.
Login to RMAN and run the following commands:
RMAN> CONNECT target /
connected TO target DATABASE: INSTANT (DBID=1234567890)
RMAN> crosscheck archivelog ALL;
Note: You can schedule this command once a week.
The crosscheck command does NOT DELETE the information about the logs that it
could NOT find ON disk, it just updates their STATUS IN the repository AS 'EXPIRED'.
TO obtain a list OF logs marked AS 'EXPIRED' USE the following command:
RMAN> list expired archivelog ALL;
IF it IS NOT necessary TO keep the information about these
logs IN the repository, DELETE them WITH command:
RMAN> DELETE expired archivelog ALL;
RMAN> delete archivelog all completed before 'SYSDATE-1';
The above RMAN command will delete old archive logs past 24 hours.
Now you should be able to connect to the database.
Below are some useful queries to check on FRA available space:
SQL> SELECT name, free_mb, total_mb, free_mb/total_mb*100 "%" FROM v$asm_diskgroup;
NAME FREE_MB TOTAL_MB %
------------------------------ ---------- ---------- ----------
DATA 823641 1044450 78.8588252
FRA 207743 511992 40.5754387
LOG 306958 307191 99.9241514
SQL>
set lines 100
col name format a60
select name
, floor(space_limit / 1024 / 1024) "Size MB"
, ceil(space_used / 1024 / 1024) "Used MB"
from v$recovery_file_dest
order by name
/
NAME Size MB Used MB
------------------------------------------------------------ ---------- ----------
+FRA 102400 302965
If you want to change the default size for the FRA you can issue the following command from SQL*PLUS:
SQL> alter system set db_recovery_file_dest_size=600G;
System altered.
Then you should archive logs:
SQL> archive log all;
3 logs archived.
Sunday, October 16, 2011
My review Oracle OpenWorld 2011
Oracle OpenWorld 2011 was a fantastic event with many new product announcements including Oracle Public Cloud as well as Oracle Big Data Appliances along with Exalytics for Business Intelligence. Before the conference, I had the pleasure of attending the Oracle ACE Director briefing at Oracle HQ in Redwood Shores under NDA to learn of new developments at Oracle.
After the ACED briefing, I headed to San Francisco with my fellow Oracle ACE Director, Arup Nanda and we checked into the Hilton Hotel. The computers failed during checkin so we had the keep the patience of a saint which paid off with a free complimentary full breakfast. My hotel room had the most amazing view of the entire San Francisco Bay area including views of the Golden Gate bridge and Alcatraz island!

My favorite part of the conference was meeting new people and the special interest groups (SIG) for virtualization with Oracle. I had a great time meeting a lot of Oracle gurus including Jonathan Lewis, Tim Gorman, Doug Burns, James Morle, Michael Abbey, and my co-author Syed Jaffar Hussein of my Oracle 11g RAC book as well as James and Kerry from PACKT Press. Some more cool things at OOW this year was the America's cup winning yacht:
This year we had a great time at OTN night and the appreciation event was wonderful with Sting in concert. Funny because it seemed like Sting was singing in slow motion on songs like Roxanne!
The lightshow with Sting was incredible!
On Tuesday, I delivered my session on database security and would like to thank the 400+ attendees who came to my session. I do hope it was informative and useful to your journey to implementing security for your Oracle environments. The weather was a bit challenging with lots of rain and cold this year.
Fortunately I had an umbrella to keep me dry.
I would like to thank Justin, Lillian, and Vikki for all their hard work to make the OOW conference and ACE briefing a total victory and success this year.
Wednesday, September 28, 2011
OpenWorld 2011
It is rapidly approaching time again for one of the best technical conferences, Oracle OpenWorld!
Come learn how to secure your Oracle database environment with new features at my session next week.
Session ID: 02180
Session Title: Lockdown: Secure Your Database in a Day
Room: Moscone South- 303
Date/Time: 10/4/11, 11:45-12:45
I will also be hosting a book signing after the event in the Oracle bookstore at OpenWorld. Hope to see everyone there.
Cheers,
Ben
Tuesday, September 20, 2011
Howto- Install Oracle 11g Grid Control Enterprise Manager (OEM) Server
Preparation
OEM 11g Grid Control server is a resource hog and requires A LOT more memory than 10g OEM Grid Control so make sure you allocate a new server with at least 16GB of RAM or you will have performance issues.
Summary of Steps
1. Download the OEM 11g Grid Control software from Oracle Edelivery or OTN site (otn.oracle.com)
2. Download Oracle Weblogic 10.3.2 Server
Note; DO NOT use later version of Weblogic as this will NOT work with setup for 11g Grid Control Server!
3. Download Java JDK version 1.6 or later
4. Download required packages for Linux
Setup Java and Weblogic:
1. install and configure java
export PATH=$PATH:/usr/local/jdk/jdk1.5.0_10/bin
2. install and setup oem grid database
In our case, we used Oracle 11.2.0.2 database for the repository with Grid Control
3. post-db config tasks
$ emca -deconfig dbcontrol db -repos drop -SYS_PWD
Login to the OEM repository Oracle 11gR2 database
Make the following initialization parameter changes.
sqlplus / AS SYSDBA
ALTER SYSTEM SET log_buffer=10485760 SCOPE=SPFILE;
ALTER SYSTEM SET processes=500 SCOPE=SPFILE;
ALTER SYSTEM SET session_cached_cursors=200 SCOPE=SPFILE;
In another shell window, un-install the database control:
$ emca -deconfig dbcontrol db -repos drop -SYS_PWD oracle -SYSMAN_PWD oracle
STARTED EMCA at Sep 3, 2011 11:43:41 AM
EM Configuration Assistant, Version 11.2.0.0.2 Production
Copyright (c) 2003, 2005, Oracle. All rights reserved.
Enter the following information:
Database SID: oem
Listener port number: 1521
----------------------------------------------------------------------
WARNING : While repository is dropped the database will be put in quiesce mode.
----------------------------------------------------------------------
Do you wish to continue? [yes(Y)/no(N)]: Y
Sep 3, 2011 11:43:54 AM oracle.sysman.emcp.EMConfig perform
INFO: This operation is being logged at /u01/app/oracle/cfgtoollogs/emca/oem/emca_2011_09_03_11_43_41.log.
Sep 3, 2011 11:43:54 AM oracle.sysman.emcp.util.DBControlUtil stopOMS
INFO: Stopping Database Control (this may take a while) ...
Sep 3, 2011 11:44:15 AM oracle.sysman.emcp.EMReposConfig invoke
INFO: Dropping the EM repository (this may take a while) ...
Sep 3, 2011 11:45:23 AM oracle.sysman.emcp.EMReposConfig invoke
INFO: Repository successfully dropped
Enterprise Manager configuration completed successfully
FINISHED EMCA at Sep 3, 2011 11:45:25 AM
4. Install weblogic server 10.3.2
$ java -d64 -jar wls1032_generic.jar
5. Apply patch for weblogic
Next we must apply the "WDJ7" patch to the installation by running the Smart Update utility.
Notes:
The installation will complete without this patch, so if you have any trouble you can ignore it.
Newer versions of the Smart Update utility no longer allow you to download patches using the utility itself. For the revised patch download instructions see MOS Notes 876004.1 and 1302053.1.
download patch from MOS site
unzip p9438213_10320_Generic.zip -d /u01/weblogic/utils/bsu/cache_dir
Execute the following command to apply the Weblogic patch with the BSU utility:
Syntax:
./bsu.sh -prod_dir=
For example:
./bsu.sh -prod_dir=/opt/bea/weblogic92 -patch_download_dir=/opt/bea/utils/bsu/cache_dir -status=downloaded -view -verbose
This is the syntax for the command to install a patch:
./bsu.sh -prod_dir=/u01/weblogic -patchlist=WDJ7 -verbose -install
This is the syntax for the command to check if the patch is installed:
./bsu.sh -prod_dir=
For example:
./bsu.sh -prod_dir=/opt/bea/weblogic92 -status=applied -verbose -view
$ cd /u01/app/oracle/Middleware/utils/bsu
$ sh bsu.sh
Update the patch client by clicking the "OK" button on the message dialog.
./bsu.sh -prod_dir=/u01/weblogic/wlserver_10.3 -patch_download_dir=/u01/weblogic/utils/bsu -patchlist=4D53,NIXN,XLXA -verbose -install
./bsu.sh -prod_dir=/u01/weblogic/wlserver_10.3 -patch_download_dir=/u01/weblogic/utils/bsu -patchlist=WDJ7 -verbose -install
Unable to apply patch for weblogic 10.3.2 server
Oh no Mr. Bill! We hit a Weblogic bug! Seems like the BSU utility does not play well with some online systems. So, we tried the following approach:
$ ./bsu.sh -prod_dir=/u01/weblogic/wlserver_10.3 -patch_download_dir=/u01/weblogic/utils/bsu -patchlist=4D53,NIXN,XLXA -verbose -install
Encountered unrecognized patch ID: 4D53
Encountered unrecognized patch ID: NIXN
Encountered unrecognized patch ID: XLXA
$ ./bsu.sh -view -patch_download_dir=/u01/weblogic/utils/bsu/cache_dir -status=applied -verbose -prod_dir=/u01/weblogic/wlserver_10.3
ProductName: WebLogic Server
ProductVersion: 10.3 MP2
Components: WebLogic Server/Core Application Server,WebLogic Server/Admi
nistration Console,WebLogic Server/Configuration Wizard and
Upgrade Framework,WebLogic Server/Web 2.0 HTTP Pub-Sub Serve
r,WebLogic Server/WebLogic SCA,WebLogic Server/WebLogic JDBC
Drivers,WebLogic Server/Third Party JDBC Drivers,WebLogic S
erver/WebLogic Server Clients,WebLogic Server/WebLogic Web S
erver Plugins,WebLogic Server/UDDI and Xquery Support,WebLog
ic Server/Workshop Code Completion Support
BEAHome: /u01/weblogic
ProductHome: /u01/weblogic/wlserver_10.3
PatchSystemDir: /u01/weblogic/utils/bsu
PatchDir: /u01/weblogic/patch_wls1032
Profile: Default
DownloadDir: /u01/weblogic/utils/bsu/cache_dir
JavaVersion: 1.6.0_14
JavaVendor: Sun
Let's Fix this issue with apply Weblogic Patch:
Solution: rename patch-catalog-
I found the answer on the OTN Oracle forums:
https://forums.oracle.com/forums/thread.jspa?threadID=1064396
Now, this is not supported by Oracle so you should not do this in a production environment without guidance of Oracle support! Since we are doing a test setup, we can experiment!
$ mv patch-catalog_15563.xml patch-catalog.xml
$ cd ..
$ ./bsu.sh -prod_dir=/u01/weblogic/wlserver_10.3 -patch_download_dir=/u01/weblogic/utils/bsu/cache_dir -patchlist=4D53,NIXN,XLXA -verbose -install
Checking for conflicts..
No conflict(s) detected
Starting installation of Patch ID: 4D53
Installing /u01/weblogic/utils/bsu/cache_dir/4D53.jar
Extracting /u01/weblogic/patch_wls1032/patch_jars/Bug9100465_1032.jar
Updating /u01/weblogic/patch_wls1032/profiles/default/sys_manifest_classpath/weblogic_patch.jar
Old manifest value: Class-Path=
New manifest value: Class-Path=../../../patch_jars/Bug9100465_1032.jar
Result: Success
Starting installation of Patch ID: NIXN
Installing /u01/weblogic/utils/bsu/cache_dir/NIXN.jar
Extracting /u01/weblogic/patch_wls1032/patch_jars/bug8990616_1032.jar
Updating /u01/weblogic/patch_wls1032/profiles/default/sys_manifest_classpath/weblogic_patch.jar
Old manifest value: Class-Path=../../../patch_jars/Bug9100465_1032.jar
New manifest value: Class-Path=../../../patch_jars/bug8990616_1032.jar ../../../patch_jars/Bug9100465_1032.jar
Result: Success
Starting installation of Patch ID: XLXA
Installing /u01/weblogic/utils/bsu/cache_dir/XLXA.jar
Extracting /u01/weblogic/patch_wls1032/patch_jars/Bug9221722_1032.jar
Updating /u01/weblogic/patch_wls1032/profiles/default/sys_manifest_classpath/weblogic_patch.jar
Old manifest value: Class-Path=../../../patch_jars/bug8990616_1032.jar ../../../patch_jars/Bug9100465_1032.jar
New manifest value: Class-Path=../../../patch_jars/Bug9221722_1032.jar ../../../patch_jars/bug8990616_1032.jar ../../../patch_jars/Bug9100465_1032.jar
Result: Success
Verify weblogic patch was installed successfully
Ok, lets verify again that the Weblogic Patch was applied:
$ ./bsu.sh -view -patch_download_dir=/u01/weblogic/utils/bsu/cache_dir -status=applied -verbose -prod_dir=/u01/weblogic/wlserver_10.3
ProductName: WebLogic Server
ProductVersion: 10.3 MP2
Components: WebLogic Server/Core Application Server,WebLogic Server/Admi
nistration Console,WebLogic Server/Configuration Wizard and
Upgrade Framework,WebLogic Server/Web 2.0 HTTP Pub-Sub Serve
r,WebLogic Server/WebLogic SCA,WebLogic Server/WebLogic JDBC
Drivers,WebLogic Server/Third Party JDBC Drivers,WebLogic S
erver/WebLogic Server Clients,WebLogic Server/WebLogic Web S
erver Plugins,WebLogic Server/UDDI and Xquery Support,WebLog
ic Server/Workshop Code Completion Support
BEAHome: /u01/weblogic
ProductHome: /u01/weblogic/wlserver_10.3
PatchSystemDir: /u01/weblogic/utils/bsu
PatchDir: /u01/weblogic/patch_wls1032
Profile: Default
DownloadDir: /u01/weblogic/utils/bsu/cache_dir
JavaVersion: 1.6.0_14
JavaVendor: Sun
Patch ID: 4D53
PatchContainer: 4D53.jar
Checksum: -653702491
Severity: optional
Category: Web App
CR/BUG:
Restart: true
Description: weblogic http client implementation, does not handle keepali
ve conf (http1.0)
Patch ID: NIXN
PatchContainer: NIXN.jar
Checksum: -1413651003
Severity: optional
Category: Core
CR/BUG:
Restart: true
Description: Connecting to wls with non-ssl IIOP port fails if the port i
s enabled on a server running without a non-ssl IIOP channel
.
Patch ID: XLXA
PatchContainer: XLXA.jar
Checksum: 1670664663
Severity: optional
Category: Web App
CR/BUG:
Restart: true
Description: Jsp compiler shouldn't recompile jsp page when page-check-se
conds is set to -1
Whew! Now, that we have Weblogic 10.3.2 installed and patched, we need to change settings and resize the OEM 11g Grid Control repository database.
Resize undo tablespace to 500G
To avoid an error during installation for Grid Control, you need to resize the undo tablespace.
select file_name from dba_data_files where tablespace_name ='UNDOTBS1';
FILE_NAME
--------------------------------------------------------------------------------
/u01/app/oracle/oradata/oem/undotbs01.dbf
Then we alter the size of the undo tablespace file, set autoextend on and maxsize to unlimited:
SQL> select file_name from dba_data_files where tablespace_name ='UNDOTBS1';
FILE_NAME
--------------------------------------------------------------------------------
/u01/app/oracle/oradata/oem/undotbs01.dbf
SQL> alter database datafile '/u01/app/oracle/oradata/oem/undotbs01.dbf' resize 1500M;
Database altered.
SQL> alter database datafile '/u01/app/oracle/oradata/oem/undotbs01.dbf' autoextend on maxsize unlimited;
Database altered.
Start the OEM 11g Grid Control installation program with ./runInstaller
I will spare you the screenshots as these are available online in the Oracle 11g Grid Control installation documentation and also at Dr. Tim Hall's blog at http://www.oracle-base.com/articles/11g/GridControl11gR1InstallationOnOEL5.php
Once the Grid Control setup completes, take down the following information for reference:
1. Enterprise Manager Grid Control URL: https://oem:7799/em
2. Admin Server URL: https://oem:7101/console
The following details need to be provided during the additional OMS install:
1. Admin Server Hostname: oem
2. Admin Server Port: 7101
Special thanks to Dr. Tim Hall for his excellent blog spot and walk through on 11g Grid Control. As you can see, Oracle 11gR1 Grid Control is far more resource intensive to setup and run than previous versions.
Oracle 11gR1 Grid Control Enterprise Manager Collection fails

This was a cryptic error and after research, we found the solution in the following My Oracle Support:
Note: 271598.1
Grid Agent Configuration: Certain Operations from Grid Console Fail With 'ERROR: NMO Not Setuid-root (Unix-only)'
So, I performed the following steps to resolve the issue with Oracle 11g Grid Control:
./emctl stop agent
Oracle Enterprise Manager 11g Release 1 Grid Control 11.1.0.1.0
Copyright (c) 1996, 2010 Oracle Corporation. All rights reserved.
Stopping agent ... stopped.
Open a new shell window logged in as the root user:
# cd /home/oracle/agent11g/
# ./root.sh
As the oracle user, start the agent on the monitored host:
$ ./emctl start agent
Oracle Enterprise Manager 11g Release 1 Grid Control 11.1.0.1.0
Copyright (c) 1996, 2010 Oracle Corporation. All rights reserved.
Starting agent ....... started.
Now comes the secret sauce or step that solves issue:
./emctl control agent runcollection rac1:host host_storage
Oracle Enterprise Manager 11g Release 1 Grid Control 11.1.0.1.0
Copyright (c) 1996, 2010 Oracle Corporation. All rights reserved.
---------------------------------------------------------------
EMD runCollection completed successfully
In the above command emctl control agent runcollection hostname:host host_storage
you will update the details sent to the OEM 11g Grid Control server.
Perform upload to OEM 11g Grid Control Server:
$ ./emctl upload
Oracle Enterprise Manager 11g Release 1 Grid Control 11.1.0.1.0
Copyright (c) 1996, 2010 Oracle Corporation. All rights reserved.
---------------------------------------------------------------
EMD upload completed successfully
Verify it has resolved issue. You will see the current host status as shown in the below example:

Stay tuned for more Oracle 11g Grid control tips. I will share my lessons learned next for the installation process with Oracle 11gR1 Grid Control Server.
Oracle 11g Agent silent installation with OEM 11g Grid Control
After setup of a new Oracle 11gR1 Grid control Enteprise manager (OEM) server, I had to deploy agents to monitor the Oracle environments. Unfortunately, the mass push automation process via OEM 11g Grid Control kept failing and numerous support calls to Oracle Support alas yielded no success. I was able to quickly automate the process via silent installation and deployment of Oracle 11g agents with Grid Control 11g. Below is the process of steps to setup the Oracle 11g agents for Oracle 11g Grid Control:
1) Download and unzip the agent 11g software to hosts monitored. In my case, I needed to configure agents for Oracle 11g RAC on Red Hat Enterprise Linux 5 platform.
cd /home/oracle
unzip Linux_x86_64_Grid_Control_agent_download_11_1_0_1_0.zip
Configure response file for silent agent installation
Before we can kickoff the silent install for the 11g agent, you need to edit the response file called additional_agent.rsp located under the $AGENT_HOME/response directory as shown in my example below:
cd /home/oracle/linux_x64/response
Edit the file with editor of your choice, in my case, I used vi editor on Linux.
Since we don't want updates and don't care about My Oracle support details, we will comment out the email address specifics.
$ vi additional_agent.rsp
SECURITY_UPDATES_VIA_MYORACLESUPPORT=FALSE
#DECLINE_SECURITY_UPDATES=
#MYORACLESUPPORT_USERNAME=
#MYORACLESUPPORT_PASSWORD=
#COLLECTOR_SUPPORTHUB_URL=
INSTALL_UPDATES_SELECTION="skip"
#STAGE_LOCATION=
#PROXY_USER=
#PROXY_PWD=
#PROXY_HOST=
#PROXY_PORT=
ORACLE_AGENT_HOME_LOCATION=/home/oracle
b_silentInstall=true
CLUSTER_NODES=rac1,rac2,rac3,rac4
#-------------------------------------------------------------------------------
OMS_HOST=oem
OMS_PORT=4900
AGENT_REGISTRATION_PASSWORD=yourpassword
Since we have an Oracle 11g RAC environment, we set the parameter in the response file for CLUSTER_NODES to the cluster RAC node names. If you do not have RAC, you can comment this parameter out. Make sure to set the correct hostname, OMS port number and agent registration password or your silent agent installation will fail.
Perform silent installation for Oracle 11g Agent with OEM Grid Control 11g
Navigate to the AGENT_HOME/agent directory
cd /home/oracle/linux_x64/agent
Execute the silent agent installer script with the correct options and parameters as shown in the below example:
./runInstaller -silent -responseFile /home/oracle/linux_x64/response/additional_agent.rsp \
-invPtrLoc /u01/app/oracle/product/11.2.0/dbhome_1/oraInst.loc
Note: you must grant the correct location for the Oracle Inventory file (oraInst.loc) or the silent installation will fail for the agent.
Once you silent installation has succeeded you should see the following status:
Running Configuration assistant "Agent Configuration Assistant"
Performing free port detection on host=rac1
Performing targets discovery and agent configuration
Securing the agent
Starting the agent
Configuration assistant "Agent Configuration Assistant" Succeeded
AgentPlugIn:agent configuration finished with status = true
Running Configuration assistant "Agent Add-on Plug-in"
Configuration assistant "Agent Add-on Plug-in" Succeeded
Finally, we verify that we can show all of the agents deployed for the Oracle 11g RAC environment
$ cd /home/oracle/agent11g/bin
$ ./emctl status agent
Oracle Enterprise Manager 11g Release 1 Grid Control 11.1.0.1.0
Copyright (c) 1996, 2010 Oracle Corporation. All rights reserved.
---------------------------------------------------------------
Agent Version : 11.1.0.1.0
OMS Version : 11.1.0.1.0
Protocol Version : 11.1.0.0.0
Agent Home : /home/oracle/agent11g/rac1
Agent binaries : /home/oracle/agent11g
Agent Process ID : 13181
Parent Process ID : 13157
Agent URL : https://rac1:3872/emd/main
Repository URL : https://oem:4900/em/upload
Started at : 2011-09-20 20:28:06
Started by user : oracle
Last Reload : 2011-09-20 20:28:06
Last successful upload : 2011-09-20 20:38:49
Total Megabytes of XML files uploaded so far : 15.68
Number of XML files pending upload : 0
Size of XML files pending upload(MB) : 0.00
Available disk space on upload filesystem : 31.34%
Last successful heartbeat to OMS : 2011-09-20 20:39:10
---------------------------------------------------------------
Agent is Running and Ready
The beauty is when you specify the RAC cluster nodes in the first response file as we did above, is that all of the cluster nodes will have the silent install agent performed at same time!
Application Tier crashes at 100% CPU load
After much research, it was determined to be caused by
Bug 6519700 ESC: CSE: R12SIP: 6513826 FRMWEB RUNAWAY PROCESS CONSUMING 100% CPU-MIDDLE TIER
The fix was to shutdown the apps middle tier server and configure the following forms environment variables:
FORMS_RECORD_GROUP_MAXFORMS_CATCHTERM
1) First we had to update the context file on the application middle tier as follows:
s_forms_catchterm=0
2) Since the FORMS_RECORD_GROUP_MAX is not a default variable in the app tier context file, you need to create a custom template as follows.
cd $AD_TOP/admin/template
mkdir custom
cp $AD_TOP/admin/template/APPLSYS_ux.env $AD_TOP/template/custom/APPLSYS_ux.env
Edit the custom template file:
vi $AD_TOP/admin/template/custom/APPLSYS_ux.env
Set the environment variable FORMS_RECORD_GROUP_MAX to a high value such as 10000 or greater.
FORMS_RECORD_GROUP_MAX=10000
export FORMS_RECORD_GROUP_MAX
Save the file.
Run autoconfig utility for apps middle tier server.
Start app tier services and test.
Now we now longer have this issue.