Dear readers,
It is rapidly approaching time again for one of the best technical conferences, Oracle OpenWorld!
Come learn how to secure your Oracle database environment with new features at my session next week.
Session ID: 02180
Session Title: Lockdown: Secure Your Database in a Day
Room: Moscone South- 303
Date/Time: 10/4/11, 11:45-12:45
I will also be hosting a book signing after the event in the Oracle bookstore at OpenWorld. Hope to see everyone there.
Cheers,
Ben
Wednesday, September 28, 2011
Tuesday, September 20, 2011
Howto- Install Oracle 11g Grid Control Enterprise Manager (OEM) Server
With the new features for Oracle 11g Grid Control Enterprise Manager (OEM), it is valuable tool now for managing large Oracle environments. I recall from Oracle 9i OEM days that it was a buggy tool and back then years ago, I used SQL*Plus and command line tools. However, in 11gR1, OEM has emerged as an excellent monitoring tool. I still use command line tools most of the time but for monitoring, its a great product. With that said, let's get started in building a new Oracle11gR1 Grid Control Enterprise Manager (OEM) environment on Linux!
Preparation
OEM 11g Grid Control server is a resource hog and requires A LOT more memory than 10g OEM Grid Control so make sure you allocate a new server with at least 16GB of RAM or you will have performance issues.
Summary of Steps
1. Download the OEM 11g Grid Control software from Oracle Edelivery or OTN site (otn.oracle.com)
2. Download Oracle Weblogic 10.3.2 Server
Note; DO NOT use later version of Weblogic as this will NOT work with setup for 11g Grid Control Server!
3. Download Java JDK version 1.6 or later
4. Download required packages for Linux
Setup Java and Weblogic:
1. install and configure java
export PATH=$PATH:/usr/local/jdk/jdk1.5.0_10/bin
2. install and setup oem grid database
In our case, we used Oracle 11.2.0.2 database for the repository with Grid Control
3. post-db config tasks
$ emca -deconfig dbcontrol db -repos drop -SYS_PWD -SYSMAN_PWD
Login to the OEM repository Oracle 11gR2 database
Make the following initialization parameter changes.
sqlplus / AS SYSDBA
ALTER SYSTEM SET log_buffer=10485760 SCOPE=SPFILE;
ALTER SYSTEM SET processes=500 SCOPE=SPFILE;
ALTER SYSTEM SET session_cached_cursors=200 SCOPE=SPFILE;
In another shell window, un-install the database control:
$ emca -deconfig dbcontrol db -repos drop -SYS_PWD oracle -SYSMAN_PWD oracle
STARTED EMCA at Sep 3, 2011 11:43:41 AM
EM Configuration Assistant, Version 11.2.0.0.2 Production
Copyright (c) 2003, 2005, Oracle. All rights reserved.
Enter the following information:
Database SID: oem
Listener port number: 1521
----------------------------------------------------------------------
WARNING : While repository is dropped the database will be put in quiesce mode.
----------------------------------------------------------------------
Do you wish to continue? [yes(Y)/no(N)]: Y
Sep 3, 2011 11:43:54 AM oracle.sysman.emcp.EMConfig perform
INFO: This operation is being logged at /u01/app/oracle/cfgtoollogs/emca/oem/emca_2011_09_03_11_43_41.log.
Sep 3, 2011 11:43:54 AM oracle.sysman.emcp.util.DBControlUtil stopOMS
INFO: Stopping Database Control (this may take a while) ...
Sep 3, 2011 11:44:15 AM oracle.sysman.emcp.EMReposConfig invoke
INFO: Dropping the EM repository (this may take a while) ...
Sep 3, 2011 11:45:23 AM oracle.sysman.emcp.EMReposConfig invoke
INFO: Repository successfully dropped
Enterprise Manager configuration completed successfully
FINISHED EMCA at Sep 3, 2011 11:45:25 AM
4. Install weblogic server 10.3.2
$ java -d64 -jar wls1032_generic.jar
5. Apply patch for weblogic
Next we must apply the "WDJ7" patch to the installation by running the Smart Update utility.
Notes:
The installation will complete without this patch, so if you have any trouble you can ignore it.
Newer versions of the Smart Update utility no longer allow you to download patches using the utility itself. For the revised patch download instructions see MOS Notes 876004.1 and 1302053.1.
download patch from MOS site
unzip p9438213_10320_Generic.zip -d /u01/weblogic/utils/bsu/cache_dir
Execute the following command to apply the Weblogic patch with the BSU utility:
Syntax:
./bsu.sh -prod_dir= -patch_download_dir= -status=downloaded -view -verbose
For example:
./bsu.sh -prod_dir=/opt/bea/weblogic92 -patch_download_dir=/opt/bea/utils/bsu/cache_dir -status=downloaded -view -verbose
This is the syntax for the command to install a patch:
./bsu.sh -prod_dir=/u01/weblogic -patchlist=WDJ7 -verbose -install
This is the syntax for the command to check if the patch is installed:
./bsu.sh -prod_dir= -patch_download_dir= -status=applied -verbose -view
For example:
./bsu.sh -prod_dir=/opt/bea/weblogic92 -status=applied -verbose -view
$ cd /u01/app/oracle/Middleware/utils/bsu
$ sh bsu.sh
Update the patch client by clicking the "OK" button on the message dialog.
./bsu.sh -prod_dir=/u01/weblogic/wlserver_10.3 -patch_download_dir=/u01/weblogic/utils/bsu -patchlist=4D53,NIXN,XLXA -verbose -install
./bsu.sh -prod_dir=/u01/weblogic/wlserver_10.3 -patch_download_dir=/u01/weblogic/utils/bsu -patchlist=WDJ7 -verbose -install
Unable to apply patch for weblogic 10.3.2 server
Oh no Mr. Bill! We hit a Weblogic bug! Seems like the BSU utility does not play well with some online systems. So, we tried the following approach:
$ ./bsu.sh -prod_dir=/u01/weblogic/wlserver_10.3 -patch_download_dir=/u01/weblogic/utils/bsu -patchlist=4D53,NIXN,XLXA -verbose -install
Encountered unrecognized patch ID: 4D53
Encountered unrecognized patch ID: NIXN
Encountered unrecognized patch ID: XLXA
$ ./bsu.sh -view -patch_download_dir=/u01/weblogic/utils/bsu/cache_dir -status=applied -verbose -prod_dir=/u01/weblogic/wlserver_10.3
ProductName: WebLogic Server
ProductVersion: 10.3 MP2
Components: WebLogic Server/Core Application Server,WebLogic Server/Admi
nistration Console,WebLogic Server/Configuration Wizard and
Upgrade Framework,WebLogic Server/Web 2.0 HTTP Pub-Sub Serve
r,WebLogic Server/WebLogic SCA,WebLogic Server/WebLogic JDBC
Drivers,WebLogic Server/Third Party JDBC Drivers,WebLogic S
erver/WebLogic Server Clients,WebLogic Server/WebLogic Web S
erver Plugins,WebLogic Server/UDDI and Xquery Support,WebLog
ic Server/Workshop Code Completion Support
BEAHome: /u01/weblogic
ProductHome: /u01/weblogic/wlserver_10.3
PatchSystemDir: /u01/weblogic/utils/bsu
PatchDir: /u01/weblogic/patch_wls1032
Profile: Default
DownloadDir: /u01/weblogic/utils/bsu/cache_dir
JavaVersion: 1.6.0_14
JavaVendor: Sun
Let's Fix this issue with apply Weblogic Patch:
Solution: rename patch-catalog-
I found the answer on the OTN Oracle forums:
https://forums.oracle.com/forums/thread.jspa?threadID=1064396
Now, this is not supported by Oracle so you should not do this in a production environment without guidance of Oracle support! Since we are doing a test setup, we can experiment!
$ mv patch-catalog_15563.xml patch-catalog.xml
$ cd ..
$ ./bsu.sh -prod_dir=/u01/weblogic/wlserver_10.3 -patch_download_dir=/u01/weblogic/utils/bsu/cache_dir -patchlist=4D53,NIXN,XLXA -verbose -install
Checking for conflicts..
No conflict(s) detected
Starting installation of Patch ID: 4D53
Installing /u01/weblogic/utils/bsu/cache_dir/4D53.jar
Extracting /u01/weblogic/patch_wls1032/patch_jars/Bug9100465_1032.jar
Updating /u01/weblogic/patch_wls1032/profiles/default/sys_manifest_classpath/weblogic_patch.jar
Old manifest value: Class-Path=
New manifest value: Class-Path=../../../patch_jars/Bug9100465_1032.jar
Result: Success
Starting installation of Patch ID: NIXN
Installing /u01/weblogic/utils/bsu/cache_dir/NIXN.jar
Extracting /u01/weblogic/patch_wls1032/patch_jars/bug8990616_1032.jar
Updating /u01/weblogic/patch_wls1032/profiles/default/sys_manifest_classpath/weblogic_patch.jar
Old manifest value: Class-Path=../../../patch_jars/Bug9100465_1032.jar
New manifest value: Class-Path=../../../patch_jars/bug8990616_1032.jar ../../../patch_jars/Bug9100465_1032.jar
Result: Success
Starting installation of Patch ID: XLXA
Installing /u01/weblogic/utils/bsu/cache_dir/XLXA.jar
Extracting /u01/weblogic/patch_wls1032/patch_jars/Bug9221722_1032.jar
Updating /u01/weblogic/patch_wls1032/profiles/default/sys_manifest_classpath/weblogic_patch.jar
Old manifest value: Class-Path=../../../patch_jars/bug8990616_1032.jar ../../../patch_jars/Bug9100465_1032.jar
New manifest value: Class-Path=../../../patch_jars/Bug9221722_1032.jar ../../../patch_jars/bug8990616_1032.jar ../../../patch_jars/Bug9100465_1032.jar
Result: Success
Verify weblogic patch was installed successfully
Ok, lets verify again that the Weblogic Patch was applied:
$ ./bsu.sh -view -patch_download_dir=/u01/weblogic/utils/bsu/cache_dir -status=applied -verbose -prod_dir=/u01/weblogic/wlserver_10.3
ProductName: WebLogic Server
ProductVersion: 10.3 MP2
Components: WebLogic Server/Core Application Server,WebLogic Server/Admi
nistration Console,WebLogic Server/Configuration Wizard and
Upgrade Framework,WebLogic Server/Web 2.0 HTTP Pub-Sub Serve
r,WebLogic Server/WebLogic SCA,WebLogic Server/WebLogic JDBC
Drivers,WebLogic Server/Third Party JDBC Drivers,WebLogic S
erver/WebLogic Server Clients,WebLogic Server/WebLogic Web S
erver Plugins,WebLogic Server/UDDI and Xquery Support,WebLog
ic Server/Workshop Code Completion Support
BEAHome: /u01/weblogic
ProductHome: /u01/weblogic/wlserver_10.3
PatchSystemDir: /u01/weblogic/utils/bsu
PatchDir: /u01/weblogic/patch_wls1032
Profile: Default
DownloadDir: /u01/weblogic/utils/bsu/cache_dir
JavaVersion: 1.6.0_14
JavaVendor: Sun
Patch ID: 4D53
PatchContainer: 4D53.jar
Checksum: -653702491
Severity: optional
Category: Web App
CR/BUG:
Restart: true
Description: weblogic http client implementation, does not handle keepali
ve conf (http1.0)
Patch ID: NIXN
PatchContainer: NIXN.jar
Checksum: -1413651003
Severity: optional
Category: Core
CR/BUG:
Restart: true
Description: Connecting to wls with non-ssl IIOP port fails if the port i
s enabled on a server running without a non-ssl IIOP channel
.
Patch ID: XLXA
PatchContainer: XLXA.jar
Checksum: 1670664663
Severity: optional
Category: Web App
CR/BUG:
Restart: true
Description: Jsp compiler shouldn't recompile jsp page when page-check-se
conds is set to -1
Whew! Now, that we have Weblogic 10.3.2 installed and patched, we need to change settings and resize the OEM 11g Grid Control repository database.
Resize undo tablespace to 500G
To avoid an error during installation for Grid Control, you need to resize the undo tablespace.
select file_name from dba_data_files where tablespace_name ='UNDOTBS1';
FILE_NAME
--------------------------------------------------------------------------------
/u01/app/oracle/oradata/oem/undotbs01.dbf
Then we alter the size of the undo tablespace file, set autoextend on and maxsize to unlimited:
SQL> select file_name from dba_data_files where tablespace_name ='UNDOTBS1';
FILE_NAME
--------------------------------------------------------------------------------
/u01/app/oracle/oradata/oem/undotbs01.dbf
SQL> alter database datafile '/u01/app/oracle/oradata/oem/undotbs01.dbf' resize 1500M;
Database altered.
SQL> alter database datafile '/u01/app/oracle/oradata/oem/undotbs01.dbf' autoextend on maxsize unlimited;
Database altered.
Start the OEM 11g Grid Control installation program with ./runInstaller
I will spare you the screenshots as these are available online in the Oracle 11g Grid Control installation documentation and also at Dr. Tim Hall's blog at http://www.oracle-base.com/articles/11g/GridControl11gR1InstallationOnOEL5.php
Once the Grid Control setup completes, take down the following information for reference:
1. Enterprise Manager Grid Control URL: https://oem:7799/em
2. Admin Server URL: https://oem:7101/console
The following details need to be provided during the additional OMS install:
1. Admin Server Hostname: oem
2. Admin Server Port: 7101
Special thanks to Dr. Tim Hall for his excellent blog spot and walk through on 11g Grid Control. As you can see, Oracle 11gR1 Grid Control is far more resource intensive to setup and run than previous versions.
Preparation
OEM 11g Grid Control server is a resource hog and requires A LOT more memory than 10g OEM Grid Control so make sure you allocate a new server with at least 16GB of RAM or you will have performance issues.
Summary of Steps
1. Download the OEM 11g Grid Control software from Oracle Edelivery or OTN site (otn.oracle.com)
2. Download Oracle Weblogic 10.3.2 Server
Note; DO NOT use later version of Weblogic as this will NOT work with setup for 11g Grid Control Server!
3. Download Java JDK version 1.6 or later
4. Download required packages for Linux
Setup Java and Weblogic:
1. install and configure java
export PATH=$PATH:/usr/local/jdk/jdk1.5.0_10/bin
2. install and setup oem grid database
In our case, we used Oracle 11.2.0.2 database for the repository with Grid Control
3. post-db config tasks
$ emca -deconfig dbcontrol db -repos drop -SYS_PWD
Login to the OEM repository Oracle 11gR2 database
Make the following initialization parameter changes.
sqlplus / AS SYSDBA
ALTER SYSTEM SET log_buffer=10485760 SCOPE=SPFILE;
ALTER SYSTEM SET processes=500 SCOPE=SPFILE;
ALTER SYSTEM SET session_cached_cursors=200 SCOPE=SPFILE;
In another shell window, un-install the database control:
$ emca -deconfig dbcontrol db -repos drop -SYS_PWD oracle -SYSMAN_PWD oracle
STARTED EMCA at Sep 3, 2011 11:43:41 AM
EM Configuration Assistant, Version 11.2.0.0.2 Production
Copyright (c) 2003, 2005, Oracle. All rights reserved.
Enter the following information:
Database SID: oem
Listener port number: 1521
----------------------------------------------------------------------
WARNING : While repository is dropped the database will be put in quiesce mode.
----------------------------------------------------------------------
Do you wish to continue? [yes(Y)/no(N)]: Y
Sep 3, 2011 11:43:54 AM oracle.sysman.emcp.EMConfig perform
INFO: This operation is being logged at /u01/app/oracle/cfgtoollogs/emca/oem/emca_2011_09_03_11_43_41.log.
Sep 3, 2011 11:43:54 AM oracle.sysman.emcp.util.DBControlUtil stopOMS
INFO: Stopping Database Control (this may take a while) ...
Sep 3, 2011 11:44:15 AM oracle.sysman.emcp.EMReposConfig invoke
INFO: Dropping the EM repository (this may take a while) ...
Sep 3, 2011 11:45:23 AM oracle.sysman.emcp.EMReposConfig invoke
INFO: Repository successfully dropped
Enterprise Manager configuration completed successfully
FINISHED EMCA at Sep 3, 2011 11:45:25 AM
4. Install weblogic server 10.3.2
$ java -d64 -jar wls1032_generic.jar
5. Apply patch for weblogic
Next we must apply the "WDJ7" patch to the installation by running the Smart Update utility.
Notes:
The installation will complete without this patch, so if you have any trouble you can ignore it.
Newer versions of the Smart Update utility no longer allow you to download patches using the utility itself. For the revised patch download instructions see MOS Notes 876004.1 and 1302053.1.
download patch from MOS site
unzip p9438213_10320_Generic.zip -d /u01/weblogic/utils/bsu/cache_dir
Execute the following command to apply the Weblogic patch with the BSU utility:
Syntax:
./bsu.sh -prod_dir=
For example:
./bsu.sh -prod_dir=/opt/bea/weblogic92 -patch_download_dir=/opt/bea/utils/bsu/cache_dir -status=downloaded -view -verbose
This is the syntax for the command to install a patch:
./bsu.sh -prod_dir=/u01/weblogic -patchlist=WDJ7 -verbose -install
This is the syntax for the command to check if the patch is installed:
./bsu.sh -prod_dir=
For example:
./bsu.sh -prod_dir=/opt/bea/weblogic92 -status=applied -verbose -view
$ cd /u01/app/oracle/Middleware/utils/bsu
$ sh bsu.sh
Update the patch client by clicking the "OK" button on the message dialog.
./bsu.sh -prod_dir=/u01/weblogic/wlserver_10.3 -patch_download_dir=/u01/weblogic/utils/bsu -patchlist=4D53,NIXN,XLXA -verbose -install
./bsu.sh -prod_dir=/u01/weblogic/wlserver_10.3 -patch_download_dir=/u01/weblogic/utils/bsu -patchlist=WDJ7 -verbose -install
Unable to apply patch for weblogic 10.3.2 server
Oh no Mr. Bill! We hit a Weblogic bug! Seems like the BSU utility does not play well with some online systems. So, we tried the following approach:
$ ./bsu.sh -prod_dir=/u01/weblogic/wlserver_10.3 -patch_download_dir=/u01/weblogic/utils/bsu -patchlist=4D53,NIXN,XLXA -verbose -install
Encountered unrecognized patch ID: 4D53
Encountered unrecognized patch ID: NIXN
Encountered unrecognized patch ID: XLXA
$ ./bsu.sh -view -patch_download_dir=/u01/weblogic/utils/bsu/cache_dir -status=applied -verbose -prod_dir=/u01/weblogic/wlserver_10.3
ProductName: WebLogic Server
ProductVersion: 10.3 MP2
Components: WebLogic Server/Core Application Server,WebLogic Server/Admi
nistration Console,WebLogic Server/Configuration Wizard and
Upgrade Framework,WebLogic Server/Web 2.0 HTTP Pub-Sub Serve
r,WebLogic Server/WebLogic SCA,WebLogic Server/WebLogic JDBC
Drivers,WebLogic Server/Third Party JDBC Drivers,WebLogic S
erver/WebLogic Server Clients,WebLogic Server/WebLogic Web S
erver Plugins,WebLogic Server/UDDI and Xquery Support,WebLog
ic Server/Workshop Code Completion Support
BEAHome: /u01/weblogic
ProductHome: /u01/weblogic/wlserver_10.3
PatchSystemDir: /u01/weblogic/utils/bsu
PatchDir: /u01/weblogic/patch_wls1032
Profile: Default
DownloadDir: /u01/weblogic/utils/bsu/cache_dir
JavaVersion: 1.6.0_14
JavaVendor: Sun
Let's Fix this issue with apply Weblogic Patch:
Solution: rename patch-catalog-
I found the answer on the OTN Oracle forums:
https://forums.oracle.com/forums/thread.jspa?threadID=1064396
Now, this is not supported by Oracle so you should not do this in a production environment without guidance of Oracle support! Since we are doing a test setup, we can experiment!
$ mv patch-catalog_15563.xml patch-catalog.xml
$ cd ..
$ ./bsu.sh -prod_dir=/u01/weblogic/wlserver_10.3 -patch_download_dir=/u01/weblogic/utils/bsu/cache_dir -patchlist=4D53,NIXN,XLXA -verbose -install
Checking for conflicts..
No conflict(s) detected
Starting installation of Patch ID: 4D53
Installing /u01/weblogic/utils/bsu/cache_dir/4D53.jar
Extracting /u01/weblogic/patch_wls1032/patch_jars/Bug9100465_1032.jar
Updating /u01/weblogic/patch_wls1032/profiles/default/sys_manifest_classpath/weblogic_patch.jar
Old manifest value: Class-Path=
New manifest value: Class-Path=../../../patch_jars/Bug9100465_1032.jar
Result: Success
Starting installation of Patch ID: NIXN
Installing /u01/weblogic/utils/bsu/cache_dir/NIXN.jar
Extracting /u01/weblogic/patch_wls1032/patch_jars/bug8990616_1032.jar
Updating /u01/weblogic/patch_wls1032/profiles/default/sys_manifest_classpath/weblogic_patch.jar
Old manifest value: Class-Path=../../../patch_jars/Bug9100465_1032.jar
New manifest value: Class-Path=../../../patch_jars/bug8990616_1032.jar ../../../patch_jars/Bug9100465_1032.jar
Result: Success
Starting installation of Patch ID: XLXA
Installing /u01/weblogic/utils/bsu/cache_dir/XLXA.jar
Extracting /u01/weblogic/patch_wls1032/patch_jars/Bug9221722_1032.jar
Updating /u01/weblogic/patch_wls1032/profiles/default/sys_manifest_classpath/weblogic_patch.jar
Old manifest value: Class-Path=../../../patch_jars/bug8990616_1032.jar ../../../patch_jars/Bug9100465_1032.jar
New manifest value: Class-Path=../../../patch_jars/Bug9221722_1032.jar ../../../patch_jars/bug8990616_1032.jar ../../../patch_jars/Bug9100465_1032.jar
Result: Success
Verify weblogic patch was installed successfully
Ok, lets verify again that the Weblogic Patch was applied:
$ ./bsu.sh -view -patch_download_dir=/u01/weblogic/utils/bsu/cache_dir -status=applied -verbose -prod_dir=/u01/weblogic/wlserver_10.3
ProductName: WebLogic Server
ProductVersion: 10.3 MP2
Components: WebLogic Server/Core Application Server,WebLogic Server/Admi
nistration Console,WebLogic Server/Configuration Wizard and
Upgrade Framework,WebLogic Server/Web 2.0 HTTP Pub-Sub Serve
r,WebLogic Server/WebLogic SCA,WebLogic Server/WebLogic JDBC
Drivers,WebLogic Server/Third Party JDBC Drivers,WebLogic S
erver/WebLogic Server Clients,WebLogic Server/WebLogic Web S
erver Plugins,WebLogic Server/UDDI and Xquery Support,WebLog
ic Server/Workshop Code Completion Support
BEAHome: /u01/weblogic
ProductHome: /u01/weblogic/wlserver_10.3
PatchSystemDir: /u01/weblogic/utils/bsu
PatchDir: /u01/weblogic/patch_wls1032
Profile: Default
DownloadDir: /u01/weblogic/utils/bsu/cache_dir
JavaVersion: 1.6.0_14
JavaVendor: Sun
Patch ID: 4D53
PatchContainer: 4D53.jar
Checksum: -653702491
Severity: optional
Category: Web App
CR/BUG:
Restart: true
Description: weblogic http client implementation, does not handle keepali
ve conf (http1.0)
Patch ID: NIXN
PatchContainer: NIXN.jar
Checksum: -1413651003
Severity: optional
Category: Core
CR/BUG:
Restart: true
Description: Connecting to wls with non-ssl IIOP port fails if the port i
s enabled on a server running without a non-ssl IIOP channel
.
Patch ID: XLXA
PatchContainer: XLXA.jar
Checksum: 1670664663
Severity: optional
Category: Web App
CR/BUG:
Restart: true
Description: Jsp compiler shouldn't recompile jsp page when page-check-se
conds is set to -1
Whew! Now, that we have Weblogic 10.3.2 installed and patched, we need to change settings and resize the OEM 11g Grid Control repository database.
Resize undo tablespace to 500G
To avoid an error during installation for Grid Control, you need to resize the undo tablespace.
select file_name from dba_data_files where tablespace_name ='UNDOTBS1';
FILE_NAME
--------------------------------------------------------------------------------
/u01/app/oracle/oradata/oem/undotbs01.dbf
Then we alter the size of the undo tablespace file, set autoextend on and maxsize to unlimited:
SQL> select file_name from dba_data_files where tablespace_name ='UNDOTBS1';
FILE_NAME
--------------------------------------------------------------------------------
/u01/app/oracle/oradata/oem/undotbs01.dbf
SQL> alter database datafile '/u01/app/oracle/oradata/oem/undotbs01.dbf' resize 1500M;
Database altered.
SQL> alter database datafile '/u01/app/oracle/oradata/oem/undotbs01.dbf' autoextend on maxsize unlimited;
Database altered.
Start the OEM 11g Grid Control installation program with ./runInstaller
I will spare you the screenshots as these are available online in the Oracle 11g Grid Control installation documentation and also at Dr. Tim Hall's blog at http://www.oracle-base.com/articles/11g/GridControl11gR1InstallationOnOEL5.php
Once the Grid Control setup completes, take down the following information for reference:
1. Enterprise Manager Grid Control URL: https://oem:7799/em
2. Admin Server URL: https://oem:7101/console
The following details need to be provided during the additional OMS install:
1. Admin Server Hostname: oem
2. Admin Server Port: 7101
Special thanks to Dr. Tim Hall for his excellent blog spot and walk through on 11g Grid Control. As you can see, Oracle 11gR1 Grid Control is far more resource intensive to setup and run than previous versions.
Oracle 11gR1 Grid Control Enterprise Manager Collection fails
Recently we had issue with Oracle 11g Enterprise Manager (OEM) Grid control failing to collect up to data host data with following error from Grid Control:
This was a cryptic error and after research, we found the solution in the following My Oracle Support:
Note: 271598.1
Grid Agent Configuration: Certain Operations from Grid Console Fail With 'ERROR: NMO Not Setuid-root (Unix-only)'
So, I performed the following steps to resolve the issue with Oracle 11g Grid Control:
./emctl stop agent
Oracle Enterprise Manager 11g Release 1 Grid Control 11.1.0.1.0
Copyright (c) 1996, 2010 Oracle Corporation. All rights reserved.
Stopping agent ... stopped.
Open a new shell window logged in as the root user:
# cd /home/oracle/agent11g/
# ./root.sh
As the oracle user, start the agent on the monitored host:
$ ./emctl start agent
Oracle Enterprise Manager 11g Release 1 Grid Control 11.1.0.1.0
Copyright (c) 1996, 2010 Oracle Corporation. All rights reserved.
Starting agent ....... started.
Now comes the secret sauce or step that solves issue:
./emctl control agent runcollection rac1:host host_storage
Oracle Enterprise Manager 11g Release 1 Grid Control 11.1.0.1.0
Copyright (c) 1996, 2010 Oracle Corporation. All rights reserved.
---------------------------------------------------------------
EMD runCollection completed successfully
In the above command emctl control agent runcollection hostname:host host_storage
you will update the details sent to the OEM 11g Grid Control server.
Perform upload to OEM 11g Grid Control Server:
$ ./emctl upload
Oracle Enterprise Manager 11g Release 1 Grid Control 11.1.0.1.0
Copyright (c) 1996, 2010 Oracle Corporation. All rights reserved.
---------------------------------------------------------------
EMD upload completed successfully
Verify it has resolved issue. You will see the current host status as shown in the below example:
Stay tuned for more Oracle 11g Grid control tips. I will share my lessons learned next for the installation process with Oracle 11gR1 Grid Control Server.
This was a cryptic error and after research, we found the solution in the following My Oracle Support:
Note: 271598.1
Grid Agent Configuration: Certain Operations from Grid Console Fail With 'ERROR: NMO Not Setuid-root (Unix-only)'
So, I performed the following steps to resolve the issue with Oracle 11g Grid Control:
./emctl stop agent
Oracle Enterprise Manager 11g Release 1 Grid Control 11.1.0.1.0
Copyright (c) 1996, 2010 Oracle Corporation. All rights reserved.
Stopping agent ... stopped.
Open a new shell window logged in as the root user:
# cd /home/oracle/agent11g/
# ./root.sh
As the oracle user, start the agent on the monitored host:
$ ./emctl start agent
Oracle Enterprise Manager 11g Release 1 Grid Control 11.1.0.1.0
Copyright (c) 1996, 2010 Oracle Corporation. All rights reserved.
Starting agent ....... started.
Now comes the secret sauce or step that solves issue:
./emctl control agent runcollection rac1:host host_storage
Oracle Enterprise Manager 11g Release 1 Grid Control 11.1.0.1.0
Copyright (c) 1996, 2010 Oracle Corporation. All rights reserved.
---------------------------------------------------------------
EMD runCollection completed successfully
In the above command emctl control agent runcollection hostname:host host_storage
you will update the details sent to the OEM 11g Grid Control server.
Perform upload to OEM 11g Grid Control Server:
$ ./emctl upload
Oracle Enterprise Manager 11g Release 1 Grid Control 11.1.0.1.0
Copyright (c) 1996, 2010 Oracle Corporation. All rights reserved.
---------------------------------------------------------------
EMD upload completed successfully
Verify it has resolved issue. You will see the current host status as shown in the below example:
Stay tuned for more Oracle 11g Grid control tips. I will share my lessons learned next for the installation process with Oracle 11gR1 Grid Control Server.
Oracle 11g Agent silent installation with OEM 11g Grid Control
Dear readers,
After setup of a new Oracle 11gR1 Grid control Enteprise manager (OEM) server, I had to deploy agents to monitor the Oracle environments. Unfortunately, the mass push automation process via OEM 11g Grid Control kept failing and numerous support calls to Oracle Support alas yielded no success. I was able to quickly automate the process via silent installation and deployment of Oracle 11g agents with Grid Control 11g. Below is the process of steps to setup the Oracle 11g agents for Oracle 11g Grid Control:
1) Download and unzip the agent 11g software to hosts monitored. In my case, I needed to configure agents for Oracle 11g RAC on Red Hat Enterprise Linux 5 platform.
cd /home/oracle
unzip Linux_x86_64_Grid_Control_agent_download_11_1_0_1_0.zip
Configure response file for silent agent installation
Before we can kickoff the silent install for the 11g agent, you need to edit the response file called additional_agent.rsp located under the $AGENT_HOME/response directory as shown in my example below:
cd /home/oracle/linux_x64/response
Edit the file with editor of your choice, in my case, I used vi editor on Linux.
Since we don't want updates and don't care about My Oracle support details, we will comment out the email address specifics.
$ vi additional_agent.rsp
SECURITY_UPDATES_VIA_MYORACLESUPPORT=FALSE
#DECLINE_SECURITY_UPDATES=
#MYORACLESUPPORT_USERNAME=
#MYORACLESUPPORT_PASSWORD=
#COLLECTOR_SUPPORTHUB_URL=
INSTALL_UPDATES_SELECTION="skip"
#STAGE_LOCATION=
#PROXY_USER=
#PROXY_PWD=
#PROXY_HOST=
#PROXY_PORT=
ORACLE_AGENT_HOME_LOCATION=/home/oracle
b_silentInstall=true
CLUSTER_NODES=rac1,rac2,rac3,rac4
#-------------------------------------------------------------------------------
OMS_HOST=oem
OMS_PORT=4900
AGENT_REGISTRATION_PASSWORD=yourpassword
Since we have an Oracle 11g RAC environment, we set the parameter in the response file for CLUSTER_NODES to the cluster RAC node names. If you do not have RAC, you can comment this parameter out. Make sure to set the correct hostname, OMS port number and agent registration password or your silent agent installation will fail.
Perform silent installation for Oracle 11g Agent with OEM Grid Control 11g
Navigate to the AGENT_HOME/agent directory
cd /home/oracle/linux_x64/agent
Execute the silent agent installer script with the correct options and parameters as shown in the below example:
./runInstaller -silent -responseFile /home/oracle/linux_x64/response/additional_agent.rsp \
-invPtrLoc /u01/app/oracle/product/11.2.0/dbhome_1/oraInst.loc
Note: you must grant the correct location for the Oracle Inventory file (oraInst.loc) or the silent installation will fail for the agent.
Once you silent installation has succeeded you should see the following status:
Running Configuration assistant "Agent Configuration Assistant"
Performing free port detection on host=rac1
Performing targets discovery and agent configuration
Securing the agent
Starting the agent
Configuration assistant "Agent Configuration Assistant" Succeeded
AgentPlugIn:agent configuration finished with status = true
Running Configuration assistant "Agent Add-on Plug-in"
Configuration assistant "Agent Add-on Plug-in" Succeeded
Finally, we verify that we can show all of the agents deployed for the Oracle 11g RAC environment
$ cd /home/oracle/agent11g/bin
$ ./emctl status agent
Oracle Enterprise Manager 11g Release 1 Grid Control 11.1.0.1.0
Copyright (c) 1996, 2010 Oracle Corporation. All rights reserved.
---------------------------------------------------------------
Agent Version : 11.1.0.1.0
OMS Version : 11.1.0.1.0
Protocol Version : 11.1.0.0.0
Agent Home : /home/oracle/agent11g/rac1
Agent binaries : /home/oracle/agent11g
Agent Process ID : 13181
Parent Process ID : 13157
Agent URL : https://rac1:3872/emd/main
Repository URL : https://oem:4900/em/upload
Started at : 2011-09-20 20:28:06
Started by user : oracle
Last Reload : 2011-09-20 20:28:06
Last successful upload : 2011-09-20 20:38:49
Total Megabytes of XML files uploaded so far : 15.68
Number of XML files pending upload : 0
Size of XML files pending upload(MB) : 0.00
Available disk space on upload filesystem : 31.34%
Last successful heartbeat to OMS : 2011-09-20 20:39:10
---------------------------------------------------------------
Agent is Running and Ready
The beauty is when you specify the RAC cluster nodes in the first response file as we did above, is that all of the cluster nodes will have the silent install agent performed at same time!
After setup of a new Oracle 11gR1 Grid control Enteprise manager (OEM) server, I had to deploy agents to monitor the Oracle environments. Unfortunately, the mass push automation process via OEM 11g Grid Control kept failing and numerous support calls to Oracle Support alas yielded no success. I was able to quickly automate the process via silent installation and deployment of Oracle 11g agents with Grid Control 11g. Below is the process of steps to setup the Oracle 11g agents for Oracle 11g Grid Control:
1) Download and unzip the agent 11g software to hosts monitored. In my case, I needed to configure agents for Oracle 11g RAC on Red Hat Enterprise Linux 5 platform.
cd /home/oracle
unzip Linux_x86_64_Grid_Control_agent_download_11_1_0_1_0.zip
Configure response file for silent agent installation
Before we can kickoff the silent install for the 11g agent, you need to edit the response file called additional_agent.rsp located under the $AGENT_HOME/response directory as shown in my example below:
cd /home/oracle/linux_x64/response
Edit the file with editor of your choice, in my case, I used vi editor on Linux.
Since we don't want updates and don't care about My Oracle support details, we will comment out the email address specifics.
$ vi additional_agent.rsp
SECURITY_UPDATES_VIA_MYORACLESUPPORT=FALSE
#DECLINE_SECURITY_UPDATES=
#MYORACLESUPPORT_USERNAME=
#MYORACLESUPPORT_PASSWORD=
#COLLECTOR_SUPPORTHUB_URL=
INSTALL_UPDATES_SELECTION="skip"
#STAGE_LOCATION=
#PROXY_USER=
#PROXY_PWD=
#PROXY_HOST=
#PROXY_PORT=
ORACLE_AGENT_HOME_LOCATION=/home/oracle
b_silentInstall=true
CLUSTER_NODES=rac1,rac2,rac3,rac4
#-------------------------------------------------------------------------------
OMS_HOST=oem
OMS_PORT=4900
AGENT_REGISTRATION_PASSWORD=yourpassword
Since we have an Oracle 11g RAC environment, we set the parameter in the response file for CLUSTER_NODES to the cluster RAC node names. If you do not have RAC, you can comment this parameter out. Make sure to set the correct hostname, OMS port number and agent registration password or your silent agent installation will fail.
Perform silent installation for Oracle 11g Agent with OEM Grid Control 11g
Navigate to the AGENT_HOME/agent directory
cd /home/oracle/linux_x64/agent
Execute the silent agent installer script with the correct options and parameters as shown in the below example:
./runInstaller -silent -responseFile /home/oracle/linux_x64/response/additional_agent.rsp \
-invPtrLoc /u01/app/oracle/product/11.2.0/dbhome_1/oraInst.loc
Note: you must grant the correct location for the Oracle Inventory file (oraInst.loc) or the silent installation will fail for the agent.
Once you silent installation has succeeded you should see the following status:
Running Configuration assistant "Agent Configuration Assistant"
Performing free port detection on host=rac1
Performing targets discovery and agent configuration
Securing the agent
Starting the agent
Configuration assistant "Agent Configuration Assistant" Succeeded
AgentPlugIn:agent configuration finished with status = true
Running Configuration assistant "Agent Add-on Plug-in"
Configuration assistant "Agent Add-on Plug-in" Succeeded
Finally, we verify that we can show all of the agents deployed for the Oracle 11g RAC environment
$ cd /home/oracle/agent11g/bin
$ ./emctl status agent
Oracle Enterprise Manager 11g Release 1 Grid Control 11.1.0.1.0
Copyright (c) 1996, 2010 Oracle Corporation. All rights reserved.
---------------------------------------------------------------
Agent Version : 11.1.0.1.0
OMS Version : 11.1.0.1.0
Protocol Version : 11.1.0.0.0
Agent Home : /home/oracle/agent11g/rac1
Agent binaries : /home/oracle/agent11g
Agent Process ID : 13181
Parent Process ID : 13157
Agent URL : https://rac1:3872/emd/main
Repository URL : https://oem:4900/em/upload
Started at : 2011-09-20 20:28:06
Started by user : oracle
Last Reload : 2011-09-20 20:28:06
Last successful upload : 2011-09-20 20:38:49
Total Megabytes of XML files uploaded so far : 15.68
Number of XML files pending upload : 0
Size of XML files pending upload(MB) : 0.00
Available disk space on upload filesystem : 31.34%
Last successful heartbeat to OMS : 2011-09-20 20:39:10
---------------------------------------------------------------
Agent is Running and Ready
The beauty is when you specify the RAC cluster nodes in the first response file as we did above, is that all of the cluster nodes will have the silent install agent performed at same time!
Application Tier crashes at 100% CPU load
We had an issue with our Oracle R12 EBS environment when the CPU spiked constantly at 100% when load testing with lots of concurrent active users.
After much research, it was determined to be caused by
Bug 6519700 ESC: CSE: R12SIP: 6513826 FRMWEB RUNAWAY PROCESS CONSUMING 100% CPU-MIDDLE TIER
After much research, it was determined to be caused by
Bug 6519700 ESC: CSE: R12SIP: 6513826 FRMWEB RUNAWAY PROCESS CONSUMING 100% CPU-MIDDLE TIER
The fix was to shutdown the apps middle tier server and configure the following forms environment variables:
FORMS_RECORD_GROUP_MAXFORMS_CATCHTERM
1) First we had to update the context file on the application middle tier as follows:
s_forms_catchterm=0
2) Since the FORMS_RECORD_GROUP_MAX is not a default variable in the app tier context file, you need to create a custom template as follows.
cd $AD_TOP/admin/template
mkdir custom
cp $AD_TOP/admin/template/APPLSYS_ux.env $AD_TOP/template/custom/APPLSYS_ux.env
Edit the custom template file:
vi $AD_TOP/admin/template/custom/APPLSYS_ux.env
Set the environment variable FORMS_RECORD_GROUP_MAX to a high value such as 10000 or greater.
FORMS_RECORD_GROUP_MAX=10000
export FORMS_RECORD_GROUP_MAX
Save the file.
Run autoconfig utility for apps middle tier server.
Start app tier services and test.
Now we now longer have this issue.
Load Balancing Part I: Oracle R12 EBS and Oracle 11g RAC- Manual process
I had to configure an Oracle R12 (12.1.1/12.1.3) E-Business Suite and two node Oracle 11gR2 RAC environment recently to take advantage of load balancing.
However, instead of using a hardware load balancer such as Cisco ACE, we had to find a manual process to implement this goal.
First stop was to review the documentation listed below on My Oracle Support (http://support.oracle.com):
Note # 380489.1 Using Load Balancers with Oracle E-Business Suite Release 12
Note # 603325.1 Using Cisco ACE Series Application Control Engine with Oracle E-Business Suite 12
Note # 226880.1 Configuration of Load Balancing and Transparent Application Failover
Note # 262298.1 Load Balancing Doesn't Balance the Number of Sessions Across All Available Nodes
Note # 887522.1 11gR2 Grid Infrastructure Single Client Access Name (SCAN) Explained
Note #1080674.1 JDBC Drivers and Failover Mechanisms (FCF, TAF, SCAN )
Note #1100024.1 Master Note for Oracle JDBC High Availability Features
Note #247135.1 How to Implement Load Balancing With RAC Configured System Using JDBC
Now, based on my experience with load balancers and Oracle, you should in most cases take the default setup of running your application middle tier server nodes in the default servlet mode to work with Cisco ACE and other hardware load balancers.
There is no compatibility with socket mode for hardware load balancers after discussions with Oracle support, you MUST run in servlet mode for hardware load balancing.
With that said, you could also use a web cache 10gAS application server to use software load balancing instead of the expensive hardware load balancer.
Our challenge was to implement a pure software load balance setup with Oracle R12 EBS and 11gR2 RAC environment with the ability to run the application middle tier nodes in socket mode for performance and application compatibility reasons.
Configure application middle tier environment for load balancing
First, we had to edit the following context variables on the apps tier for load balancing using context editor in the Oracle Applications Manager (OAM):
s_tools_twotask=VIS_BALANCE
s_weboh_twotask=VIS_BALANCE
s_apps_jdbc_connect_alias=VIS_BALANCE
Make sure to run the Oracle R12 EBS autoconfig utility ($ADMIN_SCRIPTS_HOME/adautocfg.sh) after changing the above context variables in your app tier context file.
Note: take a backup of the context file before making changes in case you need to revert back to original configuration!
The above context variables for the apps tier should point to the load balance setup in tnsnames.ora on the two RAC cluster nodes as shown below:
VIS_BALANCE=
(DESCRIPTION=
(ADDRESS_LIST=
(LOAD_BALANCE=YES)
(FAILOVER=YES)
(ADDRESS=(PROTOCOL=tcp)(HOST=node1-vip)(PORT=1521))
(ADDRESS=(PROTOCOL=tcp)(HOST=node2-vip)(PORT=1521)) )
(CONNECT_DATA= (SERVICE_NAME=VIS)
) )
So you will need to create the entry called VIS_BALANCE for example as shown in the above tnsnames.ora file for the app tier nodes and database tier RAC Nodes.
Configure listener.ora
Sometimes even if you setup load balancing correctly, the distribution will not be sent equally across the cluster nodes. To resolve this issue, there is an undocumented parameter you can set to avoid this problem in your listener.ora file.
vi listener.ora
PREFER_LEAST_LOADED_NODE_listener_name=OFF
Register listener and services
Now that we have completed these changes, we need to register the listener services for all nodes.
Then unset and reset the remote_listener parameter for node1.
sqlplus '/as sysdba'
alter system set remote_listener='' scope=both;
alter system register;
alter system set remote_listener='VIS_REMOTE' scope=both;
alter system register;
Stop and restart the database and listeners.
lsnrctl stop
lsnrctl start
lsnrctl reload
Verify that both listeners display both instances in the status and services output.
lsnrctl services
lsnrctl status
Since load balancing is a complex topic, we just scratched the surface here in this post to highlight a quick and easy way to implement manual load balancing at the database layer with Oracle R12 EBS and Oracle 11gR2 RAC.
Future post will cover how to load balance the application middle tier.
However, instead of using a hardware load balancer such as Cisco ACE, we had to find a manual process to implement this goal.
First stop was to review the documentation listed below on My Oracle Support (http://support.oracle.com):
Note # 380489.1 Using Load Balancers with Oracle E-Business Suite Release 12
Note # 603325.1 Using Cisco ACE Series Application Control Engine with Oracle E-Business Suite 12
Note # 226880.1 Configuration of Load Balancing and Transparent Application Failover
Note # 262298.1 Load Balancing Doesn't Balance the Number of Sessions Across All Available Nodes
Note # 887522.1 11gR2 Grid Infrastructure Single Client Access Name (SCAN) Explained
Note #1080674.1 JDBC Drivers and Failover Mechanisms (FCF, TAF, SCAN )
Note #1100024.1 Master Note for Oracle JDBC High Availability Features
Note #247135.1 How to Implement Load Balancing With RAC Configured System Using JDBC
Now, based on my experience with load balancers and Oracle, you should in most cases take the default setup of running your application middle tier server nodes in the default servlet mode to work with Cisco ACE and other hardware load balancers.
There is no compatibility with socket mode for hardware load balancers after discussions with Oracle support, you MUST run in servlet mode for hardware load balancing.
With that said, you could also use a web cache 10gAS application server to use software load balancing instead of the expensive hardware load balancer.
Our challenge was to implement a pure software load balance setup with Oracle R12 EBS and 11gR2 RAC environment with the ability to run the application middle tier nodes in socket mode for performance and application compatibility reasons.
Configure application middle tier environment for load balancing
First, we had to edit the following context variables on the apps tier for load balancing using context editor in the Oracle Applications Manager (OAM):
s_tools_twotask=VIS_BALANCE
s_weboh_twotask=VIS_BALANCE
s_apps_jdbc_connect_alias=VIS_BALANCE
Make sure to run the Oracle R12 EBS autoconfig utility ($ADMIN_SCRIPTS_HOME/adautocfg.sh) after changing the above context variables in your app tier context file.
Note: take a backup of the context file before making changes in case you need to revert back to original configuration!
The above context variables for the apps tier should point to the load balance setup in tnsnames.ora on the two RAC cluster nodes as shown below:
VIS_BALANCE=
(DESCRIPTION=
(ADDRESS_LIST=
(LOAD_BALANCE=YES)
(FAILOVER=YES)
(ADDRESS=(PROTOCOL=tcp)(HOST=node1-vip)(PORT=1521))
(ADDRESS=(PROTOCOL=tcp)(HOST=node2-vip)(PORT=1521)) )
(CONNECT_DATA= (SERVICE_NAME=VIS)
) )
So you will need to create the entry called VIS_BALANCE for example as shown in the above tnsnames.ora file for the app tier nodes and database tier RAC Nodes.
Configure listener.ora
Sometimes even if you setup load balancing correctly, the distribution will not be sent equally across the cluster nodes. To resolve this issue, there is an undocumented parameter you can set to avoid this problem in your listener.ora file.
vi listener.ora
PREFER_LEAST_LOADED_NODE_listener_name=OFF
Register listener and services
Now that we have completed these changes, we need to register the listener services for all nodes.
Then unset and reset the remote_listener parameter for node1.
sqlplus '/as sysdba'
alter system set remote_listener='' scope=both;
alter system register;
alter system set remote_listener='VIS_REMOTE' scope=both;
alter system register;
Stop and restart the database and listeners.
lsnrctl stop
lsnrctl start
lsnrctl reload
Verify that both listeners display both instances in the status and services output.
lsnrctl services
lsnrctl status
Since load balancing is a complex topic, we just scratched the surface here in this post to highlight a quick and easy way to implement manual load balancing at the database layer with Oracle R12 EBS and Oracle 11gR2 RAC.
Future post will cover how to load balance the application middle tier.
Oracle 11g agent fails to upload to Oracle 11g Grid Control
Recently, I built a new Oracle 11gR1 Grid Control Enterprise Manager (OEM) server at work to monitor our complex environments.
All worked fine with agents, OMS and Grid Control until the other day when the agent failed to upload the status to Grid control.
After checking status for the agent and trying to perform a manual upload as listed below,
I received the following errors:
"emctl upload failed: collection status disabled by upload manager"
So I checked on disk space and plenty was available on the filesystem where the agent was located.
After finding a hit on My Oracle Support (MOS) site under Doc ID 396238.1,
I reviewed the following parameters in the emd.properties file under the
AGENT_HOME/sysman/emd directory:
UploadFileSize
UploadMaxBytes
UploadMaxDiskUsedPct
These were set to 2048, 50, and 98 for UploadFileSize, UploadMaxBytes, and UploadMaxDiskUsed by default when I checked the emd.properties file.
Sure enough the default values were too small thus causing the agent upload to fail.
First, I shutdown the agent with the emctl stop agent command.
Next, I changed the parameters to larger values for size of files:
$ cd /home/oracle/agent11g/sysman/config
vi emd.propertiesUploadFileSize=4096
## The maximum number of megabytes(MB) the upload manager will support in the
# upload directory before temporarily disabling collections, logging and tracing
##
UploadMaxBytesXML=500
# The maximum amount (%) of disk space that can be used on the EMD filesystem
# before the following is disabled:
# - Collection of data (upload manager)
# - Logging and tracing
# - Diagnosability traces
#@description=
#@valueType=Int
#@LOV=#@default=98#
UploadMaxDiskUsedPct=99
Then I started the agent and performed manual upload to OEM Grid OMS server:
./emctl status agent
Oracle Enterprise Manager 11g Release 1 Grid Control 11.1.0.1.0
Copyright (c) 1996, 2010 Oracle Corporation. All rights reserved.
---------------------------------------------------------------
Agent Version : 11.1.0.1.0
OMS Version : 11.1.0.1.0
Protocol Version : 11.1.0.0.0
Agent Home : /home/oracle/agent11g/node1
Agent binaries : /home/oracle/agent11g
Agent Process ID : 16864
Parent Process ID : 16828
Agent URL : https://node1:3872/emd/main
Repository URL : https://oem:4900/em/upload
Started at : 2011-09-20 11:33:05
Started by user : oracle
Last Reload : 2011-09-20 11:33:05
Last successful upload : 2011-09-20 13:22:32
Total Megabytes of XML files uploaded so far : 6.27
Number of XML files pending upload : 0
Size of XML files pending upload(MB) : 0.00
Available disk space on upload filesystem : 5.30%
Last successful heartbeat to OMS : 2011-09-20 13:36:26
---------------------------------------------------------------
Agent is Running and Ready
$ ./emctl upload
Oracle Enterprise Manager 11g Release 1 Grid Control 11.1.0.1.0
Copyright (c) 1996, 2010 Oracle Corporation. All rights reserved.
---------------------------------------------------------------
EMD upload completed successfully
Now it works and OEM Grid Control can receive the updates from the agent on the server monitored.Cheers,Ben
All worked fine with agents, OMS and Grid Control until the other day when the agent failed to upload the status to Grid control.
After checking status for the agent and trying to perform a manual upload as listed below,
I received the following errors:
"emctl upload failed: collection status disabled by upload manager"
So I checked on disk space and plenty was available on the filesystem where the agent was located.
After finding a hit on My Oracle Support (MOS) site under Doc ID 396238.1,
I reviewed the following parameters in the emd.properties file under the
AGENT_HOME/sysman/emd directory:
UploadFileSize
UploadMaxBytes
UploadMaxDiskUsedPct
These were set to 2048, 50, and 98 for UploadFileSize, UploadMaxBytes, and UploadMaxDiskUsed by default when I checked the emd.properties file.
Sure enough the default values were too small thus causing the agent upload to fail.
First, I shutdown the agent with the emctl stop agent command.
Next, I changed the parameters to larger values for size of files:
$ cd /home/oracle/agent11g/sysman/config
vi emd.propertiesUploadFileSize=4096
## The maximum number of megabytes(MB) the upload manager will support in the
# upload directory before temporarily disabling collections, logging and tracing
##
UploadMaxBytesXML=500
# The maximum amount (%) of disk space that can be used on the EMD filesystem
# before the following is disabled:
# - Collection of data (upload manager)
# - Logging and tracing
# - Diagnosability traces
#@description=
#@valueType=Int
#@LOV=#@default=98#
UploadMaxDiskUsedPct=99
Then I started the agent and performed manual upload to OEM Grid OMS server:
./emctl status agent
Oracle Enterprise Manager 11g Release 1 Grid Control 11.1.0.1.0
Copyright (c) 1996, 2010 Oracle Corporation. All rights reserved.
---------------------------------------------------------------
Agent Version : 11.1.0.1.0
OMS Version : 11.1.0.1.0
Protocol Version : 11.1.0.0.0
Agent Home : /home/oracle/agent11g/node1
Agent binaries : /home/oracle/agent11g
Agent Process ID : 16864
Parent Process ID : 16828
Agent URL : https://node1:3872/emd/main
Repository URL : https://oem:4900/em/upload
Started at : 2011-09-20 11:33:05
Started by user : oracle
Last Reload : 2011-09-20 11:33:05
Last successful upload : 2011-09-20 13:22:32
Total Megabytes of XML files uploaded so far : 6.27
Number of XML files pending upload : 0
Size of XML files pending upload(MB) : 0.00
Available disk space on upload filesystem : 5.30%
Last successful heartbeat to OMS : 2011-09-20 13:36:26
---------------------------------------------------------------
Agent is Running and Ready
$ ./emctl upload
Oracle Enterprise Manager 11g Release 1 Grid Control 11.1.0.1.0
Copyright (c) 1996, 2010 Oracle Corporation. All rights reserved.
---------------------------------------------------------------
EMD upload completed successfully
Now it works and OEM Grid Control can receive the updates from the agent on the server monitored.Cheers,Ben
Subscribe to:
Posts (Atom)
