Setup PeopleTools 8.52 PIA

Yesterday I was setting up a PIA for an CRM9.1 Demo environment and ran into below warning:


This version of PeopleSoft PeopleTools requires a 64-bit Oracle WebLogic
installation. The selected Oracle WebLogic is a 32-bit installation. Are you
sure you want to continue?

Since I didn't install the weblogic on the server, I had no idea if the weblogic that was pointed to is 32 bit or 64 bit. After tweaking a few things and still the same issue, I decided to reinstall weblogic. To ensure weblogic is 64 bit, the key is to use 64 bit java to install weblogic. Below is how to check if your java is 32 bit or 64 bit:

./java -d64 -version
Running a 64-bit JVM is not supported on this platform.

This means this java is not 64 bit.

 ./java -d64 -version
java version "1.6.0_20"
Java(TM) SE Runtime Environment (build 1.6.0_20-b02)
Oracle JRockit(R) (build R28.1.0-123-138454-1.6.0_20-20101014-1350-linux-x86_64, compiled mode)

This means this java is 64 bit.

Once I reinstalled weblogic with 64 bit java, the warning during setting up 8.52 PIA disappeared.

RMAN Duplication from Active RAC database to non-RAC database

I ran into below error when trying to RMAN duplicate from an active RAC database to a non-RAC database, the issue turned out to be oracle version mismatch between Target and Auxiliary. In my case, both Target and Auxiliary are 11.2.0.2.3, however, PSU 11.2.0.2.3 has 2 portions, one for GI_HOME and one for RDBMS_HOME. On my RAC cluster, both portions of PSU 11.2.0.2.3 were applied, while on my non-RAC host, only database portion of PSU 11.2.0.2.3 was applied. I ended up installing a new RDBMS_HOME and applying both portions of PSU 11.2.0.2.3 to this new RDBMS_HOME, the issue resolved by using this new RDBMS_HOME.

copying current control file
Oracle instance started
Segmentation fault

Another issue I ran into was as below, I was puzzled for a little bit because I know my Auxiliary's SYS password was correct and I was able to use it to startup/shutdown the instance. I had to change my Auxiliary's SYS password to be identical to my Target's SYS password. Once this was done, all went well.

contents of Memory Script:
{
   sql clone "alter system set  db_name =
 ''SA9TOLD'' comment=
 ''Reset to original value by RMAN'' scope=spfile";
   sql clone "alter system reset  db_unique_name scope=spfile";
   shutdown clone immediate;
}
executing Memory Script

sql statement: alter system set  db_name =  ''SA9TOLD'' comment= ''Reset to original value by RMAN'' scope=spfile

sql statement: alter system reset  db_unique_name scope=spfile

Oracle instance shut down
released channel: ch1
released channel: ch2
released channel: ch3
released channel: ch4
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03002: failure of Duplicate Db command at 11/09/2012 13:27:49
RMAN-05501: aborting duplication of target database
RMAN-03015: error occurred in stored script Memory Script
RMAN-03009: failure of backup command on ch1 channel at 11/09/2012 13:27:39
ORA-17629: Cannot connect to the remote database server
ORA-17627: ORA-01017: invalid username/password; logon denied
ORA-17629: Cannot connect to the remote database server

Log Corruption on Standby Database.

Oracle 11.2.0.2 on Oracle Enterprise Linux 6.1, kernel bug caused log corruption on standby database. It was not a certified configuration by Oracle (historical reason beyond my control). Upgraded to OEL 6.2, log corruption disappeared. I had to use below note to roll forward standby database at times before the kernel was upgraded.

Steps to perform for Rolling forward a standby database using RMAN incremental backup when primary and standby are in ASM filesystem [ID 836986.1]

This worked well in my Non-ASM filesystem with OMF.

Run 10.2.0.3 and 11.1.0.7 RDBMS on 11.2.0.2 Grid Infrastructure

Recently I've been tasked to a project that needs to run 10.2.0.3 and 11.1.0.7 databases on the latest 11.2.0.2 Grid Infrastructure, I thought that would be easy, as the latest Grid Infrastructure should support the older RDBMS well, but it turns out it's not as easy as I thought. Here are the notes that describe what needs to be done to make it work:

1. Install 10.2.0.3 RDBMS: I got an error stating that it's not a correct oracle clusterware version, I ignored this error and the installation went just fine.

2. Create 10.2.0.3 database using dbca: I ran into "ORA-29702: error occurred in Cluster Group Service operation", had to "crsctl pin css -n node1 node2" to get past this, then I got "DBCA could not startup the ASM instance configured on this node. To processd with database creation using ASM you need the ASM instance to be up and running. Do you want to recreate the ASM instance on this node?" I applied patch 8288940 to get past this. Then I got "Encountered file error when copying listeners from home=/opt/app/11.2.0/grid", had to create a symbolic link "listener.ora" under the RDBMS $ORACLE_HOME/network/admin to point to the endpoints_listener.ora under $GI_HOME/network/admin, and setup TNS_ADMIN variable to point to $ORACLE_HOME/network/admin. Once all these are done, I was finally able to create databases using dbca.

3. To use srvctl to start/stop 10.2.0.3 database, I ran into "/opt/app/oracle/product/10.2.0/db_1/jdk/jre/bin/java: error while loading shared libraries: libpthread.so.0: cannot open shared object file: No such file or directory". I had to comment out "LD_ASSUME_KERNEL" in $ORACLE_HOME/bin/srvctl to fix this issue.

4. Install 11.1.0.7 RDBMS: This worked flawlessly.

5. Create 11.1.0.7 database using dbca: Same issues with creating 10.2.0.3 database, apply the same fixes, I was able to create the database successfully, but at the end it fails to start the database with error PRKP-1001 and CRS-0215. However you can start and stop the database in SQL/Plus.

6. To use srvctl to start/stop 11.1.0.7 database, I ran into the same error PRKP-1001 and CRS-0215. I had to apply patch 9294495 to fix it.

Install 11.1.0.1.0 Grid Control Agent on Windows 2008R2 using agentDownload.vbs

First download wget for windows, you can find it easily on the internet. Suppose you put wget.exe under C:\, you need to add this to the PATH environment variable; otherwise you will run into the following error:

cmd is:WGET.exe http://mksllc01p:4889/agent_download/10.2.0.4.0/agent_download.rsp
C:\WGET\agentDownload.vbs(77, 4) (null): The system cannot find the file specified.

Now you can use wget to download the agentDownload.vbs file from your OMS server as below:

Wget http://:/agentdownload/11.1.0.1.0/windows_x64/agentDownload.vbs

Once this is done, you are ready to install:

C:\>cscript agentDownload.vbs m r b y

It’ll download the response file and get the agent installed.

After the installation is finished, the agent fails to start. After looking at the logs, it turns out to be time zone setting mismatch between the agent and the setting on the OMS. Looking at \sysman\config\emd.properties, the timezone is set to agentTZRegion=America/Karachi, after changing it to the correct time zone, agent started successfully.

A Different Way to Add More Space to ASM DiskGroup in a Linux/RAC Environment

Usually when you need to add more space to an ASM DiskGroup, you add more disks of same size to the DiskGroup. Here is a different way to accomplish the same goal, it sounds a little scary at first but it actually works. In a RAC environment with EMC SAN as storage, we have a DiskGroup A that contains one LUN of 100GB. To increase the size of this DiskGroup to 200GB, you can create a LUN of 200GB, in Navisphere, then migrate the existing 100GB LUN to this larger LUN. This migration process can run in the background and be invisible to OS/Oracle/End users. After the migration is done, you get a larger LUN/Disk in DiskGroup A, however, to make ASM "see" the extra 100GB, two more things will need to happen:

1. Make OS recognize the extra space. Before the LUN migration, there was one partition of 100GB on that LUN, after the migration, the partition is still 100GB even though the LUN is 200GB. Re-partition is needed to make OS aware of the added extra space on the LUN. Unfortunately, fdisk doesn't provide a method to extend the partition, which leads to the scary part: the original partition has to be deleted, and a new partition is then created to take all of the LUN space. As long as you make the start of data the same as the old partition, no data will be lost. In our case, ASM sits on top of raw device without ASMLib, so the re-partition process can be done online without downtime.
   

   
    

2. Make ASM recognize the extra space. After step 1, OS recognizes the extra space, but ASM still doesn't. You can verify it by running the following query:
   
   select name, total_mb, free_mb from v$asm_diskgroup;
   
   To make ASM recognize the extra space, you'll need to run:
   
   alter diskgroup {diskgroup name} resize all rebalance power 11;
   
   Now verify it again with the first query, you'll see that ASM now "sees" all the space from that LUN.

WebUtil 1.0.6 Installation/Configuration

Recently I have been working on upgrading an Oracle Forms/Reports application from Oracle Application Server 10g (9.0.4) to 10g Release 2(10.1.2.2.0), which requires upgrading WebUtil 1.0.5 to 1.0.6. I followed Oracle® Forms Developer WebUtil User's Guide Release 1.0.6, but had problem to make it work. First, I had this error when the application tried to use WebUtil: "oracle.forms.webutil.file.FileFunctions bean not found. WEBUTIL_FILE.FILE_SELECTION_DIALOG_INT will not work". This turned out to be a configuration issue in formsweb.conf. Once I copied all the [webutil] configuration section to the [application] configuration section, this error disappeared and it started to install/download WebUtil. During the download process, I ran into another error: "WUC-24: Error reading URL http://hostname:port/forms/webutil/jacob.dll". I was puzzled at first because the URL seemed to be correct and Jacob.dll was in that directory. Finally I re-extracted Jacob.dll from jacob_18.zip to my ORACLE_HOME/forms/webutil directory, and that resolved the error.