Oracle DBA & Data Science Enthusiast: January 2015

Thursday, 29 January 2015

Change the DB name without nid .

Problem: To change DB name from db11g to db11gnew without using nid utility ,

SQl> show parameter db_name ;

NAME TYPE VALUE
------------------------------------ ----------- -----
db_name string db11g

SQL> ARCHIVE LOG LIST ;
Database log mode Archive Mode
Automatic archival Enabled
Archive destination USE_DB_RECOVERY_FILE_DEST
Oldest online log sequence 2
Next log sequence to archive 4
Current log sequence 4

SQL> alter system switch logfile;

System altered.

SQL> alter database backup controlfile to trace as '/home/oracle/control_trace.sql' ;

SQL> SELECT DECODE(value, NULL, 'PFILE', 'SPFILE') "Init File Type" FROM sys.v_$parameter WHERE name = 'spfile';

Init File Type
----------------

SPFILE

SQL>create pfile=' /home/oracle/initdb11gnew.ora' from spfile ;

Database altered.
[oracle@oracle11g dbs]$ sqlplus / as sysdba

SQL*Plus: Release 11.2.0.3.0 Production on Thu Jan 29 18:04:37 2015

Copyright (c) 1982, 2011, Oracle. All rights reserved.

Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64bit Production
With the Partitioning, Automatic Storage Management, OLAP, Data Mining
and Real Application Testing options

SQL> shu immediate ;
Database closed.
Database dismounted.
ORACLE instance shut down.

Edit to control file trace /home/oracle/control_trace.sql . Remove everything above Set#2 : Resetlog case and edit the create statement as :

CREATE CREATE CONTROLFILE SET DATABASE "db11gnew"

[oracle@oracle11g ~]$ cat control_trace.sql

-- Set #2. RESETLOGS case
--
-- The following commands will create a new control file and use it
-- to open the database.
-- Data used by Recovery Manager will be lost.
-- The contents of online logs will be lost and all backups will
-- be invalidated. Use this only if online logs are damaged.

-- After mounting the created controlfile, the following SQL
-- statement will place the database in the appropriate
-- protection mode:
-- ALTER DATABASE SET STANDBY DATABASE TO MAXIMIZE PERFORMANCE

STARTUP NOMOUNT
CREATE CONTROLFILE SET DATABASE "db11gnew" RESETLOGS ARCHIVELOG
MAXLOGFILES 16
MAXLOGMEMBERS 3
MAXDATAFILES 100
MAXINSTANCES 8
MAXLOGHISTORY 292
LOGFILE
GROUP 1 (
'+DATA2/db11g/onlinelog/group_1.261.868740309',
'+RMAN_DISK/db11g/onlinelog/group_1.257.868740313'
) SIZE 50M BLOCKSIZE 512,
GROUP 2 (
'+DATA2/db11g/onlinelog/group_2.262.868740319',
'+RMAN_DISK/db11g/onlinelog/group_2.258.868740323'
) SIZE 50M BLOCKSIZE 512,
GROUP 3 (
'+DATA2/db11g/onlinelog/group_3.263.868740329',
'+RMAN_DISK/db11g/onlinelog/group_3.259.868740333'
) SIZE 50M BLOCKSIZE 512
-- STANDBY LOGFILE
DATAFILE
'+DATA2/db11g/datafile/system.256.868740081',
'+DATA2/db11g/datafile/sysaux.257.868740081',
'+DATA2/db11g/datafile/undotbs1.258.868740081',
'+DATA2/db11g/datafile/users.259.868740081'
CHARACTER SET WE8MSWIN1252
;
ALTER DATABASE OPEN RESETLOGS;
ALTER TABLESPACE TEMP ADD TEMPFILE '+DATA2' SIZE 20971520 REUSE AUTOEXTEND ON NEXT 655360 MAXSIZE 32767M;

Rename the current init file and spfile as Bkp files .

Copy the init file create above in $ORACLE_HOME/dbs and change the db_name to db11gnew .

[oracle@oracle11g dbs]$ cp /home/oracle/initdb11gnew.ora $ORACLE_HOME/dbs
[oracle@oracle11g dbs]$ ls

[oracle@oracle11g dbs]$ cat initdb11gnew.ora

*.audit_file_dest='/u01/app/oracle/admin/db11g/adump'
*.audit_trail='db'
*.compatible='11.2.0.0.0'
#*.control_files='+DATA2/db11g/controlfile/','+RMAN_DISK/db11g/controlfile/'
*.db_block_size=8192
*.db_create_file_dest='+DATA2'
*.db_domain=''
*.db_name='db11gnew'
*.db_recovery_file_dest='+RMAN_DISK'
*.db_recovery_file_dest_size=4322230272
*.diagnostic_dest='/u01/app/oracle'
*.dispatchers='(PROTOCOL=TCP) (SERVICE=db11gXDB)'
*.log_archive_format='%t_%s_%r.dbf'
*.memory_target=807403520
*.open_cursors=300
*.processes=150
*.remote_login_passwordfile='EXCLUSIVE'
*.undo_tablespace='UNDOTBS1'

[oracle@oracle11g ~]$ export ORACLE_SID=db11gnew
[oracle@oracle11g ~]$ ls
control_trace.sql db_11g_single_with_asm.rsp db_info.log Desktop initdb11gnew.ora
[oracle@oracle11g ~]$ sqlplus / as sysdba

SQL*Plus: Release 11.2.0.3.0 Production on Thu Jan 29 18:06:39 2015

Copyright (c) 1982, 2011, Oracle. All rights reserved.

Connected to an idle instance.

SQL> @control_trace.sql ;
ORACLE instance started.

Total System Global Area 805875712 bytes
Fixed Size 2232680 bytes
Variable Size 478154392 bytes
Database Buffers 322961408 bytes
Redo Buffers 2527232 bytes

Control file created.

Database altered.

Tablespace altered.

SQL> select NAME,LOG_MODE,CONTROLFILE_TYPE,OPEN_MODE from V$DATABASE ;

NAME LOG_MODE CONTROL OPEN_MODE
--------- ------------ ------- -------------------------------------
DB11GNEW ARCHIVELOG CURRENT READ WRITE

SQL> Show parameter control_file ;

control_files string +DATA2/db11gnew/controlfile/current.266.870286263, +RMAN_DISK/db11gnew/controlfile/current .262.870286265

Shu immediate and Edit the init file with the new control_file values . Again start the DB using pfile and create spfile .

[oracle@oracle11g ~]$ lsnrctl status

LSNRCTL for Linux: Version 11.2.0.3.0 - Production on 29-JAN-2015 18:18:07

Copyright (c) 1991, 2011, Oracle. All rights reserved.

Connecting to (ADDRESS=(PROTOCOL=tcp)(HOST=)(PORT=1521))
STATUS of the LISTENER
------------------------
Alias LISTENER2
Version TNSLSNR for Linux: Version 11.2.0.3.0 - Production
Start Date 29-JAN-2015 17:27:48
Uptime 0 days 0 hr. 50 min. 19 sec
Trace Level off
Security ON: Local OS Authentication
SNMP OFF
Listener Parameter File /u01/app/11.2.0/grid_11203/network/admin/listener.ora
Listener Log File /u01/app/grid/diag/tnslsnr/oracle11g/listener2/alert/log.xml
Listening Endpoints Summary...
(DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=oracle11g.localdomain)(PORT=1521)))
Services Summary...
Service "+ASM" has 1 instance(s).
Instance "+ASM", status READY, has 1 handler(s) for this service...
Service "db11gXDB" has 1 instance(s).
Instance "db11gnew", status READY, has 1 handler(s) for this service...
Service "db11gnew" has 1 instance(s).
Instance "db11gnew", status READY, has 1 handler(s) for this service...
The command completed successfully
[oracle@oracle11g ~]$

Check for the new path of control files created in ASM using asmcmd you can check that .
Shutdown the DB and change the new path of control_file in initdb11gnew.ora .

Start the db using initdb11gnew.ora and recreate the spfile , shutdown again and startup DB using spfile .

SQL> create spfile='+DATA2' from pfile ;

File created.

SQL> shu immediate ;
Database closed.
Database dismounted.
ORACLE instance shut down.

After this modify the initdb11gnew.ora to point to new spfile created .

ASMCMD> ls -l +DATA2/DB11GNEW/PARAMETERFILE/

Type Redund Striped Time Sys Name
PARAMETERFILE UNPROT COARSE JAN 29 18:00:00 Y spfile.268.870289035

ASMCMD> +DATA2/DB11GNEW/PARAMETERFILE/spfile.268.870289035
ASMCMD>

Edit the init file to point to the spfile :

[oracle@oracle11g dbs]$ cat initdb11gnew.ora

SPFILE='+DATA2/DB11GNEW/PARAMETERFILE/spfile.268.870289035'

Now start the Db using spfile :

SQL> startup ;
ORACLE instance started.

Total System Global Area 805875712 bytes
Fixed Size 2232680 bytes
Variable Size 478154392 bytes
Database Buffers 322961408 bytes
Redo Buffers 2527232 bytes
Database mounted.
Database opened.

SQL> show parameter pfile ;

NAME TYPE VALUE
------------------------------------ ----------- ------------------------------
spfile string +DATA2/db11gnew/parameterfile/
spfile.268.870289035

SQL> show parameter control_file;

NAME TYPE VALUE
------------------------------------ ----------- ------------------------------
control_file_record_keep_time integer 7
control_files string +DATA2/db11gnew/controlfile/current.266.870286263, +RMAN_DISK/db11gnew/controlfile/current .262.870286265

change the settings in oratab , tns file etc .

Checkpoint # of all datafiles

set linesize 200;

set pagesize 100;

col inst_id for 9999999 heading 'Instance #'

col file_nr for 9999999 heading 'File #'

col file_name for A50 heading 'File name'

col checkpoint_change_nr for 99999999999999 heading 'Checkpoint #'

col checkpoint_change_time for A20 heading 'Checkpoint time'

col last_change_nr for 99999999999999 heading 'Last change #'

SELECT

fe.inst_id,

fe.fenum file_nr,

fn.fnnam file_name,

TO_NUMBER (fe.fecps) checkpoint_change_nr,

fe.fecpt checkpoint_change_time,

-- fe.fests last_change_nr,

DECODE (

fe.fetsn,

0, DECODE (BITAND (fe.festa, 2), 0, 'SYSOFF', 'SYSTEM'),

DECODE (BITAND (fe.festa, 18),

0, 'OFFLINE',

2, 'ONLINE',

'RECOVER')

) status

FROM x$kccfe fe,

x$kccfn fn

WHERE ( (fe.fepax != 65535 AND fe.fepax != 0 )

OR (fe.fepax = 65535 OR fe.fepax = 0)

)

AND fn.fnfno = fe.fenum

AND fe.fefnh = fn.fnnum

AND fe.fedup != 0

AND fn.fntyp = 4

AND fn.fnnam IS NOT NULL

AND BITAND (fn.fnflg, 4) != 4

ORDER BY fe.fenum

Instance # File # File name Checkpoint # Checkpoint time STATUS
--------------------------------------- --------------- -------------------- --------------- ---------------

1 1 +DATA2/db11g/system.256.868 1026049 01/29/2015 17:39:04 SYSTEM
1 2 +DATA2/db11g/sysaux.257.868 1026049 01/29/2015 17:39:04 ONLINE
1 3 +DATA2/db11g/undotbs.258.868 1026049 01/29/2015 17:39:04 ONLINE
1 4 +DATA2/db11g/users.259.868 1026049 01/29/2015 17:39:04 ONLINE

To check Datafile fuzziness:
=======================

set numwidth 30;
set pagesize 50000;
alter session set nls_date_format = 'DD-MON-RRRR HH24:MI:SS';
set line 1000
SQL> select status,checkpoint_change#,checkpoint_time, resetlogs_change#, resetlogs_time, count(*), fuzzy from v$datafile_header group by status,checkpoint_change#,checkpoint_time, resetlogs_change#, resetlogs_time, fuzzy;

STATUS CHECKPOINT_CHANGE# CHECKPOINT_TIME RESETLOGS_CHANGE# RESETLOGS_TIME COUNT(*) FUZ
------- ------------------ -------------------------- ------------------------------ -------------------------- ------------------------------ ---
ONLINE 6798623301 03-DEC-2008 10:47:36 3264889930 06-APR-2006 16:39:38 1 NO
ONLINE 7116772157 17-FEB-2009 15:49:37 3264889930 06-APR-2006 16:39:38 1 NO
ONLINE 7180206429 02-MAR-2009 12:08:35 3264889930 06-APR-2006 16:39:38 1 NO
ONLINE 7340089977 07-APR-2009 00:28:19 3264889930 06-APR-2006 16:39:38 1 NO
ONLINE 18205667200 13-DEC-2014 14:45:30 3264889930 06-APR-2006 16:39:38 4 NO
ONLINE 18205667200 13-DEC-2014 14:45:30 3264889930 06-APR-2006 16:39:38 10 YES
ONLINE 18207657100 14-DEC-2014 05:18:47 3264889930 06-APR-2006 16:39:38 1 NO
ONLINE 18207657100 14-DEC-2014 05:18:47 3264889930 06-APR-2006 16:39:38 1 YES
ONLINE 10983307652 05-JAN-2011 09:19:51 3264889930 06-APR-2006 16:39:38 1 NO
ONLINE 18187780810 09-DEC-2014 15:08:55 3264889930 06-APR-2006 16:39:38 1 NO
ONLINE 18205667203 13-DEC-2014 14:45:30 3264889930 06-APR-2006 16:39:38 3 NO
ONLINE 18205667203 13-DEC-2014 14:45:30 3264889930 06-APR-2006 16:39:38 2 YES
ONLINE 18205696549 13-DEC-2014 15:11:27 3264889930 06-APR-2006 16:39:38 2 NO
ONLINE 18205696549 13-DEC-2014 15:11:27 3264889930 06-APR-2006 16:39:38 1 YES
ONLINE 18205701941 13-DEC-2014 15:15:52 3264889930 06-APR-2006 16:39:38 2 NO
ONLINE 18205722195 13-DEC-2014 15:35:18 3264889930 06-APR-2006 16:39:38 1 NO
ONLINE 18205722195 13-DEC-2014 15:35:18 3264889930 06-APR-2006 16:39:38 1 YES
ONLINE 18205727227 13-DEC-2014 15:40:24 3264889930 06-APR-2006 16:39:38 2 NO
ONLINE 18205809760 13-DEC-2014 16:00:01 3264889930 06-APR-2006 16:39:38 2 NO
ONLINE 18205822343 13-DEC-2014 16:04:56 3264889930 06-APR-2006 16:39:38 1 NO
ONLINE 18205822343 13-DEC-2014 16:04:56 3264889930 06-APR-2006 16:39:38 1 YES
ONLINE 18205850552 13-DEC-2014 16:24:22 3264889930 06-APR-2006 16:39:38 1 NO
ONLINE 18205850552 13-DEC-2014 16:24:22 3264889930 06-APR-2006 16:39:38 1 YES
ONLINE 18205854452 13-DEC-2014 16:29:28 3264889930 06-APR-2006 16:39:38 2 NO
ONLINE 18205902094 13-DEC-2014 16:48:44 3264889930 06-APR-2006 16:39:38 2 NO

Friday, 23 January 2015

Upgrade Oracle 11.2.0.3 RAC to 11.2.0.4 - ( Out-of-Place Manual Upgrade )

The different upgrade methods you can use to upgrade your database to the new Oracle Database release or higher are:

1) Database Upgrade Assistant (DBUA)
2) Manual Upgrade
3) Export/Import
4) Data Copying
5) Golden Gate

1) DBUA (Database Upgrade Assistant)

The Database Upgrade Assistant (DBUA) interactively steps you through the upgrade process and configures the database for the new Oracle Database release. The DBUA automates the upgrade process by performing all of the tasks normally performed manually. The DBUA makes appropriate recommendations for configuration options such as tablespaces and redo logs. You can then act on these recommendations. This method is very easy and user friendly. But if any error occurs it will take time to diagnose the error as the upgrade process is done automatically by the upgrade assistant.
For more information, refer to the following link:

10.2=> http://docs.oracle.com/cd/B19306_01/server.102/b14238/upgrade.htm#i1011482

11.1=> http://download.oracle.com/docs/cd/B28359_01/server.111/b28300/upgrade.htm#i1011482

11.2 => http://docs.oracle.com/cd/E11882_01/server.112/e23633.pdf

Refer to the following oracle support notes for DBUA upgrade:
================================================
NOTE 556477.1 Complete Checklist for Upgrades to 11gR1 using DBUA
NOTE 870814.1 Complete checklist to upgrade the database to 11g R2 using DBUA
NOTE 1516557.1 Complete Checklist for Upgrading to Oracle Database 12c Release 1 (12.1) using DBUA

Upgrade Advisor: Database from 10.2 to 11.2 (Doc ID 251.1)
How to Download and Run Oracle's Database Pre-Upgrade Utility (Doc ID 884522.1)

2) Manual upgrade

A manual upgrade consists of running SQL scripts and utilities from a command line to upgrade a database to the new Oracle Database release. A manual upgrade gives you finer control over the upgrade process as it is done step by step manually. So if any error occurs, it is easier to diagnose the error. While a manual upgrade gives you finer control over the upgrade process, it is more susceptible to error if any of the upgrade or pre-upgrade steps are either not followed or are performed out of order.
When manually upgrading a database, perform the following pre-upgrade steps:

Analyze the database using the Pre-Upgrade Information Tool. The Upgrade Information Tool is a SQL script that ships with the new Oracle Database release, and must be run in the environment of the database being upgraded.
The Upgrade Information Tool displays warnings about possible upgrade issues with the database. It also displays information about required initialization parameters for the new Oracle Database release.
Prepare the new Oracle Home.
Perform a backup of the database.
Depending on the release of the database being upgraded, you may need to perform additional pre-upgrade steps (adjust the parameter file for the upgrade, remove obsolete initialization parameters and adjust initialization parameters that might cause upgrade problems).

Refer to the following oracle support notes for manual upgrade:
================================================
NOTE 1503653.1 Complete Checklist for Manual Upgrades to Oracle Database 12c Release 1 (12.1)
NOTE 837570.1 Complete Checklist for Manual Upgrades to 11gR2
NOTE 429825.1 Complete Checklist for Manual Upgrades to 11gR1
NOTE 263809.1 Complete checklist for manual upgrades to 10gR1 (10.1.0.x).
NOTE 316889.1 Complete checklist for manual upgrades to 10gR2.
NOTE 466181.1 10g Upgrade Companion
NOTE 601807.1 Oracle 11gR1 Upgrade Companion

Complete Checklist for out-of-Place Manual Upgrade from Previous 11.2.0.N Version to the Latest 11.2.0.N Patchset (Doc ID 1276368.1)
Complete Checklist for Manual Upgrades to 11gR2 (Doc ID 837570.1)

Note: DBUA can be used only if the source & target Oracle homes are on the same Server.
But manual upgrade will work even if the source and target Home are on different servers provided that the Hardware architecture and operating system on source and target Home are the same.

Example :
- You cannot manually upgrade a database from an AIX Operating System to Solaris Operating System.
- You cannot manually upgrade a database from a Solaris X86-64 Machine to Solaris SPARC 64-bit.

3) Export/Import

The Export/Import upgrade method does not change the current database, which enables the database to remain available throughout the upgrade process. However, if a consistent snapshot of the database is required (for data integrity or other purposes), then the database must run in restricted mode or must otherwise be protected from changes during the export procedure. Because the current database can remain available, you can, for example, keep an existing production database running while the new Oracle Database is being built at the same time by Export/Import. During the upgrade, to maintain complete database consistency, changes to the data in the database cannot be permitted without the same changes to the data in the new Oracle Database.

Most importantly, the Export/Import operation results in a completely new database. Although the current database ultimately contains a copy of the specified data, the upgraded database may perform differently from the original database. For example, although Export/Import creates an identical copy of the database, other factors, such as disk placement of data and unset tuning parameters, may cause unexpected performance problems.

Upgrading an entire database by using Export/Import can take a long time, especially compared to using the DBUA or performing a manual upgrade. Therefore, you may need to schedule the upgrade during non-peak hours or make provisions for propagating to the new database any changes that are made to the current database during the upgrade.

For more information, refer to the following link:
http://docs.oracle.com/cd/B19306_01/server.102/b14238/expimp.htm#i262247

NOTE: Export/Import works for all versions. But for 10g and above, DataPump or Transportable Table Spaces methods are better.

NOTE 351598.1 Export/Import DataPump: The Minimum Requirements to Use Export DataPump and Import DataPump (System Privileges)

4) Data Copying

You can copy data from one Oracle Database to another using database links. For example, you can create new tables and fill the tables with data by using the INSERT INTO statement and the CREATE TABLE ... AS statement. Copying data and Export/Import offer the same advantages for upgrading. Using either method, you can defragment data files and restructure the database by creating new tablespaces or modifying existing tables or tablespaces. In addition, you can copy only specified database objects or users.
Copying data, however, unlike Export/Import, enables the selection of specific rows of tables to be placed into the new database. Copying data is a good method for copying only part of a database table. In contrast, using Export/Import, you can copy only entire tables

Oracle Database Upgrade Guide
10g Release 2 (10.2)
Part Number B14238-02
http://docs.oracle.com/cd/B19306_01/server.102/b14238/upgrade.htm

5) Golden Gate

Check on the OTN site with the following url for how to upgrade with zero downtime using Golden Gate:

http://www.oracle.com/technetwork/middleware/goldengate/overview/index.html

In this article we will do , out-of-Place Manual Upgrade from Previous 11.2.0.N Version to the Latest 11.2.0.N Patchset:

The 11.2.0.2 and later patchsets are a full release. The 11.2 Patchset Installer does not update existing 11.2 installations.The install process performs a new installation whether you are doing out-of-place upgrade or in-place upgrade.

Starting with 11.2.0.2, you have two ways to apply a patchset:

Out-of-place upgrade (Recommended)
In-place upgrade

In the below example we are going to upgrade 11.2.0.3 DB to 11.2.0.4 using out-of-place manual upgrade method :

Current GRID Version : 11.2.0.3 ---> Upgrade to 11.2.0.4
Current GRID_HOME : /data01/app/11.2.0/grid_11203
Upgrade to GRID_HOME : /data01/app/11.2.0/grid_11204

Current RDBMS Version : 11.2.0.3 ---> Upgrade to 11.2.0.4
Current ORACLE_HOME : /data01/app/oracle/product/11.2.0.3/db_1
Upgrade to ORACLE_HOME : /data01/app/oracle/product/11.2.0.4/db_1

Step- 1; Upgrade the Grid Infra

http://ora10gadmin.blogspot.in/2015/01/upgrading-grid-infrastructure-from.html

Step 2: Upgrade RDBMS

1. Install 11.2.0.4 binaries in new ORACLE_HOME /data01/app/oracle/product/11.2.0.4/db_1

Download the binaries from oracle support website and unzip in a directory as below:

unzip p13390677_112040_Linux-x86-64_1of7 -d /home/oracle/software
unzip p13390677_112040_Linux-x86-64_2of7 -d /home/oracle/software

STEP 2
======
Install the latest 11.2 RDBMS Software into a new ORACLE_HOME.

Please note that with the 11.2 installation forward, all base RDBMS components are installed. The only options are if a component is linked on or off (active and able to be used). Custom installations are not possible.

You can run "opatch lsinventory -detail" against previous and new ORACLE_HOME to compare installed products.

Also please note:

rootupgrade.sh will fail if you are using /opt/oracle as the ORACLE_BASE. For further information please see:

Note: 1281913.1 Root Script Fails if ORACLE_BASE is set to /opt/oracle

Start the runInstaller :

[oracle@node1 database]$ ls -lrth
total 60K
drwxrwxr-x 2 oracle oinstall 4.0K Aug 27 2013 sshsetup
-rwxrwxr-x 1 oracle oinstall 3.2K Aug 27 2013 runInstaller
drwxrwxr-x 2 oracle oinstall 4.0K Aug 27 2013 rpm
drwxrwxr-x 2 oracle oinstall 4.0K Aug 27 2013 response
-rwxrwxr-x 1 oracle oinstall 30K Aug 27 2013 readme.html
drwxrwxr-x 14 oracle oinstall 4.0K Aug 27 2013 stage
-rwxrwxr-x 1 oracle oinstall 500 Aug 27 2013 welcome.html
drwxrwxr-x 4 oracle oinstall 4.0K Aug 27 2013 install

[oracle@node1 database]$ ./runInstaller
Starting Oracle Universal Installer...

Checking Temp space: must be greater than 120 MB. Actual 4702 MB Passed
Checking swap space: must be greater than 150 MB. Actual 1285 MB Passed
Checking monitor: must be configured to display at least 256 colors. Actual 16777216 Passed
Preparing to launch Oracle Universal Installer from /tmp/OraInstall2015-01-16_10-11-15AM. Please wait ...
[oracle@node1 database]$ You can find the log of this install session at:
/data01/app/oraInventory/logs/installActions2015-01-16_10-11-15AM.log

When you click next it will promt you to run root.sh of node1 & node2 . So execute the root.sh using root on both nodes :

[root@node1 ~]# sh /data01/app/oracle/product/11.2.0.4/db_1/root.sh
Performing root user operation for Oracle 11g

The following environment variables are set as:
ORACLE_OWNER= oracle
ORACLE_HOME= /data01/app/oracle/product/11.2.0.4/db_1

Enter the full pathname of the local bin directory: [/usr/local/bin]:
The contents of "dbhome" have not changed. No need to overwrite.
The contents of "oraenv" have not changed. No need to overwrite.
The contents of "coraenv" have not changed. No need to overwrite.

Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Finished product-specific root actions.

Check the current version of the 11.2.0.1 Oracle RDBMS time zone definitions -
If this reports version 14 then there is no action to take for the Oracle time zone definitions, you can upgrade from 10.1.0.5 / 10.2.0.x or 11.1.0.x to 11.2.0.2 without any action on TSLTZ or TSTZ data.

If it is not 14 follow the doc : Actions For DST Updates When Upgrading To Or Applying The 11.2.0.2 Patchset (Doc ID 1201253.1)

SQL>SELECT version FROM v$timezone_file;

VERSION
----------
14

Now run the Pre-upgrade Utility tool and follow the recommendations:

[oracle@node1 ~]$ cp /data01/app/oracle/product/11.2.0.4/db_1/rdbms/admin/utlu112i.sql /home/oracle/upgrade/
[oracle@node1 ~]$ echo $ORACLE_HOME
/data01/app/oracle/product/11.2.0.3/db_1
[oracle@node1 ~]$
[oracle@node1 ~]$ /data01/app/oracle/product/11.2.0.3/db_1/bin/sqlplus / as sysdba

SQL*Plus: Release 11.2.0.3.0 Production on Fri Jan 16 12:19:13 2015

Copyright (c) 1982, 2011, Oracle. All rights reserved.

Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64bit Production
With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP,
Data Mining and Real Application Testing options

SQL> spool pre_upgrade_utlu112i_log.log
SQL> @/home/oracle/upgrade/utlu112i.sql
Oracle Database 11.2 Pre-Upgrade Information Tool 01-16-2015 12:21:01
Script Version: 11.2.0.4.0 Build: 001
.
**********************************************************************
Database:
**********************************************************************
--> name: RACDB
--> version: 11.2.0.3.0
--> compatible: 11.2.0.0.0
--> blocksize: 8192
--> platform: Linux x86 64-bit
--> timezone file: V14
.
**********************************************************************
Tablespaces: [make adjustments in the current environment]
**********************************************************************
--> SYSTEM tablespace is adequate for the upgrade.
.... minimum required size: 932 MB
--> SYSAUX tablespace is adequate for the upgrade.
.... minimum required size: 724 MB
--> UNDOTBS1 tablespace is adequate for the upgrade.
.... minimum required size: 400 MB
--> TEMP tablespace is adequate for the upgrade.
.... minimum required size: 60 MB
.
**********************************************************************
Flashback: OFF
**********************************************************************
**********************************************************************
Update Parameters: [Update Oracle Database 11.2 init.ora or spfile]
Note: Pre-upgrade tool was run on a lower version 64-bit database.
**********************************************************************
--> If Target Oracle is 32-Bit, refer here for Update Parameters:
-- No update parameter changes are required.
.

--> If Target Oracle is 64-Bit, refer here for Update Parameters:
-- No update parameter changes are required.
.
**********************************************************************
Renamed Parameters: [Update Oracle Database 11.2 init.ora or spfile]
**********************************************************************
-- No renamed parameters found. No changes are required.
.
**********************************************************************
Obsolete/Deprecated Parameters: [Update Oracle Database 11.2 init.ora or spfile]
**********************************************************************
-- No obsolete parameters found. No changes are required
.

**********************************************************************
Components: [The following database components will be upgraded or installed]
**********************************************************************
--> Oracle Catalog Views [upgrade] VALID
--> Oracle Packages and Types [upgrade] VALID
--> JServer JAVA Virtual Machine [upgrade] VALID
--> Oracle XDK for Java [upgrade] VALID
--> Real Application Clusters [upgrade] VALID
--> Oracle Workspace Manager [upgrade] VALID
--> OLAP Analytic Workspace [upgrade] VALID
--> OLAP Catalog [upgrade] VALID
--> EM Repository [upgrade] VALID
--> Oracle Text [upgrade] VALID
--> Oracle XML Database [upgrade] VALID
--> Oracle Java Packages [upgrade] VALID
--> Oracle interMedia [upgrade] VALID
--> Spatial [upgrade] VALID
--> Expression Filter [upgrade] VALID
--> Rule Manager [upgrade] VALID
--> Oracle Application Express [upgrade] VALID
... APEX will only be upgraded if the version of APEX in
... the target Oracle home is higher than the current one.
--> Oracle OLAP API [upgrade] VALID
.
**********************************************************************
Miscellaneous Warnings
**********************************************************************
WARNING: --> The "cluster_database" parameter is currently "TRUE"
.... and must be set to "FALSE" prior to running a manual upgrade.
WARNING: --> Your recycle bin contains 2 object(s).
.... It is REQUIRED that the recycle bin is empty prior to upgrading
.... your database. The command:
PURGE DBA_RECYCLEBIN
.... must be executed immediately prior to executing your upgrade.
WARNING: --> Database contains schemas with objects dependent on DBMS_LDAP package.
.... Refer to the 11g Upgrade Guide for instructions to configure Network ACLs.
.... USER APEX_030200 has dependent objects.
.
**********************************************************************
Recommendations
**********************************************************************
Oracle recommends gathering dictionary statistics prior to
upgrading the database.
To gather dictionary statistics execute the following command
while connected as SYSDBA:

EXECUTE dbms_stats.gather_dictionary_stats;

**********************************************************************
Oracle recommends reviewing any defined events prior to upgrading.

To view existing non-default events execute the following commands
while connected AS SYSDBA:
Events:
SELECT (translate(value,chr(13)||chr(10),' ')) FROM sys.v$parameter2
WHERE UPPER(name) ='EVENT' AND isdefault='FALSE'

Trace Events:
SELECT (translate(value,chr(13)||chr(10),' ')) from sys.v$parameter2
WHERE UPPER(name) = '_TRACE_EVENTS' AND isdefault='FALSE'

Changes will need to be made in the init.ora or spfile.

**********************************************************************

Backup database for a Manual Upgrade:

run {
Backup database format 'someformatdirectory%U' TAG BEFORE_UPGRADE ;
Backup current controlfile format '/location/control_backup_name' ;
}

Prepare the New Oracle Home for the Upgrade:

1. Copy configuration files from 11.2.0.3 home to 11.2.0.4 home $ORACLE_HOME/dbs ( pfile , spfile)

2. Take backup of spfile also by creating a pfile at some non-default location

3. copy the password file from old home to new home

4. If you are doing a cluster database upgrade perform above steps on all nodes

5. In the pfile make necessary changes like :

--- Remove obsolete parameters

--- Make sure compatibility parameter is 11.2

-- Adjust other parameters as suggested by pre-upgrade utility

-- Make sure all paths are fully specified in the pfile

-- If you are upgrading a cluster database , make sure you make cluster_database=FALSE and after upgrade

you must set this back to TRUE

Manually Upgrade the database :

1. Shutdown the database

2. change the PATH and bash profiles to point to new $ORACLE_HOME

3. If there us a different user and group for GRID and Oracle RDBMS then run the setasmgidwrap script which is located in $GRID_HOME/bin

against the new $ORACLE_HOME/bin/oracle directory with the -o option as shown below:

Run on all nodes :

[root@node1 ~]# ls -lrth /data01/app/oracle/product/11.2.0.3/db_1/bin/oracle*

-rwxr-x--- 1 oracle oinstall 0 Sep 17 2011 /data01/app/oracle/product/11.2.0.3/db_1/bin/oracleO

-rwsr-s--x 1 oracle asmdba 222M Dec 21 11:57 /data01/app/oracle/product/11.2.0.3/db_1/bin/oracle

[root@node1 ~]# ls -lrth /data01/app/oracle/product/11.2.0.4/db_1/bin/oracle*

-rwxr-x--- 1 oracle oinstall 0 Aug 24 2013 /data01/app/oracle/product/11.2.0.4/db_1/bin/oracleO

-rwsr-s--x 1 oracle oinstall 229M Jan 16 11:10 /data01/app/oracle/product/11.2.0.4/db_1/bin/oracle

[grid@node1 ~]$ $GI_HOME/bin/setasmgidwrap o=/data01/app/oracle/product/11.2.0.4/db_1/bin/oracle

4. From the new $ORACLE_HOME/rdbms/admin

$ORACLE_HOME/bin/sqlplus / as sysdba

SQL>spool startup_upgrade.log

SQL> Startup upgrade pfile='/location of pfile for new home with cluster_databse=false' ;

sql>spool off;

5. Once the DB is open in upgrade mode:

SQL> SPool catupgrade.log

SQL> @$ORACLE_HOME/rdbms/admin/catupgrd.sql ;

SQL> spool off ;

The catupgrd.sql script determines which upgrade scripts must be run , runs them and then shut down the database.

if this script fails for some reason , then you can run the script again after you correct the issue.

6. Now start the database .

7.Run SQL> @$ORACLE_HOME/rdbms/admin/utlu112s.sql "The Post Upgrade status tool" which provides the summary of the upgrade at the end of spool log.

SQL> @$ORACLE_HOME/rdbms/admin/utlu112s.sql ;

Oracle Database 11.2 Post-Upgrade Status Tool 01-23-2015 22:05:02

Component Current Version Elapsed Time

Name Status Number HH:MM:SS

Oracle Server

. VALID 11.2.0.4.0 02:30:04

JServer JAVA Virtual Machine

. VALID 11.2.0.4.0 01:41:39

Oracle Real Application Clusters

. VALID 11.2.0.4.0 00:00:10

Oracle Workspace Manager

. VALID 11.2.0.4.0 00:01:57

OLAP Analytic Workspace

. VALID 11.2.0.4.0 00:04:28

OLAP Catalog

. VALID 11.2.0.4.0 00:04:53

Oracle OLAP API

. VALID 11.2.0.4.0 00:02:59

Oracle Enterprise Manager

. VALID 11.2.0.4.0 00:08:52

Oracle XDK

. VALID 11.2.0.4.0 00:02:50

Oracle Text

. VALID 11.2.0.4.0 00:02:12

Oracle XML Database

. VALID 11.2.0.4.0 00:10:46

Oracle Database Java Packages

. VALID 11.2.0.4.0 00:02:01

Oracle Multimedia

. VALID 11.2.0.4.0 03:59:06

Spatial

. VALID 11.2.0.4.0 00:10:10

Oracle Expression Filter

. VALID 11.2.0.4.0 00:01:53

Oracle Rules Manager

. VALID 11.2.0.4.0 00:01:05

Oracle Application Express

. VALID 3.2.1.00.12

Final Actions

. 00:00:01

Total Upgrade Time: 09:05:24

PL/SQL procedure successfully completed.

8. Run @CATuppst.sql to perform upgrade actions that do not require the database to be in upgrade mode.

9. To reccompile any remaining stored procedure or java code

SQl> @$ORACLE_HOME/rdbms/admin/utlrp.sql ;

Verify the invalid objects count :

select count(*) from dba_invalid_objects ;

If the pre-upgrade information tool detected INVALID objects and populated teh registry$sys_inv_objs and registry$nonsys_inv_objs tables

then execute $ORACLE_HOME/rdbms/admin/utluiobj.sql to display only those objects which are newly invalid because of the upgrade process;

10. If you are upgrading a cluster database , then upgrade the database configuration in clusterware

srvctl upgrade database -d db-unique-name -o /new_oracle_home

11.For cluster database change the CLUSTER_DATABASE=TRUE and stop DB and start the database using the srvctl utility

SQL> select * from v$version ;

BANNER
--------------------------------------------------------------------------------
Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production
PL/SQL Release 11.2.0.4.0 - Production
CORE 11.2.0.4.0 Production
TNS for Linux: Version 11.2.0.4.0 - Production
NLSRTL Version 11.2.0.4.0 - Production

Monday, 19 January 2015

Changing the DBID & DBNAME of a database using nid

Follow the below steps to change only the dbid :
====================================
1. Backup the database
2. Shutdown all instances in the cluster make sure that this is a consistent shutdown done using shutdown immediate or shutdown normal command.
3. Make the parameter cluster_database=false
4. Startup mount only on one of the instances
5. Invoke the DBNEWID utility (nid) from the command line using a user with SYSDBA privilege.Do not specify a new DBNAME:

$ORACLE_HOME/bin/nid TARGET=SYS/password@DB11G

6. Shutdown IMMEDIATE of the database (Note: This step is not required in 10g)
7. Open the database with resetlogs
8 . Make the parameter cluster_database=true and shutdown the current instance and startup all the instances
9. Take full backup of the database .

Follow the below steps to change only the DBNAME :
=========================================

1. Backup the database
2. Shutdown all instances in the cluster make sure that this is a consistent shutdown done using shutdown immediate or shutdown normal command.
3. Set the parameter CLUSTER_DATABASE=false
4. Startup mount only on one of the instances
5. Invoke the utility on the command line, specifying a valid user with the SYSDBA privilege.You must specify both the DBNAME and SETNAME parameters.
The SETNAME parameter tells the DBNEWID utility to only alter the database name.

$ORACLE_HOME/bin/nid TARGET=SYS/password@DB11G DBNAME=DB_NEW SETNAME=YES

6. Shutdown IMMEDIATE of the database
7. Set the DB_NAME initialization parameter in the initialization parameter file (PFILE) to the new database name
8. Create a new password file.
9. Open the database. When opening the database the RESETLOGS option is not needed so the database can be started using the STARTUP command.
11. Make the parameter CLUSTER_DATABASE=true and shutdown the current instance and startup all the instances.

Follow the below steps to Change Both DBID and DBNAME :
===============================================
1. Backup the database
2. Shutdown all instances in the cluster make sure that this is a consistent shutdown done using shutdown immediate or shutdown normal command.
3. Make the parameter CLUSTER_DATABASE=false
4. Startup mount only on one of the instances
5. Invoke the DBNEWID utility on the command line, specifying a valid user with the SYSDBA privilege

$ORACLE_HOME/bin/nid TARGET=SYS/password@DB11G DBNAME=DB_NEW

6. Shutdown IMMEDIATE of the database
7. Set the DB_NAME initialization parameter in the initialization parameter file (PFILE) to the new database name
8. Create a new password file.
10. Mount and Open the database with resetlogs
11. Make the parameter CLUSTER_DATABASE=true and shutdown the current instance and startup all the instances

Some points to consider :
===================

1. The DBNEWID utility does not change the server parameter file (SPFILE). Therefore, if you use SPFILE to start your Oracle database, you must recreate the initialization parameter file from the server parameter file, remove the server parameter file, change the DB_NAME in the initialization parameter file, and then recreate the server parameter file.

2. DBNEWID utility makes the changes in the controlfile and datafile headers. This utility is not RAC aware. So it will not update the OCR when the database is renamed. Hence, the user needs to invoke SRVCTL to remove and again add the database information to OCR

Example Below is for changing just the DBID:

$ORACLE_HOME/bin/nid TARGET=SYS/password@DB11G

DBNEWID: Release 11.2.0.4.0 - Production on Mon Jan 19 03:53:54 2015

Copyright (c) 1982, 2011, Oracle and/or its affiliates. All rights reserved.

Connected to database DB11G (DBID=338404813)

Connected to server version 11.2.0

Control Files in database:
+DATA/db11g/controlfile/current.261.868735245
+DATA/db11g/controlfile/current.260.868735247

Change database ID of database DB11G? (Y/[N]) => Y

Proceeding with operation
Changing database ID from 338404813 to 339029651
Control File +DATA/db11g/controlfile/current.261.868735245 - modified
Control File +DATA/db11g/controlfile/current.260.868735247 - modified
Datafile +DATA/db11g/datafile/system.256.86873510 - dbid changed
Datafile +DATA/db11g/datafile/sysaux.257.86873510 - dbid changed
Datafile +DATA/db11g/datafile/undotbs1.258.86873510 - dbid changed
Datafile +DATA/db11g/datafile/users.259.86873510 - dbid changed
Datafile +DATA/db11g/tempfile/temp.268.86873527 - dbid changed
Control File +DATA/db11g/controlfile/current.261.868735245 - dbid changed
Control File +DATA/db11g/controlfile/current.260.868735247 - dbid changed
Instance shut down

Database ID for database DB11G changed to 339029651.
All previous backups and archived redo logs for this database are unusable.
Database is not aware of previous backups and archived logs in Recovery Area.
Database has been shutdown, open database with RESETLOGS option.
Succesfully changed database ID.
DBNEWID - Completed succesfully.

Troubleshooting a DBID Change Operation
=================================

If the DBNEWID utility succeeds in its validation stage but detects an error while changing the DBID, then the utility stops and leaves the database in the middle of the change. In this case, you cannot open the database until the DBNEWID operation is either completed or reverted. DBNEWID displays messages indicating the status of the operation.

Before continuing or reverting, fix the underlying cause of the error. Sometimes the only solution is to restore the whole database from a recent backup and perform recovery to the point in time before DBNEWID was started. This underscores the importance of having a recent backup available before running DBNEWID.

If you choose to continue the DBID change operation rather than revert it, reexecute your original command. The DBNEWID utility resumes and attempts to continue the change until all datafiles and control files have the new DBID. At this point, the database is left mounted. You should shut it down and then mount it again prior to opening it with the RESETLOGS option.

If you choose to revert a DBNEWID operation, and if the reversion succeeds, then DBNEWID reverts all performed changes and leaves the database in a mounted state.

To revert a stalled DBID change operation, run the DBNEWID utility again, specifying the REVERT keyword. For example:

% nid TARGET=SYS/oracle REVERT=YES LOGFILE=$HOME/nid.log

Troubleshooting a Database Name Change Operation
=========================================

If you specify that only the database name should be changed (and not the DBID), then the validation process is the same as for a DBID change except that DBNEWID checks only the control files. It does not read the datafiles. If the validation encounters a problem, then the database is left mounted.

It is possible for validation to succeed, but for the actual database name change to fail. The possible failure scenarios depend on how many control files are in the database, as follows:

If you have one or more control files and DBNEWID fails on the first control file, then the database name is not changed in the control file. You can either try the operation again or open the database and resume normal database use.
If you have more than one control file and DBNEWID fails on the second control file or on any one thereafter, then some control files will have the old DBNAME and some will have the new DBNAME. In this case, you must either manually copy the first changed control file to all CONTROL_FILES locations, or revert by copying the unchanged control files to all CONTROL_FILES locations.

Sunday, 18 January 2015

[FATAL] [INS-00001] Unknown irrecoverable error during silent installation

If the response file for an 11.2 silent install uses a password that contains the character '%' (percent sign) , then the installation will fail with the following errors:

SEVERE:Unknown environment variable: %1
[FATAL] [INS-00001] Unknown irrecoverable error
CAUSE: No additional information available.

Workaround:

Change any passwords in the response file that use the '%' character.

Saturday, 10 January 2015

Upgrading Grid Infrastructure from 11.2.0.3 to 11.2.0.4

Refer the below docs for pre upgrade considerations:

--> Script to Collect DB Upgrade/Migrate Diagnostic Information (dbupgdiag.sql) (Doc ID 556610.1)

---> Things to Consider Before Upgrading to 11.2.0.3/11.2.0.4 Grid Infrastructure/ASM (Doc ID 1363369.1)

Make sure all CRS components are healthy before you start the Upgrade.
---------------------------------------------------------------------------------------

[grid@node1 grid] crsctl stat res -t | tee /home/grid/logs/crsctl_stat_p_bef_patch.txt

<= Is anything other than gsd OFFLINE ?

--------------------------------------------------------------------------------
NAME TARGET STATE SERVER STATE_DETAILS
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.DATA.dg
ONLINE ONLINE node1
ONLINE ONLINE node2
ora.FRA.dg
ONLINE ONLINE node1
ONLINE ONLINE node2
ora.LISTENER.lsnr
ONLINE ONLINE node1
ONLINE ONLINE node2
ora.USER_DATA.dg
ONLINE ONLINE node1
ONLINE ONLINE node2
ora.asm
ONLINE ONLINE node1 Started
ONLINE ONLINE node2 Started
ora.gsd
OFFLINE OFFLINE node1
OFFLINE OFFLINE node2
ora.net1.network
ONLINE ONLINE node1
ONLINE ONLINE node2
ora.ons
ONLINE ONLINE node1
ONLINE ONLINE node2
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
1 ONLINE ONLINE node2
ora.LISTENER_SCAN2.lsnr
1 ONLINE ONLINE node1
ora.LISTENER_SCAN3.lsnr
1 ONLINE ONLINE node1
ora.cvu
1 ONLINE ONLINE node1
ora.node1.vip
1 ONLINE ONLINE node1
ora.node2.vip
1 ONLINE ONLINE node2
ora.oc4j
1 ONLINE ONLINE node2
ora.racdb.db
1 ONLINE ONLINE node1 Open
2 ONLINE ONLINE node2 Open
ora.scan1.vip
1 ONLINE ONLINE node2
ora.scan2.vip
1 ONLINE ONLINE node1
ora.scan3.vip
1 ONLINE ONLINE node1

[grid@node1 grid] crsctl query crs activeversion | tee /home/grid/logs/crsversion_bef_patch.txt

Oracle Clusterware active version on the cluster is [11.2.0.3.0]

[grid@node1 grid] crsctl query crs softwareversion | tee /home/grid/logs/crsversion_software_bef_patch.txt

Oracle Clusterware version on node [node1] is [11.2.0.3.0]

[grid@node1 grid] crsctl stat res -p | tee /home/grid/logs/crs_stat_p_bef_patch.txt

[grid@node1 grid] crsctl query css votedisk | tee /home/grid/logs/qry_css_bef_patch.txt

## STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
1. ONLINE 8648b62898a24f65bf5ac6becf11922b (/dev/oracleasm/disks/OCR_VOTE) [DATA]
Located 1 voting disk(s).

[grid@node1 grid] ocrcheck | tee /home/grid/logs/ocrchk_bef_patch.txt

Status of Oracle Cluster Registry is as follows :
Version : 3
Total space (kbytes) : 262120
Used space (kbytes) : 2944
Available space (kbytes) : 259176
ID : 1499390533
Device/File Name : +DATA
Device/File integrity check succeeded

Device/File not configured

Device/File not configured

Device/File not configured

Device/File not configured

Cluster registry integrity check succeeded

Logical corruption check bypassed due to non-privileged user

[grid@node1 grid] crsctl check cluster -all | tee /home/grid/logs/check_cluster.txt
**************************************************************
node1:
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
**************************************************************
node2:
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
**************************************************************

From oracle user check instance status :
------------------------------------------------

[oracle@node1 oracle] srvctl status database -d racdb | tee /home/oracle/upgrade/check_cluster.txt

Instance racdb1 is running on node node1
Instance racdb2 is running on node node2

[oracle@node1 oracle] srvctl config database -d racdb | tee /home/oracle/upgrade/check_cluster.txt

Database unique name: racdb
Database name: racdb
Oracle home: /data01/app/oracle/product/11.2.0.3/db_1
Oracle user: oracle
Spfile: +DATA/racdb/spfileracdb.ora
Domain: localdomain
Start options: open
Stop options: immediate
Database role: PRIMARY
Management policy: AUTOMATIC
Server pools: racdb
Database instances: racdb1,racdb2
Disk Groups: DATA,USER_DATA
Mount point paths:
Services:
Type: RAC
Database is administrator managed

On both node1 and node2 make directory for 11.2.0.4 ORACLE_HOME :
------------------------------------------------------------------------------------------------

mkdir -p /data01/app/11.2.0/grid_11204

root@node1 ~]# mkdir -p /data01/app/11.2.0/grid_11204
[root@node1 ~]# cd /data01/app/11.2.0
[root@node1 11.2.0]# ls -lrth grid_11204

drwxr-xr-x 2 root root 4.0K Jan 9 14:32 grid_11204

[root@node1 11.2.0]# chown -R grid:oinstall grid_11204/
[root@node1 11.2.0]# chmod -R 775 grid_11204/
[root@node1 11.2.0]# ls -lrth grid_11204

drwxrwxr-x 2 grid oinstall 4.0K Jan 9 14:32 grid_11204
[root@node1 11.2.0]#

Verify the qosadmin users :

Known bug related to qosadmin user AUTH failure : So check before upgrade only :

GI Upgrade from 11.2.0.3.6+ to 11.2.0.4 or 12.1.0.1 Fails with User(qosadmin) is deactivated. AUTH FAILURE. (Doc ID 1577072.1)
otherwise rootupgrade.sh will fail with error "Failed to perform J2EE (OC4J) Container Resource upgrade at /haclu/64bit/11.2.0.4/grid/crs/install/crsconfig_lib.pm line 9323"

[root@node1 ~]# /data01/app/11.2.0/grid_11203/bin/qosctl qosadmin -listusers
AbstractLoginModule password: ---> Default password is oracle112
oc4jadmin
JtaAdmin
qosadmin

Run Runcluvfy utility pre crs upgrade :
----------------------------------------

./runcluvfy.sh stage -pre crsinst -upgrade -n node1,node2 -rolling -src_crshome /data01/app/11.2.0/grid_11203 -dest_crshome /data01/app/11.2.0/grid_11204 -dest_version 11.2.0.4.0 -fixup -fixupdir /home/grid/logs -verbose | tee /home/grid/logs/runcluvfy.out

======================start: Run cluster verify output ========================

Performing pre-checks for cluster services setup

Checking node reachability...

Check: Node reachability from node "node1"
Destination Node Reachable?
------------------------------------ ------------------------
node1 yes
node2 yes
Result: Node reachability check passed from node "node1"

Checking user equivalence...

Check: User equivalence for user "grid"
Node Name Status
------------------------------------ ------------------------
node2 passed
node1 passed
Result: User equivalence check passed for user "grid"

Checking CRS user consistency
Result: CRS user consistency check successful

Checking node connectivity...

Checking hosts config file...
Node Name Status
------------------------------------ ------------------------
node2 passed
node1 passed

Verification of the hosts config file successful

Interface information for node "node2"
Name IP Address Subnet Gateway Def. Gateway HW Address MTU
------ --------------- --------------- --------------- --------------- ----------------- ------
eth0 192.168.56.72 192.168.56.0 0.0.0.0 UNKNOWN 08:00:27:C4:E7:ED 1500
eth0 192.168.56.91 192.168.56.0 0.0.0.0 UNKNOWN 08:00:27:C4:E7:ED 1500
eth0 192.168.56.82 192.168.56.0 0.0.0.0 UNKNOWN 08:00:27:C4:E7:ED 1500
eth1 192.168.10.2 192.168.10.0 0.0.0.0 UNKNOWN 08:00:27:53:6F:4D 1500
eth1 169.254.231.223 169.254.0.0 0.0.0.0 UNKNOWN 08:00:27:53:6F:4D 1500

Interface information for node "node1"
Name IP Address Subnet Gateway Def. Gateway HW Address MTU
------ --------------- --------------- --------------- --------------- ----------------- ------
eth0 192.168.56.71 192.168.56.0 0.0.0.0 UNKNOWN 08:00:27:C4:E7:E6 1500
eth0 192.168.56.93 192.168.56.0 0.0.0.0 UNKNOWN 08:00:27:C4:E7:E6 1500
eth0 192.168.56.92 192.168.56.0 0.0.0.0 UNKNOWN 08:00:27:C4:E7:E6 1500
eth0 192.168.56.81 192.168.56.0 0.0.0.0 UNKNOWN 08:00:27:C4:E7:E6 1500
eth1 192.168.10.1 192.168.10.0 0.0.0.0 UNKNOWN 08:00:27:53:6F:46 1500
eth1 169.254.204.171 169.254.0.0 0.0.0.0 UNKNOWN 08:00:27:53:6F:46 1500

Check: Node connectivity for interface "eth0"
Source Destination Connected?
------------------------------ ------------------------------ ----------------
node2[192.168.56.72] node2[192.168.56.91] yes
node2[192.168.56.72] node2[192.168.56.82] yes
node2[192.168.56.72] node1[192.168.56.71] yes
node2[192.168.56.72] node1[192.168.56.93] yes
node2[192.168.56.72] node1[192.168.56.92] yes
node2[192.168.56.72] node1[192.168.56.81] yes
node2[192.168.56.91] node2[192.168.56.82] yes
node2[192.168.56.91] node1[192.168.56.71] yes
node2[192.168.56.91] node1[192.168.56.93] yes
node2[192.168.56.91] node1[192.168.56.92] yes
node2[192.168.56.91] node1[192.168.56.81] yes
node2[192.168.56.82] node1[192.168.56.71] yes
node2[192.168.56.82] node1[192.168.56.93] yes
node2[192.168.56.82] node1[192.168.56.92] yes
node2[192.168.56.82] node1[192.168.56.81] yes
node1[192.168.56.71] node1[192.168.56.93] yes
node1[192.168.56.71] node1[192.168.56.92] yes
node1[192.168.56.71] node1[192.168.56.81] yes
node1[192.168.56.93] node1[192.168.56.92] yes
node1[192.168.56.93] node1[192.168.56.81] yes
node1[192.168.56.92] node1[192.168.56.81] yes
Result: Node connectivity passed for interface "eth0"

Check: TCP connectivity of subnet "192.168.56.0"
Source Destination Connected?
------------------------------ ------------------------------ ----------------
node1:192.168.56.71 node2:192.168.56.72 passed
node1:192.168.56.71 node2:192.168.56.91 passed
node1:192.168.56.71 node2:192.168.56.82 passed
node1:192.168.56.71 node1:192.168.56.93 passed
node1:192.168.56.71 node1:192.168.56.92 passed
node1:192.168.56.71 node1:192.168.56.81 passed
Result: TCP connectivity check passed for subnet "192.168.56.0"

Check: Node connectivity for interface "eth1"
Source Destination Connected?
------------------------------ ------------------------------ ----------------
node2[192.168.10.2] node1[192.168.10.1] yes
Result: Node connectivity passed for interface "eth1"

Check: TCP connectivity of subnet "192.168.10.0"
Source Destination Connected?
------------------------------ ------------------------------ ----------------
node1:192.168.10.1 node2:192.168.10.2 passed
Result: TCP connectivity check passed for subnet "192.168.10.0"

Checking subnet mask consistency...
Subnet mask consistency check passed for subnet "192.168.56.0".
Subnet mask consistency check passed for subnet "192.168.10.0".
Subnet mask consistency check passed.
Checking multicast communication...

Checking subnet "192.168.56.0" for multicast communication with multicast group "230.0.1.0"...
Check of subnet "192.168.56.0" for multicast communication with multicast group "230.0.1.0" passed.

Checking subnet "192.168.10.0" for multicast communication with multicast group "230.0.1.0"...
Check of subnet "192.168.10.0" for multicast communication with multicast group "230.0.1.0" passed.

Check of multicast communication passed.

Checking OCR integrity...

OCR integrity check passed

Checking ASMLib configuration.
Node Name Status
------------------------------------ ------------------------
node2 passed
node1 passed
Result: Check for ASMLib configuration passed.

Check: Total memory
Node Name Available Required Status
------------ ------------------------ ------------------------ ----------
node2 1.9598GB (2055048.0KB) 1.5GB (1572864.0KB) passed
node1 1.9598GB (2055048.0KB) 1.5GB (1572864.0KB) passed
Result: Total memory check passed
Check: Available memory
Node Name Available Required Status
------------ ------------------------ ------------------------ ----------
node2 774.3477MB (792932.0KB) 50MB (51200.0KB) passed
node1 913.3008MB (935220.0KB) 50MB (51200.0KB) passed
Result: Available memory check passed

Check: Swap space
Node Name Available Required Status
------------ ------------------------ ------------------------ ----------
node2 1.7578GB (1843196.0KB) 2.9398GB (3082572.0KB) failed
node1 1.7578GB (1843196.0KB) 2.9398GB (3082572.0KB) failed
Result: Swap space check failed

Check: Free disk space for "node2:/data01/app/11.2.0/grid_11204,node2:/tmp"
Path Node Name Mount point Available Required Status
---------------- ------------ ------------ ------------ ------------ ------------
/data01/app/11.2.0/grid_11204 node2 / 11.376GB 7.5GB passed
/tmp node2 / 11.376GB 7.5GB passed
Result: Free disk space check passed for "node2:/data01/app/11.2.0/grid_11204,node2:/tmp"

Check: Free disk space for "node1:/data01/app/11.2.0/grid_11204,node1:/tmp"
Path Node Name Mount point Available Required Status
---------------- ------------ ------------ ------------ ------------ ------------
/data01/app/11.2.0/grid_11204 node1 / 11.1307GB 7.5GB passed
/tmp node1 / 11.1307GB 7.5GB passed
Result: Free disk space check passed for "node1:/data01/app/11.2.0/grid_11204,node1:/tmp"

Check: User existence for "grid"
Node Name Status Comment
------------ ------------------------ ------------------------
node2 passed exists(201)
node1 passed exists(201)

Checking for multiple users with UID value 201
Result: Check for multiple users with UID value 201 passed
Result: User existence check passed for "grid"

Check: Group existence for "oinstall"
Node Name Status Comment
------------ ------------------------ ------------------------
node2 passed exists
node1 passed exists
Result: Group existence check passed for "oinstall"

Check: Membership of user "grid" in group "oinstall" [as Primary]
Node Name User Exists Group Exists User in Group Primary Status
---------------- ------------ ------------ ------------ ------------ ------------
node2 yes yes yes yes passed
node1 yes yes yes yes passed
Result: Membership check for user "grid" in group "oinstall" [as Primary] passed

Check: Run level
Node Name run level Required Status
------------ ------------------------ ------------------------ ----------
node2 5 3,5 passed
node1 5 3,5 passed
Result: Run level check passed

Check: Hard limits for "maximum open file descriptors"
Node Name Type Available Required Status
---------------- ------------ ------------ ------------ ----------------
node2 hard 65536 65536 passed
node1 hard 65536 65536 passed
Result: Hard limits check passed for "maximum open file descriptors"

Check: Soft limits for "maximum open file descriptors"
Node Name Type Available Required Status
---------------- ------------ ------------ ------------ ----------------
node2 soft 65536 1024 passed
node1 soft 65536 1024 passed
Result: Soft limits check passed for "maximum open file descriptors"

Check: Hard limits for "maximum user processes"
Node Name Type Available Required Status
---------------- ------------ ------------ ------------ ----------------
node2 hard 16384 16384 passed
node1 hard 16384 16384 passed
Result: Hard limits check passed for "maximum user processes"

Check: Soft limits for "maximum user processes"
Node Name Type Available Required Status
---------------- ------------ ------------ ------------ ----------------
node2 soft 16384 2047 passed
node1 soft 16384 2047 passed
Result: Soft limits check passed for "maximum user processes"

There are no oracle patches required for home "/data01/app/11.2.0/grid_11203".

There are no oracle patches required for home "/data01/app/11.2.0/grid_11204".

Check: System architecture
Node Name Available Required Status
------------ ------------------------ ------------------------ ----------
node2 x86_64 x86_64 passed
node1 x86_64 x86_64 passed
Result: System architecture check passed

Check: Kernel version
Node Name Available Required Status
------------ ------------------------ ------------------------ ----------
node2 2.6.39-400.17.1.el6uek.x86_64 2.6.32 passed
node1 2.6.39-400.17.1.el6uek.x86_64 2.6.32 passed
Result: Kernel version check passed

Check: Kernel parameter for "semmsl"
Node Name Current Configured Required Status Comment
---------------- ------------ ------------ ------------ ------------ ------------
node2 250 250 250 passed
node1 250 250 250 passed
Result: Kernel parameter check passed for "semmsl"

Check: Kernel parameter for "semmns"
Node Name Current Configured Required Status Comment
---------------- ------------ ------------ ------------ ------------ ------------
node2 32000 32000 32000 passed
node1 32000 32000 32000 passed
Result: Kernel parameter check passed for "semmns"

Check: Kernel parameter for "semopm"
Node Name Current Configured Required Status Comment
---------------- ------------ ------------ ------------ ------------ ------------
node2 100 100 100 passed
node1 100 100 100 passed
Result: Kernel parameter check passed for "semopm"

Check: Kernel parameter for "semmni"
Node Name Current Configured Required Status Comment
---------------- ------------ ------------ ------------ ------------ ------------
node2 128 128 128 passed
node1 128 128 128 passed
Result: Kernel parameter check passed for "semmni"

Check: Kernel parameter for "shmmax"
Node Name Current Configured Required Status Comment
---------------- ------------ ------------ ------------ ------------ ------------
node2 4294967295 4294967295 1052184576 passed
node1 4294967295 4294967295 1052184576 passed
Result: Kernel parameter check passed for "shmmax"

Check: Kernel parameter for "shmmni"
Node Name Current Configured Required Status Comment
---------------- ------------ ------------ ------------ ------------ ------------
node2 4096 4096 4096 passed
node1 4096 4096 4096 passed
Result: Kernel parameter check passed for "shmmni"

Check: Kernel parameter for "shmall"
Node Name Current Configured Required Status Comment
---------------- ------------ ------------ ------------ ------------ ------------
node2 2097152 2097152 2097152 passed
node1 2097152 2097152 2097152 passed
Result: Kernel parameter check passed for "shmall"

Check: Kernel parameter for "file-max"
Node Name Current Configured Required Status Comment
---------------- ------------ ------------ ------------ ------------ ------------
node2 6815744 6815744 6815744 passed
node1 6815744 6815744 6815744 passed
Result: Kernel parameter check passed for "file-max"

Check: Kernel parameter for "ip_local_port_range"
Node Name Current Configured Required Status Comment
---------------- ------------ ------------ ------------ ------------ ------------
node2 between 9000.0 & 65500.0 between 9000.0 & 65500.0 between 9000.0 & 65500.0 passed
node1 between 9000.0 & 65500.0 between 9000.0 & 65500.0 between 9000.0 & 65500.0 passed
Result: Kernel parameter check passed for "ip_local_port_range"

Check: Kernel parameter for "rmem_default"
Node Name Current Configured Required Status Comment
---------------- ------------ ------------ ------------ ------------ ------------
node2 262144 262144 262144 passed
node1 262144 262144 262144 passed
Result: Kernel parameter check passed for "rmem_default"

Check: Kernel parameter for "rmem_max"
Node Name Current Configured Required Status Comment
---------------- ------------ ------------ ------------ ------------ ------------
node2 4194304 4194304 4194304 passed
node1 4194304 4194304 4194304 passed
Result: Kernel parameter check passed for "rmem_max"

Check: Kernel parameter for "wmem_default"
Node Name Current Configured Required Status Comment
---------------- ------------ ------------ ------------ ------------ ------------
node2 262144 262144 262144 passed
node1 262144 262144 262144 passed
Result: Kernel parameter check passed for "wmem_default"

Node Name Current Configured Required Status Comment
---------------- ------------ ------------ ------------ ------------ ------------
node2 1048576 1048576 1048576 passed
node1 1048576 1048576 1048576 passed
Result: Kernel parameter check passed for "wmem_max"

Check: Kernel parameter for "aio-max-nr"
Node Name Current Configured Required Status Comment
---------------- ------------ ------------ ------------ ------------ ------------
node2 1048576 1048576 1048576 passed
node1 1048576 1048576 1048576 passed
Result: Kernel parameter check passed for "aio-max-nr"

Check: Package existence for "binutils"
Node Name Available Required Status
------------ ------------------------ ------------------------ ----------
node2 binutils-2.20.51.0.2-5.36.el6 binutils-2.20.51.0.2 passed
node1 binutils-2.20.51.0.2-5.36.el6 binutils-2.20.51.0.2 passed
Result: Package existence check passed for "binutils"

Check: Package existence for "compat-libcap1"
Node Name Available Required Status
------------ ------------------------ ------------------------ ----------
node2 compat-libcap1-1.10-1 compat-libcap1-1.10 passed
node1 compat-libcap1-1.10-1 compat-libcap1-1.10 passed
Result: Package existence check passed for "compat-libcap1"

Check: Package existence for "compat-libstdc++-33(x86_64)"
Node Name Available Required Status
------------ ------------------------ ------------------------ ----------
node2 compat-libstdc++-33(x86_64)-3.2.3-69.el6 compat-libstdc++-33(x86_64)-3.2.3 passed
node1 compat-libstdc++-33(x86_64)-3.2.3-69.el6 compat-libstdc++-33(x86_64)-3.2.3 passed
Result: Package existence check passed for "compat-libstdc++-33(x86_64)"

Check: Package existence for "libgcc(x86_64)"
Node Name Available Required Status
------------ ------------------------ ------------------------ ----------
node2 libgcc(x86_64)-4.4.7-3.el6 libgcc(x86_64)-4.4.4 passed
node1 libgcc(x86_64)-4.4.7-3.el6 libgcc(x86_64)-4.4.4 passed
Result: Package existence check passed for "libgcc(x86_64)"

Check: Package existence for "libstdc++(x86_64)"
Node Name Available Required Status
------------ ------------------------ ------------------------ ----------
node2 libstdc++(x86_64)-4.4.7-3.el6 libstdc++(x86_64)-4.4.4 passed
node1 libstdc++(x86_64)-4.4.7-3.el6 libstdc++(x86_64)-4.4.4 passed
Result: Package existence check passed for "libstdc++(x86_64)"

Check: Package existence for "libstdc++-devel(x86_64)"
Node Name Available Required Status
------------ ------------------------ ------------------------ ----------
node2 libstdc++-devel(x86_64)-4.4.7-3.el6 libstdc++-devel(x86_64)-4.4.4 passed
node1 libstdc++-devel(x86_64)-4.4.7-3.el6 libstdc++-devel(x86_64)-4.4.4 passed
Result: Package existence check passed for "libstdc++-devel(x86_64)"

Check: Package existence for "sysstat"
Node Name Available Required Status
------------ ------------------------ ------------------------ ----------
node2 sysstat-9.0.4-20.el6 sysstat-9.0.4 passed
node1 sysstat-9.0.4-20.el6 sysstat-9.0.4 passed
Result: Package existence check passed for "sysstat"

Check: Package existence for "gcc"
Node Name Available Required Status
------------ ------------------------ ------------------------ ----------
node2 gcc-4.4.7-3.el6 gcc-4.4.4 passed
node1 gcc-4.4.7-3.el6 gcc-4.4.4 passed
Result: Package existence check passed for "gcc"

Check: Package existence for "gcc-c++"
Node Name Available Required Status
------------ ------------------------ ------------------------ ----------
node2 gcc-c++-4.4.7-3.el6 gcc-c++-4.4.4 passed
node1 gcc-c++-4.4.7-3.el6 gcc-c++-4.4.4 passed
Result: Package existence check passed for "gcc-c++"

Check: Package existence for "ksh"
Node Name Available Required Status
------------ ------------------------ ------------------------ ----------
node2 ksh-20100621-19.el6 ksh-20100621 passed
node1 ksh-20100621-19.el6 ksh-20100621 passed
Result: Package existence check passed for "ksh"

Check: Package existence for "make"
Node Name Available Required Status
------------ ------------------------ ------------------------ ----------
node2 make-3.81-20.el6 make-3.81 passed
node1 make-3.81-20.el6 make-3.81 passed
Result: Package existence check passed for "make"

Check: Package existence for "glibc(x86_64)"
Node Name Available Required Status
------------ ------------------------ ------------------------ ----------
node2 glibc(x86_64)-2.12-1.107.el6 glibc(x86_64)-2.12 passed
node1 glibc(x86_64)-2.12-1.107.el6 glibc(x86_64)-2.12 passed
Result: Package existence check passed for "glibc(x86_64)"

Check: Package existence for "glibc-devel(x86_64)"
Node Name Available Required Status
------------ ------------------------ ------------------------ ----------
node2 glibc-devel(x86_64)-2.12-1.107.el6 glibc-devel(x86_64)-2.12 passed
node1 glibc-devel(x86_64)-2.12-1.107.el6 glibc-devel(x86_64)-2.12 passed
Result: Package existence check passed for "glibc-devel(x86_64)"

Check: Package existence for "libaio(x86_64)"
Node Name Available Required Status
------------ ------------------------ ------------------------ ----------
node2 libaio(x86_64)-0.3.107-10.el6 libaio(x86_64)-0.3.107 passed
node1 libaio(x86_64)-0.3.107-10.el6 libaio(x86_64)-0.3.107 passed
Result: Package existence check passed for "libaio(x86_64)"

Check: Package existence for "libaio-devel(x86_64)"
Node Name Available Required Status
------------ ------------------------ ------------------------ ----------
node2 libaio-devel(x86_64)-0.3.107-10.el6 libaio-devel(x86_64)-0.3.107 passed
node1 libaio-devel(x86_64)-0.3.107-10.el6 libaio-devel(x86_64)-0.3.107 passed
Result: Package existence check passed for "libaio-devel(x86_64)"

Checking for multiple users with UID value 0
Result: Check for multiple users with UID value 0 passed

Check: Current group ID
Result: Current group ID check passed

Starting check for consistency of primary group of root user
Node Name Status
------------------------------------ ------------------------
node2 passed
node1 passed

Check for consistency of root user's primary group passed

Check: Package existence for "cvuqdisk"
Node Name Available Required Status
------------ ------------------------ ------------------------ ----------
node2 cvuqdisk-1.0.9-1 cvuqdisk-1.0.9-1 passed
node1 cvuqdisk-1.0.9-1 cvuqdisk-1.0.9-1 passed
Result: Package existence check passed for "cvuqdisk"

Starting Clock synchronization checks using Network Time Protocol(NTP)...
NTP Configuration file check started...
Network Time Protocol(NTP) configuration file not found on any of the nodes. Oracle Cluster Time Synchronization Service(CTSS) can be used instead of NTP for time sync
hronization on the cluster nodes

Checking daemon liveness...

Check: Liveness for "ntpd"
Node Name Running?
------------------------------------ ------------------------
node2 yes
node1 yes
Result: Liveness check passed for "ntpd"
Check for NTP daemon or service alive passed on all nodes

Checking NTP daemon command line for slewing option "-x"
Check: NTP daemon command line
Node Name Slewing Option Set?
------------------------------------ ------------------------
node2 no
node1 no
Result:
NTP daemon slewing option check failed on some nodes
PRVF-5436 : The NTP daemon running on one or more nodes lacks the slewing option "-x"
Result: Clock synchronization check using Network Time Protocol(NTP) failed

Checking Core file name pattern consistency...
Core file name pattern consistency check passed.

Checking to make sure user "grid" is not in "root" group
Node Name Status Comment
------------ ------------------------ ------------------------
node2 passed does not exist
node1 passed does not exist
Result: User "grid" is not part of "root" group. Check passed

Check default user file creation mask
Node Name Available Required Comment
------------ ------------------------ ------------------------ ----------
node2 0022 0022 passed
node1 0022 0022 passed
Result: Default user file creation mask check passed
Checking consistency of file "/etc/resolv.conf" across nodes

Checking the file "/etc/resolv.conf" to make sure only one of domain and search entries is defined
File "/etc/resolv.conf" does not have both domain and search entries defined
Checking if domain entry in file "/etc/resolv.conf" is consistent across the nodes...
domain entry in file "/etc/resolv.conf" is consistent across nodes
Checking if search entry in file "/etc/resolv.conf" is consistent across the nodes...
search entry in file "/etc/resolv.conf" is consistent across nodes
Checking file "/etc/resolv.conf" to make sure that only one search entry is defined
All nodes have one search entry defined in file "/etc/resolv.conf"
Checking all nodes to make sure that search entry is "localdomain" as found on node "node2"
All nodes of the cluster have same value for 'search'
Checking DNS response time for an unreachable node
Node Name Status
------------------------------------ ------------------------
node2 failed
node1 failed
PRVF-5636 : The DNS response time for an unreachable node exceeded "15000" ms on following nodes: node2,node1

File "/etc/resolv.conf" is not consistent across nodes

UDev attributes check for OCR locations started...
Result: UDev attributes check passed for OCR locations

UDev attributes check for Voting Disk locations started...
Result: UDev attributes check passed for Voting Disk locations

Check: Time zone consistency
Result: Time zone consistency check passed
Checking VIP configuration.
Checking VIP Subnet configuration.
Check for VIP Subnet configuration passed.
Checking VIP reachability
Check for VIP reachability passed.

Checking Oracle Cluster Voting Disk configuration...

ASM Running check passed. ASM is running on all specified nodes

Oracle Cluster Voting Disk configuration check passed
Clusterware version consistency passed

Starting check for Reverse path filter setting ...
Reverse path filter setting is correct for all private interconnect network interfaces on node "node2.localdomain".
Reverse path filter setting is correct for all private interconnect network interfaces on node "node1.localdomain".

Check for Reverse path filter setting passed

Pre-check for cluster services setup was unsuccessful on all the nodes.

========================= END: Runcluvfy output =================

check for failed parameters in the log file :

[root@node1 logs]# cat runcluvfy.out |grep -i failed

Check: Swap space

Node Name Available Required Status
------------ ------------------------ ------------------------ ----------
node2 1.7578GB (1843196.0KB) 2.9398GB (3082572.0KB) failed
node1 1.7578GB (1843196.0KB) 2.9398GB (3082572.0KB) failed
Result: Swap space check failed
NTP daemon slewing option check failed on some nodes
Result: Clock synchronization check using Network Time Protocol(NTP) failed
node2 failed
node1 failed

In our case above failed parameters can be ignored as this is only a test server demo installation and swap and NTP failure can be ignored
-----------------------

Unset ORALCE env variables and start the runInstaller or you can use silent installation option . For silent upgrade response file is provided at the end of this article. I am using the GUI runInstaller in this demo.

unset ORACLE_HOME
unset ORACLE_BASE
unset ORACLE_SID

Now start runInstaller GUI using the grid user :

Solution to the above issue:

Password connectivity Issue: 11.2.0.4 runInstaller: [INS-06006] Passwordless SSH connectivity not set up between the following nodes(s) (Doc ID 1597212.1)

Solution : Check hostname update node list
set up ssh connectivity using sshsetup script

[grid@node1 ~]$ /data01/app/11.2.0/grid_11203/oui/bin/runInstaller -updateNodelist ORACLE_HOME=/data01/app/11.2.0/grid_11203 "CLUSTER_NODES={node1,node2}" CRS=true Starting Oracle Universal Installer... Checking swap space: must be greater than 500 MB. Actual 1328 MB Passed The inventory pointer is located at /etc/oraInst.loc The inventory is located at /data01/app/oraInventory 'UpdateNodeList' was successful.

[grid@node1 ~]$

[grid@node1 sshsetup]$ sh sshUserSetup.sh -user grid -hosts node2
The output of this script is also logged into /tmp/sshUserSetup_2015-01-10-14-31-22.log
Hosts are node2
user is grid
Platform:- Linux
Checking if the remote hosts are reachable
PING node2.localdomain (192.168.56.72) 56(84) bytes of data.
64 bytes from node2.localdomain (192.168.56.72): icmp_seq=1 ttl=64 time=0.250 ms
64 bytes from node2.localdomain (192.168.56.72): icmp_seq=2 ttl=64 time=0.393 ms
64 bytes from node2.localdomain (192.168.56.72): icmp_seq=3 ttl=64 time=0.357 ms
64 bytes from node2.localdomain (192.168.56.72): icmp_seq=4 ttl=64 time=0.435 ms
64 bytes from node2.localdomain (192.168.56.72): icmp_seq=5 ttl=64 time=0.417 ms

--- node2.localdomain ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 4002ms
rtt min/avg/max/mdev = 0.250/0.370/0.435/0.067 ms
Remote host reachability check succeeded.
The following hosts are reachable: node2.
The following hosts are not reachable: .
All hosts are reachable. Proceeding further...
firsthost node2
numhosts 1
The script will setup SSH connectivity from the host node1.localdomain to all
the remote hosts. After the script is executed, the user can use SSH to run
commands on the remote hosts or copy files between this host node1.localdomain
and the remote hosts without being prompted for passwords or confirmations.

NOTE 1:
As part of the setup procedure, this script will use ssh and scp to copy
files between the local host and the remote hosts. Since the script does not
store passwords, you may be prompted for the passwords during the execution of
the script whenever ssh or scp is invoked.

NOTE 2:
AS PER SSH REQUIREMENTS, THIS SCRIPT WILL SECURE THE USER HOME DIRECTORY
AND THE .ssh DIRECTORY BY REVOKING GROUP AND WORLD WRITE PRIVILEDGES TO THESE
directories.

Do you want to continue and let the script make the above mentioned changes (yes/no)?
yes

The user chose yes
Please specify if you want to specify a passphrase for the private key this script will create for the local host. Passphrase is used to encrypt the private key and makes SSH much more secure. Type 'yes' or 'no' and then press enter. In case you press 'yes', you would need to enter the passphrase whenever the script executes ssh or scp.
The estimated number of times the user would be prompted for a passphrase is 2. In addition, if the private-public files are also newly created, the user would have to specify the passphrase on one additional occasion.
Enter 'yes' or 'no'.
no

The user chose no
Creating .ssh directory on local host, if not present already
Creating authorized_keys file on local host
Changing permissions on authorized_keys to 644 on local host
Creating known_hosts file on local host
Changing permissions on known_hosts to 644 on local host
Creating config file on local host
If a config file exists already at /home/grid/.ssh/config, it would be backed up to /home/grid/.ssh/config.backup.
Removing old private/public keys on local host
Running SSH keygen on local host with empty passphrase
Generating public/private rsa key pair.
Your identification has been saved in /home/grid/.ssh/id_rsa.
Your public key has been saved in /home/grid/.ssh/id_rsa.pub.
The key fingerprint is:
43:5f:93:1e:88:96:fd:8f:28:a4:ce:47:6e:2a:6e:a6 grid@node1.localdomain
The key's randomart image is:
+--[ RSA 1024]----+
| |
| + . . |
| = o = |
| o . + o |
| S . o |
| o.. . o |
| .o. . . . |
| +o +. |
| E=..++ |
+-----------------+
Creating .ssh directory and setting permissions on remote host node2
THE SCRIPT WOULD ALSO BE REVOKING WRITE PERMISSIONS FOR group AND others ON THE HOME DIRECTORY FOR grid. THIS IS AN SSH REQUIREMENT.
The script would create ~grid/.ssh/config file on remote host node2. If a config file exists already at ~grid/.ssh/config, it would be backed up to ~grid/.ssh/config.backup.
The user may be prompted for a password here since the script would be running SSH on host node2.
Warning: Permanently added 'node2,192.168.56.72' (RSA) to the list of known hosts.
grid@node2's password:
Done with creating .ssh directory and setting permissions on remote host node2.
Copying local host public key to the remote host node2
The user may be prompted for a password or passphrase here since the script would be using SCP for host node2.
grid@node2's password:
Done copying local host public key to the remote host node2
SSH setup is complete.

------------------------------------------------------------------------
Verifying SSH setup
===================
The script will now run the date command on the remote nodes using ssh
to verify if ssh is setup correctly. IF THE SETUP IS CORRECTLY SETUP,
THERE SHOULD BE NO OUTPUT OTHER THAN THE DATE AND SSH SHOULD NOT ASK FOR
PASSWORDS. If you see any output other than date or are prompted for the
password, ssh is not setup correctly and you will need to resolve the
issue and set up ssh again.
The possible causes for failure could be:
1. The server settings in /etc/ssh/sshd_config file do not allow ssh
for user grid.
2. The server may have disabled public key based authentication.
3. The client public key on the server may be outdated.
4. ~grid or ~grid/.ssh on the remote host may not be owned by grid.
5. User may not have passed -shared option for shared remote users or
may be passing the -shared option for non-shared remote users.
6. If there is output in addition to the date, but no password is asked,
it may be a security alert shown as part of company policy. Append the
additional text to the <OMS HOME>/sysman/prov/resources/ignoreMessages.txt file.
------------------------------------------------------------------------
--node2:--
Running /usr/bin/ssh -x -l grid node2 date to verify SSH connectivity has been setup from local host to node2.
IF YOU SEE ANY OTHER OUTPUT BESIDES THE OUTPUT OF THE DATE COMMAND OR IF YOU ARE PROMPTED FOR A PASSWORD HERE, IT MEANS SSH SETUP HAS NOT BEEN SUCCESSFUL. Please note that being prompted for a passphrase may be OK but being prompted for a password is ERROR.
Sat Jan 10 14:31:59 IST 2015
------------------------------------------------------------------------
SSH verification complete.

[grid@node1 sshsetup]$ ssh node2 date
Sat Jan 10 14:32:03 IST 2015
[grid@node1 sshsetup]$ ssh node2
Last login: Fri Jan 9 17:19:25 2015 from node1.localdomain
[grid@node2 ~]$ ssh node1 date
Sat Jan 10 14:44:05 IST 2015

Now dont setup using runinstllaler . Just click next ->.

rootupgrade.sh will stop the cluster on node1 and will replace the old clusterware entries with new clusterware entries in cluster upstart. after node one .run rootupgrade.sh on node2 .

[root@node1 ~]# sh /data01/app/11.2.0/grid_11204/rootupgrade.sh
Performing root user operation for Oracle 11g

The following environment variables are set as:
ORACLE_OWNER= grid
ORACLE_HOME= /data01/app/11.2.0/grid_11204

Enter the full pathname of the local bin directory: [/usr/local/bin]:
The contents of "dbhome" have not changed. No need to overwrite.
The contents of "oraenv" have not changed. No need to overwrite.
The contents of "coraenv" have not changed. No need to overwrite.

Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Using configuration parameter file: /data01/app/11.2.0/grid_11204/crs/install/crsconfig_params
Creating trace directory
User ignored Prerequisites during installation
Installing Trace File Analyzer
ASM upgrade has started on first node.

CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'node1'
CRS-2673: Attempting to stop 'ora.crsd' on 'node1'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on 'node1'
CRS-2673: Attempting to stop 'ora.LISTENER.lsnr' on 'node1'
CRS-2673: Attempting to stop 'ora.cvu' on 'node1'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN3.lsnr' on 'node1'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN2.lsnr' on 'node1'
CRS-2673: Attempting to stop 'ora.FRA.dg' on 'node1'
CRS-2673: Attempting to stop 'ora.racdb.db' on 'node1'
CRS-2677: Stop of 'ora.cvu' on 'node1' succeeded
CRS-2672: Attempting to start 'ora.cvu' on 'node2'
CRS-2677: Stop of 'ora.LISTENER.lsnr' on 'node1' succeeded
CRS-2673: Attempting to stop 'ora.node1.vip' on 'node1'
CRS-2677: Stop of 'ora.LISTENER_SCAN3.lsnr' on 'node1' succeeded
CRS-2673: Attempting to stop 'ora.scan3.vip' on 'node1'
CRS-2677: Stop of 'ora.LISTENER_SCAN2.lsnr' on 'node1' succeeded
CRS-2673: Attempting to stop 'ora.scan2.vip' on 'node1'
CRS-2677: Stop of 'ora.scan2.vip' on 'node1' succeeded
CRS-2672: Attempting to start 'ora.scan2.vip' on 'node2'
CRS-2677: Stop of 'ora.node1.vip' on 'node1' succeeded
CRS-2672: Attempting to start 'ora.node1.vip' on 'node2'
CRS-2677: Stop of 'ora.scan3.vip' on 'node1' succeeded
CRS-2672: Attempting to start 'ora.scan3.vip' on 'node2'
CRS-2676: Start of 'ora.cvu' on 'node2' succeeded
CRS-2677: Stop of 'ora.racdb.db' on 'node1' succeeded
CRS-2673: Attempting to stop 'ora.DATA.dg' on 'node1'
CRS-2673: Attempting to stop 'ora.USER_DATA.dg' on 'node1'
CRS-2677: Stop of 'ora.FRA.dg' on 'node1' succeeded
CRS-2677: Stop of 'ora.USER_DATA.dg' on 'node1' succeeded
CRS-2676: Start of 'ora.scan3.vip' on 'node2' succeeded
CRS-2676: Start of 'ora.scan2.vip' on 'node2' succeeded
CRS-2676: Start of 'ora.node1.vip' on 'node2' succeeded
CRS-2672: Attempting to start 'ora.LISTENER_SCAN3.lsnr' on 'node2'
CRS-2672: Attempting to start 'ora.LISTENER_SCAN2.lsnr' on 'node2'
CRS-2676: Start of 'ora.LISTENER_SCAN3.lsnr' on 'node2' succeeded
CRS-2676: Start of 'ora.LISTENER_SCAN2.lsnr' on 'node2' succeeded
CRS-2677: Stop of 'ora.DATA.dg' on 'node1' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'node1'
CRS-2677: Stop of 'ora.asm' on 'node1' succeeded
CRS-2673: Attempting to stop 'ora.ons' on 'node1'
CRS-2677: Stop of 'ora.ons' on 'node1' succeeded
CRS-2673: Attempting to stop 'ora.net1.network' on 'node1'
CRS-2677: Stop of 'ora.net1.network' on 'node1' succeeded
CRS-2792: Shutdown of Cluster Ready Services-managed resources on 'node1' has completed
CRS-2677: Stop of 'ora.crsd' on 'node1' succeeded
CRS-2673: Attempting to stop 'ora.crf' on 'node1'
CRS-2673: Attempting to stop 'ora.ctssd' on 'node1'
CRS-2673: Attempting to stop 'ora.evmd' on 'node1'
CRS-2673: Attempting to stop 'ora.asm' on 'node1'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'node1'
CRS-2677: Stop of 'ora.crf' on 'node1' succeeded
CRS-2677: Stop of 'ora.evmd' on 'node1' succeeded
CRS-2677: Stop of 'ora.mdnsd' on 'node1' succeeded
CRS-2677: Stop of 'ora.ctssd' on 'node1' succeeded
CRS-2677: Stop of 'ora.asm' on 'node1' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'node1'
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'node1' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'node1'
CRS-2677: Stop of 'ora.cssd' on 'node1' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'node1'
CRS-2677: Stop of 'ora.gipcd' on 'node1' succeeded
CRS-2673: Attempting to stop 'ora.gpnpd' on 'node1'
CRS-2677: Stop of 'ora.gpnpd' on 'node1' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'node1' has completed
CRS-4133: Oracle High Availability Services has been stopped.
OLR initialization - successful
Replacing Clusterware entries in upstart
clscfg: EXISTING configuration version 5 detected.
clscfg: version 5 is 11g Release 2.
Successfully accumulated necessary OCR keys.
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
Configure Oracle Grid Infrastructure for a Cluster ... succeeded

[grid@node2 ~]$ crsctl stat res -t
--------------------------------------------------------------------------------
NAME TARGET STATE SERVER STATE_DETAILS
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.DATA.dg
ONLINE ONLINE node1
ONLINE OFFLINE node2
ora.FRA.dg
ONLINE ONLINE node1
ONLINE OFFLINE node2
ora.LISTENER.lsnr
ONLINE ONLINE node1
ONLINE ONLINE node2
ora.USER_DATA.dg
ONLINE ONLINE node1
ONLINE OFFLINE node2
ora.asm
ONLINE ONLINE node1 Started
ONLINE ONLINE node2 Started
ora.gsd
OFFLINE OFFLINE node1
OFFLINE OFFLINE node2
ora.net1.network
ONLINE ONLINE node1
ONLINE ONLINE node2
ora.ons
ONLINE ONLINE node1
ONLINE ONLINE node2
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
1 ONLINE ONLINE node1
ora.LISTENER_SCAN2.lsnr
1 ONLINE ONLINE node2
ora.LISTENER_SCAN3.lsnr
1 ONLINE ONLINE node2
ora.cvu
1 ONLINE ONLINE node2
ora.node1.vip
1 ONLINE ONLINE node1
ora.node2.vip
1 ONLINE ONLINE node2
ora.oc4j
1 OFFLINE ONLINE node2
ora.racdb.db
1 ONLINE ONLINE node1 Open
2 ONLINE ONLINE node2 Open
ora.scan1.vip
1 ONLINE ONLINE node1
ora.scan2.vip
1 ONLINE ONLINE node2
ora.scan3.vip
1 ONLINE ONLINE node2

[grid@node1 sshsetup]$ crsctl query crs softwareversion node1
Oracle Clusterware version on node [node1] is [11.2.0.4.0]
[grid@node1 sshsetup]$ crsctl query crs softwareversion node2
Oracle Clusterware version on node [node2] is [11.2.0.3.0]

[root@node2 ~]# sh /data01/app/11.2.0/grid_11204/rootupgrade.sh
Performing root user operation for Oracle 11g

The following environment variables are set as:
ORACLE_OWNER= grid
ORACLE_HOME= /data01/app/11.2.0/grid_11204

Enter the full pathname of the local bin directory: [/usr/local/bin]:
The contents of "dbhome" have not changed. No need to overwrite.
The contents of "oraenv" have not changed. No need to overwrite.
The contents of "coraenv" have not changed. No need to overwrite.

Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Using configuration parameter file: /data01/app/11.2.0/grid_11204/crs/install/crsconfig_params
Creating trace directory
User ignored Prerequisites during installation
Installing Trace File Analyzer
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'node2'
CRS-2673: Attempting to stop 'ora.crsd' on 'node2'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on 'node2'
CRS-2673: Attempting to stop 'ora.LISTENER.lsnr' on 'node2'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN2.lsnr' on 'node2'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN3.lsnr' on 'node2'
CRS-2673: Attempting to stop 'ora.oc4j' on 'node2'
CRS-2673: Attempting to stop 'ora.cvu' on 'node2'
CRS-2673: Attempting to stop 'ora.racdb.db' on 'node2'
CRS-2677: Stop of 'ora.LISTENER_SCAN3.lsnr' on 'node2' succeeded
CRS-2673: Attempting to stop 'ora.scan3.vip' on 'node2'
CRS-2677: Stop of 'ora.LISTENER_SCAN2.lsnr' on 'node2' succeeded
CRS-2673: Attempting to stop 'ora.scan2.vip' on 'node2'
CRS-2677: Stop of 'ora.LISTENER.lsnr' on 'node2' succeeded
CRS-2673: Attempting to stop 'ora.node2.vip' on 'node2'
CRS-2677: Stop of 'ora.node2.vip' on 'node2' succeeded
CRS-2672: Attempting to start 'ora.node2.vip' on 'node1'
CRS-2677: Stop of 'ora.scan3.vip' on 'node2' succeeded
CRS-2672: Attempting to start 'ora.scan3.vip' on 'node1'
CRS-2677: Stop of 'ora.scan2.vip' on 'node2' succeeded
CRS-2672: Attempting to start 'ora.scan2.vip' on 'node1'
CRS-2677: Stop of 'ora.racdb.db' on 'node2' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'node2'
CRS-2676: Start of 'ora.scan2.vip' on 'node1' succeeded
CRS-2676: Start of 'ora.scan3.vip' on 'node1' succeeded
CRS-2676: Start of 'ora.node2.vip' on 'node1' succeeded
CRS-2677: Stop of 'ora.asm' on 'node2' succeeded
CRS-2672: Attempting to start 'ora.LISTENER_SCAN2.lsnr' on 'node1'
CRS-2672: Attempting to start 'ora.LISTENER_SCAN3.lsnr' on 'node1'
CRS-2677: Stop of 'ora.cvu' on 'node2' succeeded
CRS-2672: Attempting to start 'ora.cvu' on 'node1'
CRS-2675: Stop of 'ora.oc4j' on 'node2' failed
CRS-2679: Attempting to clean 'ora.oc4j' on 'node2'
CRS-2681: Clean of 'ora.oc4j' on 'node2' succeeded
CRS-2676: Start of 'ora.cvu' on 'node1' succeeded
CRS-2676: Start of 'ora.LISTENER_SCAN3.lsnr' on 'node1' succeeded
CRS-2676: Start of 'ora.LISTENER_SCAN2.lsnr' on 'node1' succeeded
CRS-2673: Attempting to stop 'ora.ons' on 'node2'
CRS-2677: Stop of 'ora.ons' on 'node2' succeeded
CRS-2673: Attempting to stop 'ora.net1.network' on 'node2'
CRS-2677: Stop of 'ora.net1.network' on 'node2' succeeded
CRS-2792: Shutdown of Cluster Ready Services-managed resources on 'node2' has completed
CRS-2677: Stop of 'ora.crsd' on 'node2' succeeded
CRS-2673: Attempting to stop 'ora.mdnsd' on 'node2'
CRS-2673: Attempting to stop 'ora.crf' on 'node2'
CRS-2673: Attempting to stop 'ora.ctssd' on 'node2'
CRS-2673: Attempting to stop 'ora.evmd' on 'node2'
CRS-2673: Attempting to stop 'ora.asm' on 'node2'
CRS-2677: Stop of 'ora.crf' on 'node2' succeeded
CRS-2677: Stop of 'ora.evmd' on 'node2' succeeded
CRS-2677: Stop of 'ora.ctssd' on 'node2' succeeded
CRS-2677: Stop of 'ora.mdnsd' on 'node2' succeeded
CRS-2677: Stop of 'ora.asm' on 'node2' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'node2'
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'node2' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'node2'
CRS-2677: Stop of 'ora.cssd' on 'node2' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'node2'
CRS-2677: Stop of 'ora.gipcd' on 'node2' succeeded
CRS-2673: Attempting to stop 'ora.gpnpd' on 'node2'
CRS-2677: Stop of 'ora.gpnpd' on 'node2' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'node2' has completed
CRS-4133: Oracle High Availability Services has been stopped.
OLR initialization - successful
Replacing Clusterware entries in upstart
clscfg: EXISTING configuration version 5 detected.
clscfg: version 5 is 11g Release 2.
Successfully accumulated necessary OCR keys.
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
Started to upgrade the Oracle Clusterware. This operation may take a few minutes.
Started to upgrade the CSS.
Started to upgrade the CRS.
The CRS was successfully upgraded.
Successfully upgraded the Oracle Clusterware.
Oracle Clusterware operating version was successfully set to 11.2.0.4.0

ASM upgrade has finished on last node.
Configure Oracle Grid Infrastructure for a Cluster ... succeeded

[grid@node2 ~]$ crsctl stat res -t

--------------------------------------------------------------------------------

NAME TARGET STATE SERVER STATE_DETAILS

--------------------------------------------------------------------------------

Local Resources

--------------------------------------------------------------------------------

ora.DATA.dg

ONLINE ONLINE node1

ONLINE ONLINE node2

ora.FRA.dg

ONLINE ONLINE node1

ONLINE ONLINE node2

ora.LISTENER.lsnr

ONLINE ONLINE node1

ONLINE ONLINE node2

ora.USER_DATA.dg

ONLINE ONLINE node1

ONLINE ONLINE node2

ora.asm

ONLINE ONLINE node1 Started

ONLINE ONLINE node2 Started

ora.gsd

OFFLINE OFFLINE node1

OFFLINE OFFLINE node2

ora.net1.network

ONLINE ONLINE node1

ONLINE ONLINE node2

ora.ons

ONLINE ONLINE node1

ONLINE ONLINE node2

ora.registry.acfs

ONLINE ONLINE node1

ONLINE ONLINE node2

--------------------------------------------------------------------------------

Cluster Resources

--------------------------------------------------------------------------------

ora.LISTENER_SCAN1.lsnr

1 ONLINE ONLINE node2

ora.LISTENER_SCAN2.lsnr

1 ONLINE ONLINE node1

ora.LISTENER_SCAN3.lsnr

1 ONLINE ONLINE node1

ora.cvu

1 ONLINE ONLINE node1

ora.node1.vip

1 ONLINE ONLINE node1

ora.node2.vip

1 ONLINE ONLINE node2

ora.oc4j

1 ONLINE ONLINE node2

ora.racdb.db

1 ONLINE ONLINE node1 Open

2 ONLINE ONLINE node2 Open

ora.scan1.vip

1 ONLINE ONLINE node2

ora.scan2.vip

1 ONLINE ONLINE node1

ora.scan3.vip

1 ONLINE ONLINE node1

Check the error for failed status and if related to scan/NTP it can be ignored :

INFO: PRVF-4657 : Name resolution setup check for "node-scan.localdomain" (IP address: 192.168.56.91) failed

INFO: ERROR:

INFO: PRVF-4657 : Name resolution setup check for "node-scan.localdomain" (IP address: 192.168.56.92) failed

INFO: ERROR:

INFO: PRVF-4657 : Name resolution setup check for "node-scan.localdomain" (IP address: 192.168.56.93) failed

INFO: ERROR:

INFO: PRVF-4664 : Found inconsistent name resolution entries for SCAN name "node-scan.localdomain"

INFO: Verification of SCAN VIP and Listener setup failed

[grid@node1 grid]$ ps -ef|grep /data01 | grep -v java | grep -v grep

root 6552 1 1 18:26 ? 00:01:11 /data01/app/11.2.0/grid_11204/bin/ohasd.bin reboot

grid 19211 1 0 18:35 ? 00:00:16 /data01/app/11.2.0/grid_11204/bin/oraagent.bin

grid 19226 1 0 18:36 ? 00:00:00 /data01/app/11.2.0/grid_11204/bin/mdnsd.bin

grid 19240 1 0 18:36 ? 00:00:03 /data01/app/11.2.0/grid_11204/bin/gpnpd.bin

root 19288 1 0 18:36 ? 00:00:08 /data01/app/11.2.0/grid_11204/bin/cssdmonitor

grid 19290 1 0 18:36 ? 00:00:31 /data01/app/11.2.0/grid_11204/bin/gipcd.bin

root 19340 1 0 18:36 ? 00:00:07 /data01/app/11.2.0/grid_11204/bin/cssdagent

grid 19382 1 0 18:36 ? 00:00:42 /data01/app/11.2.0/grid_11204/bin/ocssd.bin

root 19460 1 0 18:37 ? 00:00:17 /data01/app/11.2.0/grid_11204/bin/orarootagent.bin

root 19471 1 0 18:37 ? 00:00:54 /data01/app/11.2.0/grid_11204/bin/octssd.bin

root 19498 1 1 18:37 ? 00:02:01 /data01/app/11.2.0/grid_11204/bin/osysmond.bin

root 19516 1 2 18:37 ? 00:02:29 /data01/app/11.2.0/grid_11204/bin/ologgerd -M -d /data01/app/11.2.0/grid_11204/crf/db/node1

root 20388 1 1 18:43 ? 00:01:36 /data01/app/11.2.0/grid_11204/bin/crsd.bin reboot

grid 20459 1 0 18:43 ? 00:00:54 /data01/app/11.2.0/grid_11204/bin/evmd.bin

grid 20542 20459 0 18:43 ? 00:00:00 /data01/app/11.2.0/grid_11204/bin/evmlogger.bin -o /data01/app/11.2.0/grid_11204/evm/log/evmlogger.info -l /data01/app/11.2.0/grid_11204/evm/log/evmlogger.log

grid 20590 1 0 18:44 ? 00:00:13 /data01/app/11.2.0/grid_11204/bin/oraagent.bin

root 20598 1 0 18:44 ? 00:00:30 /data01/app/11.2.0/grid_11204/bin/orarootagent.bin

grid 20799 1 0 18:44 ? 00:00:00 /data01/app/11.2.0/grid_11204/opmn/bin/ons -d

grid 20801 20799 0 18:44 ? 00:00:01 /data01/app/11.2.0/grid_11204/opmn/bin/ons -d

grid 20845 1 0 18:44 ? 00:00:00 /data01/app/11.2.0/grid_11204/bin/tnslsnr LISTENER -inherit

oracle 20968 1 0 18:45 ? 00:00:19 /data01/app/11.2.0/grid_11204/bin/oraagent.bin

grid 26692 1 0 19:32 ? 00:00:00 /data01/app/11.2.0/grid_11204/bin/tnslsnr LISTENER_SCAN3 -inherit

grid 26693 1 0 19:32 ? 00:00:00 /data01/app/11.2.0/grid_11204/bin/tnslsnr LISTENER_SCAN2 -inherit

grid 26837 1 0 19:34 ? 00:00:00 /data01/app/11.2.0/grid_11204/bin/scriptagent.bin

[grid@node2 ~]$ crsctl query crs activeversion

Oracle Clusterware active version on the cluster is [11.2.0.4.0]

[grid@node2 ~]$ crsctl query crs softwareversion

Oracle Clusterware version on node [node2] is [11.2.0.4.0]

[grid@node2 ~]$ crsctl query crs softwareversion node1

Oracle Clusterware version on node [node1] is [11.2.0.4.0]

[grid@node2 ~]$ crsctl query crs activeversion node1

[oracle@node1 ~]$ srvctl status database -d racdb

Instance racdb1 is running on node node1

Instance racdb2 is running on node node2

Now make changes in bash profile for grid user and enjoy :)

Response file for silent installation:

[grid@node1 grid]$ ./runInstaller -silent -showProgress -ignoreSysPrereqs -ignorePrereq

-responseFile /home/grid/grid_upgrade_to_11204.rsp

[grid@node1 ~]$ cat grid_upgrade_to_11204.rsp

###############################################################################

## ##

## Specify values for the variables listed below to customize ##

## your installation. ##

## ##

## Each variable is associated with a comment. The comment ##

## can help to populate the variables with the appropriate ##

## values. ##

## ##

## IMPORTANT NOTE: This file contains plain text passwords and ##

## should be secured to have read permission only by oracle user ##

## or db administrator who owns this installation. ##

## ##

###############################################################################

## ##

## Instructions to fill this response file ##

## To install and configure 'Grid Infrastructure for Cluster' ##

## - Fill out sections A,B,C,D,E,F and G ##

## - Fill out section G if OCR and voting disk should be placed on ASM ##

## ##

## To install and configure 'Grid Infrastructure for Standalone server' ##

## - Fill out sections A,B and G ##

## ##

## To install software for 'Grid Infrastructure' ##

## - Fill out sections A,B and C ##

## ##

## To upgrade clusterware and/or Automatic storage management of earlier ##

## releases ##

## - Fill out sections A,B,C,D and H ##

## ##

###############################################################################

#------------------------------------------------------------------------------

# Do not change the following system generated value.

#------------------------------------------------------------------------------

oracle.install.responseFileVersion=/oracle/install/rspfmt_crsinstall_response_schema_v11_2_0

###############################################################################

# #

# SECTION A - BASIC #

# #

###############################################################################

#-------------------------------------------------------------------------------

# Specify the hostname of the system as set during the install. It can be used

# to force the installation to use an alternative hostname rather than using the

# first hostname found on the system. (e.g., for systems with multiple hostnames

# and network interfaces)

#-------------------------------------------------------------------------------

ORACLE_HOSTNAME=node1.localdomain

#-------------------------------------------------------------------------------

# Specify the location which holds the inventory files.

# This is an optional parameter if installing on

# Windows based Operating System.

#-------------------------------------------------------------------------------

INVENTORY_LOCATION=/data01/app/oraInventory

#-------------------------------------------------------------------------------

# Specify the languages in which the components will be installed.

# en : English ja : Japanese

# fr : French ko : Korean

# ar : Arabic es : Latin American Spanish

# bn : Bengali lv : Latvian

# pt_BR: Brazilian Portuguese lt : Lithuanian

# bg : Bulgarian ms : Malay

# fr_CA: Canadian French es_MX: Mexican Spanish

# ca : Catalan no : Norwegian

# hr : Croatian pl : Polish

# cs : Czech pt : Portuguese

# da : Danish ro : Romanian

# nl : Dutch ru : Russian

# ar_EG: Egyptian zh_CN: Simplified Chinese

# en_GB: English (Great Britain) sk : Slovak

# et : Estonian sl : Slovenian

# fi : Finnish es_ES: Spanish

# de : German sv : Swedish

# el : Greek th : Thai

# iw : Hebrew zh_TW: Traditional Chinese

# hu : Hungarian tr : Turkish

# is : Icelandic uk : Ukrainian

# in : Indonesian vi : Vietnamese

# it : Italian

# all_langs : All languages

# Specify value as the following to select any of the languages.

# Example : SELECTED_LANGUAGES=en,fr,ja

# Specify value as the following to select all the languages.

# Example : SELECTED_LANGUAGES=all_langs

#-------------------------------------------------------------------------------

SELECTED_LANGUAGES=en

#-------------------------------------------------------------------------------

# Specify the installation option.

# Allowed values: CRS_CONFIG or HA_CONFIG or UPGRADE or CRS_SWONLY

# - CRS_CONFIG : To configure Grid Infrastructure for cluster

# - HA_CONFIG : To configure Grid Infrastructure for stand alone server

# - UPGRADE : To upgrade clusterware software of earlier release

# - CRS_SWONLY : To install clusterware files only (can be configured for cluster

# or stand alone server later)

#-------------------------------------------------------------------------------

oracle.install.option=UPGRADE

#-------------------------------------------------------------------------------

# Specify the complete path of the Oracle Base.

#-------------------------------------------------------------------------------

ORACLE_BASE=/data01/app/grid

#-------------------------------------------------------------------------------

# Specify the complete path of the Oracle Home.

#-------------------------------------------------------------------------------

ORACLE_HOME=/data01/app/11.2.0/grid_11204

################################################################################

# #

# SECTION B - GROUPS #

# #

# The following three groups need to be assigned for all GI installations. #

# OSDBA and OSOPER can be the same or different. OSASM must be different #

# than the other two. #

# The value to be specified for OSDBA, OSOPER and OSASM group is only for #

# Unix based Operating System. #

# #

################################################################################

#-------------------------------------------------------------------------------

# The DBA_GROUP is the OS group which is to be granted OSDBA privileges.

#-------------------------------------------------------------------------------

oracle.install.asm.OSDBA=asmdba

#-------------------------------------------------------------------------------

# The OPER_GROUP is the OS group which is to be granted OSOPER privileges.

# The value to be specified for OSOPER group is optional.

#-------------------------------------------------------------------------------

oracle.install.asm.OSOPER=asmoper

#-------------------------------------------------------------------------------

# The OSASM_GROUP is the OS group which is to be granted OSASM privileges. This

# must be different than the previous two.

#-------------------------------------------------------------------------------

oracle.install.asm.OSASM=asmdba

################################################################################

# #

# SECTION C - SCAN #

# #

################################################################################

#-------------------------------------------------------------------------------

# Specify a name for SCAN

#-------------------------------------------------------------------------------

oracle.install.crs.config.gpnp.scanName=

#-------------------------------------------------------------------------------

# Specify a unused port number for SCAN service

#-------------------------------------------------------------------------------

oracle.install.crs.config.gpnp.scanPort=

################################################################################

# #

# SECTION D - CLUSTER & GNS #

# #

################################################################################

#-------------------------------------------------------------------------------

# Specify a name for the Cluster you are creating.

# The maximum length allowed for clustername is 15 characters. The name can be

# any combination of lower and uppercase alphabets (A - Z), (0 - 9), hyphen(-)

# and underscore(_).

#-------------------------------------------------------------------------------

oracle.install.crs.config.clusterName=node-cluster

#-------------------------------------------------------------------------------

# Specify 'true' if you would like to configure Grid Naming Service(GNS), else

# specify 'false'

#-------------------------------------------------------------------------------

oracle.install.crs.config.gpnp.configureGNS=false

#-------------------------------------------------------------------------------

# Applicable only if you choose to configure GNS

# Specify the GNS subdomain and an unused virtual hostname for GNS service

# Additionally you may also specify if VIPs have to be autoconfigured

# Value for oracle.install.crs.config.autoConfigureClusterNodeVIP should be true

# if GNS is being configured(oracle.install.crs.config.gpnp.configureGNS), false

# otherwise.

#-------------------------------------------------------------------------------

oracle.install.crs.config.gpnp.gnsSubDomain=

oracle.install.crs.config.gpnp.gnsVIPAddress=

oracle.install.crs.config.autoConfigureClusterNodeVIP=false

#-------------------------------------------------------------------------------

# Specify a list of public node names, and virtual hostnames that have to be

# part of the cluster.

# The list should a comma-separated list of nodes. Each entry in the list

# should be a colon-separated string that contains 2 fields.

# The fields should be ordered as follows:

# 1. The first field is for public node name.

# 2. The second field is for virtual host name

# (specify as AUTO if you have chosen 'auto configure for VIP'

# i.e. autoConfigureClusterNodeVIP=true)

# Example: oracle.install.crs.config.clusterNodes=node1:node1-vip,node2:node2-vip

#-------------------------------------------------------------------------------

oracle.install.crs.config.clusterNodes=node1:,node2:

#-------------------------------------------------------------------------------

# The value should be a comma separated strings where each string is as shown below

# InterfaceName:SubnetAddress:InterfaceType

# where InterfaceType can be either "1", "2", or "3"

# (1 indicates public, 2 indicates private, and 3 indicates the interface is not used)

# For example: eth0:140.87.24.0:1,eth1:10.2.1.0:2,eth2:140.87.52.0:3

#-------------------------------------------------------------------------------

oracle.install.crs.config.networkInterfaceList=

################################################################################

# #

# SECTION E - STORAGE #

# #

################################################################################

#-------------------------------------------------------------------------------

# Specify the type of storage to use for Oracle Cluster Registry(OCR) and Voting

# Disks files

# - ASM_STORAGE

# - FILE_SYSTEM_STORAGE

#-------------------------------------------------------------------------------

oracle.install.crs.config.storageOption=

#-------------------------------------------------------------------------------

# THIS PROPERTY NEEDS TO BE FILLED ONLY IN CASE OF WINDOWS INSTALL.

# Specify a comma separated list of strings where each string is as shown below:

# Disk Number:Partition Number:Drive Letter:Format Option

# The Disk Number and Partition Number should refer to the location which has to

# be formatted. The Drive Letter should refer to the drive letter that has to be

# assigned. "Format Option" can be either of the following -

# - SOFTWARE : Format to place software binaries.

# - DATA : Format to place the OCR/VDSK files.

# For example: 1:2:P:DATA,1:3:Q:SOFTWARE,1:4:R:DATA,1:5:S:DATA

#-------------------------------------------------------------------------------

oracle.install.crs.config.sharedFileSystemStorage.diskDriveMapping=

#-------------------------------------------------------------------------------

# These properties are applicable only if FILE_SYSTEM_STORAGE is chosen for

# storing OCR and voting disk

# Specify the location(s) and redundancy for OCR and voting disks

# Multiple locations can be specified, separated by commas.

# In case of windows, mention the drive location that is specified to be

# formatted for DATA in the above property.

# Redundancy can be one of these:

# EXTERNAL - one(1) location should be specified for OCR and voting disk

# NORMAL - three(3) locations should be specified for OCR and voting disk

# Example:

# For Unix based Operating System:

# oracle.install.crs.config.sharedFileSystemStorage.votingDiskLocations=/oradbocfs/storage/vdsk1,/oradbocfs/storage/vdsk2,/oradbocfs/storage/vdsk3

# oracle.install.crs.config.sharedFileSystemStorage.ocrLocations=/oradbocfs/storage/ocr1,/oradbocfs/storage/ocr2,/oradbocfs/storage/ocr3

# For Windows based Operating System:

# oracle.install.crs.config.sharedFileSystemStorage.votingDiskLocations=P:\vdsk1,R:\vdsk2,S:\vdsk3

# oracle.install.crs.config.sharedFileSystemStorage.ocrLocations=P:\ocr1,R:\ocr2,S:\ocr3

#-------------------------------------------------------------------------------

oracle.install.crs.config.sharedFileSystemStorage.votingDiskLocations=

oracle.install.crs.config.sharedFileSystemStorage.votingDiskRedundancy=NORMAL

oracle.install.crs.config.sharedFileSystemStorage.ocrLocations=

oracle.install.crs.config.sharedFileSystemStorage.ocrRedundancy=NORMAL

################################################################################

# #

# SECTION F - IPMI #

# #

################################################################################

#-------------------------------------------------------------------------------

# Specify 'true' if you would like to configure Intelligent Power Management interface

# (IPMI), else specify 'false'

#-------------------------------------------------------------------------------

oracle.install.crs.config.useIPMI=false

#-------------------------------------------------------------------------------

# Applicable only if you choose to configure IPMI

# i.e. oracle.install.crs.config.useIPMI=true

# Specify the username and password for using IPMI service

#-------------------------------------------------------------------------------

oracle.install.crs.config.ipmi.bmcUsername=

oracle.install.crs.config.ipmi.bmcPassword=

################################################################################

# #

# SECTION G - ASM #

# #

################################################################################

#-------------------------------------------------------------------------------

# Specify a password for SYSASM user of the ASM instance

#-------------------------------------------------------------------------------

oracle.install.asm.SYSASMPassword=

#-------------------------------------------------------------------------------

# The ASM DiskGroup

# Example: oracle.install.asm.diskGroup.name=data

#-------------------------------------------------------------------------------

oracle.install.asm.diskGroup.name=

#-------------------------------------------------------------------------------

# Redundancy level to be used by ASM.

# It can be one of the following

# - NORMAL

# - HIGH

# - EXTERNAL

# Example: oracle.install.asm.diskGroup.redundancy=NORMAL

#-------------------------------------------------------------------------------

oracle.install.asm.diskGroup.redundancy=

#-------------------------------------------------------------------------------

# Allocation unit size to be used by ASM.

# It can be one of the following values

# - 1

# - 2

# - 4

# - 8

# - 16

# - 32

# - 64

# Example: oracle.install.asm.diskGroup.AUSize=4

# size unit is MB

#-------------------------------------------------------------------------------

oracle.install.asm.diskGroup.AUSize=1

#-------------------------------------------------------------------------------

# List of disks to create a ASM DiskGroup

# Example:

# For Unix based Operating System:

# oracle.install.asm.diskGroup.disks=/oracle/asm/disk1,/oracle/asm/disk2

# For Windows based Operating System:

# oracle.install.asm.diskGroup.disks=\\.\ORCLDISKDATA0,\\.\ORCLDISKDATA1

#-------------------------------------------------------------------------------

oracle.install.asm.diskGroup.disks=

#-------------------------------------------------------------------------------

# The disk discovery string to be used to discover the disks used create a ASM DiskGroup

# Example:

# For Unix based Operating System:

# oracle.install.asm.diskGroup.diskDiscoveryString=/oracle/asm/*

# For Windows based Operating System:

# oracle.install.asm.diskGroup.diskDiscoveryString=\\.\ORCLDISK*

#-------------------------------------------------------------------------------

oracle.install.asm.diskGroup.diskDiscoveryString=

#-------------------------------------------------------------------------------

# oracle.install.asm.monitorPassword=password

#-------------------------------------------------------------------------------

oracle.install.asm.monitorPassword=

################################################################################

# #

# SECTION H - UPGRADE #

# #

################################################################################

#-------------------------------------------------------------------------------

# Specify nodes for Upgrade.

# For upgrade on Windows, installer overrides the value of this parameter to include

# all the nodes of the cluster. However, the stack is upgraded one node at a time.

# Hence, this parameter may be left blank for Windows.

# Example: oracle.install.crs.upgrade.clusterNodes=node1,node2

#-------------------------------------------------------------------------------

oracle.install.crs.upgrade.clusterNodes=node1,node2

#-------------------------------------------------------------------------------

# For RAC-ASM only. oracle.install.asm.upgradeASM=true/false

# Value should be 'true' while upgrading Cluster ASM of version 11gR2(11.2.0.1.0) and above

#-------------------------------------------------------------------------------

oracle.install.asm.upgradeASM=true

#------------------------------------------------------------------------------

# Specify the auto-updates option. It can be one of the following:

# - MYORACLESUPPORT_DOWNLOAD

# - OFFLINE_UPDATES

# - SKIP_UPDATES

#------------------------------------------------------------------------------

oracle.installer.autoupdates.option=SKIP_UPDATES

#------------------------------------------------------------------------------

# In case MYORACLESUPPORT_DOWNLOAD option is chosen, specify the location where

# the updates are to be downloaded.

# In case OFFLINE_UPDATES option is chosen, specify the location where the updates

# are present.

#------------------------------------------------------------------------------

oracle.installer.autoupdates.downloadUpdatesLoc=

#------------------------------------------------------------------------------

# Specify the My Oracle Support Account Username which has the patches download privileges

# to be used for software updates.

# Example : AUTOUPDATES_MYORACLESUPPORT_USERNAME=abc@oracle.com

#------------------------------------------------------------------------------

AUTOUPDATES_MYORACLESUPPORT_USERNAME=

#------------------------------------------------------------------------------

# Specify the My Oracle Support Account Username password which has the patches download privileges

# to be used for software updates.

# Example : AUTOUPDATES_MYORACLESUPPORT_PASSWORD=password

#------------------------------------------------------------------------------

AUTOUPDATES_MYORACLESUPPORT_PASSWORD=

#------------------------------------------------------------------------------

# Specify the Proxy server name. Length should be greater than zero.

# Example : PROXY_HOST=proxy.domain.com

#------------------------------------------------------------------------------

PROXY_HOST=

#------------------------------------------------------------------------------

# Specify the proxy port number. Should be Numeric and at least 2 chars.

# Example : PROXY_PORT=25

#------------------------------------------------------------------------------

PROXY_PORT=0

#------------------------------------------------------------------------------

# Specify the proxy user name. Leave PROXY_USER and PROXY_PWD

# blank if your proxy server requires no authentication.

# Example : PROXY_USER=username

#------------------------------------------------------------------------------

PROXY_USER=

#------------------------------------------------------------------------------

# Specify the proxy password. Leave PROXY_USER and PROXY_PWD

# blank if your proxy server requires no authentication.

# Example : PROXY_PWD=password

#------------------------------------------------------------------------------

PROXY_PWD=

#------------------------------------------------------------------------------

# Specify the proxy realm.

# Example : PROXY_REALM=metalink

#------------------------------------------------------------------------------

PROXY_REALM=[grid@node1 ~]$

Total Pageviews

Thursday, 29 January 2015

Change the DB name without nid .

Checkpoint # of all datafiles

Friday, 23 January 2015

Upgrade Oracle 11.2.0.3 RAC to 11.2.0.4 - ( Out-of-Place Manual Upgrade )

In the below example we are going to upgrade 11.2.0.3 DB to 11.2.0.4 using out-of-place manual upgrade method :

Monday, 19 January 2015

Changing the DBID & DBNAME of a database using nid

Sunday, 18 January 2015

[FATAL] [INS-00001] Unknown irrecoverable error during silent installation

Saturday, 10 January 2015

Upgrading Grid Infrastructure from 11.2.0.3 to 11.2.0.4