Oracle DBA & Data Science Enthusiast

Saturday, 1 October 2016

Patch 23274134 - Oracle Grid Infrastructure Patch Set Update 11.2.0.4.160719 (Jul2016)

The GI PSU patch 23274134 includes updates for both the Clusterware home and Database home that can be applied in a rolling fashion.

Configuration and PSU Mapping

Configuration : GI Home in conjunction with RAC

GI Version : 11.2.0.4

Database Versions : 11.2.0.4

PSU Patch (to Apply) : GI PSU including rdbms patch

Comments : GI Home and all the Database Homes will be patched

Patch Numbers Getting Installed as Part of this GI PSU Patch

23054359 ---> DB PSU 11.2.0.4.160719 (INCLUDES CPUJUL2016) ---> Both DB and Grid Home
23054319 ---> OCW PSU 11.2.0.4.160719 ---> Both DB Homes and Grid Home
22502505 ----> ACFS PSU 11.2.0.4.160419 ---> Only Grid Home

Download and Unzip the Patch

[grid@node1 PSU_PATCH]# pwd
/data01/PSU_PATCH
[grid@node1 PSU_PATCH]# ls -lrth
total 116K
drwxrwxr-x 5 grid oinstall 4.0K Jul 15 17:50 23274134
-rwxrwxr-x 1 grid oinstall 109K Jul 19 12:50 PatchSearch.xml

[grid@node1 23274134]$ pwd
/data01/PSU_PATCH/23274134
[grid@node1 23274134]$ ls -lrt
total 156
drwxrwxrwx 5 grid oinstall 4096 Feb 17 2016 22502505
drwxrwxrwx 13 grid oinstall 4096 Jun 15 12:10 23054359
drwxrwxrwx 5 grid oinstall 4096 Jul 15 17:38 23054319
-rwxrwxrwx 1 grid oinstall 72207 Jun 28 10:30 PatchSearch.xml
-rwxrwxrwx 1 grid oinstall 549 Jul 15 17:49 bundle.xml
-rwxrwxrwx 1 grid oinstall 64093 Jul 19 10:17 README.html
-rwxrwxrwx 1 grid oinstall 24 Jul 19 10:17 README.txt

Patch Installation Prerequisites

OPatch Utility version should be 11.2.0.3.6 or later to apply this patch . Check on both node1 and node2 under both $GRID_HOME and $ORACLE_HOME:

[grid@node1 OPatch]$ ./opatch version
OPatch Version: 11.2.0.3.15
OPatch succeeded.

[oracle@node1 OPatch]$ ./opatch version
OPatch Version: 11.2.0.3.15
OPatch succeeded.

Validation of Oracle Inventory

Before beginning patch application, check the consistency of inventory information for GI home and each database home to be patched. Run the following command as respective Oracle home owner to check the consistency.

[grid@node1 23274134]$ /data01/app/11.2.0/grid_11204/OPatch/opatch lsinventory -oh /data01/app/11.2.0/grid_11204
Oracle Interim Patch Installer version 11.2.0.3.15
Copyright (c) 2016, Oracle Corporation. All rights reserved.
Oracle Home : /data01/app/11.2.0/grid_11204
Central Inventory : /data01/app/oraInventory
from : /data01/app/11.2.0/grid_11204/oraInst.loc
OPatch version : 11.2.0.3.15
OUI version : 11.2.0.4.0
Log file location : /data01/app/11.2.0/grid_11204/cfgtoollogs/opatch/opatch2016-10-01_03-33-01AM_1.log
Lsinventory Output file location : /data01/app/11.2.0/grid_11204/cfgtoollogs/opatch/lsinv/lsinventory2016-10-01_03-33-01AM.txt
--------------------------------------------------------------------------------
Local Machine Information::
Hostname: node1.localdomain
ARU platform id: 226
ARU platform description:: Linux x86-64

Installed Top-level Products (1):
Oracle Grid Infrastructure 11g 11.2.0.4.0
There are 1 products installed in this Oracle Home.
There are no Interim patches installed in this Oracle Home.
Rac system comprising of multiple nodes
Local node = node1
Remote node = node2
--------------------------------------------------------------------------------
OPatch succeeded.

[oracle@node1 ~]$ /data01/app/oracle/product/11.2.0.4/db_1/OPatch/opatch lsinventory -oh /data01/app/oracle/product/11.2.0.4/db_1
Oracle Interim Patch Installer version 11.2.0.3.15
Copyright (c) 2016, Oracle Corporation. All rights reserved.

Oracle Home : /data01/app/oracle/product/11.2.0.4/db_1
Central Inventory : /data01/app/oraInventory
from : /data01/app/oracle/product/11.2.0.4/db_1/oraInst.loc
OPatch version : 11.2.0.3.15
OUI version : 11.2.0.4.0
Log file location : /data01/app/oracle/product/11.2.0.4/db_1/cfgtoollogs/opatch/opatch2016-10-01_03-34-34AM_1.log
Lsinventory Output file location : /data01/app/oracle/product/11.2.0.4/db_1/cfgtoollogs/opatch/lsinv/lsinventory2016-10-01_03-34-34AM.txt
--------------------------------------------------------------------------------
Local Machine Information::
Hostname: node1.localdomain
ARU platform id: 226
ARU platform description:: Linux x86-64

Installed Top-level Products (1):
Oracle Database 11g 11.2.0.4.0
There are 1 products installed in this Oracle Home.
There are no Interim patches installed in this Oracle Home.

Rac system comprising of multiple nodes
Local node = node1
Remote node = node2
--------------------------------------------------------------------------------
OPatch succeeded.

Stop EM Agent Processes Prior to Patching and Prior to Rolling Back the Patch
$ORACLE_HOME/bin/emctl stop dbconsole

One-off Patch Conflict Detection and Resolution

[grid@node1 23054359]$ cd /data01/PSU_PATCH/23274134/22502505

[grid@node1 22502505]$ /data01/app/11.2.0/grid_11204/OPatch/opatch prereq CheckConflictAgainstOHWithDetail -ph ./

Oracle Interim Patch Installer version 11.2.0.3.15
Copyright (c) 2016, Oracle Corporation. All rights reserved.
PREREQ session
Oracle Home : /data01/app/11.2.0/grid_11204
Central Inventory : /data01/app/oraInventory
from : /data01/app/11.2.0/grid_11204/oraInst.loc
OPatch version : 11.2.0.3.15
OUI version : 11.2.0.4.0
Log file location : /data01/app/11.2.0/grid_11204/cfgtoollogs/opatch/opatch2016-10-01_03-47-49AM_1.log
Invoking prereq "checkconflictagainstohwithdetail"
Prereq "checkConflictAgainstOHWithDetail" passed.
OPatch succeeded.

[grid@node1 23054359]$ cd /data01/PSU_PATCH/23274134/23054359
[grid@node1 23054359]$ /data01/app/11.2.0/grid_11204/OPatch/opatch prereq CheckConflictAgainstOHWithDetail -ph ./

Oracle Interim Patch Installer version 11.2.0.3.15
Copyright (c) 2016, Oracle Corporation. All rights reserved.

PREREQ session
Oracle Home : /data01/app/11.2.0/grid_11204
Central Inventory : /data01/app/oraInventory
from : /data01/app/11.2.0/grid_11204/oraInst.loc
OPatch version : 11.2.0.3.15
OUI version : 11.2.0.4.0
Log file location : /data01/app/11.2.0/grid_11204/cfgtoollogs/opatch/opatch2016-10-01_03-49-25AM_1.log
Invoking prereq "checkconflictagainstohwithdetail"
Prereq "checkConflictAgainstOHWithDetail" passed.
OPatch succeeded.

[grid@node1 23054319]$ cd /data01/PSU_PATCH/23274134/23054319
[grid@node1 23054319]$ /data01/app/11.2.0/grid_11204/OPatch/opatch prereq CheckConflictAgainstOHWithDetail -ph ./
Oracle Interim Patch Installer version 11.2.0.3.15
Copyright (c) 2016, Oracle Corporation. All rights reserved.
PREREQ session
Oracle Home : /data01/app/11.2.0/grid_11204
Central Inventory : /data01/app/oraInventory
from : /data01/app/11.2.0/grid_11204/oraInst.loc
OPatch version : 11.2.0.3.15
OUI version : 11.2.0.4.0
Log file location : /data01/app/11.2.0/grid_11204/cfgtoollogs/opatch/opatch2016-10-01_03-50-07AM_1.log
Invoking prereq "checkconflictagainstohwithdetail"
Prereq "checkConflictAgainstOHWithDetail" passed.
OPatch succeeded.

[oracle@node1 23054359]$ cd /data01/PSU_PATCH/23274134/23054359
[oracle@node1 23054359]$ /data01/app/oracle/product/11.2.0.4/db_1/OPatch/opatch prereq CheckConflictAgainstOHWithDetail -ph ./
Oracle Interim Patch Installer version 11.2.0.3.15
Copyright (c) 2016, Oracle Corporation. All rights reserved.
PREREQ session
Oracle Home : /data01/app/oracle/product/11.2.0.4/db_1
Central Inventory : /data01/app/oraInventory
from : /data01/app/oracle/product/11.2.0.4/db_1/oraInst.loc
OPatch version : 11.2.0.3.15
OUI version : 11.2.0.4.0
Log file location : /data01/app/oracle/product/11.2.0.4/db_1/cfgtoollogs/opatch/opatch2016-10-01_03-52-32AM_1.log
Invoking prereq "checkconflictagainstohwithdetail"
Prereq "checkConflictAgainstOHWithDetail" passed.
OPatch succeeded.

[oracle@node1 23054319]$ cd /data01/PSU_PATCH/23274134/23054319
[oracle@node1 23054319]$ /data01/app/oracle/product/11.2.0.4/db_1/OPatch/opatch prereq CheckConflictAgainstOHWithDetail -ph ./
Oracle Interim Patch Installer version 11.2.0.3.15
Copyright (c) 2016, Oracle Corporation. All rights reserved.
PREREQ session
Oracle Home : /data01/app/oracle/product/11.2.0.4/db_1
Central Inventory : /data01/app/oraInventory
from : /data01/app/oracle/product/11.2.0.4/db_1/oraInst.loc
OPatch version : 11.2.0.3.15
OUI version : 11.2.0.4.0
Log file location : /data01/app/oracle/product/11.2.0.4/db_1/cfgtoollogs/opatch/opatch2016-10-01_03-54-14AM_1.log
Invoking prereq "checkconflictagainstohwithdetail"
Prereq "checkConflictAgainstOHWithDetail" passed.
OPatch succeeded.

Verify Server, Cluster info

[grid@node1 OPatch]$ cemutlo -n
node-cluster

[grid@node1 OPatch]$ ps -ef | grep ora_pmon | grep -v grep | awk -F_ '{ print $3 }' | sort -u
orcl1

[grid@node1 OPatch]$ srvctl status scan_listener
SCAN Listener LISTENER_SCAN1 is enabled
SCAN listener LISTENER_SCAN1 is running on node node2
SCAN Listener LISTENER_SCAN2 is enabled
SCAN listener LISTENER_SCAN2 is running on node node1
SCAN Listener LISTENER_SCAN3 is enabled
SCAN listener LISTENER_SCAN3 is running on node node1

ACFS considerations

Oracle Grid Infra 11.2.0.4.x Patch Set Update SUPPLEMENTAL README (Doc ID 1641136.1)

1494652.1 How to Mount or Unmount ACFS File System While Applying GI Patches?

When patching the GI Home, a shared location on ACFS only needs to be unmounted on the node where the GI Home is being patched.

[grid@node1 grid_11204]$ df -kh
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/vg_node1-lv_root
28G 19G 7.5G 72% /
tmpfs 1004M 394M 610M 40% /dev/shm
/dev/sda1 485M 55M 405M 12% /boot
Softwares 84G 14G 71G 17% /media/sf_Softwares
/dev/sr0 57M 57M 0 100% /media/VBOXADDITIONS_5.1.6_110634
/dev/asm/asmvol-260 4.9G 151M 4.7G 4% /data01/app/grid/acfsmounts/asmvol_asmvol

SQL> SELECT dg.name AS diskgroup, v.volume_name, v.volume_device, v.mountpath,v.state FROM V$ASM_DISKGROUP dg, V$ASM_VOLUME v WHERE dg.group_number = v.group_number;

DISKGROUP VOLUME_NAME VOLUME_DEVICE MOUNTPATH STATE
--------------- -------------------- -----------------------------------------------------------------------
ASMVOL ASMVOL /dev/asm/asmvol-260 /data01/app/grid/acfsmounts/asmvol_asmvol ENABLED

For each node, perform the following steps:

[root@node1 ~]# /sbin/acfsutil registry
Mount Object:
Device: /dev/asm/asmvol-260
Mount Point: /data01/app/grid/acfsmounts/asmvol_asmvol
Disk Group: ASMVOL
Volume: ASMVOL
Options: none
Nodes: all

[grid@node2 ~]$ /sbin/acfsutil info fs
/data01/app/grid/acfsmounts/asmvol_asmvol
ACFS Version: 11.2.0.4.0
flags: MountPoint,Available
mount time: Sat Oct 1 01:05:54 2016
volumes: 1
total size: 5167382528
total free: 5009920000
primary volume: /dev/asm/asmvol-260
label:
flags: Primary,Available,ADVM
on-disk version: 39.0
allocation unit: 4096
major, minor: 251, 133121
size: 5167382528
free: 5009920000
ADVM diskgroup ASMVOL
ADVM resize increment: 33554432
ADVM redundancy: unprotected
ADVM stripe columns: 4
ADVM stripe width: 131072
number of snapshots: 0
snapshot space usage: 0
replication status: DISABLED

[root@node1 ~]# /bin/umount /data01/app/grid/acfsmounts/asmvol_asmvol

[root@node1 ~]# /sbin/acfsutil info fs
acfsutil info fs: ACFS-03036: no mounted ACFS file systems -------> Mount point successfully un-mounted

Perform same step on node2 for acfs :

[root@node2 ~]# /bin/umount /data01/app/grid/acfsmounts/asmvol_asmvol

[root@node2 ~]# /sbin/acfsutil info fs
acfsutil info fs: ACFS-03036: no mounted ACFS file systems
[root@node2 ~]#

** Roll services from Node1 to Node2 and stop oracle on node1 :

[oracle@node1 ~]$ srvctl status service -d orcl
Service node1_app is running on instance(s) orcl1
Service node2_app is running on instance(s) orcl2

[oracle@node1 ~]$ srvctl relocate service -s node1_app -d orcl -i orcl1 -t orcl2
[oracle@node1 ~]$ srvctl status service -d orcl
Service node1_app is running on instance(s) orcl2
Service node2_app is running on instance(s) orcl2

As Oracle you'll need to stop the services running out of the database home. Shutdown all instances :

[oracle@node1 ~]$ srvctl stop instance -d orcl -i orcl1 -o immediate

[oracle@node1 ~]$ srvctl status database -d orcl -v
Instance orcl1 is not running on node node1
Instance orcl2 is running on node node2 with online services node1_app,node2_app. Instance status: Open.

[oracle@node1 ~]$ srvctl stop home -o /data01/app/oracle/product/11.2.0.4/db_1 -s /tmp/db_1_stop -n node1

Unlock Grid home using root :

[root@node1 ~]# ps -ef | grep d.bin
root 2644 1 0 00:58 ? 00:00:44 /data01/app/11.2.0/grid_11204/bin/ohasd.bin reboot
grid 2968 1 0 00:58 ? 00:00:00 /data01/app/11.2.0/grid_11204/bin/mdnsd.bin
grid 2979 1 0 00:58 ? 00:00:04 /data01/app/11.2.0/grid_11204/bin/gpnpd.bin
grid 2989 1 0 00:58 ? 00:00:38 /data01/app/11.2.0/grid_11204/bin/gipcd.bin
root 3003 1 1 00:58 ? 00:02:30 /data01/app/11.2.0/grid_11204/bin/osysmond.bin
grid 3047 1 0 00:58 ? 00:00:45 /data01/app/11.2.0/grid_11204/bin/ocssd.bin
root 4051 1 0 01:02 ? 00:00:23 /data01/app/11.2.0/grid_11204/bin/octssd.bin reboot
grid 4074 1 0 01:02 ? 00:00:22 /data01/app/11.2.0/grid_11204/bin/evmd.bin
root 4500 1 0 01:03 ? 00:01:18 /data01/app/11.2.0/grid_11204/bin/crsd.bin reboot
root 15459 15355 0 04:48 pts/0 00:00:00 grep d.bin
[root@node1 ~]# ps -ef | grep pmon
grid 4329 1 0 01:03 ? 00:00:02 asm_pmon_+ASM1

[root@node1 ~]# /data01/app/11.2.0/grid_11204/crs/install/rootcrs.pl -unlock
Using configuration parameter file: /data01/app/11.2.0/grid_11204/crs/install/crsconfig_params
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'node1'
CRS-2673: Attempting to stop 'ora.crsd' on 'node1'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on 'node1'
CRS-2673: Attempting to stop 'ora.cvu' on 'node1'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN2.lsnr' on 'node1'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN3.lsnr' on 'node1'
CRS-2673: Attempting to stop 'ora.OCR.dg' on 'node1'
CRS-2673: Attempting to stop 'ora.registry.acfs' on 'node1'
CRS-2673: Attempting to stop 'ora.DATA.dg' on 'node1'
CRS-2673: Attempting to stop 'ora.oc4j' on 'node1'
CRS-2673: Attempting to stop 'ora.LISTENER.lsnr' on 'node1'
CRS-2677: Stop of 'ora.cvu' on 'node1' succeeded
CRS-2672: Attempting to start 'ora.cvu' on 'node2'
CRS-2677: Stop of 'ora.LISTENER_SCAN3.lsnr' on 'node1' succeeded
CRS-2673: Attempting to stop 'ora.scan3.vip' on 'node1'
CRS-2677: Stop of 'ora.LISTENER_SCAN2.lsnr' on 'node1' succeeded
CRS-2673: Attempting to stop 'ora.scan2.vip' on 'node1'
CRS-2677: Stop of 'ora.LISTENER.lsnr' on 'node1' succeeded
CRS-2673: Attempting to stop 'ora.node1.vip' on 'node1'
CRS-2677: Stop of 'ora.node1.vip' on 'node1' succeeded
CRS-2672: Attempting to start 'ora.node1.vip' on 'node2'
CRS-2677: Stop of 'ora.scan3.vip' on 'node1' succeeded
CRS-2672: Attempting to start 'ora.scan3.vip' on 'node2'
CRS-2677: Stop of 'ora.registry.acfs' on 'node1' succeeded
CRS-2673: Attempting to stop 'ora.ASMVOL.dg' on 'node1'
CRS-2677: Stop of 'ora.scan2.vip' on 'node1' succeeded
CRS-2672: Attempting to start 'ora.scan2.vip' on 'node2'
CRS-2676: Start of 'ora.cvu' on 'node2' succeeded
CRS-2676: Start of 'ora.node1.vip' on 'node2' succeeded
CRS-2676: Start of 'ora.scan2.vip' on 'node2' succeeded
CRS-2676: Start of 'ora.scan3.vip' on 'node2' succeeded
CRS-2672: Attempting to start 'ora.LISTENER_SCAN2.lsnr' on 'node2'
CRS-2672: Attempting to start 'ora.LISTENER_SCAN3.lsnr' on 'node2'
CRS-2676: Start of 'ora.LISTENER_SCAN2.lsnr' on 'node2' succeeded
CRS-2676: Start of 'ora.LISTENER_SCAN3.lsnr' on 'node2' succeeded
CRS-2677: Stop of 'ora.OCR.dg' on 'node1' succeeded
CRS-2677: Stop of 'ora.DATA.dg' on 'node1' succeeded
CRS-2677: Stop of 'ora.ASMVOL.dg' on 'node1' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'node1'
CRS-2677: Stop of 'ora.asm' on 'node1' succeeded
CRS-2677: Stop of 'ora.oc4j' on 'node1' succeeded
CRS-2672: Attempting to start 'ora.oc4j' on 'node2'
CRS-2676: Start of 'ora.oc4j' on 'node2' succeeded
CRS-2673: Attempting to stop 'ora.ons' on 'node1'
CRS-2677: Stop of 'ora.ons' on 'node1' succeeded
CRS-2673: Attempting to stop 'ora.net1.network' on 'node1'
CRS-2677: Stop of 'ora.net1.network' on 'node1' succeeded
CRS-2792: Shutdown of Cluster Ready Services-managed resources on 'node1' has completed
CRS-2677: Stop of 'ora.crsd' on 'node1' succeeded
CRS-2673: Attempting to stop 'ora.ctssd' on 'node1'
CRS-2673: Attempting to stop 'ora.evmd' on 'node1'
CRS-2673: Attempting to stop 'ora.asm' on 'node1'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'node1'
CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'node1'
CRS-2677: Stop of 'ora.evmd' on 'node1' succeeded
CRS-2677: Stop of 'ora.mdnsd' on 'node1' succeeded
CRS-2677: Stop of 'ora.ctssd' on 'node1' succeeded
CRS-2677: Stop of 'ora.drivers.acfs' on 'node1' succeeded
CRS-2677: Stop of 'ora.asm' on 'node1' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'node1'
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'node1' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'node1'
CRS-2677: Stop of 'ora.cssd' on 'node1' succeeded
CRS-2673: Attempting to stop 'ora.crf' on 'node1'
CRS-2677: Stop of 'ora.crf' on 'node1' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'node1'
CRS-2677: Stop of 'ora.gipcd' on 'node1' succeeded
CRS-2673: Attempting to stop 'ora.gpnpd' on 'node1'
CRS-2677: Stop of 'ora.gpnpd' on 'node1' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'node1' has completed
CRS-4133: Oracle High Availability Services has been stopped.
Successfully unlock /data01/app/11.2.0/grid_11204

[root@node1 ~]# ps -ef | grep d.bin
root 16401 15355 0 04:53 pts/0 00:00:00 grep d.bin
[root@node1 ~]# ps -ef | grep pmon
root 16419 15355 0 04:53 pts/0 00:00:00 grep pmon

Switch to grid user and Patch GI Home:

[grid@node1 PSU_PATCH]$ which opatch
/data01/app/11.2.0/grid_11204/OPatch/opatch
[grid@node1 PSU_PATCH]$ whoami
grid
[grid@node1 PSU_PATCH]$

[grid@node1 23274134]$ /data01/app/11.2.0/grid_11204/OPatch/opatch apply -oh /data01/app/11.2.0/grid_11204 -local /data01/PSU_PATCH/23274134/22502505
Oracle Interim Patch Installer version 11.2.0.3.15
Copyright (c) 2016, Oracle Corporation. All rights reserved.

Oracle Home : /data01/app/11.2.0/grid_11204
Central Inventory : /data01/app/oraInventory
from : /data01/app/11.2.0/grid_11204/oraInst.loc
OPatch version : 11.2.0.3.15
OUI version : 11.2.0.4.0
Log file location : /data01/app/11.2.0/grid_11204/cfgtoollogs/opatch/opatch2016-10-01_04-59-00AM_1.log
Verifying environment and performing prerequisite checks...
OPatch continues with these patches: 22502505

Do you want to proceed? [y|n]
y
User Responded with: Y
All checks passed.
Provide your email address to be informed of security issues, install and
initiate Oracle Configuration Manager. Easier for you if you use your My
Oracle Support Email address/User Name.
Visit http://www.oracle.com/support/policies.html for details.
Email address/User Name:

You have not provided an email address for notification of security issues.
Do you wish to remain uninformed of security issues ([Y]es, [N]o) [N]: Y

Please shutdown Oracle instances running out of this ORACLE_HOME on the local system.
(Oracle Home = '/data01/app/11.2.0/grid_11204')
Is the local system ready for patching? [y|n]
y
User Responded with: Y
Backing up files...
Applying interim patch '22502505' to OH '/data01/app/11.2.0/grid_11204'

Patching component oracle.usm, 11.2.0.4.0...
Patch 22502505 successfully applied.
Log file location: /data01/app/11.2.0/grid_11204/cfgtoollogs/opatch/opatch2016-10-01_04-59-00AM_1.log

OPatch succeeded.

[grid@node1 23274134]$ opatch apply -oh /data01/app/11.2.0/grid_11204 -local /data01/PSU_PATCH/23274134/23054359
Oracle Interim Patch Installer version 11.2.0.3.15
Copyright (c) 2016, Oracle Corporation. All rights reserved.

Oracle Home : /data01/app/11.2.0/grid_11204
Central Inventory : /data01/app/oraInventory
from : /data01/app/11.2.0/grid_11204/oraInst.loc
OPatch version : 11.2.0.3.15
OUI version : 11.2.0.4.0
Log file location : /data01/app/11.2.0/grid_11204/cfgtoollogs/opatch/opatch2016-10-01_05-10-40AM_1.log

Verifying environment and performing prerequisite checks...
OPatch continues with these patches: 17478514 18031668 18522509 19121551 19769489 20299013 20760982 21352635 21948347 22502456 23054359

Do you want to proceed? [y|n]
y
User Responded with: Y
All checks passed.
Provide your email address to be informed of security issues, install and
initiate Oracle Configuration Manager. Easier for you if you use your My
Oracle Support Email address/User Name.
Visit http://www.oracle.com/support/policies.html for details.
Email address/User Name:

You have not provided an email address for notification of security issues.
Do you wish to remain uninformed of security issues ([Y]es, [N]o) [N]: Y

Please shutdown Oracle instances running out of this ORACLE_HOME on the local system.
(Oracle Home = '/data01/app/11.2.0/grid_11204')

Is the local system ready for patching? [y|n]
y
User Responded with: Y
Backing up files...
Applying sub-patch '17478514' to OH '/data01/app/11.2.0/grid_11204'
ApplySession: Optional component(s) [ oracle.sdo, 11.2.0.4.0 ] , [ oracle.sysman.agent, 10.2.0.4.5 ] , [ oracle.xdk, 11.2.0.4.0 ] not present in the Oracle Home or a higher version is found.

Patching component oracle.rdbms, 11.2.0.4.0...

Patching component oracle.rdbms.rsf, 11.2.0.4.0...

Patching component oracle.rdbms.dbscripts, 11.2.0.4.0...

Patching component oracle.sdo.locator, 11.2.0.4.0...

Patching component oracle.nlsrtl.rsf, 11.2.0.4.0...

Patching component oracle.xdk.rsf, 11.2.0.4.0...

Patching component oracle.rdbms.rman, 11.2.0.4.0...
Applying sub-patch '18031668' to OH '/data01/app/11.2.0/grid_11204'
ApplySession: Optional component(s) [ oracle.precomp.common, 11.2.0.4.0 ] not present in the Oracle Home or a higher version is found.

Patching component oracle.rdbms, 11.2.0.4.0...

Patching component oracle.rdbms.rsf, 11.2.0.4.0...

Patching component oracle.ldap.rsf, 11.2.0.4.0...

Patching component oracle.rdbms.crs, 11.2.0.4.0...

Patching component oracle.ldap.rsf.ic, 11.2.0.4.0...

Patching component oracle.rdbms.deconfig, 11.2.0.4.0...

Patching component oracle.rdbms.dbscripts, 11.2.0.4.0...

Patching component oracle.rdbms.rman, 11.2.0.4.0...
Applying sub-patch '18522509' to OH '/data01/app/11.2.0/grid_11204'
ApplySession: Optional component(s) [ oracle.precomp.common, 11.2.0.4.0 ] not present in the Oracle Home or a higher version is found.

Patching component oracle.rdbms.rsf, 11.2.0.4.0...

Patching component oracle.rdbms, 11.2.0.4.0...

Patching component oracle.rdbms.rman, 11.2.0.4.0...

Patching component oracle.rdbms.dbscripts, 11.2.0.4.0...

Patching component oracle.rdbms.deconfig, 11.2.0.4.0...
Applying sub-patch '19121551' to OH '/data01/app/11.2.0/grid_11204'
ApplySession: Optional component(s) [ oracle.precomp.common, 11.2.0.4.0 ] , [ oracle.sysman.console.db, 11.2.0.4.0 ] not present in the Oracle Home or a higher version is found.

Patching component oracle.rdbms.rsf, 11.2.0.4.0...

Patching component oracle.rdbms.rman, 11.2.0.4.0...

Patching component oracle.rdbms, 11.2.0.4.0...

Patching component oracle.rdbms.dbscripts, 11.2.0.4.0...

Patching component oracle.ordim.client, 11.2.0.4.0...

Patching component oracle.ordim.jai, 11.2.0.4.0...
Applying sub-patch '19769489' to OH '/data01/app/11.2.0/grid_11204'
ApplySession: Optional component(s) [ oracle.precomp.common, 11.2.0.4.0 ] , [ oracle.ovm, 11.2.0.4.0 ] , [ oracle.xdk, 11.2.0.4.0 ] , [ oracle.oraolap, 11.2.0.4.0 ] , [ oracle.sysman.agent, 11.2.0.4.0 ] not present in the Oracle Home or a higher version is found.

Patching component oracle.rdbms.util, 11.2.0.4.0...

Patching component oracle.rdbms, 11.2.0.4.0...

Patching component oracle.rdbms.dbscripts, 11.2.0.4.0...

Patching component oracle.xdk.parser.java, 11.2.0.4.0...

Patching component oracle.rdbms.rsf, 11.2.0.4.0...

Patching component oracle.xdk.rsf, 11.2.0.4.0...

Patching component oracle.rdbms.rman, 11.2.0.4.0...

Patching component oracle.rdbms.deconfig, 11.2.0.4.0...
Applying sub-patch '20299013' to OH '/data01/app/11.2.0/grid_11204'
ApplySession: Optional component(s) [ oracle.rdbms.dv, 11.2.0.4.0 ] , [ oracle.rdbms.oci, 11.2.0.4.0 ] , [ oracle.precomp.common, 11.2.0.4.0 ] , [ oracle.sysman.agent, 10.2.0.4.5 ] , [ oracle.xdk, 11.2.0.4.0 ] , [ oracle.sysman.console.db, 11.2.0.4.0 ] not present in the Oracle Home or a higher version is found.

Patching component oracle.sysman.common, 10.2.0.4.5...

Patching component oracle.rdbms, 11.2.0.4.0...

Patching component oracle.rdbms.dbscripts, 11.2.0.4.0...

Patching component oracle.xdk.parser.java, 11.2.0.4.0...

Patching component oracle.xdk.rsf, 11.2.0.4.0...

Patching component oracle.rdbms.rsf, 11.2.0.4.0...

Patching component oracle.sysman.common.core, 10.2.0.4.5...

Patching component oracle.rdbms.rman, 11.2.0.4.0...

Patching component oracle.rdbms.deconfig, 11.2.0.4.0...
Applying sub-patch '20760982' to OH '/data01/app/11.2.0/grid_11204'
ApplySession: Optional component(s) [ oracle.sysman.console.db, 11.2.0.4.0 ] not present in the Oracle Home or a higher version is found.

Patching component oracle.rdbms, 11.2.0.4.0...

Patching component oracle.rdbms.dbscripts, 11.2.0.4.0...

Patching component oracle.rdbms.rsf, 11.2.0.4.0...
Applying sub-patch '21352635' to OH '/data01/app/11.2.0/grid_11204'
ApplySession: Optional component(s) [ oracle.sysman.agent, 10.2.0.4.5 ] not present in the Oracle Home or a higher version is found.

Patching component oracle.rdbms.rsf, 11.2.0.4.0...

Patching component oracle.rdbms.rman, 11.2.0.4.0...

Patching component oracle.rdbms, 11.2.0.4.0...
Applying sub-patch '21948347' to OH '/data01/app/11.2.0/grid_11204'
ApplySession: Optional component(s) [ oracle.sysman.agent, 10.2.0.4.5 ] , [ oracle.ovm, 11.2.0.4.0 ] , [ oracle.xdk, 11.2.0.4.0 ] , [ oracle.tfa, 11.2.0.4.0 ] , [ oracle.sysman.console.db, 11.2.0.4.0 ] , [ oracle.sysman.oms.core, 10.2.0.4.5 ] not present in the Oracle Home or a higher version is found.

Patching component oracle.rdbms, 11.2.0.4.0...

Patching component oracle.nlsrtl.rsf, 11.2.0.4.0...

Patching component oracle.xdk.parser.java, 11.2.0.4.0...

Patching component oracle.xdk.rsf, 11.2.0.4.0...

Patching component oracle.rdbms.rsf, 11.2.0.4.0...
Applying sub-patch '22502456' to OH '/data01/app/11.2.0/grid_11204'
ApplySession: Optional component(s) [ oracle.precomp.common, 11.2.0.4.0 ] , [ oracle.tfa, 11.2.0.4.0 ] , [ oracle.rdbms.olap, 11.2.0.4.0 ] , [ oracle.oraolap, 11.2.0.4.0 ] not present in the Oracle Home or a higher version is found.

Patching component oracle.oraolap.dbscripts, 11.2.0.4.0...

Patching component oracle.rdbms.rsf, 11.2.0.4.0...

Patching component oracle.rdbms.rman, 11.2.0.4.0...

Patching component oracle.rdbms, 11.2.0.4.0...

Patching component oracle.rdbms.dbscripts, 11.2.0.4.0...
Applying sub-patch '23054359' to OH '/data01/app/11.2.0/grid_11204'
ApplySession: Optional component(s) [ oracle.rdbms.dv, 11.2.0.4.0 ] not present in the Oracle Home or a higher version is found.

Patching component oracle.rdbms, 11.2.0.4.0...

Patching component oracle.rdbms.dbscripts, 11.2.0.4.0...

Patching component oracle.rdbms.rsf, 11.2.0.4.0...
Composite patch 23054359 successfully applied.
Log file location: /data01/app/11.2.0/grid_11204/cfgtoollogs/opatch/opatch2016-10-01_05-10-40AM_1.log

OPatch succeeded.

[grid@node1 23274134]$ opatch apply -oh /data01/app/11.2.0/grid_11204 -local /data01/PSU_PATCH/23274134/23054319
Oracle Interim Patch Installer version 11.2.0.3.15
Copyright (c) 2016, Oracle Corporation. All rights reserved.

Oracle Home : /data01/app/11.2.0/grid_11204
Central Inventory : /data01/app/oraInventory
from : /data01/app/11.2.0/grid_11204/oraInst.loc
OPatch version : 11.2.0.3.15
OUI version : 11.2.0.4.0
Log file location : /data01/app/11.2.0/grid_11204/cfgtoollogs/opatch/opatch2016-10-01_05-23-42AM_1.log

Verifying environment and performing prerequisite checks...
OPatch continues with these patches: 23054319

Do you want to proceed? [y|n]
y
User Responded with: Y
All checks passed.
Provide your email address to be informed of security issues, install and
initiate Oracle Configuration Manager. Easier for you if you use your My
Oracle Support Email address/User Name.
Visit http://www.oracle.com/support/policies.html for details.
Email address/User Name:
You have not provided an email address for notification of security issues.
Do you wish to remain uninformed of security issues ([Y]es, [N]o) [N]: Y

Please shutdown Oracle instances running out of this ORACLE_HOME on the local system.
(Oracle Home = '/data01/app/11.2.0/grid_11204')

Is the local system ready for patching? [y|n]
y
User Responded with: Y
Backing up files...
Applying interim patch '23054319' to OH '/data01/app/11.2.0/grid_11204'

Patching component oracle.crs, 11.2.0.4.0...
Patch 23054319 successfully applied.
Log file location: /data01/app/11.2.0/grid_11204/cfgtoollogs/opatch/opatch2016-10-01_05-23-42AM_1.log

OPatch succeeded.

Start CRS Services As Root

# whoami
root

[root@node1 ~]# /data01/app/11.2.0/grid_11204/crs/install/rootcrs.pl -patch
Using configuration parameter file: /data01/app/11.2.0/grid_11204/crs/install/crsconfig_params
Installing Trace File Analyzer
CRS-4123: Oracle High Availability Services has been started.
[root@node1 ~]# ps -ef | grep d.bin
root 25999 1 1 05:43 ? 00:00:01 /data01/app/11.2.0/grid_11204/bin/ohasd.bin reboot
grid 26156 1 0 05:44 ? 00:00:00 /data01/app/11.2.0/grid_11204/bin/mdnsd.bin
grid 26170 1 0 05:44 ? 00:00:00 /data01/app/11.2.0/grid_11204/bin/gpnpd.bin
grid 26180 1 0 05:44 ? 00:00:00 /data01/app/11.2.0/grid_11204/bin/gipcd.bin
root 26183 1 1 05:44 ? 00:00:01 /data01/app/11.2.0/grid_11204/bin/osysmond.bin
grid 26267 1 0 05:44 ? 00:00:01 /data01/app/11.2.0/grid_11204/bin/ocssd.bin
root 26419 1 0 05:44 ? 00:00:00 /data01/app/11.2.0/grid_11204/bin/octssd.bin reboot
grid 26447 1 0 05:44 ? 00:00:00 /data01/app/11.2.0/grid_11204/bin/evmd.bin
root 26702 1 1 05:45 ? 00:00:00 /data01/app/11.2.0/grid_11204/bin/crsd.bin reboot
root 27313 23786 0 05:46 pts/0 00:00:00 grep d.bin
[root@node1 ~]# ps -ef | grep pmon
grid 26582 1 0 05:44 ? 00:00:00 asm_pmon_+ASM1
root 27316 23786 0 05:46 pts/0 00:00:00 grep pmon

Switch to Oracle. Patch Oracle Home:

[oracle@node1 23054319]$ which opatch
/data01/app/oracle/product/11.2.0.4/db_1/OPatch/opatch

[oracle@node1 23054319]$ opatch apply -oh /data01/app/oracle/product/11.2.0.4/db_1 -local /data01/PSU_PATCH/23274134/23054359
Oracle Interim Patch Installer version 11.2.0.3.15
Copyright (c) 2016, Oracle Corporation. All rights reserved.

Oracle Home : /data01/app/oracle/product/11.2.0.4/db_1
Central Inventory : /data01/app/oraInventory
from : /data01/app/oracle/product/11.2.0.4/db_1/oraInst.loc
OPatch version : 11.2.0.3.15
OUI version : 11.2.0.4.0
Log file location : /data01/app/oracle/product/11.2.0.4/db_1/cfgtoollogs/opatch/opatch2016-10-01_05-49-44AM_1.log

Verifying environment and performing prerequisite checks...
OPatch continues with these patches: 17478514 18031668 18522509 19121551 19769489 20299013 20760982 21352635 21948347 22502456 23054359

Do you want to proceed? [y|n]
y
User Responded with: Y
All checks passed.
Provide your email address to be informed of security issues, install and
initiate Oracle Configuration Manager. Easier for you if you use your My
Oracle Support Email address/User Name.
Visit http://www.oracle.com/support/policies.html for details.
Email address/User Name:

You have not provided an email address for notification of security issues.
Do you wish to remain uninformed of security issues ([Y]es, [N]o) [N]: Y

Please shutdown Oracle instances running out of this ORACLE_HOME on the local system.
(Oracle Home = '/data01/app/oracle/product/11.2.0.4/db_1')

Is the local system ready for patching? [y|n]
y
User Responded with: Y
Backing up files...
Applying sub-patch '17478514' to OH '/data01/app/oracle/product/11.2.0.4/db_1'

Patching component oracle.rdbms, 11.2.0.4.0...

Patching component oracle.rdbms.rsf, 11.2.0.4.0...

Patching component oracle.sdo, 11.2.0.4.0...

Patching component oracle.sysman.agent, 10.2.0.4.5...

Patching component oracle.xdk, 11.2.0.4.0...

Patching component oracle.rdbms.dbscripts, 11.2.0.4.0...

Patching component oracle.sdo.locator, 11.2.0.4.0...

Patching component oracle.nlsrtl.rsf, 11.2.0.4.0...

Patching component oracle.xdk.rsf, 11.2.0.4.0...

Patching component oracle.rdbms.rman, 11.2.0.4.0...
Applying sub-patch '18031668' to OH '/data01/app/oracle/product/11.2.0.4/db_1'

Patching component oracle.rdbms, 11.2.0.4.0...

Patching component oracle.rdbms.rsf, 11.2.0.4.0...

Patching component oracle.ldap.rsf, 11.2.0.4.0...

Patching component oracle.rdbms.crs, 11.2.0.4.0...

Patching component oracle.precomp.common, 11.2.0.4.0...

Patching component oracle.ldap.rsf.ic, 11.2.0.4.0...

Patching component oracle.rdbms.deconfig, 11.2.0.4.0...

Patching component oracle.rdbms.dbscripts, 11.2.0.4.0...

Patching component oracle.rdbms.rman, 11.2.0.4.0...
Applying sub-patch '18522509' to OH '/data01/app/oracle/product/11.2.0.4/db_1'

Patching component oracle.rdbms.rsf, 11.2.0.4.0...

Patching component oracle.rdbms, 11.2.0.4.0...

Patching component oracle.precomp.common, 11.2.0.4.0...

Patching component oracle.rdbms.rman, 11.2.0.4.0...

Patching component oracle.rdbms.dbscripts, 11.2.0.4.0...

Patching component oracle.rdbms.deconfig, 11.2.0.4.0...
Applying sub-patch '19121551' to OH '/data01/app/oracle/product/11.2.0.4/db_1'

Patching component oracle.precomp.common, 11.2.0.4.0...

Patching component oracle.sysman.console.db, 11.2.0.4.0...

Patching component oracle.rdbms.rsf, 11.2.0.4.0...

Patching component oracle.rdbms.rman, 11.2.0.4.0...

Patching component oracle.rdbms, 11.2.0.4.0...

Patching component oracle.rdbms.dbscripts, 11.2.0.4.0...

Patching component oracle.ordim.client, 11.2.0.4.0...

Patching component oracle.ordim.jai, 11.2.0.4.0...
Applying sub-patch '19769489' to OH '/data01/app/oracle/product/11.2.0.4/db_1'
ApplySession: Optional component(s) [ oracle.sysman.agent, 11.2.0.4.0 ] not present in the Oracle Home or a higher version is found.

Patching component oracle.precomp.common, 11.2.0.4.0...

Patching component oracle.ovm, 11.2.0.4.0...

Patching component oracle.xdk, 11.2.0.4.0...

Patching component oracle.rdbms.util, 11.2.0.4.0...

Patching component oracle.rdbms, 11.2.0.4.0...

Patching component oracle.rdbms.dbscripts, 11.2.0.4.0...

Patching component oracle.xdk.parser.java, 11.2.0.4.0...

Patching component oracle.oraolap, 11.2.0.4.0...

Patching component oracle.rdbms.rsf, 11.2.0.4.0...

Patching component oracle.xdk.rsf, 11.2.0.4.0...

Patching component oracle.rdbms.rman, 11.2.0.4.0...

Patching component oracle.rdbms.deconfig, 11.2.0.4.0...
Applying sub-patch '20299013' to OH '/data01/app/oracle/product/11.2.0.4/db_1'

Patching component oracle.rdbms.dv, 11.2.0.4.0...

Patching component oracle.rdbms.oci, 11.2.0.4.0...

Patching component oracle.precomp.common, 11.2.0.4.0...

Patching component oracle.sysman.agent, 10.2.0.4.5...

Patching component oracle.xdk, 11.2.0.4.0...

Patching component oracle.sysman.common, 10.2.0.4.5...

Patching component oracle.rdbms, 11.2.0.4.0...

Patching component oracle.rdbms.dbscripts, 11.2.0.4.0...

Patching component oracle.xdk.parser.java, 11.2.0.4.0...

Patching component oracle.sysman.console.db, 11.2.0.4.0...

Patching component oracle.xdk.rsf, 11.2.0.4.0...

Patching component oracle.rdbms.rsf, 11.2.0.4.0...

Patching component oracle.sysman.common.core, 10.2.0.4.5...

Patching component oracle.rdbms.rman, 11.2.0.4.0...

Patching component oracle.rdbms.deconfig, 11.2.0.4.0...
Applying sub-patch '20760982' to OH '/data01/app/oracle/product/11.2.0.4/db_1'

Patching component oracle.sysman.console.db, 11.2.0.4.0...

Patching component oracle.rdbms, 11.2.0.4.0...

Patching component oracle.rdbms.dbscripts, 11.2.0.4.0...

Patching component oracle.rdbms.rsf, 11.2.0.4.0...
Applying sub-patch '21352635' to OH '/data01/app/oracle/product/11.2.0.4/db_1'

Patching component oracle.sysman.agent, 10.2.0.4.5...

Patching component oracle.rdbms.rsf, 11.2.0.4.0...

Patching component oracle.rdbms.rman, 11.2.0.4.0...

Patching component oracle.rdbms, 11.2.0.4.0...
Applying sub-patch '21948347' to OH '/data01/app/oracle/product/11.2.0.4/db_1'
ApplySession: Optional component(s) [ oracle.tfa, 11.2.0.4.0 ] not present in the Oracle Home or a higher version is found.

Patching component oracle.sysman.agent, 10.2.0.4.5...

Patching component oracle.ovm, 11.2.0.4.0...

Patching component oracle.xdk, 11.2.0.4.0...

Patching component oracle.rdbms, 11.2.0.4.0...

Patching component oracle.nlsrtl.rsf, 11.2.0.4.0...

Patching component oracle.xdk.parser.java, 11.2.0.4.0...

Patching component oracle.sysman.console.db, 11.2.0.4.0...

Patching component oracle.xdk.rsf, 11.2.0.4.0...

Patching component oracle.rdbms.rsf, 11.2.0.4.0...

Patching component oracle.sysman.oms.core, 10.2.0.4.5...
Applying sub-patch '22502456' to OH '/data01/app/oracle/product/11.2.0.4/db_1'
ApplySession: Optional component(s) [ oracle.tfa, 11.2.0.4.0 ] not present in the Oracle Home or a higher version is found.

Patching component oracle.precomp.common, 11.2.0.4.0...

Patching component oracle.oraolap.dbscripts, 11.2.0.4.0...

Patching component oracle.rdbms.olap, 11.2.0.4.0...

Patching component oracle.oraolap, 11.2.0.4.0...

Patching component oracle.rdbms.rsf, 11.2.0.4.0...

Patching component oracle.rdbms.rman, 11.2.0.4.0...

Patching component oracle.rdbms, 11.2.0.4.0...

Patching component oracle.rdbms.dbscripts, 11.2.0.4.0...
Applying sub-patch '23054359' to OH '/data01/app/oracle/product/11.2.0.4/db_1'

Patching component oracle.rdbms.dv, 11.2.0.4.0...

Patching component oracle.rdbms, 11.2.0.4.0...

Patching component oracle.rdbms.dbscripts, 11.2.0.4.0...

Patching component oracle.rdbms.rsf, 11.2.0.4.0...
Patching component oracle.rdbms.rman, 11.2.0.4.0...

Patching component oracle.rdbms, 11.2.0.4.0...

Patching component oracle.rdbms.dbscripts, 11.2.0.4.0...
Applying sub-patch '23054359' to OH '/data01/app/oracle/product/11.2.0.4/db_1'

Patching component oracle.rdbms.dv, 11.2.0.4.0...

Patching component oracle.rdbms, 11.2.0.4.0...

Patching component oracle.rdbms.dbscripts, 11.2.0.4.0...

Patching component oracle.rdbms.rsf, 11.2.0.4.0...
Composite patch 23054359 successfully applied.
Log file location: /data01/app/oracle/product/11.2.0.4/db_1/cfgtoollogs/opatch/opatch2016-10-01_05-49-44AM_1.log

OPatch succeeded.

[oracle@node1 23054319]$ opatch apply -oh /data01/app/oracle/product/11.2.0.4/db_1 -local /data01/PSU_PATCH/23274134/23054319
Oracle Interim Patch Installer version 11.2.0.3.15
Copyright (c) 2016, Oracle Corporation. All rights reserved.

Oracle Home : /data01/app/oracle/product/11.2.0.4/db_1
Central Inventory : /data01/app/oraInventory
from : /data01/app/oracle/product/11.2.0.4/db_1/oraInst.loc
OPatch version : 11.2.0.3.15
OUI version : 11.2.0.4.0
Log file location : /data01/app/oracle/product/11.2.0.4/db_1/cfgtoollogs/opatch/opatch2016-10-01_06-17-49AM_1.log

Verifying environment and performing prerequisite checks...
UtilSession failed: Patch 23054319 requires component(s) that are not installed in OracleHome. These not-installed components are oracle.crs:11.2.0.4.0,
Log file location: /data01/app/oracle/product/11.2.0.4/db_1/cfgtoollogs/opatch/opatch2016-10-01_06-17-49AM_1.log

OPatch failed with error code 73

Solution : Above issue is a known bug . Reason is duplicate custom/server folder under the patch directory as shown below .

[oracle@node1 23054319]$ pwd
/data01/PSU_PATCH/23274134/23054319/custom/server/23054319
[oracle@node1 23054319]$ ls -lrth
total 32K
-rwxrwxrwx 1 grid oinstall 17K Jul 15 17:38 README.txt
drwxrwxrwx 11 grid oinstall 4.0K Jul 15 17:38 files
drwxrwxrwx 4 grid oinstall 4.0K Jul 15 17:38 etc
drwxrwxrwx 3 grid oinstall 4.0K Jul 15 17:38 custom

As a workaround run the opatch from folder : /data01/PSU_PATCH/23274134/23054319/custom/server/23054319

[oracle@node1 ~]$ opatch apply -oh /data01/app/oracle/product/11.2.0.4/db_1 -local /data01/PSU_PATCH/23274134/23054319/custom/server/23054319
Oracle Interim Patch Installer version 11.2.0.3.15
Copyright (c) 2016, Oracle Corporation. All rights reserved.

Oracle Home : /data01/app/oracle/product/11.2.0.4/db_1
Central Inventory : /data01/app/oraInventory
from : /data01/app/oracle/product/11.2.0.4/db_1/oraInst.loc
OPatch version : 11.2.0.3.15
OUI version : 11.2.0.4.0
Log file location : /data01/app/oracle/product/11.2.0.4/db_1/cfgtoollogs/opatch/opatch2016-10-01_06-26-54AM_1.log

Verifying environment and performing prerequisite checks...
OPatch continues with these patches: 23054319

Do you want to proceed? [y|n]
y
User Responded with: Y
All checks passed.
Provide your email address to be informed of security issues, install and
initiate Oracle Configuration Manager. Easier for you if you use your My
Oracle Support Email address/User Name.
Visit http://www.oracle.com/support/policies.html for details.
Email address/User Name:

You have not provided an email address for notification of security issues.
Do you wish to remain uninformed of security issues ([Y]es, [N]o) [N]: Y

Please shutdown Oracle instances running out of this ORACLE_HOME on the local system.
(Oracle Home = '/data01/app/oracle/product/11.2.0.4/db_1')

Is the local system ready for patching? [y|n]
y
User Responded with: Y
Backing up files...
Applying interim patch '23054319' to OH '/data01/app/oracle/product/11.2.0.4/db_1'

Patching component oracle.rdbms, 11.2.0.4.0...
Patch 23054319 successfully applied.
Log file location: /data01/app/oracle/product/11.2.0.4/db_1/cfgtoollogs/opatch/opatch2016-10-01_06-26-54AM_1.log

OPatch succeeded.

** Start Oracle home and relocate service to node1 :

[oracle@node1 ~]$ srvctl start home -o /data01/app/oracle/product/11.2.0.4/db_1 -s /tmp/db_1_stop -n node1

[oracle@node1 ~]$ srvctl start instance -d orcl -i orcl1
[oracle@node1 ~]$ srvctl relocate service -s node1_app -d orcl -i orcl2 -t orcl1

[oracle@node1 ~]$ srvctl status database -d orcl -v
Instance orcl1 is running on node node1 with online services node1_app. Instance status: Open.
Instance orcl2 is running on node node2 with online services node2_app. Instance status: Open.

[oracle@node1 ~]$ srvctl relocate service -s node2_app -d orcl -i orcl2 -t orcl1
[oracle@node1 ~]$ srvctl status database -d orcl -v
Instance orcl1 is running on node node1 with online services node1_app,node2_app. Instance status: Open.
Instance orcl2 is running on node node2. Instance status: Open.

Remove /tmp file /tmp/db_1_stop

==== Start Steps for node2 ========

[oracle@node1 ~]$ srvctl stop instance -d orcl -i orcl2 -o immediate

[oracle@node2 ~]$ srvctl status database -d orcl -v
Instance orcl1 is running on node node1 with online services node1_app,node2_app. Instance status: Open.
Instance orcl2 is not running on node node2

[oracle@node2 ~]$ srvctl stop home -o /data01/app/oracle/product/11.2.0.4/db_1 -s /tmp/db_1_stop -n node2

Unlock Grid home using root :

[root@node2 ~]# /data01/app/11.2.0/grid_11204/crs/install/rootcrs.pl -unlock
Using configuration parameter file: /data01/app/11.2.0/grid_11204/crs/install/crsconfig_params
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'node2'
CRS-2673: Attempting to stop 'ora.crsd' on 'node2'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on 'node2'
CRS-2673: Attempting to stop 'ora.oc4j' on 'node2'
CRS-2673: Attempting to stop 'ora.LISTENER.lsnr' on 'node2'
CRS-2673: Attempting to stop 'ora.cvu' on 'node2'
CRS-2673: Attempting to stop 'ora.OCR.dg' on 'node2'
CRS-2673: Attempting to stop 'ora.registry.acfs' on 'node2'
CRS-2673: Attempting to stop 'ora.DATA.dg' on 'node2'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN3.lsnr' on 'node2'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN2.lsnr' on 'node2'
CRS-2677: Stop of 'ora.cvu' on 'node2' succeeded
CRS-2672: Attempting to start 'ora.cvu' on 'node1'
CRS-2677: Stop of 'ora.LISTENER_SCAN2.lsnr' on 'node2' succeeded
CRS-2673: Attempting to stop 'ora.scan2.vip' on 'node2'
CRS-2677: Stop of 'ora.LISTENER_SCAN3.lsnr' on 'node2' succeeded
CRS-2673: Attempting to stop 'ora.scan3.vip' on 'node2'
CRS-2677: Stop of 'ora.LISTENER.lsnr' on 'node2' succeeded
CRS-2673: Attempting to stop 'ora.node2.vip' on 'node2'
CRS-2677: Stop of 'ora.scan2.vip' on 'node2' succeeded
CRS-2672: Attempting to start 'ora.scan2.vip' on 'node1'
CRS-2676: Start of 'ora.cvu' on 'node1' succeeded
CRS-2677: Stop of 'ora.scan3.vip' on 'node2' succeeded
CRS-2672: Attempting to start 'ora.scan3.vip' on 'node1'
CRS-2677: Stop of 'ora.registry.acfs' on 'node2' succeeded
CRS-2673: Attempting to stop 'ora.ASMVOL.dg' on 'node2'
CRS-2677: Stop of 'ora.node2.vip' on 'node2' succeeded
CRS-2672: Attempting to start 'ora.node2.vip' on 'node1'
CRS-2676: Start of 'ora.scan3.vip' on 'node1' succeeded
CRS-2676: Start of 'ora.scan2.vip' on 'node1' succeeded
CRS-2672: Attempting to start 'ora.LISTENER_SCAN3.lsnr' on 'node1'
CRS-2676: Start of 'ora.node2.vip' on 'node1' succeeded
CRS-2672: Attempting to start 'ora.LISTENER_SCAN2.lsnr' on 'node1'
CRS-2676: Start of 'ora.LISTENER_SCAN3.lsnr' on 'node1' succeeded
CRS-2676: Start of 'ora.LISTENER_SCAN2.lsnr' on 'node1' succeeded
CRS-2677: Stop of 'ora.DATA.dg' on 'node2' succeeded
CRS-2677: Stop of 'ora.ASMVOL.dg' on 'node2' succeeded
CRS-2677: Stop of 'ora.OCR.dg' on 'node2' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'node2'
CRS-2677: Stop of 'ora.asm' on 'node2' succeeded
CRS-2677: Stop of 'ora.oc4j' on 'node2' succeeded
CRS-2672: Attempting to start 'ora.oc4j' on 'node1'
CRS-2676: Start of 'ora.oc4j' on 'node1' succeeded
CRS-2673: Attempting to stop 'ora.ons' on 'node2'
CRS-2677: Stop of 'ora.ons' on 'node2' succeeded
CRS-2673: Attempting to stop 'ora.net1.network' on 'node2'
CRS-2677: Stop of 'ora.net1.network' on 'node2' succeeded
CRS-2792: Shutdown of Cluster Ready Services-managed resources on 'node2' has completed
CRS-2677: Stop of 'ora.crsd' on 'node2' succeeded
CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'node2'
CRS-2673: Attempting to stop 'ora.crf' on 'node2'
CRS-2673: Attempting to stop 'ora.ctssd' on 'node2'
CRS-2673: Attempting to stop 'ora.evmd' on 'node2'
CRS-2673: Attempting to stop 'ora.asm' on 'node2'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'node2'
CRS-2677: Stop of 'ora.crf' on 'node2' succeeded
CRS-2677: Stop of 'ora.evmd' on 'node2' succeeded
CRS-2677: Stop of 'ora.mdnsd' on 'node2' succeeded
CRS-2677: Stop of 'ora.ctssd' on 'node2' succeeded
CRS-2677: Stop of 'ora.drivers.acfs' on 'node2' succeeded
CRS-2677: Stop of 'ora.asm' on 'node2' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'node2'
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'node2' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'node2'
CRS-2677: Stop of 'ora.cssd' on 'node2' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'node2'
CRS-2677: Stop of 'ora.gipcd' on 'node2' succeeded
CRS-2673: Attempting to stop 'ora.gpnpd' on 'node2'
CRS-2677: Stop of 'ora.gpnpd' on 'node2' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'node2' has completed
CRS-4133: Oracle High Availability Services has been stopped.
Successfully unlock /data01/app/11.2.0/grid_11204

Switch to grid user and Patch GI Home:

[grid@node1 PSU_PATCH]$ which opatch
/data01/app/11.2.0/grid_11204/OPatch/opatch
[grid@node1 PSU_PATCH]$ whoami
grid
[grid@node1 PSU_PATCH]$

[grid@node2 ~]$ /data01/app/11.2.0/grid_11204/OPatch/opatch apply -oh /data01/app/11.2.0/grid_11204 -local /data01/PSU_PATCH/23274134/22502505
Oracle Interim Patch Installer version 11.2.0.3.15
Copyright (c) 2016, Oracle Corporation. All rights reserved.

Oracle Home : /data01/app/11.2.0/grid_11204
Central Inventory : /data01/app/oraInventory
from : /data01/app/11.2.0/grid_11204/oraInst.loc
OPatch version : 11.2.0.3.15
OUI version : 11.2.0.4.0
Log file location : /data01/app/11.2.0/grid_11204/cfgtoollogs/opatch/opatch2016-10-01_07-05-20AM_1.log

Verifying environment and performing prerequisite checks...
OPatch continues with these patches: 22502505

Do you want to proceed? [y|n]
y
User Responded with: Y
All checks passed.
Provide your email address to be informed of security issues, install and
initiate Oracle Configuration Manager. Easier for you if you use your My
Oracle Support Email address/User Name.
Visit http://www.oracle.com/support/policies.html for details.
Email address/User Name:

You have not provided an email address for notification of security issues.
Do you wish to remain uninformed of security issues ([Y]es, [N]o) [N]: Y

Please shutdown Oracle instances running out of this ORACLE_HOME on the local system.
(Oracle Home = '/data01/app/11.2.0/grid_11204')

Is the local system ready for patching? [y|n]
y
User Responded with: Y
Backing up files...
Applying interim patch '22502505' to OH '/data01/app/11.2.0/grid_11204'

Patching component oracle.usm, 11.2.0.4.0...
Patch 22502505 successfully applied.
Log file location: /data01/app/11.2.0/grid_11204/cfgtoollogs/opatch/opatch2016-10-01_07-05-20AM_1.log

OPatch succeeded

[grid@node2 ~]$ opatch apply -oh /data01/app/11.2.0/grid_11204 -local /data01/PSU_PATCH/23274134/23054359
Oracle Interim Patch Installer version 11.2.0.3.15
Copyright (c) 2016, Oracle Corporation. All rights reserved.

Oracle Home : /data01/app/11.2.0/grid_11204
Central Inventory : /data01/app/oraInventory
from : /data01/app/11.2.0/grid_11204/oraInst.loc
OPatch version : 11.2.0.3.15
OUI version : 11.2.0.4.0
Log file location : /data01/app/11.2.0/grid_11204/cfgtoollogs/opatch/opatch2016-10-01_07-14-41AM_1.log

Verifying environment and performing prerequisite checks...
OPatch continues with these patches: 17478514 18031668 18522509 19121551 19769489 20299013 20760982 21352635 21948347 22502456 23054359

Do you want to proceed? [y|n]
y
User Responded with: Y
All checks passed.
Provide your email address to be informed of security issues, install and
initiate Oracle Configuration Manager. Easier for you if you use your My
Oracle Support Email address/User Name.
Visit http://www.oracle.com/support/policies.html for details.
Email address/User Name:

You have not provided an email address for notification of security issues.
Do you wish to remain uninformed of security issues ([Y]es, [N]o) [N]: Y

Please shutdown Oracle instances running out of this ORACLE_HOME on the local system.
(Oracle Home = '/data01/app/11.2.0/grid_11204')

Is the local system ready for patching? [y|n]
y
User Responded with: Y
Backing up files...
Applying sub-patch '17478514' to OH '/data01/app/11.2.0/grid_11204'
ApplySession: Optional component(s) [ oracle.sdo, 11.2.0.4.0 ] , [ oracle.sysman.agent, 10.2.0.4.5 ] , [ oracle.xdk, 11.2.0.4.0 ] not present in the Oracle Home or a higher version is found.

Patching component oracle.rdbms, 11.2.0.4.0...

Patching component oracle.rdbms.rsf, 11.2.0.4.0...

Patching component oracle.rdbms.dbscripts, 11.2.0.4.0...

Patching component oracle.sdo.locator, 11.2.0.4.0...

Patching component oracle.nlsrtl.rsf, 11.2.0.4.0...

Patching component oracle.xdk.rsf, 11.2.0.4.0...

Patching component oracle.rdbms.rman, 11.2.0.4.0...
Applying sub-patch '18031668' to OH '/data01/app/11.2.0/grid_11204'
ApplySession: Optional component(s) [ oracle.precomp.common, 11.2.0.4.0 ] not present in the Oracle Home or a higher version is found.

Patching component oracle.rdbms, 11.2.0.4.0...

Patching component oracle.rdbms.rsf, 11.2.0.4.0...

Patching component oracle.ldap.rsf, 11.2.0.4.0...

Patching component oracle.rdbms.crs, 11.2.0.4.0...

Patching component oracle.ldap.rsf.ic, 11.2.0.4.0...

Patching component oracle.rdbms.deconfig, 11.2.0.4.0...

Patching component oracle.rdbms.dbscripts, 11.2.0.4.0...

Patching component oracle.rdbms.rman, 11.2.0.4.0...
Applying sub-patch '18522509' to OH '/data01/app/11.2.0/grid_11204'
ApplySession: Optional component(s) [ oracle.precomp.common, 11.2.0.4.0 ] not present in the Oracle Home or a higher version is found.

Patching component oracle.rdbms.rsf, 11.2.0.4.0...

Patching component oracle.rdbms, 11.2.0.4.0...

Patching component oracle.rdbms.rman, 11.2.0.4.0...

Patching component oracle.rdbms.dbscripts, 11.2.0.4.0...

Patching component oracle.rdbms.deconfig, 11.2.0.4.0...
Applying sub-patch '19121551' to OH '/data01/app/11.2.0/grid_11204'
ApplySession: Optional component(s) [ oracle.precomp.common, 11.2.0.4.0 ] , [ oracle.sysman.console.db, 11.2.0.4.0 ] not present in the Oracle Home or a higher version is found.

Patching component oracle.rdbms.rsf, 11.2.0.4.0...

Patching component oracle.rdbms.rman, 11.2.0.4.0...

Patching component oracle.rdbms, 11.2.0.4.0...

Patching component oracle.rdbms.dbscripts, 11.2.0.4.0...

Patching component oracle.ordim.client, 11.2.0.4.0...

Patching component oracle.ordim.jai, 11.2.0.4.0...
Applying sub-patch '19769489' to OH '/data01/app/11.2.0/grid_11204'
ApplySession: Optional component(s) [ oracle.precomp.common, 11.2.0.4.0 ] , [ oracle.ovm, 11.2.0.4.0 ] , [ oracle.xdk, 11.2.0.4.0 ] , [ oracle.oraolap, 11.2.0.4.0 ] , [ oracle.sysman.agent, 11.2.0.4.0 ] not present in the Oracle Home or a higher version is found.

Patching component oracle.rdbms.util, 11.2.0.4.0...

Patching component oracle.rdbms, 11.2.0.4.0...

Patching component oracle.rdbms.dbscripts, 11.2.0.4.0...

Patching component oracle.xdk.parser.java, 11.2.0.4.0...

Patching component oracle.rdbms.rsf, 11.2.0.4.0...

Patching component oracle.xdk.rsf, 11.2.0.4.0...

Patching component oracle.rdbms.rman, 11.2.0.4.0...

Patching component oracle.rdbms.deconfig, 11.2.0.4.0...
Applying sub-patch '20299013' to OH '/data01/app/11.2.0/grid_11204'
ApplySession: Optional component(s) [ oracle.rdbms.dv, 11.2.0.4.0 ] , [ oracle.rdbms.oci, 11.2.0.4.0 ] , [ oracle.precomp.common, 11.2.0.4.0 ] , [ oracle.sysman.agent, 10.2.0.4.5 ] , [ oracle.xdk, 11.2.0.4.0 ] , [ oracle.sysman.console.db, 11.2.0.4.0 ] not present in the Oracle Home or a higher version is found.

Patching component oracle.sysman.common, 10.2.0.4.5...

Patching component oracle.rdbms, 11.2.0.4.0...

Patching component oracle.rdbms.dbscripts, 11.2.0.4.0...

Patching component oracle.xdk.parser.java, 11.2.0.4.0...

Patching component oracle.xdk.rsf, 11.2.0.4.0...

Patching component oracle.rdbms.rsf, 11.2.0.4.0...

Patching component oracle.sysman.common.core, 10.2.0.4.5...

Patching component oracle.rdbms.rman, 11.2.0.4.0...

Patching component oracle.rdbms.deconfig, 11.2.0.4.0...
Applying sub-patch '20760982' to OH '/data01/app/11.2.0/grid_11204'
ApplySession: Optional component(s) [ oracle.sysman.console.db, 11.2.0.4.0 ] not present in the Oracle Home or a higher version is found.

Patching component oracle.rdbms, 11.2.0.4.0...

Patching component oracle.rdbms.dbscripts, 11.2.0.4.0...

Patching component oracle.rdbms.rsf, 11.2.0.4.0...
Applying sub-patch '21352635' to OH '/data01/app/11.2.0/grid_11204'
ApplySession: Optional component(s) [ oracle.sysman.agent, 10.2.0.4.5 ] not present in the Oracle Home or a higher version is found.

Patching component oracle.rdbms.rsf, 11.2.0.4.0...

Patching component oracle.rdbms.rman, 11.2.0.4.0...

Patching component oracle.rdbms, 11.2.0.4.0...
Applying sub-patch '21948347' to OH '/data01/app/11.2.0/grid_11204'
ApplySession: Optional component(s) [ oracle.sysman.agent, 10.2.0.4.5 ] , [ oracle.ovm, 11.2.0.4.0 ] , [ oracle.xdk, 11.2.0.4.0 ] , [ oracle.tfa, 11.2.0.4.0 ] , [ oracle.sysman.console.db, 11.2.0.4.0 ] , [ oracle.sysman.oms.core, 10.2.0.4.5 ] not present in the Oracle Home or a higher version is found.

Patching component oracle.rdbms, 11.2.0.4.0...

Patching component oracle.nlsrtl.rsf, 11.2.0.4.0...

Patching component oracle.xdk.parser.java, 11.2.0.4.0...

Patching component oracle.xdk.rsf, 11.2.0.4.0...

Patching component oracle.rdbms.rsf, 11.2.0.4.0...
Applying sub-patch '22502456' to OH '/data01/app/11.2.0/grid_11204'
ApplySession: Optional component(s) [ oracle.precomp.common, 11.2.0.4.0 ] , [ oracle.tfa, 11.2.0.4.0 ] , [ oracle.rdbms.olap, 11.2.0.4.0 ] , [ oracle.oraolap, 11.2.0.4.0 ] not present in the Oracle Home or a higher version is found.

Patching component oracle.oraolap.dbscripts, 11.2.0.4.0...

Patching component oracle.rdbms.rsf, 11.2.0.4.0...

Patching component oracle.rdbms.rman, 11.2.0.4.0...

Patching component oracle.rdbms, 11.2.0.4.0...

Patching component oracle.rdbms.dbscripts, 11.2.0.4.0...
Applying sub-patch '23054359' to OH '/data01/app/11.2.0/grid_11204'
ApplySession: Optional component(s) [ oracle.rdbms.dv, 11.2.0.4.0 ] not present in the Oracle Home or a higher version is found.

Patching component oracle.rdbms, 11.2.0.4.0...

Patching component oracle.rdbms.dbscripts, 11.2.0.4.0...

Patching component oracle.rdbms.rsf, 11.2.0.4.0...
Composite patch 23054359 successfully applied.
Log file location: /data01/app/11.2.0/grid_11204/cfgtoollogs/opatch/opatch2016-10-01_07-14-41AM_1.log

OPatch succeeded.

[grid@node2 ~]$ opatch apply -oh /data01/app/11.2.0/grid_11204 -local /data01/PSU_PATCH/23274134/23054319
Oracle Interim Patch Installer version 11.2.0.3.15
Copyright (c) 2016, Oracle Corporation. All rights reserved.

Oracle Home : /data01/app/11.2.0/grid_11204
Central Inventory : /data01/app/oraInventory
from : /data01/app/11.2.0/grid_11204/oraInst.loc
OPatch version : 11.2.0.3.15
OUI version : 11.2.0.4.0
Log file location : /data01/app/11.2.0/grid_11204/cfgtoollogs/opatch/opatch2016-10-01_07-30-57AM_1.log

Verifying environment and performing prerequisite checks...
OPatch continues with these patches: 23054319

Do you want to proceed? [y|n]
y
User Responded with: Y
All checks passed.
Provide your email address to be informed of security issues, install and
initiate Oracle Configuration Manager. Easier for you if you use your My
Oracle Support Email address/User Name.
Visit http://www.oracle.com/support/policies.html for details.
Email address/User Name:

You have not provided an email address for notification of security issues.
Do you wish to remain uninformed of security issues ([Y]es, [N]o) [N]: Y

Please shutdown Oracle instances running out of this ORACLE_HOME on the local system.
(Oracle Home = '/data01/app/11.2.0/grid_11204')

Is the local system ready for patching? [y|n]
y
User Responded with: Y
Backing up files...
Applying interim patch '23054319' to OH '/data01/app/11.2.0/grid_11204'

Patching component oracle.crs, 11.2.0.4.0...
Patch 23054319 successfully applied.
Log file location: /data01/app/11.2.0/grid_11204/cfgtoollogs/opatch/opatch2016-10-01_07-30-57AM_1.log

OPatch succeeded.

** Start CRS Services As Root
----------------------------------
# whoami
root

[root@node2 ~]# /data01/app/11.2.0/grid_11204/crs/install/rootcrs.pl -patch
Using configuration parameter file: /data01/app/11.2.0/grid_11204/crs/install/crsconfig_params
Installing Trace File Analyzer
CRS-4123: Oracle High Availability Services has been started.

** Switch to Oracle. Patch Oracle Home:
--------------------------------------

[oracle@node1 23054319]$ which opatch
/data01/app/oracle/product/11.2.0.4/db_1/OPatch/opatch

[oracle@node2 ~]$ opatch apply -oh /data01/app/oracle/product/11.2.0.4/db_1 -local /data01/PSU_PATCH/23274134/23054359
Oracle Interim Patch Installer version 11.2.0.3.15
Copyright (c) 2016, Oracle Corporation. All rights reserved.

Oracle Home : /data01/app/oracle/product/11.2.0.4/db_1
Central Inventory : /data01/app/oraInventory
from : /data01/app/oracle/product/11.2.0.4/db_1/oraInst.loc
OPatch version : 11.2.0.3.15
OUI version : 11.2.0.4.0
Log file location : /data01/app/oracle/product/11.2.0.4/db_1/cfgtoollogs/opatch/opatch2016-10-01_07-51-43AM_1.log

Verifying environment and performing prerequisite checks...
OPatch continues with these patches: 17478514 18031668 18522509 19121551 19769489 20299013 20760982 21352635 21948347 22502456 23054359

Do you want to proceed? [y|n]
y
User Responded with: Y
All checks passed.
Provide your email address to be informed of security issues, install and
initiate Oracle Configuration Manager. Easier for you if you use your My
Oracle Support Email address/User Name.
Visit http://www.oracle.com/support/policies.html for details.
Email address/User Name:

You have not provided an email address for notification of security issues.
Do you wish to remain uninformed of security issues ([Y]es, [N]o) [N]: Y

Please shutdown Oracle instances running out of this ORACLE_HOME on the local system.
(Oracle Home = '/data01/app/oracle/product/11.2.0.4/db_1')

Is the local system ready for patching? [y|n]
y
User Responded with: Y
Backing up files...
Applying sub-patch '17478514' to OH '/data01/app/oracle/product/11.2.0.4/db_1'
Patching component oracle.rdbms, 11.2.0.4.0...
Patching component oracle.rdbms.rsf, 11.2.0.4.0...
Patching component oracle.sdo, 11.2.0.4.0...
Patching component oracle.sysman.agent, 10.2.0.4.5...
Patching component oracle.xdk, 11.2.0.4.0...
Patching component oracle.rdbms.dbscripts, 11.2.0.4.0...
Patching component oracle.sdo.locator, 11.2.0.4.0...
Patching component oracle.nlsrtl.rsf, 11.2.0.4.0...
Patching component oracle.xdk.rsf, 11.2.0.4.0...
Patching component oracle.rdbms.rman, 11.2.0.4.0...
Applying sub-patch '18031668' to OH '/data01/app/oracle/product/11.2.0.4/db_1'
Patching component oracle.rdbms, 11.2.0.4.0...
Patching component oracle.rdbms.rsf, 11.2.0.4.0...
Patching component oracle.ldap.rsf, 11.2.0.4.0...
Patching component oracle.rdbms.crs, 11.2.0.4.0...
Patching component oracle.precomp.common, 11.2.0.4.0...
Patching component oracle.ldap.rsf.ic, 11.2.0.4.0...
Patching component oracle.rdbms.deconfig, 11.2.0.4.0...
Patching component oracle.rdbms.dbscripts, 11.2.0.4.0...
Patching component oracle.rdbms.rman, 11.2.0.4.0...
Applying sub-patch '18522509' to OH '/data01/app/oracle/product/11.2.0.4/db_1'
Patching component oracle.rdbms.rsf, 11.2.0.4.0...
Patching component oracle.rdbms, 11.2.0.4.0...
Patching component oracle.precomp.common, 11.2.0.4.0...
Patching component oracle.rdbms.rman, 11.2.0.4.0...
Patching component oracle.rdbms.dbscripts, 11.2.0.4.0...
Patching component oracle.rdbms.deconfig, 11.2.0.4.0...
Applying sub-patch '19121551' to OH '/data01/app/oracle/product/11.2.0.4/db_1'
Patching component oracle.precomp.common, 11.2.0.4.0...
Patching component oracle.sysman.console.db, 11.2.0.4.0...
Patching component oracle.rdbms.rsf, 11.2.0.4.0...
Patching component oracle.rdbms.rman, 11.2.0.4.0...
Patching component oracle.rdbms, 11.2.0.4.0...
Patching component oracle.rdbms.dbscripts, 11.2.0.4.0...
Patching component oracle.ordim.client, 11.2.0.4.0...
Patching component oracle.ordim.jai, 11.2.0.4.0...
Applying sub-patch '19769489' to OH '/data01/app/oracle/product/11.2.0.4/db_1'
ApplySession: Optional component(s) [ oracle.sysman.agent, 11.2.0.4.0 ] not present in the Oracle Home or a higher version is found.
Patching component oracle.precomp.common, 11.2.0.4.0...
Patching component oracle.ovm, 11.2.0.4.0...
Patching component oracle.xdk, 11.2.0.4.0...
Patching component oracle.rdbms.util, 11.2.0.4.0...
Patching component oracle.rdbms, 11.2.0.4.0...
Patching component oracle.rdbms.dbscripts, 11.2.0.4.0...
Patching component oracle.xdk.parser.java, 11.2.0.4.0...
Patching component oracle.oraolap, 11.2.0.4.0...
Patching component oracle.rdbms.rsf, 11.2.0.4.0...
Patching component oracle.xdk.rsf, 11.2.0.4.0...
Patching component oracle.rdbms.rman, 11.2.0.4.0...
Patching component oracle.rdbms.deconfig, 11.2.0.4.0...
Applying sub-patch '20299013' to OH '/data01/app/oracle/product/11.2.0.4/db_1'
Patching component oracle.rdbms.dv, 11.2.0.4.0...
Patching component oracle.rdbms.oci, 11.2.0.4.0...
Patching component oracle.precomp.common, 11.2.0.4.0...
Patching component oracle.sysman.agent, 10.2.0.4.5...
Patching component oracle.xdk, 11.2.0.4.0...
Patching component oracle.sysman.common, 10.2.0.4.5...
Patching component oracle.rdbms, 11.2.0.4.0...
Patching component oracle.rdbms.dbscripts, 11.2.0.4.0...
Patching component oracle.xdk.parser.java, 11.2.0.4.0...
Patching component oracle.sysman.console.db, 11.2.0.4.0...
Patching component oracle.xdk.rsf, 11.2.0.4.0...
Patching component oracle.rdbms.rsf, 11.2.0.4.0...
Patching component orace.sysman.common.core, 10.2.0.4.5...
Patching component oracle.rdbms.rman, 11.2.0.4.0...
Patching component oracle.rdbms.deconfig, 11.2.0.4.0...
Applying sub-patch '20760982' to OH '/data01/app/oracle/product/11.2.0.4/db_1'
Patching component oracle.sysman.console.db, 11.2.0.4.0...
Patching component oracle.rdbms, 11.2.0.4.0...
Patching component oracle.rdbms.dbscripts, 11.2.0.4.0...
Patching component oracle.rdbms.rsf, 11.2.0.4.0...
Applying sub-patch '21352635' to OH '/data01/app/oracle/product/11.2.0.4/db_1'
Patching component oracle.sysman.agent, 10.2.0.4.5...
Patching component oracle.rdbms.rsf, 11.2.0.4.0...
Patching component oracle.rdbms.rman, 11.2.0.4.0...
Patching component oracle.rdbms, 11.2.0.4.0...
Applying sub-patch '21948347' to OH '/data01/app/oracle/product/11.2.0.4/db_1'
ApplySession: Optional component(s) [ oracle.tfa, 11.2.0.4.0 ] not present in the Oracle Home or a higher version is found.
Patching component oracle.sysman.agent, 10.2.0.4.5...
Patching component oracle.ovm, 11.2.0.4.0...
Patching component oracle.xdk, 11.2.0.4.0...
Patching component oracle.rdbms, 11.2.0.4.0...
Patching componet oracle.nlsrtl.rsf, 11.2.0.4.0...
Patching componnt oracle.xdk.parser.java, 11.2.0.4.0...
Patching component oracle.sysman.console.db, 11.2.0.4.0...
Patching component oracle.xdk.rsf, 11.2.0.4.0...
Patching component oracle.rdbms.rsf, 11.2.0.4.0...
Patching component oracle.sysman.oms.core, 10.2.0.4.5...
Applying sub-patch '22502456' to OH '/data01/app/oracle/product/11.2.0.4/db_1'
ApplySession: Optional component(s) [ oracle.tfa, 11.2.0.4.0 ] not present in the Oracle Home or a higher version is found.
Patching component oracle.precomp.common, 11.2.0.4.0...
Patching component oracle.oraolap.dbscripts, 11.2.0.4.0...
Patching component oracle.rdbms.olap, 11.2.0.4.0...
Patching component oracle.oraolap, 11.2.0.4.0...
Patching component oracle.rdbms.rsf, 11.2.0.4.0...
Patching component oracle.rdbms.rman, 11.2.0.4.0...
Patching component oracle.rdbms, 11.2.0.4.0...
Patching component oracle.rdbms.dbscripts, 11.2.0.4.0...
Applying sub-patch '23054359' to OH '/data01/app/oracle/product/11.2.0.4/db_1'
Patching component oracle.rdbms.dv, 11.2.0.4.0...
Patching component oracle.rdbms, 11.2.0.4.0...
Patching component oracle.rdbms.dbscripts, 11.2.0.4.0...
Patching component oracle.rdbms.rsf, 11.2.0.4.0...
Composite patch 23054359 successfully applied.
Log file location: /data01/app/oracle/product/11.2.0.4/db_1/cfgtoollogs/opatch/opatch2016-10-01_07-51-43AM_1.log

OPatch succeeded.

[oracle@node2 ~]$ opatch apply -oh /data01/app/oracle/product/11.2.0.4/db_1 -local /data01/PSU_PATCH/23274134/23054319/custom/server/23054319
Oracle Interim Patch Installer version 11.2.0.3.15
Copyright (c) 2016, Oracle Corporation. All rights reserved.
Oracle Home : /data01/app/oracle/product/11.2.0.4/db_1
Central Inventory : /data01/app/oraInventory
from : /data01/app/oracle/product/11.2.0.4/db_1/oraInst.loc
OPatch version : 11.2.0.3.15
OUI version : 11.2.0.4.0
Log file location : /data01/app/oracle/product/11.2.0.4/db_1/cfgtoollogs/opatch/opatch2016-10-01_08-28-25AM_1.log
Verifying environment and performing prerequisite checks...
OPatch continues with these patches: 23054319

Do you want to proceed? [y|n]
y
User Responded with: Y
All checks passed.
Provide your email address to be informed of security issues, install and
initiate Oracle Configuration Manager. Easier for you if you use your My
Oracle Support Email address/User Name.
Visit http://www.oracle.com/support/policies.html for details.
Email address/User Name:

You have not provided an email address for notification of security issues.
Do you wish to remain uninformed of security issues ([Y]es, [N]o) [N]: Y

Please shutdown Oracle instances running out of this ORACLE_HOME on the local system.
(Oracle Home = '/data01/app/oracle/product/11.2.0.4/db_1')

Is the local system ready for patching? [y|n]
y
User Responded with: Y
Backing up files...
Applying interim patch '23054319' to OH '/data01/app/oracle/product/11.2.0.4/db_1'
Patching component oracle.rdbms, 11.2.0.4.0...
Patch 23054319 successfully applied.
Log file location: /data01/app/oracle/product/11.2.0.4/db_1/cfgtoollogs/opatch/opatch2016-10-01_08-28-25AM_1.log
OPatch succeeded.

** Start Oracle home and relocate service node2_app back to node2 :

[oracle@node2 ~]$ srvctl start home -o /data01/app/oracle/product/11.2.0.4/db_1 -s /tmp/db_1_stop -n node2
[oracle@node2 ~]$ srvctl start instance -d orcl -i orcl2

[oracle@node2 ~]$ srvctl relocate service -s node2_app -d orcl -i orcl1 -t orcl2

[oracle@node2 ~]$ srvctl status service -d orcl
Service node1_app is running on instance(s) orcl1
Service node2_app is running on instance(s) orcl2
[oracle@node2 ~]$ srvctl status database -d orcl -v
Instance orcl1 is running on node node1 with online services node1_app. Instance status: Open.
Instance orcl2 is running on node node2 with online services node2_app. Instance status: Open.

Remove /tmp file /tmp/db_1_stop

Relocate Scan listeners to proper node as per the configuration before the patching activity :

srvctl relocate scan_listener -i 1 -n node2
srvctl relocate scan_listener -i 2 -n node1
srvctl relocate scan_listener -i 3 -n node1

[oracle@node2 ~]$ srvctl status scan_listener
SCAN Listener LISTENER_SCAN1 is enabled
SCAN listener LISTENER_SCAN1 is running on node node2
SCAN Listener LISTENER_SCAN2 is enabled
SCAN listener LISTENER_SCAN2 is running on node node1
SCAN Listener LISTENER_SCAN3 is enabled
SCAN listener LISTENER_SCAN3 is running on node node1

Remount ACFS file systems :

[root@node1 ~]# /sbin/acfsutil registry
Mount Object:
Device: /dev/asm/asmvol-260
Mount Point: /data01/app/grid/acfsmounts/asmvol_asmvol
Disk Group: ASMVOL
Volume: ASMVOL
Options: none
Nodes: all
[root@node1 ~]# /bin/mount /data01/app/grid/acfsmounts/asmvol_asmvol
[root@node1 ~]# /sbin/acfsutil info fs
/data01/app/grid/acfsmounts/asmvol_asmvol
ACFS Version: 11.2.0.4.0
flags: MountPoint,Available
mount time: Sat Oct 1 05:45:56 2016
volumes: 1
total size: 5167382528
total free: 5009920000
primary volume: /dev/asm/asmvol-260
label:
flags: Primary,Available,ADVM
on-disk version: 39.0
allocation unit: 4096
major, minor: 251, 133121
size: 5167382528
free: 5009920000
ADVM diskgroup ASMVOL
ADVM resize increment: 33554432
ADVM redundancy: unprotected
ADVM stripe columns: 4
ADVM stripe width: 131072
number of snapshots: 0
snapshot space usage: 0
replication status: DISABLED

Verify the mount status on node2 also :

[root@node2 ~]# /sbin/acfsutil info fs
/data01/app/grid/acfsmounts/asmvol_asmvol
ACFS Version: 11.2.0.4.0
flags: MountPoint,Available
mount time: Sat Oct 1 07:44:35 2016
volumes: 1
total size: 5167382528
total free: 5009920000
primary volume: /dev/asm/asmvol-260
label:
flags: Primary,Available,ADVM
on-disk version: 39.0
allocation unit: 4096
major, minor: 251, 133121
size: 5167382528
free: 5009920000
ADVM diskgroup ASMVOL
ADVM resize increment: 33554432
ADVM redundancy: unprotected
ADVM stripe columns: 4
ADVM stripe width: 131072
number of snapshots: 0
snapshot space usage: 0
replication status: DISABLED

Patch Post-Installation Instructions

For each database instance running on the Oracle home being patched, connect to the database using SQL*Plus. Connect as SYSDBA and run the catbundle.sql script as follows:

cd $ORACLE_HOME/rdbms/admin
sqlplus /nolog
SQL> CONNECT / AS SYSDBA
SQL> STARTUP
SQL> @catbundle.sql psu apply
SQL> QUIT

SQL> select action , version , id , comments from dba_registry_history ;

ACTION VERSION ID COMMENTS
------------------------------ ---------- ---------- --------------------
APPLY 11.2.0.4 0 Patchset 11.2.0.2.0
APPLY 11.2.0.4 0 Patchset 11.2.0.2.0
APPLY 11.2.0.4 160719 PSU 11.2.0.4.160719

SQL> select count(*) from dba_objects where status='INVALID' ;

COUNT(*)
----------
0

--------> upgrade rman catalog for each databas

rman target username/***** CATALOG rman_user/******@ORCL

RMAN> upgrade catalog;

Wednesday, 21 September 2016

Create and Monitor ACFS using cmd

1. Configure a initialization script to load the ACFS modules at startup.

[root@node1 ~]# vi /etc/init.d/acfsload
#!/bin/sh

# chkconfig: 2345 30 21
# description: Load Oracle ACFS drivers at system boot
/data01/app/11.2.0/grid_1/bin/acfsload start -s

[root@node1 ~]# chmod u+x /etc/init.d/acfsload

[root@node1 ~]# chkconfig --list acfsload
acfsload 0:off 1:off 2:on 3:on 4:on 5:on 6:off

[root@node1 ~]# /data01/app/11.2.0/grid_1/bin/acfsload start -s

[root@node1 ~]# lsmod | grep -i oracle
oracleacfs 1976568 0
oracleadvm 243152 0
oracleoks 426742 2 oracleacfs,oracleadvm
oracleasm 53865 1

2. Create diskgroups for acfs volumes :

SQL> CREATE DISKGROUP ACFS_DG1 EXTERNAL REDUNDANCY DISK 'ORCL:DISK3'
ATTRIBUTE 'compatible.asm' = '11.2.0.2.0', 'compatible.rdbms' = '11.2.0.2.0', 'compatible.advm' = '11.2.0.2.0' ;

Diskgroup created.

SQL> CREATE DISKGROUP ACFS_DG2 EXTERNAL REDUNDANCY DISK 'ORCL:DISK1'
ATTRIBUTE 'compatible.asm' = '11.2.0.2.0', 'compatible.rdbms' = '11.2.0.2.0', 'compatible.advm' = '11.2.0.2.0' ;

Diskgroup created.

SQL> select name,STATE , TYPE , TOTAL_MB , FREE_MB from v$asm_diskgroup where name in ('ACFS_DG1' ,'ACFS_DG2' );

NAME STATE TYPE TOTAL_MB FREE_MB
-------------------- --------------- -------------------- ---------- ----------
ACFS_DG1 MOUNTED EXTERN 5114 5064
ACFS_DG2 MOUNTED EXTERN 3067 3017

SQL> select name,compatibility, database_compatibility from v$asm_diskgroup where name in ('ACFS_DG1' ,'ACFS_DG2' );;

NAME COMPATIBILITY DATABASE_COMPATIBILITY
------------ -------------------- -------------------------------------
ACFS_DG1 11.2.0.2.0 11.2.0.2.0
ACFS_DG2 11.2.0.2.0 11.2.0.2.0

3. Create/Add volumes to the diskgroups :

SQL> alter diskgroup ACFS_DG1 add volume 'ACFS_vol1' size 5000M;

Diskgroup altered.

SQL> alter diskgroup ACFS_DG2 add volume 'ACFS_vol2' size 3000M;

Diskgroup altered.

4. Enable the Volumes

SQL> alter diskgroup ACFS_DG2 enable volume 'ACFS_vol2';

Diskgroup altered.

SQL> alter diskgroup ACFS_DG1 enable volume 'ACFS_vol1' ;

Diskgroup altered.

SQL> SELECT dg.name AS diskgroup, v.volume_name, v.volume_device, v.mountpath,v.state FROM V$ASM_DISKGROUP dg, V$ASM_VOLUME v WHERE dg.group_number = v.group_number;

DISKGROUP   VOLUME_NAME VOLUME_DEVICE MOUNTPATH STATE
------------------------------ --------------- ------------------------------ ---------- ------------------------
ACFS_DG1   ACFS_VOL1 /dev/asm/acfs_vol1-454 ENABLED
ACFS_DG2   ACFS_VOL2 /dev/asm/acfs_vol2-444   ENABLED

volinfo -G ASMCMD> volinfo -G ACFS_DG1 ACFS_VOL1
Diskgroup Name: ACFS_DG1

Volume Name: ACFS_VOL1
Volume Device: /dev/asm/acfs_vol1-454
State: ENABLED
Size (MB): 5024
Resize Unit (MB): 32
Redundancy: UNPROT
Stripe Columns: 4
Stripe Width (K): 128
Usage:
Mountpath:

ASMCMD> volinfo -G ACFS_DG2 ACFS_VOL2
Diskgroup Name: ACFS_DG2

Volume Name: ACFS_VOL2
Volume Device: /dev/asm/acfs_vol2-444
State: ENABLED
Size (MB): 3008
Resize Unit (MB): 32
Redundancy: UNPROT
Stripe Columns: 4
Stripe Width (K): 128
Usage:
Mountpath:

ASMCMD> volinfo -a
Diskgroup Name: ACFS_DG1

Volume Name: ACFS_VOL1
Volume Device: /dev/asm/acfs_vol1-454
State: ENABLED
Size (MB): 5024
Resize Unit (MB): 32
Redundancy: UNPROT
Stripe Columns: 4
Stripe Width (K): 128
Usage:
Mountpath:

Diskgroup Name: ACFS_DG2

Volume Name: ACFS_VOL2
Volume Device: /dev/asm/acfs_vol2-444
State: ENABLED
Size (MB): 3008
Resize Unit (MB): 32
Redundancy: UNPROT
Stripe Columns: 4
Stripe Width (K): 128
Usage:
Mountpath:

5. Register mount point

[root@node1 ~]# mkdir -p /acfs_mount/acfs_vol1_DG1

[root@node1 ~]# acfsutil registry -a -f /dev/asm/acfs_vol1-454 /acfs_mount/acfs_vol1_DG1
acfsutil registry: mount point /acfs_mount/acfs_vol1_DG1 successfully added to Oracle Registry

[root@node1 ~]# mkdir -p /acfs_mount/acfs_vol2_DG2
[root@node1 ~]# acfsutil registry -a -f /dev/asm/acfs_vol2-444 /acfs_mount/acfs_vol2_DG2
acfsutil registry: mount point /acfs_mount/acfs_vol2_DG2 successfully added to Oracle Registry

[root@node1 ~]# df -kh
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/vg_node1-lv_root
28G 16G 11G 61% /
tmpfs 1004M 326M 679M 33% /dev/shm
/dev/sda1 485M 55M 405M 12% /boot
Softwares 84G 11G 73G 13% /media/sf_Softwares
[root@node1 ~]#

6. Now mount the acfs_mounts created above :

[root@node1 ~]# mkfs.acfs -f /dev/asm/acfs_vol1-454
mkfs.acfs: version = 11.2.0.4.0
mkfs.acfs: on-disk version = 39.0
mkfs.acfs: volume = /dev/asm/acfs_vol1-454
mkfs.acfs: volume size = 5268045824
mkfs.acfs: Format complete.

[root@node1 ~]# mkfs.acfs -f /dev/asm/acfs_vol2-444
mkfs.acfs: version = 11.2.0.4.0
mkfs.acfs: on-disk version = 39.0
mkfs.acfs: volume = /dev/asm/acfs_vol2-444
mkfs.acfs: volume size = 3154116608
mkfs.acfs: Format complete.

[root@node1 ~]# mount.acfs /dev/asm/acfs_vol1-454 /acfs_mount/acfs_vol1_DG1

[root@node1 ~]# mount.acfs /dev/asm/acfs_vol2-444 /acfs_mount/acfs_vol2_DG2

[root@node1 ~]# df -kh
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/vg_node1-lv_root
28G 16G 11G 61% /
tmpfs 1004M 326M 679M 33% /dev/shm
/dev/sda1 485M 55M 405M 12% /boot
Softwares 84G 11G 73G 13% /media/sf_Softwares
/dev/asm/acfs_vol1-454
5.0G 49M 4.9G 1% /acfs_mount/acfs_vol1_DG1
/dev/asm/acfs_vol2-444
3.0G 45M 2.9G 2% /acfs_mount/acfs_vol2_DG2

[root@node1 ~]# chown -R oracle:oinstall /acfs_vol1/
[root@node1 ~]# chown -R oracle:oinstall /acfs_vol2/
[root@node1 ~]# chmod -R 775 /acfs_vol1/
[root@node1 ~]# chmod -R 775 /acfs_vol2/

7. Verify the configuration

ASMCMD> volinfo -a
Diskgroup Name: ACFS_DG1

Volume Name: ACFS_VOL1
Volume Device: /dev/asm/acfs_vol1-454
State: ENABLED
Size (MB): 5024
Resize Unit (MB): 32
Redundancy: UNPROT
Stripe Columns: 4
Stripe Width (K): 128
Usage: ACFS
Mountpath: /acfs_mount/acfs_vol1_DG1

Diskgroup Name: ACFS_DG2

Volume Name: ACFS_VOL2
Volume Device: /dev/asm/acfs_vol2-444
State: ENABLED
Size (MB): 3008
Resize Unit (MB): 32
Redundancy: UNPROT
Stripe Columns: 4
Stripe Width (K): 128
Usage: ACFS
Mountpath: /acfs_mount/acfs_vol2_DG2

SQL>SELECT dg.name AS diskgroup, v.volume_name, v.volume_device, v.mountpath,v.state FROM V$ASM_DISKGROUP dg, V$ASM_VOLUME v WHERE dg.group_number = v.group_number;

DISKGROUP VOLUME_NAM VOLUME_DEVICE MOUNTPATH STATE
---------- ---------- ------------------------------ ------------------------------ ---------------
ACFS_DG1 ACFS_VOL1 /dev/asm/acfs_vol1-454 /acfs_mount/acfs_vol1_DG1 ENABLED
ACFS_DG2 ACFS_VOL2 /dev/asm/acfs_vol2-444 /acfs_mount/acfs_vol2_DG2 ENABLED

Tuesday, 20 September 2016

How to find LUN assigned to Oracle ASM Library

[root@node1 ~]# oracleasm listdisks
DISK1
DISK2
DISK3
DISK4

[root@node1 ~]# /etc/init.d/oracleasm querydisk -p DISK1
Disk "DISK1" is a valid ASM disk
/dev/sdb1: LABEL="DISK1" TYPE="oracleasm"
[root@node1 ~]# /etc/init.d/oracleasm querydisk -p DISK2
Disk "DISK2" is a valid ASM disk
/dev/sdc1: LABEL="DISK2" TYPE="oracleasm"
[root@node1 ~]# /etc/init.d/oracleasm querydisk -p DISK3
Disk "DISK3" is a valid ASM disk
/dev/sdd1: LABEL="DISK3" TYPE="oracleasm"
[root@node1 ~]# /etc/init.d/oracleasm querydisk -p DISK4
Disk "DISK4" is a valid ASM disk
/dev/sde1: LABEL="DISK4" TYPE="oracleasm"

Saturday, 17 September 2016

Configure DG BROKER 12c

Primary Database :
DB_NAME DATABASE_ROLE DB_UNIQUE_NAME VERSION
---------------- -------------------- ---------------- --------------- --------------- ----------
orcl PRIMARY orcl 12.1.0.2.0

Standby Database :
DB_NAME DATABASE_ROLE DB_UNIQUE_NAME VERSION
---------------- -------------------- ---------------- --------------- --------------- -------
orcl PHYSICAL STANDBY orcl_dg 12.1.0.2.0

1. . On both primary and standby set db_broker_start to TRUE

SQL> alter system set dg_broker_start=true scope=both;
System altered.

SQL> show parameter dg_broker_start ;

NAME TYPE VALUE
------------------------------------ ----------- ------------------------------
dg_broker_start boolean TRUE

Step 2 : Edit the listener.ora file on both primary and standby :

Include the db_unique_name_DGMGRL.db_domain values for the GLOBAL_DBNAME in both primary and standby database .
To set the value, lets check the db_domain and db_unique_name value .

primary :
============

SQL> show parameter db_unique_name ;

NAME TYPE VALUE
------------------------------------ ----------- ------------------------------
db_unique_name string orcl

SQL> show parameter db_domain ;

NAME TYPE VALUE
------------------------------------ ----------- ------------------------------
db_domain string

Since the value of db_domain is null so the the value of GLOBAL_DBNAME = orcl_DGMGRL for primary database

[grid@node1 ~]$ cat /data01/app/grid/product/12.1.0/grid/network/admin/listener.ora

SID_LIST_LISTENER =
(SID_LIST =
(SID_DESC =
(GLOBAL_DBNAME = orcl)
(ORACLE_HOME = /data01/app/oracle/product/12c/db_1)
(SID_NAME = orcl)
)
(SID_DESC =
(GLOBAL_DBNAME = orcl_DGMGRL)
(ORACLE_HOME = /data01/app/oracle/product/12c/db_1)
(SID_NAME = orcl)
)
)

LISTENER =
(DESCRIPTION_LIST =
(DESCRIPTION =
(ADDRESS = (PROTOCOL = TCP)(HOST = node1.localdomain)(PORT = 1521))
(ADDRESS = (PROTOCOL = IPC)(KEY = EXTPROC1521))
)
)

standby :
=========
SQL> show parameter db_unique_name ;

NAME TYPE VALUE
------------------------------------ ----------- ------------------------------
db_unique_name string orcl_dg

SQL> show parameter db_domain ;

NAME TYPE VALUE
------------------------------------ ----------- ------------------------------
db_domain string

Since the value of db_domain is null so GLOBAL_DBNAME = orcl_dg_DGMGRL for standby database .

[grid@node2 ~]$ cat /data01/app/grid/product/12.1.0/grid/network/admin/listener.ora

SID_LIST_LISTENER =
(SID_LIST =
(SID_DESC =
(GLOBAL_DBNAME = orcl)
(ORACLE_HOME = /data01/app/oracle/product/12c/db_1)
(SID_NAME = orcl)
)
(SID_DESC =
(GLOBAL_DBNAME = orcl_dg_DGMGRL)
(ORACLE_HOME = /data01/app/oracle/product/12c/db_1)
(SID_NAME = orcl)
)
)

LISTENER =
(DESCRIPTION_LIST =
(DESCRIPTION =
(ADDRESS = (PROTOCOL = TCP)(HOST = node2.localdomain)(PORT = 1521))
(ADDRESS = (PROTOCOL = IPC)(KEY = EXTPROC1521))
)
)

Configure the Data Guard Configuration

DGMGRL> connect sys/******@orcl
Connected as SYSDBA.
DGMGRL>
DGMGRL> CREATE CONFIGURATION 'ORCLCONF' AS PRIMARY DATABASE IS 'ORCL' CONNECT IDENTIFIER IS ORCL;
Error: ORA-16698: LOG_ARCHIVE_DEST_n parameter set for object to be added

Failed.

Reason : (For 12.1, for SERVICE= for Remote lad's if the NOREGISTER attribute is Not set, then you must remove/clear the lad before creating the broker configuration. Doc ID 1582179.1)

SQL> show parameter log_Archive_dest_2 ;

NAME TYPE VALUE
------------------------------------ ----------- ----------------------------------------------------------------------------------
log_archive_dest_2 string SERVICE=orcl_dg VALID_FOR=(ONLINE_LOGFILES,PRIMARY_ROLE) DB_UNIQUE_NAME=orcl_dg

SQL> alter system set log_Archive_dest_2='';

System altered.

SQL> show parameter log_Archive_dest_2 ;

NAME TYPE VALUE
------------------------------------ ----------- ------------------------------
log_archive_dest_2 string

DGMGRL> connect sys/****@orcl
Connected as SYSDBA.

DGMGRL> CREATE CONFIGURATION 'ORCLCONF' AS PRIMARY DATABASE IS 'orcl' CONNECT IDENTIFIER IS orcl;
Configuration "ORCLCONF" created with primary database "orcl"

DGMGRL> ADD DATABASE 'orcl_dg' AS CONNECT IDENTIFIER IS orcl_dg MAINTAINED AS PHYSICAL ;
Database "orcl_dg" added

DGMGRL> show configuration

Configuration - ORCLCONF
Protection Mode: MaxPerformance
Members:
orcl - Primary database
orcl_dg - Physical standby database
Fast-Start Failover: DISABLED
Configuration Status:
DISABLED
DGMGRL> enable configuration
Enabled.

DGMGRL> show configuration

Configuration - ORCLCONF
Protection Mode: MaxPerformance
Members:
orcl - Primary database
orcl_dg - Physical standby database
Fast-Start Failover: DISABLED
Configuration Status:
SUCCESS (status updated 10 seconds ago)

DGMGRL> show database verbose orcl

Database - orcl

Role: PRIMARY
Intended State: TRANSPORT-ON
Instance(s):
orcl
Properties:
DGConnectIdentifier = 'orcl'
ObserverConnectIdentifier = ''
LogXptMode = 'ASYNC'
RedoRoutes = ''
DelayMins = '0'
Binding = 'optional'
MaxFailure = '0'
MaxConnections = '1'
ReopenSecs = '300'
NetTimeout = '30'
RedoCompression = 'DISABLE'
LogShipping = 'ON'
PreferredApplyInstance = ''
ApplyInstanceTimeout = '0'
ApplyLagThreshold = '0'
TransportLagThreshold = '0'
TransportDisconnectedThreshold = '30'
ApplyParallel = 'AUTO'
StandbyFileManagement = 'AUTO'
ArchiveLagTarget = '0'
LogArchiveMaxProcesses = '4'
LogArchiveMinSucceedDest = '1'
DbFileNameConvert = ''
LogFileNameConvert = ''
FastStartFailoverTarget = ''
InconsistentProperties = '(monitor)'
InconsistentLogXptProps = '(monitor)'
SendQEntries = '(monitor)'
LogXptStatus = '(monitor)'
RecvQEntries = '(monitor)'
StaticConnectIdentifier = '(DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=192.168.56.73)(PORT=1521))(CONNECT_DATA=(SERVICE_NAME=orcl_DGMGRL)(INSTANCE_NAME=orcl)(SERVER=DEDICATED)))'
StandbyArchiveLocation = '+DATA'
AlternateLocation = ''
LogArchiveTrace = '0'
LogArchiveFormat = '%t_%s_%r.dbf'
TopWaitEvents = '(monitor)'

Database Status:
SUCCESS

DGMGRL> show database verbose orcl_dg

Database - orcl_dg
Role: PHYSICAL STANDBY
Intended State: APPLY-ON
Transport Lag: 0 seconds (computed 0 seconds ago)
Apply Lag: 0 seconds (computed 0 seconds ago)
Average Apply Rate: 9.00 KByte/s
Active Apply Rate: 24.00 KByte/s
Maximum Apply Rate: 247.00 KByte/s
Real Time Query: OFF
Instance(s):
orcl
Properties:
DGConnectIdentifier = 'orcl_dg'
ObserverConnectIdentifier = ''
LogXptMode = 'ASYNC'
RedoRoutes = ''
DelayMins = '0'
Binding = 'optional'
MaxFailure = '0'
MaxConnections = '1'
ReopenSecs = '300'
NetTimeout = '30'
RedoCompression = 'DISABLE'
LogShipping = 'ON'
PreferredApplyInstance = ''
ApplyInstanceTimeout = '0'
ApplyLagThreshold = '0'
TransportLagThreshold = '0'
TransportDisconnectedThreshold = '30'
ApplyParallel = 'AUTO'
StandbyFileManagement = 'AUTO'
ArchiveLagTarget = '0'
LogArchiveMaxProcesses = '4'
LogArchiveMinSucceedDest = '1'
DbFileNameConvert = '+DATA, +DATA2'
LogFileNameConvert = '+DATA, +DATA2'
FastStartFailoverTarget = ''
InconsistentProperties = '(monitor)'
InconsistentLogXptProps = '(monitor)'
SendQEntries = '(monitor)'
LogXptStatus = '(monitor)'
RecvQEntries = '(monitor)'
StaticConnectIdentifier = '(DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=node2.localdomain)(PORT=1521))(CONNECT_DATA=(SERVICE_NAME=orcl_dg_DGMGRL)(INSTANCE_NAME=orcl)(SERVER=DEDICATED)))'
StandbyArchiveLocation = '+DATA'
AlternateLocation = ''
LogArchiveTrace = '0'
LogArchiveFormat = '%t_%s_%r.dbf'
TopWaitEvents = '(monitor)'

Database Status:
SUCCESS

DGMGRL> validate database verbose orcl

Database Role: Primary database
Ready for Switchover: Yes
Capacity Information:
Database Instances Threads
orcl 1 1
Temporary Tablespace File Information:
orcl TEMP Files: 1
Flashback Database Status:
orcl: Off
Data file Online Move in Progress:
orcl: No
Transport-Related Information:
Transport On: Yes
Log Files Cleared:
orcl Standby Redo Log Files: Cleared
Automatic Diagnostic Repository Errors:
Error orcl
No logging operation NO
Control file corruptions NO
System data file missing NO
System data file corrupted NO
System data file offline NO
User data file missing NO
User data file corrupted NO
User data file offline NO
Block Corruptions found NO

DGMGRL> validate database verbose orcl_dg

Database Role: Physical standby database
Primary Database: orcl
Ready for Switchover: Yes
Ready for Failover: Yes (Primary Running)
Capacity Information:
Database Instances Threads
orcl 1 1
orcl_dg 1 1
Temporary Tablespace File Information:
orcl TEMP Files: 1
orcl_dg TEMP Files: 1
Flashback Database Status:
orcl: Off
orcl_dg: Off
Data file Online Move in Progress:
orcl: No
orcl_dg: No
Standby Apply-Related Information:
Apply State: Running
Apply Lag: 0 seconds (computed 1 second ago)
Apply Delay: 0 minutes
Transport-Related Information:
Transport On: Yes
Gap Status: No Gap
Transport Lag: 0 seconds (computed 1 second ago)
Transport Status: Success
Log Files Cleared:
orcl Standby Redo Log Files: Cleared
orcl_dg Online Redo Log Files: Cleared
orcl_dg Standby Redo Log Files: Available
Current Log File Groups Configuration:
Thread # Online Redo Log Groups Standby Redo Log Groups Status
(orcl) (orcl_dg)
1 3 2 Insufficient SRLs
Future Log File Groups Configuration:
Thread # Online Redo Log Groups Standby Redo Log Groups Status
(orcl_dg) (orcl)
1 3 0 Insufficient SRLs
Warning: standby redo logs not configured for thread 1 on orcl

Current Configuration Log File Sizes:
Thread # Smallest Online Redo Smallest Standby Redo
Log File Size Log File Size
(orcl) (orcl_dg)
1 50 MBytes 50 MBytes
Apply-Related Property Settings:
Property orcl Value orcl_dg Value
DelayMins 0 0
ApplyParallel AUTO AUTO
Transport-Related Property Settings:
Property orcl Value orcl_dg Value
LogXptMode ASYNC ASYNC
RedoRoutes <empty> <empty>
Dependency <empty> <empty>
DelayMins 0 0
Binding optional optional
MaxFailure 0 0
MaxConnections 1 1
ReopenSecs 300 300
NetTimeout 30 30
RedoCompression DISABLE DISABLE
LogShipping ON ON
Automatic Diagnostic Repository Errors:
Error orcl orcl_dg
No logging operation NO NO
Control file corruptions NO NO
SRL Group Unavailable NO NO
System data file missing NO NO
System data file corrupted NO NO
System data file offline NO NO
User data file missing NO NO
User data file corrupted NO NO
User data file offline NO NO
Block Corruptions found NO NO

How to set up data guard broker for RAC

https://martincarstenbach.wordpress.com/2012/06/29/how-to-set-up-data-guard-broker-for-rac/

Saturday, 10 September 2016

RMAN duplicate database having multiple directories to assign to BACKUP LOCATION

By default RMAN picks one backup location to do duplicate with clause 'BACKUP LOCATION' .

Many times user don't have much space in single location ,hence, the backup piece distributed

to different location and at same time want to use 'BACKUP LOCATION' feature for duplicate.

How to perform the RMAN duplication using the BACKUP LOCATION clause when the RMAN backuppieces are stored on several mount points or directories?

1/ Create the softlink on all of the individual RMAN backuppieces required for the RMAN duplicate, as shown in the example here. You may change the '/tmp/rmanbkup' on a different directory where you want to create those softlinks.

mkdir /tmp/rmanbkup

ln -s /o/zfssa/backup01/scs/84oqpfif_1_1 /tmp/rmanbkup/84oqpfif_1_1

ln -s /o/zfssa/backup02/scs/87oqpfif_1_1 /tmp/rmanbkup/87oqpfif_1_1

-- You can run the following SQL statements to dynamically generate the Unix command "ln -s" for the

-- individual RMAN backuppieces required for the RMAN duplicate, as shown here.

-- In the following SQL statement, just modify the start date and end date that corresponds to when

-- the backup started and backup completed.

set linesize 200 pagesize 0 verify off echo off feedback off

select 'ln -s ' || handle || ' /tmp/rmanbkup' || substr(handle, instr( handle, '/', -1 ))

from v$backup_piece

where completion_time between to_date ('05-DEC-2013 20:00:00','DD-MON-RRRR HH24:MI:SS')

and to_date('05-DEC-2013 22:00:00','DD-MON-RRRR HH24:MI:SS')

and device_type = 'DISK'

order by handle;

2/ In the RMAN duplicate, the BACKUP LOCATION is '/tmp/rmanbkup', as shown here.

DUPLICATE DATABASE TO gvlt03

SPFILE

SET CONTROL_FILES='+DATA_GXPB'

SET DB_FILE_NAME_CONVERT='+DATA_GXTB','+DATA_GXPB','RECO_GXTB','RECO_GXPB','DBFS_DG','DBFS_DG'

SET LOG_FILE_NAME_CONVERT='+DATA_GXTB','+DATA_GXPB','RECO_GXTB','RECO_GXPB','DBFS_DG','DBFS_DG'

SET DB_CREATE_ONLINE_LOG_DEST_1='+DATA_GXPB'

SET DB_CREATE_ONLINE_LOG_DEST_2='+DATA_GXPB'

SET DB_RECOVERY_FILE_DEST='+RECO_GXPB'

SET instance_number='1'

SET DIAGNOSTIC_DEST='/u01/app/oowner'

set audit_file_dest='/u01/app/oowner/admin/gvlt03/adump'

BACKUP LOCATION '/tmp/rmanbkup'

NOFILENAMECHECK;

Sunday, 4 September 2016

How to check that Database is consistent after incomplete recovery before OPEN RESETLOGS

For cold/offline backups, no archivelogs/recovery is necessary. You can simply open the database with resetlogs.
However for HOT/ONLINE backups, ALL archivelogs from backup start to backup end must be applied before the database can be opened this is the MINIMUM amount of recovery needed.

To determine which log was current at the time the backup completed, note the COMPLETION time of the database backup take this from the backup log.
If this is an RMAN backup you can also query the RMAN metadata. Ensure that the environment variable NLS_DATE_FORMAT is set before invoking rman so that timestamps as well as date are returned:

For unix:
export NLS_DATE_FORMAT='dd-mon-rr hh24:mi:ss'
rman target /

For windows:
set nls_date_format='dd-mon-rr hh24:mi:ss'

SQL> alter session set nls_date_format='DD-MON-YYYY HH24:MI:SS' ;

check 1. Checkpoint Time and Fuzziness

SQL> select fuzzy, status, error, recover, checkpoint_change#, checkpoint_time, count(*) from v$datafile_header group by fuzzy, status, error, recover, checkpoint_change#, checkpoint_time ;

FUZZY STATUS ERROR REC CHECKPOINT_CHANGE# CHECKPOINT_TIME COUNT(*)
-------------------------------------------------------------------------------------
NO ONLINE 5311260 31-AUG-2011 23:10:14 6
YES ONLINE 5311260 31-AUG-2011 23:10:14 1

a) Verify that the checkpoint_time / checkpoint_change# is in line with your intended UNTIL TIME/SCN. If not, recover the database further if you have more archived logs available.
b) If FUZZY=YES for some datafiles, it means more recovery is required.

Above check can be considered PASSED when :
a) Verified that all the datafiles are at the some checkpoint_time, and this is your intended Point in time.
b) Fuzzy=NO for SYSTEM, UNDO and all intended datafiles. For datafiles with Fuzzy=YES, either recover them further or bring them OFFLINE if no further archived logs are available.

Check 2: Absolute Fuzzy

Occasionally, it is possible to see Fuzzy=NO and same checkpoint_change# for all the intended datafiles but OPEN RESETLOGS still fails

select fuzzy, status, error, recover, checkpoint_change#, checkpoint_time, count(*) from v$datafile_header group by fuzzy, status, error, recover, checkpoint_change#, checkpoint_time ;

FUZZY STATUS ERROR REC CHECKPOINT_CHANGE# CHECKPOINT_TIME COUNT(*)
-------------------------------------------------------------------------------------
NO ONLINE 5311260 31-AUG-2011 23:10:14 7

SQL> ALTER DATABASE OPEN RESETLOGS ;
ORA01194:file 4 needs more recovery to be consistent
ORA01110:data file 3: '/u01/app/oracle/oradata/prod111/undotbs02.dbf'

Hence, we should perform additional fuzzy check known as Absolute Fuzzy Check:

SQL> select hxfil file#, substr(hxfnm, 1, 50) name, fhscn checkpoint_change#, fhafs Absolute_Fuzzy_SCN, max(fhafs) over () Min_PIT_SCN from x$kcvfh where fhafs!=0 ;

FILE# NAME CHECKPOINT_CHANGE ABSOLUTE_FUZZY_SCN MIN_PIT_SCN
------------------------------------------------------------------------------------------------------
4 /u01/app/oracle/oradata/prod111/undotbs01.dbf 5311260 5311524 5311524
6 /u01/app/oracle/oradata/prod111/system01.dbf 5311260 5311379 5311524

Note: Column Min_PIT_SCN will return same value even for multiple rows as we have applied ANALYTICAL "MAX() OVER ()" function on it.
Above query indicates that the recovery must be performed at least UNTIL SCN 5311524 to make datafiles consistent and ready to OPEN. Since the checkpoint_change# is smaller than Min_PIT_SCN, the datafiles will ask for more recovery.

Above check can be considered PASSED when:
a) No rows selected from above query (i.e. Min_PIT_SCN is 0 (Zero) for all the datafiles)
b) Min_PIT_SCN is returned less than Checkpoint_Change#

Check 3: Archive Logs Required:

Query the controlfile to find the latest archivelog required for recovery. Lets say the backup completed at 31-AUG-2011 23:20:14:

SQL> V$ARCHIVED_LOG
SQL> ALTER SESSION SET NLS_DATE_FORMAT='DD-MON-RR HH24:MI:SS';
SQL> SELECT THREAD#, SEQUENCE#, FIRST_TIME, NEXT_TIME FROM V$ARCHIVED_LOG WHERE '31-AUG-11 23:20:14' BETWEEN FIRST_TIME AND NEXT_TIME;

If the above query does not return any rows, it may be that the information has aged out of the controlfile run
the following query against v$log_history.

SQL> V$ LOG_HISTORY view does not have a column NEXT_TIME
SQL> ALTER SESSION SET NLS_DATE_FORMAT='DD-MON-RR HH24:MI:SS';
SQL> select a.THREAD#, a.SEQUENCE#, a.FIRST_TIME from V$LOG_HISTORY a
where FIRST_TIME =
( SELECT MAX(b.FIRST_TIME) FROM V$LOG_HISTORY b
WHERE b.FIRST_TIME < to_date('31-AUG-11 23:20:14', 'DD-MON-RR HH24:MI:SS')
) ;

The sequence# returned by the above query is the log sequence current at the time the backup ended let say 530 thread 1.

For minimum recovery use: (Sequence# as returned +1 ) :

RMAN> RUN
{
SET UNTIL SEQUENCE 531 THREAD 1;
RECOVER DATABASE;
}

If this is a RAC implementation the use this SQL instead to query the controlfile:

SQL> SELECT THREAD#, SEQUENCE#, FIRST_CHANGE#, NEXT_CHANGE# FROM V$ARCHIVED_LOG WHERE '31-AUG-11 23:20:14' BETWEEN FIRST_TIME AND NEXT_TIME;

For minimum recovery use the log sequence and thread that has the lowest NEXT_CHANGE# returned by the above query.

Above check can be considered PASSED when:
All archivelogs from the time of the backup to the end of the backup is available for use during recovery

Saturday, 3 September 2016

RMAN Backup Script

cat /backups2/rman_backup.sh

DATESTAMP=`date '+%y%m%d%H%M'`
NLS_DATE_FORMAT='Mon DD YYYY HH24:MI:SS'
export NLS_DATE_FORMAT
export ORACLE_SID=ORCL
export BACKUP_TYPE=LVL0
MSGLOG=backup_${BACKUP_TYPE}_${ORACLE_SID}_${DATESTAMP}.log
export BACKUP_LOCATION=/data01/BACKUP

$ORACLE_HOME/bin/rman msglog $MSGLOG append << EOF
connect target
run
{
allocate channel c1 type disk;
allocate channel c2 type disk;
allocate channel c3 type disk;
allocate channel c4 type disk;
allocate channel c5 type disk;
allocate channel c6 type disk;
allocate channel c7 type disk;
allocate channel c8 type disk;
backup as compressed backupset incremental level 0 format '${BACKUP_LOCATION}/${ORACLE_SID}_${BACKUP_TYPE}_%U.bak' database;
backup archivelog all format '${BACKUP_LOCATION}/${ORACLE_SID}_arch_%U.bak' ;
backup as copy current controlfile format '${BACKUP_LOCATION}/${ORACLE_SID}_Control_%U.bak';
release channel c1;
release channel c2;
release channel c3;
release channel c4;
release channel c5;
release channel c6;
release channel c7;
release channel c8;
}
EOF

Full backup CMD FILE :

nohup rman target / cmdfile=/home/oracle/backup_full.cmd log=/home/oracle/backup_rmanlog &

Tuesday, 9 August 2016

Recover Datafile never backed up

SQL> CREATE TABLESPACE sumit DATAFILE SIZE 150M;

Tablespace created.

SQL> create table sumit_Table (id int ) tablespace sumit;

Table created.

Insert data and check :

SQL> select count(*) from sumit_table ;

COUNT(*)
----------
2048

[oracle@node2 trace]$ strings /data01/app/oracle/oradata/ORCL2/controlfile/o1_mf_cfxpjjql_.ctl | grep -i sumit
/data01/app/oracle/oradata/ORCL2/datafile/o1_mf_sumit_cfxwfx8b_.dbf
/data01/app/oracle/oradata/ORCL2/datafile/o1_mf_sumit_cfxwfx8b_.dbf
SUMIT
SUMIT

[oracle@node2 trace]$ strings /data01/app/oracle/fast_recovery_area/ORCL2/controlfile/o1_mf_cfxpjk7m_.ctl | grep -i sumit
/data01/app/oracle/oradata/ORCL2/datafile/o1_mf_sumit_cfxwfx8b_.dbf
/data01/app/oracle/oradata/ORCL2/datafile/o1_mf_sumit_cfxwfx8b_.dbf
SUMIT
SUMIT

Drop Control files :
==================
[oracle@node2 trace]$ rm /data01/app/oracle/oradata/ORCL2/controlfile/o1_mf_cfxpjjql_.ctl
[oracle@node2 trace]$ rm /data01/app/oracle/fast_recovery_area/ORCL2/controlfile/o1_mf_cfxpjk7m_.ctl

Drop Datafile for tbs sumit :
==============================
[oracle@node2 trace]$ rm /data01/app/oracle/oradata/ORCL2/datafile/o1_mf_sumit_cfxwfx8b_.dbf

[oracle@node2 trace]$ sqlplus / as sysdba

SQL*Plus: Release 12.1.0.1.0 Production on Tue Mar 8 11:19:14 2016

Copyright (c) 1982, 2013, Oracle. All rights reserved.

Connected to:
Oracle Database 12c Enterprise Edition Release 12.1.0.1.0 - 64bit Production
With the Partitioning, OLAP, Advanced Analytics and Real Application Testing options

SQL> select count(*) from sumit_table ;

COUNT(*)
----------
2048

Alert log :
==========

Tue Mar 08 11:18:44 2016
Errors in file /data01/app/oracle/diag/rdbms/orcl2/orcl2/trace/orcl2_m000_16057.trc:
ORA-00210: cannot open the specified control file
ORA-00202: control file: '/data01/app/oracle/oradata/ORCL2/controlfile/o1_mf_cfxpjjql_.ctl'
ORA-27041: unable to open file
Linux-x86_64 Error: 2: No such file or directory
Additional information: 3
Tue Mar 08 11:19:44 2016
Errors in file /data01/app/oracle/diag/rdbms/orcl2/orcl2/trace/orcl2_m000_16076.trc:
ORA-00210: cannot open the specified control file
ORA-00202: control file: '/data01/app/oracle/oradata/ORCL2/controlfile/o1_mf_cfxpjjql_.ctl'
ORA-27041: unable to open file
Linux-x86_64 Error: 2: No such file or directory
Additional information: 3
Tue Mar 08 11:19:50 2016

SQL> shu immediate ;
ORA-00210: cannot open the specified control file
ORA-00202: control file: '/data01/app/oracle/oradata/ORCL2/controlfile/o1_mf_cfxpjjql_.ctl'
ORA-27041: unable to open file
Linux-x86_64 Error: 2: No such file or directory
Additional information: 3
SQL>

SQL> startup nomount ;
ORACLE instance started.

Total System Global Area 1135747072 bytes
Fixed Size 2287528 bytes
Variable Size 788531288 bytes
Database Buffers 335544320 bytes
Redo Buffers 9383936 bytes
SQL>

RMAN> restore controlfile from autobackup ;

Starting restore at 20160308.112954
using target database control file instead of recovery catalog
allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=1 device type=DISK

recovery area destination: /data01/app/oracle/fast_recovery_area
database name (or database unique name) used for search: ORCL2
channel ORA_DISK_1: AUTOBACKUP /data01/app/oracle/fast_recovery_area/ORCL2/autobackup/2016_03_08/o1_mf_s_905942221_cfxwdxkn_.bkp found in the recovery area
AUTOBACKUP search with format "%F" not attempted because DBID was not set
channel ORA_DISK_1: restoring control file from AUTOBACKUP /data01/app/oracle/fast_recovery_area/ORCL2/autobackup/2016_03_08/o1_mf_s_905942221_cfxwdxkn_.bkp
channel ORA_DISK_1: control file restore from AUTOBACKUP complete
output file name=/data01/app/oracle/oradata/ORCL2/controlfile/o1_mf_cfxpjjql_.ctl
output file name=/data01/app/oracle/fast_recovery_area/ORCL2/controlfile/o1_mf_cfxpjk7m_.ctl
Finished restore at 20160308.112956

RMAN> exit;

[oracle@node2 controlfile]$ strings /data01/app/oracle/fast_recovery_area/ORCL2/controlfile/o1_mf_cfxpjk7m_.ctl | grep -i sumit
[oracle@node2 controlfile]$ strings /data01/app/oracle/oradata/ORCL2/controlfile/o1_mf_cfxpjjql_.ctl | grep -i sumit
[oracle@node2 controlfile]$

[oracle@node2 controlfile]$ rman target /

Recovery Manager: Release 12.1.0.1.0 - Production on Tue Mar 8 11:31:44 2016

Copyright (c) 1982, 2013, Oracle and/or its affiliates. All rights reserved.

connected to target database: ORCL2 (not mounted)

RMAN> alter database mount;

using target database control file instead of recovery catalog
Statement processed

RMAN> report schema;

Starting implicit crosscheck backup at 20160308.113157
allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=20 device type=DISK
Crosschecked 4 objects
Finished implicit crosscheck backup at 20160308.113158

Starting implicit crosscheck copy at 20160308.113158
using channel ORA_DISK_1
Finished implicit crosscheck copy at 20160308.113158

searching for all files in the recovery area
cataloging files...
cataloging done

List of Cataloged Files
=======================
File Name: /data01/app/oracle/fast_recovery_area/ORCL2/autobackup/2016_03_08/o1_mf_s_905942221_cfxwdxkn_.bkp
File Name: /data01/app/oracle/fast_recovery_area/ORCL2/archivelog/2016_03_08/o1_mf_1_6_cfxy1r1d_.arc
File Name: /data01/app/oracle/fast_recovery_area/ORCL2/archivelog/2016_03_08/o1_mf_1_7_cfxyx64v_.arc
File Name: /data01/app/oracle/fast_recovery_area/ORCL2/archivelog/2016_03_08/o1_mf_1_8_cfxyxv1m_.arc
File Name: /data01/app/oracle/fast_recovery_area/ORCL2/archivelog/2016_03_08/o1_mf_1_5_cfxy1pxh_.arc

RMAN-06139: WARNING: control file is not current for REPORT SCHEMA
Report of database schema for database with db_unique_name ORCL2

List of Permanent Datafiles
===========================
File Size(MB) Tablespace RB segs Datafile Name
---- -------- -------------------- ------- ------------------------
1 770 SYSTEM *** /data01/app/oracle/oradata/ORCL2/datafile/o1_mf_system_cfxpdor9_.dbf
2 150 TSH_DATA *** /data01/app/oracle/oradata/ORCL2/datafile/o1_mf_tsh_data_cfxvz9do_.dbf
3 700 SYSAUX *** /data01/app/oracle/oradata/ORCL2/datafile/o1_mf_sysaux_cfxpb0jb_.dbf
4 55 UNDOTBS1 *** /data01/app/oracle/oradata/ORCL2/datafile/o1_mf_undotbs1_cfxpj05f_.dbf
6 5 USERS *** /data01/app/oracle/oradata/ORCL2/datafile/o1_mf_users_cfxphz0d_.dbf

List of Temporary Files
=======================
File Size(MB) Tablespace Maxsize(MB) Tempfile Name
---- -------- -------------------- ----------- --------------------
1 60 TEMP 32767 /data01/app/oracle/oradata/ORCL2/datafile/o1_mf_temp_cfxpk4nq_.tmp

The "RMAN-20202: Tablespace not found in the recovery catalog" and "RMAN-06019: could not translate tablespace name "SUMIT"" errors are thrown, because the controlfile used has no information about that tablespace as you can see in the following output:

RMAN> restore tablespace sumit ;

Starting restore at 20160308.113247
using channel ORA_DISK_1
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03002: failure of restore command at 03/08/2016 11:32:47
RMAN-20202: Tablespace not found in the recovery catalog
RMAN-06019: could not translate tablespace name "SUMIT"

To recover the missing and "never backed up" tablespace you must restore and recover the entire database.

RMAN> restore database;

Starting restore at 20160308.113325
using channel ORA_DISK_1

channel ORA_DISK_1: starting datafile backup set restore
channel ORA_DISK_1: specifying datafile(s) to restore from backup set
channel ORA_DISK_1: restoring datafile 00001 to /data01/app/oracle/oradata/ORCL2/datafile/o1_mf_system_cfxpdor9_.dbf
channel ORA_DISK_1: restoring datafile 00002 to /data01/app/oracle/oradata/ORCL2/datafile/o1_mf_tsh_data_cfxvz9do_.dbf
channel ORA_DISK_1: restoring datafile 00003 to /data01/app/oracle/oradata/ORCL2/datafile/o1_mf_sysaux_cfxpb0jb_.dbf
channel ORA_DISK_1: restoring datafile 00004 to /data01/app/oracle/oradata/ORCL2/datafile/o1_mf_undotbs1_cfxpj05f_.dbf
channel ORA_DISK_1: restoring datafile 00006 to /data01/app/oracle/oradata/ORCL2/datafile/o1_mf_users_cfxphz0d_.dbf
channel ORA_DISK_1: reading from backup piece /data01/app/oracle/fast_recovery_area/ORCL2/backupset/2016_03_08/o1_mf_nnndf_TAG20160308T103603_cfxwc4ds_.bkp
channel ORA_DISK_1: piece handle=/data01/app/oracle/fast_recovery_area/ORCL2/backupset/2016_03_08/o1_mf_nnndf_TAG20160308T103603_cfxwc4ds_.bkp tag=TAG20160308T103603
channel ORA_DISK_1: restored backup piece 1
channel ORA_DISK_1: restore complete, elapsed time: 00:00:55
Finished restore at 20160308.113420

RMAN> recover database;

Starting recover at 20160308.113501

using channel ORA_DISK_1

starting media recovery

archived log for thread 1 with sequence 4 is already on disk as file /data01/app/oracle/fast_recovery_area/ORCL2/archivelog/2016_03_08/o1_mf_1_4_cfxwdvgt_.arc

archived log for thread 1 with sequence 5 is already on disk as file /data01/app/oracle/fast_recovery_area/ORCL2/archivelog/2016_03_08/o1_mf_1_5_cfxy1pxh_.arc

archived log for thread 1 with sequence 6 is already on disk as file /data01/app/oracle/fast_recovery_area/ORCL2/archivelog/2016_03_08/o1_mf_1_6_cfxy1r1d_.arc

archived log for thread 1 with sequence 7 is already on disk as file /data01/app/oracle/fast_recovery_area/ORCL2/archivelog/2016_03_08/o1_mf_1_7_cfxyx64v_.arc

archived log for thread 1 with sequence 8 is already on disk as file /data01/app/oracle/fast_recovery_area/ORCL2/archivelog/2016_03_08/o1_mf_1_8_cfxyxv1m_.arc

archived log for thread 1 with sequence 9 is already on disk as file /data01/app/oracle/fast_recovery_area/ORCL2/onlinelog/o1_mf_2_cfxs974b_.log

archived log file name=/data01/app/oracle/fast_recovery_area/ORCL2/archivelog/2016_03_08/o1_mf_1_4_cfxwdvgt_.arc thread=1 sequence=4

archived log file name=/data01/app/oracle/fast_recovery_area/ORCL2/archivelog/2016_03_08/o1_mf_1_5_cfxy1pxh_.arc thread=1 sequence=5

creating datafile file number=5 name=/data01/app/oracle/oradata/ORCL2/datafile/o1_mf_sumit_cfxwfx8b_.dbf

archived log file name=/data01/app/oracle/fast_recovery_area/ORCL2/archivelog/2016_03_08/o1_mf_1_5_cfxy1pxh_.arc thread=1 sequence=5

archived log file name=/data01/app/oracle/fast_recovery_area/ORCL2/archivelog/2016_03_08/o1_mf_1_6_cfxy1r1d_.arc thread=1 sequence=6

archived log file name=/data01/app/oracle/fast_recovery_area/ORCL2/archivelog/2016_03_08/o1_mf_1_7_cfxyx64v_.arc thread=1 sequence=7

archived log file name=/data01/app/oracle/fast_recovery_area/ORCL2/archivelog/2016_03_08/o1_mf_1_8_cfxyxv1m_.arc thread=1 sequence=8

archived log file name=/data01/app/oracle/fast_recovery_area/ORCL2/onlinelog/o1_mf_2_cfxs974b_.log thread=1 sequence=9

media recovery complete, elapsed time: 00:00:02

Finished recover at 20160308.113508

[oracle@node2 datafile]$ ls -l
total 1875304
-rw-r----- 1 oracle oper 157294592 Mar 8 11:35 o1_mf_sumit_cfxzsqx7_.dbf
-rw-r----- 1 oracle oper 734011392 Mar 8 11:35 o1_mf_sysaux_cfxpb0jb_.dbf
-rw-r----- 1 oracle oper 807411712 Mar 8 11:35 o1_mf_system_cfxpdor9_.dbf
-rw-r----- 1 oracle oper 62922752 Mar 8 10:35 o1_mf_temp_cfxpk4nq_.tmp
-rw-r----- 1 oracle oper 157294592 Mar 8 11:35 o1_mf_tsh_data_cfxvz9do_.dbf
-rw-r----- 1 oracle oper 57679872 Mar 8 11:35 o1_mf_undotbs1_cfxpj05f_.dbf
-rw-r----- 1 oracle oper 5251072 Mar 8 11:35 o1_mf_users_cfxphz0d_.dbf

Total Pageviews