Completing an Oracle Clusterware Upgrade when Nodes Become Unreachable

If some nodes become unreachable in the middle of an upgrade, then you cannot complete the upgrade, because the upgrade script (rootupgrade.sh) did not run on the unreachable nodes. Because the upgrade is incomplete, Oracle Clusterware remains in the previous release. You can confirm that the upgrade is incomplete by entering the command crsctl query crs activeversion.

To resolve this problem, run the rootupgrade command with the -force flag on any of the nodes where the rootupgrade.sh script has already completed as follows:

Grid_home/rootupgrade.sh -force

For example:

# /u01/app/12.1.0/grid/rootupgrade.sh -force
This command forces the upgrade to complete. Verify that the upgrade has completed by using the command crsctl query crs activeversion. The active release should be the upgrade release.

The force cluster upgrade has the following limitations:

All active nodes must be upgraded to the newer release.

All inactive nodes (accessible or inaccessible) may be either upgraded or not upgraded.

For inaccessible nodes, after patch set upgrades, you can delete the node from the cluster. If the node becomes accessible later, and the patch version upgrade path is supported, then you can upgrade it to the new patch version.

If the cluster was previously forcibly upgraded, then ensure that all inaccessible nodes have been deleted from the cluster or joined to the cluster before starting the upgrade.

 Upgrading Inaccessible Nodes After Forcing an Upgrade

Starting with Oracle Grid Infrastructure 12c, after you complete a force cluster upgrade, you can join inaccessible nodes to the cluster as an alternative to deleting the nodes, which was required in earlier releases. To use this option, Oracle Grid Infrastructure 12c Release 1 (12.1) software must already be installed on the nodes.

To complete the upgrade of nodes that were inaccessible or unreachable:

Log in as the Grid user on the node that is to be joined to the cluster.

Change directory to the /crs/install directory in the Oracle Grid Infrastructure 12c Release 1 (12.1) Grid home. For example:

$ cd /u01/12.1.0/grid/crs/install

Run the following PERL command, where existingnode is the name of the option and upgraded_node is any node that was successfully upgraded and is currently part of the cluster:

$ rootupgrade.sh -join -existingnode upgraded_node

B.8.4 Changing the First Node for Install and Upgrade

If the first node becomes inaccessible, you can force another node to be the first node for installation or upgrade. During installation, if root.sh fails to complete on the first node, run the following command on another node using the -force option:

Note:

The -join operation is not supported for Oracle Clusterware releases earlier than 11.2.0.1.0. In such cases, delete the node and add it to the clusterware using the addNode command.