Failed to perform J2EE (OC4J) Container Resource upgrade at crsconfig_lib.pm line 9323.

During Grid upgrade from 11.2.0.3 to 11.2.0.4 rootupgrade.sh fail with below error messgae

Failed to perform J2EE (OC4J) Container Resource upgrade at /data01/app/11.2.0/grid_11204/crs/install/crsconfig_lib.pm line 9323.

/data01/app/11.2.0/grid_11204/perl/bin/perl -I/data01/app/11.2.0/grid_11204/perl/lib -I/data01/app/11.2.0/grid_11204/crs/install /data01/app/11.2.0/grid_11204/crs/install/rootcrs.pl execution failed

This issue is caused by Bug 17284386, it is due to qosadmin user being deactivated by 11.2.0.3 PSU6 or higher PSU, it is fixed in 12.1.0.2.

<11.2.0.4 GI_HOME>/cfgtoologs/crsconfig/rootcrs_<node>.log shows:

2014-03-02 12:05:26: Error encountered in the command /data01/app/11.2.0/grid_11203/bin/qosctl -autogenerate

>  Mar 2, 2014 12:05:26 PM oracle.security.jazn.spi.xml.XMLRealmUser authenticate

>  INFO: User(qosadmin) is deactivated. AUTH FAILURE.

>  Change password operation failed. The old password entered is incorrect.

>  End Command output

2014-03-02 12:05:26: Failed to perform J2EE (OC4J) Container Resource upgrade

1. Activate the qosadmin user from 11.2.0.3 GRID_HOME as grid user:

$ export ORACLE_HOME=<11.2.0.3 GRID_HOME>

$ export PATH=$ORACLE_HOME/bin:$PATH

$ qosctl qosadmin -setpasswd qosadmin <old passwd> <new passwd>, for example:

If qosadmin's passwd has never been changed before, please do:

  $ qosctl qosadmin -setpasswd qosadmin oracle112 <new qosadmin passwd>

  AbstractLoginModule password:             << enter "oracle112" here

  Please re-enter the password entered on the command line above

  Enter Password:                           << re-enter "oracle112"

  $

- Replace <new qosadmin passwd> with any passwd you choose

If qosadmin's passwd has been reset before, please use its passwd instead of the default "oracle112".

2. To verify the change:

$ qosctl qosadmin -listusers

AbstractLoginModule password:     << enter <new qosadmin passwd>, it should then display:

oc4jadmin

JtaAdmin

qosadmin

It should not display:

$ qosctl qosadmin -listusers

AbstractLoginModule password:      << enter <new qosadmin passwd>

Mar 2, 2014 11:45:26 AM oracle.security.jazn.spi.xml.XMLRealmUser authenticate

INFO: User(jazn.com/qosadmin) is deactivated. AUTH FAILURE.

Mar 2, 2014 11:45:26 AM oracle.security.jazn.login.module.RealmLoginModule authenticate

SEVERE:                 [RealmLoginModule] authentication failed

Authentication Failed

3. Rerun rootupgrade.sh from 11.2.0.4 GRID_HOME, as root user:

# cd <11.2.0.4 GRID_HOME>

# ./rootupgrade.sh

This should complete the upgrade.