How to Debug CVU / Collect CVU Trace Generated by RUNCLUVFY.SH

For 11gR2/12c:

Set environment variable CV_TRACELOC to a directory that's writable by grid user; trace should be generated in there once runcluvfy.sh starts. For example, as grid user:


rm -rf /tmp/cvutrace
mkdir /tmp/cvutrace
export CV_TRACELOC=/tmp/cvutrace
export SRVM_TRACE=true
export SRVM_TRACE_LEVEL=1
<STAGE_AREA>/runcluvfy.sh stage -pre crsinst -n <node1>,<node2> -verbose


Note:

A. STAGE_AREA refers to the location where Oracle Clusterware is unzipped.
B. Replace above "runcluvfy.sh stage -pre crsinst -n <node1>,<node2> -verbose" if other stage/comp needs to be traced, i.e. "./runcluvfy.sh comp ocr -verbose"
C. For 12.1.0.2, the following can be set for additional tracing from exectask command:

export EXECTASK_TRACE=true
The trace will be in <TMP>/CVU_<version>_<user>/exectask.trc, i.e. /tmp/CVU_12.1.0.2.0_grid/exectask.trc



For 10gR2, 11gR1 or 11gR2:

1. As crs/grid user, backup runcluvfy.sh. For 10gR2, it's located in <STAGE_AREA>/cluvfy/runcluvfy.sh; and for 11gR1 and 11gR2, <STAGE_AREA>/runcluvfy.sh


cd <STAGE_AREA>
cp runcluvfy.sh runcluvfy.debug

2. Locate the following lines in runcluvfy.debug:


# Cleanup the home for cluster verification software
$RM -rf $CV_HOME

Comment out the remove command so runtime files including trace won't be removed once CVU finishes.


# Cleanup the home for cluster verification software
# $RM -rf $CV_HOME

3. As crs/grid user, set environment variable CV_HOME to anywhere as long as the location is writable by crs/grid user and has 400MB of free space:


mkdir /tmp/cvdebug
CV_HOME=/tmp/cvdebug
export CV_HOME

This step is optional, if CV_HOME is unset, CVU files will be generated in /tmp.

4. As crs/grid user, execute runcluvfy.debug:


export SRVM_TRACE=true
export SRVM_TRACE_LEVEL=1
cd <STAGE_AREA>
./runcluvfy.debug stage -pre crsinst -n <node1>,<node2> -verbose

Note:

A. SRVM_TRACE_LEVEL is effective for 11gR2 only.
B. Replace above "runcluvfy.sh stage -pre crsinst -n <node1>,<node2> -verbose" if other stage/comp needs to be traced, i.e. "./runcluvfy.sh comp ocr -verbose"


5. Regardless whether above command finishes or not, CVU trace should be generated in:

10gR2: $CV_HOME/<pid>/cv/log
11gR1: $CV_HOME/bootstrap/cv/log
11gR2: $CV_HOME/bootstrap/cv/log

If CV_HOME is unset, trace will be in /tmp/<pid>/cv/log or /tmp/bootstrap/cv/log depend on CVU version.

6. Clean up temporary files generated by above runcluvfy.debug:


rm -rf $CV_HOME/bootstrap