srvctl fails on one node but works on another

./srvctl

./srvctl: line 179: $JRE $JRE_OPTIONS -classpath $CLASSPATH $TRACE oracle.ops.opsctl.OPSCTLDriver “$@”

the obvious checks were already done :

check timestamp of srvctl and its size against another node

diff srvctl against another server

check version of java & path to version of java

environment variables match

o/s strace of srvctl did not give any clues to the problem.

other crs commands worked e.g. crsctl

no failures in /var/log/messages

To enable oracle tracing of srvctl:

oracle +ASM1 > export SRVM_TRACE=true (MOS 178683.1)

oracle +ASM1 > srvctl

Error occurred during initialization of VM

java.lang.OutOfMemoryError: unable to create new native thread

oracle +ASM1 > free -m

ps -ef | grep java

(pid 17377 is process for the oc4j CRS process)

oracle   17377     1  0 09:22 ?        00:00:30 /oracle/GRID/11203/jdk/jre//bin/java -server -Xcheck:jni -Xms128M -Xmx384M -Djava.awt.headless=true -Ddisable.checkForUpdate=true -Dstdstream.filesize=100 -Dstdstream.filenumber=10 -DTRACING.ENABLED=false -Doracle.wlm.dbwlmlogger.logging.level=INFO -Dport.rmi=23792 -jar /oracle/GRID/11203/oc4j/j2ee/home/oc4j.jar -config /oracle/GRID/11203/oc4j/j2ee/home/OC4J_DBWLM_config/server.xml -out /oracle/GRID/11203/oc4j/j2ee/home/log/oc4j.out -err /oracle/GRID/11203/oc4j/j2ee/home/log/oc4j.err

JRE_OPTIONS is a variable in the srvctl script where the default is JRE_OPTIONS=””

New Java process when running srvctl from another window with JRE_OPTIONS=”” and SRVM_TRACE=true

(the DTRACING is due to SRVM_TRACE set to TRUE)

oracle   15025 15019  0 13:26 pts/0    00:00:00 /oracle/GRID/11203/jdk/jre//bin/java -DORACLE_HOME=/oracle/GRID/11203 -classpath /oracle/GRID/11203/jlib/netcfg.jar:/oracle/GRID/11203/jdk/jre//jre/lib/rt.jar:/oracle/GRID/11203/jdk/jre//jre/lib/i18n.jar:/oracle/GRID/11203/jlib/srvm.jar:/oracle/GRID/11203/jlib/srvmhas.jar:/oracle/GRID/11203/jlib/srvmasm.jar:/oracle/GRID/11203/jlib/supercluster.jar:/oracle/GRID/11203/jlib/supercluster-common.jar:/oracle/GRID/11203/ons/lib/ons.jar:/oracle/GRID/11203/srvm/jlib/srvctl.jar:/oracle/GRID/11203/jlib/gns.jar -DTRACING.ENABLED=true -DTRACING.LEVEL=2 -Djava.net.preferIPv4Stack=true -Djava.util.logging.config.file=/oracle/GRID/11203/srvm/admin/logging.properties oracle.ops.opsctl.OPSCTLDriver

oracle   15942 15905  0 13:26 pts/0    00:00:00 /oracle/GRID/11203/jdk/jre//bin/java -DORACLE_HOME=/oracle/GRID/11203 -classpath /oracle/GRID/11203/jlib/netcfg.jar:/oracle/GRID/11203/jdk/jre//jre/lib/rt.jar:/oracle/GRID/11203/jdk/jre//jre/lib/i18n.jar:/oracle/GRID/11203/jlib/srvm.jar:/oracle/GRID/11203/jlib/srvmhas.jar:/oracle/GRID/11203/jlib/srvmasm.jar:/oracle/GRID/11203/jlib/supercluster.jar:/oracle/GRID/11203/jlib/supercluster-common.jar:/oracle/GRID/11203/ons/lib/ons.jar:/oracle/GRID/11203/srvm/jlib/srvctl.jar:/oracle/GRID/11203/jlib/gns.jar –DTRACING.ENABLED=true -DTRACING.LEVEL=2 -Djava.net.preferIPv4Stack=true -Djava.util.logging.config.file=/oracle/GRID/11203/srvm/admin/logging.properties oracle.ops.opsctl.OPSCTLDriver

New Java process when running srvctl from another window with JRE_OPTIONS=”” and SRVM_TRACE unset

oracle   26374 26366  0 13:28 pts/0    00:00:00 /oracle/GRID/11203/jdk/jre//bin/java -DORACLE_HOME=/oracle/GRID/11203 -classpath /oracle/GRID/11203/jlib/netcfg.jar:/oracle/GRID/11203/jdk/jre//jre/lib/rt.jar:/oracle/GRID/11203/jdk/jre//jre/l                                     ib/i18n.jar:/oracle/GRID/11203/jlib/srvm.jar:/oracle/GRID/11203/jlib/srvmhas.jar:/oracle/GRID/11203/jlib/srvmasm.jar:/or                                     acle/GRID/11203/jlib/supercluster.jar:/oracle/GRID/11203/jlib/supercluster-common.jar:/oracle/GRID/11203/ons/lib/ons.jar                                     :/oracle/GRID/11203/srvm/jlib/srvctl.jar:/oracle/GRID/11203/jlib/gns.jar -Djava.net.preferIPv4Stack=true -Djava.util.log                                     ging.config.file=/oracle/GRID/11203/srvm/admin/logging.properties oracle.ops.opsctl.OPSCTLDriver

with JRE_OPTIONS=”-Xms256M -Xmx1024M” and SRVM_TRACE unset set

oracle     695   685  0 13:35 pts/0    00:00:00 /oracle/GRID/11203/jdk/jre//bin/java -Xms256M -Xmx1024M -DORACLE_HOME=/oracle/GRID/11203 -classpath /oracle/GRID/11203/jlib/netcfg.jar:/oracle/GRID/11203/jdk/jre//jre/lib/rt.jar:/oracle/GRID/11203/jdk/jre//jre/lib/i18n.jar:/oracle/GRID/11203/jlib/srvm.jar:/oracle/GRID/11203/jlib/srvmhas.jar:/oracle/GRID/11203/jlib/srvmasm.jar:/oracle/GRID/11203/jlib/supercluster.jar:/oracle/GRID/11203/jlib/supercluster-common.jar:/oracle/GRID/11203/ons/lib/ons.jar:/oracle/GRID/11203/srvm/jlib/srvctl.jar:/oracle/GRID/11203/jlib/gns.jar -Djava.net.preferIPv4Stack=true -Djava.util.logging.config.file=/oracle/GRID/11203/srvm/admin/logging.properties oracle.ops.opsctl.OPSCTLDriver


Conclusion: The o/s was unable to satisfy memory requirement for srvctl intermitently. JRE_OPTIONS was set with a small and large memory value which worked but this was just a co-incidence. Even after JRE_OPTIONS was set back to null , srvctl carried on working.