Trouble Shooting in WebLogic

 

Fails to start weblogic instances maintenance of Admin server as well as all managed servers etc. Troubleshooting in weblogic is found out the cause(Server fail, Server shutdown, SSL issue, Thread stuck, etc…) of WebLogic server failure if it occurs by servers logs

Troubleshooting Common Problems

• Before You Start the Cluster

You can do several things to help prevent problems before you boot the cluster.

• Check for a Cluster License

Your WebLogic Server license must include the clustering feature. If you try to start a cluster without a clustering license, you will see the error message Unable to find a license for clustering.

• Check the Server Version Numbers

All Managed Servers in a cluster and the cluster's Administration Server should run under the same version of WebLogic Server. The major and minor version numbers (e.g 9.1), service packs, and attached patch levels should be the same across the cluster.

• Check the Multicast Address

a) A problem with the multicast address is one of the most common reasons a cluster does not start or a server fails to join a cluster.

b) A multicast address is required for each cluster. The multicast address can be an IP number between 224.0.0.0 and 239.255.255.255, or a hostname with an IP address within that range.

c) You can check a cluster's multicast address and port on its Configuration-->Multicast tab in the Administration Console.

d) For each cluster on a network, the combination of a multicast address and port must be unique. If two clusters on a network use the same multicast address, they should use different ports. If the clusters use different multicast addresses, they can use the same port or accept the default port, 7001.

e) Before booting the cluster, make sure the cluster's multicast address and port are correct and do not conflict with the multicast address and port of any other clusters on the network.

f) The errors you are most likely to see if the multicast address is bad are:

• Unable to create a multicast socket for clustering
• Multicast socket send error
• Multicast socket receive an error

Inclined to build a profession as Weblogic Developer? Then here is the blog post on, explore Weblogic Training

Check the CLASSPATH Value

Make sure the value of CLASSPATH is the same on all managed servers in the cluster. CLASSPATH is set by the setEnv script, which you run before you run startManagedWebLogic to start the managed servers.

By default, setEnv sets this value for CLASSPATH (as represented on Windows systems):

set WL_HOME=C:beaweblogic700

set JAVA_HOME=C:beajdk131

.

.

set CLASSPATH=%JAVA_HOME%libtools.jar;

%WL_HOME%serverlibweblogic_sp.jar;

%WL_HOME%serverlibweblogic.jar;

%CLASSPATH%

If you change the value of CLASSPATH on one managed server or change how setEnv sets CLASSPATH, you must change it on all managed servers in the cluster.

Check the Thread Count

Each server instance in the cluster has a default execute queue, configured with a fixed number of executing threads. To view the thread count for the default execute queue, choose the Configure Execute Queue command on the Advanced Options portion of the Configuration> General tab for the server. The default thread count for the default queue is 15, and the minimum value is 5. If the value of Thread Count is below 5, change it to a higher value so that the Managed Server does not hang on startup.

After You Start the Cluster

This section describes the first troubleshooting steps to perform if you have problems trying to start a cluster.

Check Your Commands

If the cluster fails to start, or a server fails to join the cluster, the first step is to check any commands you have entered, such as startManagedWebLogic or a java interpreter command, for errors and misspellings.

Getting a JRockit Thread Dump under Linux

If you use the JRockit JVM under Linux, use one of the following methods to generate a thread dump.

• Use the weblogic.admin THREAD_DUMP command.
• If the JVM's management server is enabled (by starting the JVM with the -Xmanagement option), you can generate a thread dump using the JRockit Management Console.
• Use Kill -3 PID, where PID is the root of the process tree.

To obtain the root PID, perform a: ps -efHl | grep 'java' **. **

Using a grep argument that is a string that will be found in the process stack that matches the server startup command. The first PID reported will be the root process, assuming that the ps command has not been piped to another routine.

Under Linux, each executing thread appears as a separate process under the Linux process stack. To use Kill -3 on Linux you supply must match the PID of the main WebLogic execute thread, otherwise, no thread dump will be produced.

Check Garbage Collection

If you are experiencing cluster problems, you should also check the garbage collection on the managed servers. If garbage collection is taking too long, the servers will not be able to make the frequent heartbeat signals that tell the other cluster members they are running and available.

If garbage collection (either first or second-generation) is taking 10 or more seconds, you need to tune heap allocation (the ms mx parameter) on your system.

Check MultiCast Address:

• You can verify that multicast is working by running utils.MulticastTest from one of the managed servers.
• Run utils.MulticastTest

Monitor Weblogic through the command line

Everything visible through the Weblogic admin console (http://localhost:7001/console) can be accessed through a command line java tool. This tool can be used to gather data about the weblogic servers via scripting.  There are at least two ways to get runtime monitoring data about Weblogic processes. This document covers the use of the java classes that get information from management beans (mbeans). There is also a java tool that allows for browsing the mbean tree like an ftp client: Weblogic Scripting Tool (WLST).

Set the Environment:

Set the java environment

There is a script for setting the CLASSPATH and PATH so that this tool can work. On pitblade (WL Server), the setWLSEnv script is /dsk2/local/bea81/weblogic81/server/bin/setWLSEnv.sh

pitblade:II:root: > source setWLSEnv.sh

CLASSPATH=/dsk2/local/bea81/jdk141_02/lib/tools.jar:/dsk2/local/bea81/weblogic81/server/lib/weblogic_sp.jar:

/dsk2/local/bea81/weblogic81/server/lib/weblogic.jar:/dsk2/local/bea81/weblogic81/server/lib/ojdbc14.jar:

:/dsk1/AdvIngres/ing26/ingres/lib/edbc.jar:/dsk1/AdvIngres/ing26/ingres/lib/edbc.jar

 

PATH=/dsk2/local/bea81/weblogic81/server/bin:/dsk2/local/bea81/jdk141_02/jre/bin:/dsk2/local/bea81/jdk141_02/bin:

/dsk1/AdvIngres/ing26/ingres/bin:/dsk1/AdvIngres/ing26/ingres/utility:/dsk1/AdvIngres/ing26/ingres/files:

/dsk1/AdvIngres/ing26/ingres/lib:/dsk1/AdvIngres/ing26/ingres/SUNWspro/bin:/sbin:/usr/sbin:/bin:/usr/bin:

/usr/ucb:/etc:/usr/etc:/opt/fw/bin:/var/adm/psmfc/bin:/usr/openwin/bin:/usr/local/bin:/usr/psmfc/bin:

/usr/ccs/bin:/usr/lib/nis:/opt/gnu/bin:/usr/local/bin

Your environment has been set.

Verify that the java environment is set properly Success

pitblade:II:root: > java weblogic.Admin

weblogic.Admin is a command-line utility for managing WebLogic Server. Try:

        weblogic.Admin help LIFECYCLE   Starting, stopping, discovering servers

        weblogic.Admin help INFO        Retrieving info about WebLogic Server

        weblogic.Admin help JDBC        Working with JDBC connection pools

        weblogic.Admin help MBEAN       Working with WebLogic Server MBeans

        weblogic.Admin help CLUSTER     Working with clusters

        weblogic.Admin help ALL         Help for all commands

Usage: java [<SSL trust options>] weblogic.Admin

        [ [-url | -adminurl] [<protocol>://]<listen-address>:<port>]

        -username <username> -password <password>

        <COMMAND> <ARGUMENTS>

More info available at: http://e-docs.bea.com/wls/docs81/admin_ref/cli.html

Failure

pitblade:II:root: > java weblogic.Admin

Exception in thread "main" java.lang.NoClassDefFoundError: weblogic/Admin

 get state

pitblade:II:root: > java weblogic.Admin -username user -password pass GETSTATE myserver

Current state of "myserver" : RUNNING

 get server config

pitblade:II:root: > java weblogic.Admin -username user -password pass GET -pretty -type Server

---------------------------

MBeanName: "mydomain:Name=myserver,Type=Server"

        AcceptBacklog: 50

        AdministrationPort: 0

        AutoKillIfFailed: false

        AutoRestart: true

        COM: myserver

        COMEnabled: false

        CachingDisabled: true

 

Test if weblogic is running:

On command prompt issue this command:

ps -if | grep java

You should get a response like this:

reedi.psmfc.org:C1:root: > ps -eaf | grep java

    root  6233  6221  0   Jan 31 ?        2:50 /usr/local/bea81/jdk141_02/bin/java -server -Xms32m -Xmx200m -XX:MaxPermSize=12

    root  6273  6261  0   Jan 31 ?       361:59 /usr/local/bea81/jdk141_02/bin/java -server -Xms256m -Xmx512m -Djava.awt.headle

    root  1121  6273  0   Feb 08 ?       15:29 /usr/local/bea81/jdk141_02/jre/bin/java -Xms256m -Xmx1024m -cp /global/ds1/pitw

    root  2087 20643  0 13:44:11 pts/1    0:00 grep java

 Restart weblogic on the cluster

• Use the admin console to request a graceful shutdown of the server
• start a bash shell
• Use this command to start:

cd /usr/local/bea81/user_projects/mydomain2

nohup ./startManagedWebLogic.sh &

• Tail –f nohup.out until the server enters the running state
• repeat for each server in the cluster

Restart weblogic on the development server

• Use the admin console to request a graceful shutdown
• start a bash shell on the server
• Use this command to start:

cd /dsk2/local/bea81/user_projects/mydomain

nohup ./startWeblogic.sh &

Weblogic and apache debug info

On pitblade(server), the file /tmp/wlproxy.log is filling the filesystem and crashing the server. Apparently, this file is debugged information generated by apache about the weblogic proxy plug-in. It probably causes a severe drain on resources and should not be generally active. Here is the relevant stanza from httpd.conf:

Note: The specified location of the log file seems to be ignored. There would not have been a problem if the log file was actually located where it was directed in the configuration.

<IfModule mod_weblogic.c>

  WebLogicHost pitblade

  WebLogicPort 7001

  MatchExpression *.jsp

  #MatchExpression              /ptagis/home/regTagAction   

WebLogicHost=pitblade|WebLogicPort=7001|Idempotent=OFF

  Debug ON

  WLLogFile /var/log/bea/wlproxy.log

  HungServerRecoverSecs 30000

</IfModule>

After changing this setting, apache needs a restart like this:

apache2/bin/apachectl restart

Turned off this debugging output on pitblade and bay. The response time seems better as a result.

Optimization Techniques:

• Open source load generation tools
  • Apache JMeter, Grinder, OpenSTA, Dieseltest
• Starting the sun JVM with -hotspot allows it to dynamically optimize the bytecode
• If you allocate too much memory to a jvm, it can make garbage collection run slower and make the app stall for best performance, set min, and max memory for the jvm to the same value. the growing and shrinking of the jvm have overhead.
• Start with -verbosegc on the command line. This will include in stdout information about the garbage collection. It should take less than a second. If GC takes longer than 3 seconds, then reduce the heap size to make GC happen more often and thus have GC do less work each time.
• Can turn on generational GC (NewSize) must be using hotspot. Creates a nursery in the heap where partial GCs run more frequently. Assumes most objects are short-lived.
• JRockit is only supported on Intel. Beta support for solaris.
• Locate the bottleneck: CPU, IO, DB.
• Network tune: set tcp_time_wait_interval from 240 to 60 on solaris
• Execute thread pool is set to 25 in production mode and 15 on development
• Native IO socket readers are much faster than pure java threads. So check the box that says enable native IO.
• 25 to 65 are optimal for Execute threads.
• A stuck thread cannot be released without rebooting the jvm. When all the threads in the server are stuck, the server's health is flagged as critical. Nodemanager can be set to automatically restart a critical server.
• Object pooling, passivation, activation.

Common WebLogic Server Deadlocks and How to Avoid Them

While there are many possible causes of performance degradation or hangs, this article can't possibly cover them all. Instead, we'll look at three common mistakes in WebLogic Server applications that can deadlock the server or bring your performance to a screeching halt.

Thread Dumps

The best Java tool for diagnosing deadlocks is a Java virtual machine thread dump. A thread dump is a snapshot of the virtual machine's current state, including stack traces for each Java thread. Many virtual machines also include information about the Java monitors held by each thread. Monitor information is especially useful for diagnosing deadlocks and performance problems in your application. On Windows platforms, you can generate a thread dump by pressing Ctrl-Break in the virtual machine's window. On Unix systems, a SIGQUIT signal must be sent to the Java virtual machine process. This can be done with a kill -3 <process id>.

Deadly Embrace

• The classic deadlock problem is the deadly embrace: Thread 1 owns Lock A and waits on Lock B, Thread 2 owns Lock B and waits on Lock A. These threads are deadlocked and will remain blocked in this state. In many cases, the remaining threads will eventually enter the deadlock by attempting to acquire Lock A or Lock B and waiting. A thread dump is one of the best ways to discover a deadly embrace deadlock. Most virtual machines include a thread state for each Java thread in the dump. The most common thread states are: R - running; MW - monitor wait; CW - condition wait. Threads in the MW state are blocked, waiting to enter a synchronized block and acquire a Java monitor. Since the thread dump includes the Java thread's stack trace, it's also possible to determine which monitor is blocking the thread. If multiple threads are in the MW state on the same monitor, it's a good indication that there's either a lot of contention for this monitor or the server is deadlocked. In a deadlock situation, you should be able to determine the other threads blocked in MW and their held monitors.
• There are two classic techniques for solving deadly embrace deadlocks: deadlock avoidance and deadlock detection. Deadlock avoidance is merely changing or structuring your code so that it can't hit the deadlock case. A common solution is to implement lock ordering. If Lock A is always acquired before Lock B then you can't have a deadly embrace on these two locks. It's also a good idea to minimize the time that a monitor is held. Every extra line of code that runs under a monitor is another chance for someone to add a deadlock. Deadlock detection is commonly implemented by databases for their locks, but it doesn't usually find its way into programming languages. In deadlock detection, deadlocks are automatically discovered and one or more deadlock participants (known as the victims) are killed and release their locks to break the deadlock. Java virtual machines do not break deadlocks on Java monitors, so deadlock avoidance is necessary.

Out-of-Threads Deadlock

Another common way to deadlock an application is what I call the "out of threads" deadlock. Unfortunately, this deadlock often doesn't show up until a load test or, in the worst case, when your production application receives a lot of traffic. In this scenario, your WebLogic Server is running with a fixed number of threads. The application includes logic where a given request or action performs work in one thread and then blocks on work that must be done in another thread.