Performance Analysis and Troubleshooting

This is a page to assist with performance analysis and troubleshooting.

Monitoring Deadlocks in PostgreSQL

You can monitor any database locks using the Entity SQL Processor in Webtools -> Entity SQL Processor with the following query,

select pg_class.relname, pg_locks.mode, pg_locks.relation, pg_locks.transaction, pg_locks.pid
    from pg_class, pg_locks 
    where pg_class.relfilenode = pg_locks.relation 
    order by pid

This will show what kinds of locks are active on what entities. If there is an exclusive lock on a table followed by a bunch of pids that are waiting for it, then you have a deadlock.

Suspending Runaway Threads

Suppose you start a process that you realize will take forever and need to stop it. However, it can't be stopped because it was activated by an HTTP request and killing the browser session doesn't work. First, check the log to see if you can identify the thread that is running this process. For instance, suppose you have the following line in your log that corresponds to what your process is doing,

2008-01-23 18:55:47,585 (TP-Processor10) [       Something.java:1015:WARN ] Something that identifies your process

This thread is TP-Processor10. You can use the Java Thread API to suspend it by hand. The easiest way to do this is to use a bsh script or the bsh terminal. First, you will want to know the number of threads in the system. Load up Webtools -> Jobs -> Thread List and count the rough number of threads displayed. Suppose you have about 50 threads.

Once you know the rough size, run the following script, either via the bsh terminal or by hooking it up to a controller.xml request,

threads = new Thread[50];
size = Thread.enumerate(threads);
for (i = 0; i < size; i++) {
    print(i + ": " + threads[i]);
}

This will print out the index and name of each thread. Find the index of TP-Processor 10. Suppose it is index 37. You can then suspend the thread by doing this,

t = threads[37];
t.suspend();

Profiling with AspectJ

In opentaps 1.0, profiling is done using Aspect Oriented Programming. AOP provides us with the ability to specify measurement points around any part of the code without having to modify the Java code itself. This is a brief tutorial on how to take advantage of this to profile suspected problem spots in the system.

Performance monitoring is accomplished using the JETM library, which comes with opentaps.

Out of the Box Profiling

Presently, performance profiling can be triggered by running the tests. First thing is to compile the system and tests as normal. Then, you will want to compile profiling support as follows,

$ ant -f hot-deploy/opentaps-common/build.xml profiling

This will apply the profiling aspects using bytecode weaving, a process which modifies the existing jars and inserts new bytecode that represents our profiling code.

Make sure that the run-tests target in the main build.xml file does not have any dependencies. Otherwise, when you run the tests, it will recompile the codebase without the aspects. This is a more efficient setup for testing anyway, since we do not want to recompile the entire system. When ready, run the tests,

$ ant run-tests

Then when you look at the logs, you have:

• A confirmation the profiling library is loaded.

Example:

* Start JETM monitoring
* JETM 1.2.2 started.

• A table with all the profiling details.

Example:

TODO: Put an example here

Understand AspectJ code

• Some theory:

First you have to create a file with your aspect. We are not talking anymore about class, but aspect. Actually, it use a specific compiler. (Compiler errors message are not really understandable in my mind... You can activate the options verbose and showWaveInfo to have some more info.)

AOP is a way to add code to several points of the project, based on method selections or pointcut.

Creation of a poincut:

pointcut testRunContainerStart() : execution(public boolean org.ofbiz.testtools.TestRunContainer.start());

You can use wildcard: *, +, ..

* replace a name or part of the name

public  * fr.umlv.*.test.*.start*(int, String, *)

+ used to not tell the name of the parameter of the method or the name of the package

public void fr..Test.SetParams(..)
all the public method of SetParam whatever are their parameters,
returning void and available in the classes Test which are wherever in
the package fr

.. used to define whatever sub-type of a class or interface.

void fr.umlv.test.IMouseListener+.set*(..)
all the methods which begins with set in the classes implementing the
interface IMouseListener.

• Some Practical comments:

In my case I almost always give the full method name with the argument. Because ofbiz and opentaps project are big projects, side effects could happen in case of not specifing the full method name I want to work on.

• Let's go back to theory:

You can define the poincut on various events: call, execution, get, set, handler, initialization, preinitialization, staticinitialization, adviceexecution

call call of the method

execution execution of the method.

• What is the difference between call and execution?

Mainly it is about heritance. In the case of call we are working with the reference which is used in the program (it can be the interface). In the case of execution we are working on the object which is instanciated.

There is more differences.

In case of a call, the compiler modifies the invoking code. In case of an execution, the compiler modifies the target code.

It's made problematic crosslibrary using of "call", if the execution used around pointcut is inlined in target method.

I.e.

someMethod() {
  Debug.logInfo("Enter method");
  Debug.logInfo("Leave method")
}

execution(someMethod)
around(someAdvice) {
  Debug.logInfo("Enter advice");
  proceed()
  Debug.logInfo("Leave advice")
}
compiler will produce
someMethod {
  Debug.logInfo("Enter advice");
  Debug.logInfo("Enter method");
  Debug.logInfo("Leave method")
  Debug.logInfo("Leave advice")
}

A bit crazy:)

This can bring a problem with debugging and breakpoints in advises. Eclipse has special options for this case.

• During the creation of a pointcut, you can use logic and associate various call, execution with && or ||

The interesant method is args which let you access to the parameter of the method.

• When you pointcut is defined, you can realize a treatment.

before to realize the treatment before to reach the pointcut.

after to realize the treatment after to reach the pointcut.

• If the program is multithread (case of opentaps) and you have to do two

treatment before and after the pointcut, you can use around and proceed. proceed will execute the code corresponding to the poincut in a around instruction.

• This is a really quick overview, to understand the code added to

opentaps and there is some more to know, some trap (infinite loops) etc...

• Source http://www.eclipse.org/aspectj/doc/released/progguide/implementation.html
• Source http://www-igm.univ-mlv.fr/~dr/XPOSE2005/sophie/#