Performance Analysis and Troubleshooting

This is a page to assist with performance analysis and troubleshooting.

Monitoring Deadlocks in PostgreSQL

See Database_Tips#Monitoring_PostgreSQL_Deadlocks

Suspending Runaway Threads

Suppose you start a process that you realize will take forever and need to stop it. However, it can't be stopped because it was activated by an HTTP request and killing the browser session doesn't work. First, check the log to see if you can identify the thread that is running this process. For instance, suppose you have the following line in your log that corresponds to what your process is doing,

2008-01-23 18:55:47,585 (TP-Processor10) [       Something.java:1015:WARN ] Something that identifies your process

This thread is TP-Processor10. You can use the Java Thread API to suspend it by hand. The easiest way to do this is to use a bsh script or the bsh terminal. First, you will want to know the number of threads in the system. Load up Webtools -> Jobs -> Thread List and count the rough number of threads displayed. Suppose you have about 50 threads.

Once you know the rough size, run the following script, either via the bsh terminal or by hooking it up to a controller.xml request,

threads = new Thread[50];
size = Thread.enumerate(threads);
for (i = 0; i < size; i++) {
    print(i + ": " + threads[i]);
}

This will print out the index and name of each thread. Find the index of TP-Processor 10. Suppose it is index 37. You can then suspend the thread by doing this,

t = threads[37];
t.suspend();

Profiling with Aspect Oriented Programming

In opentaps 1.0, profiling is done using Aspect Oriented Programming. AOP provides us with the ability to specify measurement points around any part of the code without having to modify the Java code itself. This is a brief tutorial on how to take advantage of this to profile suspected problem spots in the system. For more information, see Aspect Oriented Programming in opentaps

Performance monitoring is accomplished using the JETM library, which comes with opentaps.

Out of the Box Profiling

Presently, performance profiling can be triggered by running the tests. First thing is to compile the system and tests as normal. Then, you will want to compile profiling support as follows,

$ ant -f hot-deploy/opentaps-common/build.xml profiling

This will apply the profiling aspects using bytecode weaving, a process which modifies the existing jars and inserts new bytecode that represents our profiling code.

Make sure that the run-tests target in the main build.xml file does not have any dependencies. Otherwise, when you run the tests, it will recompile the codebase without the aspects. This is a more efficient setup for testing anyway, since we do not want to recompile the entire system. When ready, run the tests,

$ ant run-tests

Then when you look at the logs, you have:

• A confirmation the profiling library is loaded.

Example:

* Start JETM monitoring
* JETM 1.2.2 started.

• A table with all the profiling details.

Example:

TODO: Put an example here