Extending Glassbox

Glassbox provides method-level diagnosis but the details are limited:
in the Glassbox UI, only the summarized listing of operations is shown.

However using JMX, break-up of each operation into it’s monitored nested operations and resource access if any, are displayed with average, max time and count, using Glassbox’s built-in monitors for important frameworks (includes servlets/jsp, struts, ejb calls with method name, jdbc calls with details of each prepared statement & more). So Glassbox by default would log your EJB calls, your database calls, your JSP calls, etc.


In spite of this, other details (like parameters for a request, method name in actual class used, etc) for the response to each monitored operation are recorded only if it fails some SLA. In addition, this does not anyway capture names of low-level classes or their methods.


But what if you wanted a customized Glassbox monitoring - what if you were not satisfied with just a high-level EJB call and wanted to see the actual classes and methods called?

There are 2 ways to do this:
1) Using a simple run-time configuration in an xml file, and with NO CODING
2) Much more powerful customization if you were willing to define custom Aspects and pointcuts.

This post will stick to the simple way using the xml file.
But I do want to highlight before you proceed that adding more detailed logging means more bloating of PermGen memory-usage (see here on how to increase PermGen). For example, even in Glassbox's built-in jsp monitor’s load-weaving, 1 loader is needed per Jsp, and each loader takes up 1MB.
Also, read this post about Glassbox setup first - see Steps to configure Glassbox for Sun App Server .

Ok let's see how to provide a detailed method level monitorining for all the classes in a package you specify.

First, create a file aop.xml with the following content:

<aspectj>
<weaver>
<exclude within="org.springframework.jmx..*"/>
<!--  don't reweave -->
<exclude within="glassbox.inspector..*"/>
</weaver>
<aspects>
<concrete-aspect name="YourMonitor2"
extends="glassbox.monitor.ui.TemplateOperationMonitor">
<pointcut name="methodSignatureControllerExecTarget"
expression="within(com.package.to.monitor..*)"/>
</concrete-aspect>
</aspects>
</aspectj>

Next place this in a META-INF folder in any of the directories existing in your classpath.

That's it!
Next time you start up your server, Glassbox will monitor all the methods of all classes in the package, com.package.to.monitor!

Connecting to Glassbox using JMX

The Glassbox UI is a wonderful way for anyone to quickly identify major bottlenecks. It quickly lists out most of the potential problems and even common solutions if any call exceeds the SLA.

But if you want to delve into even more details, you would need to use the excellent set of MBeans provided by Glassbox. If you have a default JMX configuration for your Glassbox installation, the following is the connection string:

service:jmx:rmi:///jndi/rmi://localhost:7232/GlassboxTroubleshooter

To connect using JConsole, provide this without username and password.

Navigate to Tree->Glassbox->stats in the left pane and click on your application for operations monitored by Glassbox - you may have to perform some activity on your application for the monitors to log anything of course.

Additional Steps if encountering frequent “java.lang.OutOfMemoryError: PermGen space” exception.

Glassbox does take up some additional memory slowing the startup. Also, it tends to place additional load in the PermGen (Permanent Generation) memory space - this would result in the Server crashing out with "java.lang.OutOfMemoryError: PermGen space" exception (see http://blogs.sun.com/fkieviet/entry/classloader_leaks_the_dreaded_java for details).

The fix is a simple matter of increasing your max PermGen size (default is 64MB in Sun Servers). Use the following options when starting the JVM:
-XX:+CMSClassUnloadingEnabled
-XX:+CMSPermGenSweepingEnabled
-XX:MaxPermSize=128m

This actually works for any genuine PermGen error. But note that in most cases, PermGen running out of space is because of inefficient or incorrect cleaning up for Garbage collection - that is, some where in your code, you are unnecessarily keeping references even after use.

Steps to configure Glassbox for Sun App Server

Prerequisites
Ensure that the steps are being carried out by the same user who created / installed the Sun App Server to avoid read / access issues.

Requirements:
Sun App Server 8.1
glassbox.war (latest file is available from Glassbox website)
Java 1.4 or above

Configuration steps:
1) Create a new base directory for Glassbox (eg: D:\Sun\glassbox). This should contain the configuration details & library files. I have gathered the neccessary files into glassbox_config.zip to make it simple - simply unzip the contents into this directory. See below on how to do this the hard way...
Additional key properties like elapsed time threshold, minimum slow threshold & thread-monitor interval can be specified in glassbox.properties in this directory.

2) Go to Sun One Admin Console and select Configuration->JVM Settings. Select the Profiler tab.

Name the profiler, Glassbox, and enable profiler adding the following details :
Classpath: D:\Sun\glassbox\lib\aspectjweaver.jar${path.separator}D:\Sun\glassbox\lib\glassboxMonitor.jar
JVM Option1: -Dglassbox.install.dir=D:\Sun\glassbox
JVM Option2: -javaagent:D:\Sun\glassbox\lib\aspectjweaver.jar
In the above settings, use whatever location chosen as base directory for Glassbox in step 1, if different from D:\Sun\glassbox.

By doing it this way, you can easily decide when you need Glassbox by coming here and turning on or off the Enable Profiler option.

3) Deploy glassbox.war on the server. Also modify server.policy to have the following additional entry:
// Following grant block is added for glassbox
grant codeBase "file:${com.sun.aas.instanceRoot}/applications/j2ee-modules/glassbox/-" {
permission java.lang.RuntimePermission "setContextClassLoader";
permission java.lang.RuntimePermission "getClassLoader";
permission java.lang.RuntimePermission "createClassLoader";
permission java.security.AllPermission;
};
The Glassbox Web Client would be available at <server-app-url>/glassbox after restart.

4) Restart the Sun App server. If the agent was installed successfully, the following should be present in the server.log:
“Glassbox Agent has started successfully.”


Obtaining Configuration files from latest glassbox.war
Extract all the files from glassbox.war located in the install/glassbox14 directory and copy the files to the target configuration directory.
Move the 2 copied jars into a lib sub-directory.

The final structure should be as below:
\--------
glassbox.properties
readme.txt
runtime.properties
\lib-----
aspectjweaver.jar
glassboxMonitor.jar


Additional steps for using Glassbox with JVM 1.4
If the Sun App Server uses a Sun / IBM 1.4 JVM, the createJavaAdapter has to be updated and added to the JVM start options –Xbootclasspath along with the unmodified file, java14Adapter.jar.

Extract aspectj14Adapter.jar and createJavaAdapter.jar from the glassbox.war located in install/glassbox14
Run java -jar createJavaAdapter.jar using the Java version you will use to run the server.
Copy the aspectj14Adapter.jar and updated createJavaAdapter.jar to the Glassfish base configuration directory.
Add the following additional JVM options in the Profiler section of Sun App Server:
-Xbootclasspath/p:\java14Adapter.jar
-Xbootclasspath/a:\createJavaAdapter.jar
Restart the Sun App server.


Additional steps for allowing Sun App Servers to give remote RMI access in Unix
If there are errors when accessing JMX on a different server than the one on which Glassbox is installed, ensure that the target remote server has the given permissions by adding the following in the server.policy file:
grant {
permission java.net.SocketPermission "*:1024-65535", "connect,accept";
};