Archive for March 2, 2010

JVM Monitoring with Oracle Application Server 10g R2

March 2, 2010 Leave a comment

A little introduction

I was recently asked to perform some stress tests on a system running Oracle Application Server 10g Release 2 installed on Windows 2003 server. Among other things, the objective was to monitor the system and profile the code in order to detect possible flaws in the code and the server configuration.

One of tasks I had to do was to find a way to monitor the application server’s JVM during the stress tests. Naively, I thought that I could easily use “visualgc” (vmstat 3.0) or even better the “jvisualvm” provided with all the latest releases of the JDK. The rest of this post shows how wrong (and ignorant) I was …

First thing to do: install a decent JVM

As you may already know, only “recent” versions of the JDK are bundled with monitoring tools (jps, jstat, jstad, jvisualvm …) and unfortunately Oracle Application Server 10g R2 is not bundled with something that can be called a “recent” JVM … JDK 1.4.2

However, this is no real problem as you can monitor an older JVM with the tools provided in a recent one: more precisely you can monitor any JVM with a version number greater or equal to 1.4.1 (see jvmstat doc). Basically, you just need to:

  • download and install the latest available JDK (for example 1.6.0_18): jps, jstat and jstatd are included starting with jdk 1.5
  • download and install jvmstat 3.0 if you wish to have the “visualgc” tool and documentation for all the monitoring tools in one bundle.

Once you have done this you can try to run jps on your Windows 2003 Server where you have your Oracle Application Server 10g R2 installed … and … no, it won’t show you any of the JVM’s of the platform 🙂

Still, you can test that everything works as supposed by writing a simple test class such as the following one and running it with the JDK bundled in the Oracle Application Server:

public class Test {
  public static void main(String[] args) throws Exception {
    while (true) {

Once you have run it, this class should output a dot every 10 seconds in your console. If you run jps from another console, you should see a Java process corresponding to your running test class listed in the output produced by the jps tool. This should be enough to reassure you and prove that the jps from a JDK 1.6 can monitor Java processes originating from a JDK 1.4.2 😉

As a matter of fact the main reason why you don’t “see” the Java processes of your Oracle Application Server listed in the output produced by the jps tool is that they are run by very different OS users. This user / permission issue is documented in each of the monitoring tools: for example for jps see towards the end of the “Description” section.

Second thing to do: run the tools with the proper user

Oracle Application Server is installed as a Windows Service and as such all its processes are owned and executed by the Local System User.

When you run the jps tool (or any other monitoring tool provided with your freshly installed JDK 1.6), the user owning and executing the monitoring processes is the one you used to log into your Windows Server. There are several ways to run a command prompt as the same “Local System User” that runs the Windows Services, 2 of them are documented here. I chose to use the the psexec tool from Sysinternals:

psexec -i -s cmd.exe

Once you have a command prompt owned by the “Local System User” all the processes run from there inherit this user. If you run jps from within this command you will find more Java processes listed in the output of the tool … but … once again no luck, the Java processes corresponding to the “OC4J Homes” running your web applications are not there 🙂

Third thing to do: set the temporary directory of the JVM’s

That is the most tricky part of the procedure and finding a solution involved a “deep diving” into the source code of the jps tool and jvmstat classes.

As far as I have understood, starting from JDK 1.4.1 all the JVM’s can produce real time performance metrics in files. These files are located in the temporary directory of the user running the Java process under a folder named “hsperfdata_<user>”. On the Windows platform, for a Java process spawned by a Windows service (and thus owned by the “Local System User”), its performance file is located under c:\WINDOWS\Temp\hsperfdata_SYSTEM and is named after the id of the process.

When you run jps (or any other monitoring tool), the OS user running the command is used to determine the directory where the performance files should be found (based on the user’s temporary directory). For example, jps will return an entry for each file present in this directory.

However, when a Java process corresponding to an “OC4J Home” is spawned by OPMN, the location of the temporary directory is overridden through an environment variable and points to the temporary directory of the user who installed the server (some Administrator user).

The problem is that:

  • on one hand, you have to run jps as the “Local System User” in order to have the sufficient privileges to monitor the Java processes of the Application Server (because it is started as a service).
  • on the other hand the performance files are not located under the temporary directory of this same “Local System User”

The solution is to override the location of the temporary directory of the “OC4J Home”. Fortunately, this is easy using the Enterprise Manager console: in the “Administration” tab of every “OC4J Home”, there is a “Server Properties” link that opens a web form where you can find an “Environment” section. In this section, you just need to add an environment variable named “TEMP” with a value set to “c:\WINDOWS\Temp”.

Once this is done and your “OC4J Home” is restarted, your jps tool run as the “Local System User” will return (among others) the Java processes corresponding to your “OC4J Homes”. Moreover, under the directory “c:\WINDOWS\Temp\hsperfdata_SYSTEM” a new file will appear for each of these Java processes.

Fourth thing to do: the final monitoring architecture

As I have 3 Oracle Application Servers 10g installed on 3 different servers, my initial idea was to able to monitor them all from a remote PC using the “jstatd” tool on the servers and the “visualgc” or “jvisualvm” tools on the PC.

Running jstatd on a server is not very different from running jps. It has to be run as the “Local System User” with a policy file allowing the embedded RMI Server to be started (see jstatd documentation for more details):

grant codebase "file:${java.home}/../lib/tools.jar" {

In order to check that everything is ok, the jps tool can be used from a remote pc passing the host name or IP of the server.

jps -l <server_host_name_or_ip>

An even better way to do this and to have it automated is to install jstatd as a Windows Service as:

  • it will run with the needed user (Local System User)
  • it can be started automatically

Instructions on how to do install jstatd as a Windows Service can be found here. A brief summary follows:

  1. Get the instsrv.exe and srvany.exe tools for example from the Windows Server 2003 Resource Kit Tools.
  2. Run the following command to install the service:
    c:\<location>\instsrv.exe jstatd c:\<location>\srvany.exe
  3. Use a Windows Registry editor to create a key named “Parameters” under HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\jstatd. Then inside the Parameters key create a new String value (REG_SZ) named Application containing the full path to jstatd.exe and the security parameter (policy file).
  4. Use the Windows Services management application to check out that the jstatd service is configured to run as the local system user.
  5. Start the jstatd service.

Once jstatd is installed as a Windows Service on every server, the “jvisualvm” tool can be used to connect to these servers (Remote Host) and monitor their instrumented JVM’s. The “visualgc” tool can either be embedded as a plug-in of the “jvisualvm” tool or be run independently against the various instrumented JVM’s on the various servers where jstatd is running.