[INFO] Graph Tomcat JMX mbeans

[ztron]

Hey guys,

I just finished working on graphing information provided by Tomcat's JMX connector. This post is a high-level write-up that will take you to the point of getting the data.

Ingredients:

• Tomcat with a JMX connector open
  Jmx4perl
  Bunch of Perl scripts
  Cacti

Note: all of this is done in a Linux environment

1. Open a JMX port on Tomcat

Edit catalina.sh and add this:

Code: Select all

JAVA_OPTS="${JAVA_OPTS} \
-Dcom.sun.management.jmxremote \
-Dcom.sun.management.jmxremote.port=${REPLACE_WITH_PORT_NUM_HIGHER_THAN_1024} \
-Dcom.sun.management.jmxremote.ssl=false \
-Dcom.sun.management.jmxremote.password.file=${PATH_THAT_EXISTS}/jmxremote.password \
-Dcom.sun.management.jmxremote.access.file=${PATH_THAT_EXISTS}/jmxremote.access"

What this does: tells tomcat to open a non-ssl jmx connector on a specified port and use authentication.

For the authentication to work create the files mentioned in the $JAVA_OPTS:

Code: Select all

host#echo 'guest readonly' > PATH_THAT_EXISTS/jmxremote.access
host#echo 'username pAssw0rd' > PATH_THAT_EXISTS/jmxremote.password

I'll let you figure out what the above does

Note that the permissions on the above files should be 600 otherwise things will break (see catalina.out for exact error messages).


Start tomcat, wait for it to fire up completely and test that the JMX connector is up and running:

Code: Select all

host#telnet localhost ${PORT_NUM_YOU_CONFIGURED_IN_JAVA_OPTS}
Trying 127.0.0.1...
Connected to localhost.localdomain (127.0.0.1).
Escape character is '^]'.

Great, the port is open! Hold down Control-] and type quit to quit telnet.


2. Get Jmx4Perl from here: http://search.cpan.org/~roland/jmx4perl/ - and by the way, huge-huge-huge kudos to Roland, this Perl module is so great I was able to (mostly) ignore all of things for which I hate Perl while I was coding in Perl.

Install the dependencies. If you're on CentOS-5 here's a cookie:

Code: Select all

CentOS5Host# yum install -y perl-Module-Build.noarch perl-JSON.noarch perl-Module-Find.noarch perl-Test-Deep.noarch perl-Config-General perl-Nagios-Plugin

Now expand the tarball and install Jmx4Perl (hint: look at the README file). Once you build it, find the j4p.war file in the source directory, it should live in the agent subdir.

Copy this file to ${TOMCAT_HOME}/webapps

WARNING: You just installed a jp4.war application into your tomcat. If you have an apache proxy that sends all of the requests to tomcat via mod_proxy then you just exposed http://yourhost/j4p to the world! This means that someone could potentially do nasty things with your JVM. Take the necessary steps to prevent this from happening before you continue!

Test it all:

Code: Select all

[root@test01 ~]# jmx4perl http://localhost:8080/j4p info
Name:      Apache Tomcat
Vendor:    Apache
Version:   6.0.26
--------------------------------------------------------------------------------
Memory:
   Heap-Memory used    : 3 MB
   Heap-Memory alloc   : 10 MB
   Heap-Memory max     : 63 MB
   NonHeap-Memory max  : 116 MB
Classes:
   Classes loaded      : 1972
   Classes total       : 57083
Threads:
   Threads current     : 23
   Threads peak        : 35
OS:
   CPU Arch            : i386
   CPU OS              : Linux 2.6.18-8.el5xen
   Memory total        : 20 MB
   Memory free         : 20 MB
   Swap total          : 1055 MB
   Swap free           : 698 MB
   FileDesc Open       : 25
   FileDesc Max        : 1024
Runtime:
   Name                : 16283@test01
   JVM                 : 1.5.0_17-b04 Java HotSpot(TM) Client VM Sun Microsystems Inc.
   Uptime              : 8 d, 7 h, 32 m, 46 s
   Starttime           : Wed Aug  4 06:26:40 2010
[root@nagiostest01 ~]# 

Got this far and everything is working? Great.

To see what information you just got access to do this:

Code: Select all

host#jmx4perl http://localhost:8080/j4p list >mbeans
host#less mbeans

3. Now. Lets suppose we want to graph Heap Memory Usage. There are a couple of way to get the required metrics. We could just wrap around jmx4perl executable like so:

Code: Select all

[root@test01 ~]# echo "heap_mem_committed: $(jmx4perl http://localhost:8080/j4p read MEMORY_HEAP_COMITTED)"
heap_mem_committed: 10694656
[root@test01 ~]# 

From here on you create a data input method and all that jazz, plenty of tutorials out there on how to do that.

However, the above method is slow if you're going for many of mbeans (which, if you know why you're graphing tomcat metrics, you are) especially if you have 6-7 tomcats. That sit in a datacenter in California. And your Cacti box is in Arizona. Connected to the tomcat servers via a VPN. Getting my drift here?

The nice thing about Jmx4Perl is that it lets you write Perl scripts that can connect to your j4p agent and get the data much faster. Here's a (pretty ugly) example:

Code: Select all

#!/usr/bin/env perl

## WARNING - THIS SCRIPT HAS BEEN PROVIDED AS AN EXAMPLE ONLY.
use strict;
use warnings;
use JMX::Jmx4Perl;
use JMX::Jmx4Perl::Alias;   # Import certains aliases for MBeans
use Getopt::Long;
use Data::Dumper;

my $url='http://localhost:8080/j4p';

my %opts = ();
my $result = GetOptions(\%opts,
                        'url=s' =>\$url,
                        "user|u=s","password|p=s",
                        "user=s","password=s",
                        "filter|f=s",
                        "verbose|v!",
                        "help|h!" => sub { &Getopt::Long::HelpMessage() }
                       );


my $jmx = new JMX::Jmx4Perl(url => $url,user => $opts{user},password => $opts{password});

my $OS=$jmx->get_attribute('java.lang:type=OperatingSystem',
                                ['AvailableProcessors',
                                'ProcessCpuTime',]
                                );
dumpVar($OS);

sub dumpVar{
               my $mbean = $_[0];
                while( my ($k, $v) = each %$mbean ) {
                print "$k:$v " ;
    }
}

The original version of the above produces something like this:

Code: Select all

ProcessCpuTime:1741800000000 AvailableProcessors:4

Good times indeed. So all of the aforementioned should get you going. Hope this helps someone out there.


A very important note for the guys with tons of tomcats:
1. You will HAVE to roll your own data collection scripts
2. You're better off using the J4P proxy mode on one dedicated tomcat that talks to the JMX connectors on all of the tomcats that you want to monitor.


Further (and required) reading:

J4P Aliases: http://search.cpan.org/~roland/jmx4perl ... l/Alias.pm <-- this page has all of the aliases J4P knows of.
J4P Module doc: http://search.cpan.org/~roland/jmx4perl ... mx4Perl.pm <-- this page has _specific_ examples that show you how to get what you want from the J4P agents. Before you ask any questions about the Perl part - read this. Seriously.


Questions and comments are welcome.

Regards,
Ztron

[roland]

Hi  Ztron,

very nice howto (and thanks for chosing jmx4perl

A minor comment: If you don't need the proxy mode (which is the case in this example) you don't need to setup a JMX remote connector at all. Simply deploy the agent, and everything's fine.

When using the proxy mode, JMX needs to be exposed on the target platforms but not on the proxy itself. However, if I would have the choice, I would always use the agent on the target platform, even for many tomcats, since this has some intrinsic advantages over the proxy mode. In other words, the proxy mode has some limitations (no real bulk requests, problems with custom data types, RMI serialization issues, ...). For more on this, see http://labs.consol.de/blog/jmx4perl/agentless-jmx4perl/

For enhanced security you could also create a policy file which allows access only to certain MBeans and/or from a dedicated IP/subnet. For the moment you have to repackage j4p.war to include this policy file, though. See http://search.cpan.org/~roland/jmx4perl ... ESS_POLICY. Setting up security will get easier in one of the next releases.

A final tip: With 0.70 there comes a jmx4perl shell (j4psh), which allows for interactive exploration of MBeans with a readline interface and context sensitive command completion (even on MBean names). This is often more comfortable to use than 'jmx4perl list'. There is no much documentation for this shell right now, though (but there is a small online help when running j4psh).

...roland

[roland]

Another remark:

Code: Select all

...
my $jmx = new JMX::Jmx4Perl(url => $url,user => $opts{user},password => $opts{password});
...

The user and password given here are not used for authenticating against the JMX remote connector but for accessing the webapp j4p. Since in this example no security has been configured for j4p.war, the user and password here are simply ignored. The agent can be accessed by everyone.

[ztron]

Another remark:

Code: Select all

...
my $jmx = new JMX::Jmx4Perl(url => $url,user => $opts{user},password => $opts{password});
...

The user and password given here are not used for authenticating against the JMX remote connector but for accessing the webapp j4p. Since in this example no security has been configured for j4p.war, the user and password here are simply ignored. The agent can be accessed by everyone.

Absolutely right. This is just an excerpt from a script in my environment (I use the proxy mode because of unrelated considerations).

On another note, Roland, (and again, thank you for this wonderful module), can you please add a feature (or maybe an alias would be more appropriate?) to jmx4perl that would output mbean info in a format that Cacti can use? For example:

Code: Select all

host#~jmx4perl http://myhost/j4p read --cacti 'memory pool whatever Usage'
HeapMemoryCommitted: 200000 HeapMemoryUsed: 200300 HeapMemoryMax: 500000
host#~

or..

Code: Select all

host#~jmx4perl http://myhost/j4p  read HEAP_MEMORY_USAGE_CACTI
HeapMemoryCommitted: 200000 HeapMemoryUsed: 200300 HeapMemoryMax: 500000
host#~

That would be incredibly super-awesome.

Regars,
Z

[roland]

On another note, Roland, (and again, thank you for this wonderful module), can you please add a feature (or maybe an alias would be more appropriate?) to jmx4perl that would output mbean info in a format that Cacti can use?

That's a fine idea, I will put it on my list. Probably an own command or integration into check_jmx4perl is more appropriate. The latter (although Nagios and Cacti are only loosely related) would have the advantage that I could use parameterized checks for Cacti out of the box. Parameterized checks are the successors of aliases in 0.70 and allows for providing one or more parameters for querying an MBean-Template (e.g. for answering the question: "HTTP request throughput of Webapplication 'bla'" where 'bla' is the parameter).

[beargod]

There seems to be a growing need for historical data from JMX stuffed into a Cacti graph, but still without a 'good' way to do it. After poking around on the web it seems "jmx4perl -> Cacti" is the most promising.

Roland, have you decided yet if/when jmx4perl will natively support a data format Cacti understands?

Thanks for all of your hard work. You have really made the lives of countless ops people better, and sometimes, fun too.

[roland]

I just uploaded a developer version jmx4perl 0.90_2 to CPAN which contains a new script 'cacti_jmx4perl'.
It is a stripped down version of 'check_jmx4perl' with all of its configuration features, but without threshold checking.

Please give it a try and let me know, whether you have any suggestions or remarks (or bug reports).
I expect to release 0.90 final in about one or two weeks.

For example:

Code: Select all

 $ cacti_jmx4perl --url http://localhost:8080/jolokia --alias MEMORY_HEAP_USED
 15308376

 $ cacti_jmx4perl -u http://localhost:8080/jolokia --config config/memory.cfg --check memory
 Heap:15341168 Non-Heap:19450312

The later sample uses a configuration file for a multi-check, which comes with the jmx4perl distribution.

enjoy it ...
...roland

[roland]

FYI, Jmx4Perl 0.90_3 has been pushed now to the CPAN Mirrors: http://search.cpan.org/~roland/jmx4perl-0.90_3