SNMP is a natural fit for publishing application statistics. The thing I don't like about existing snmp agent implementations is that they are all pretty language dependent and they are built on the old style, define a mib, crank a wheel to generate a template, then you finally get to add code for you metrics.

So firstly work has been done to have an agent publish statistics based on results from an xml stream. This has been built on top of pysnmp.

This allows us to have an agent on each device/box that can publish stats, regardless of language as long as you instrument the device in question (and you have a python 2.4 environment).

Once your stats are published, you will need a manager to gather them.
While any SNMP manager can be used, often like with jffnms, you will need to write your own poller anyway. We've created an easily configurable pollers so a manager, can poll and update rrdtool database.

While some of the leg work is alleviated, you will still need to do the following: instrument your app, define your rrd datasources and optionally design your mib.

I'm currently looking at opensourcing this very shortly.

I never really got into SNMP before and since Tiger directly supports it, the time had come.

The aim was to add our JVM to an existing monitoring system, jffnms.

Being a newbie to SNMP I had a few setbacks than anticipated.

While the jconsole is cool and useful, I wanted application/JVM monitoring to fit in with our existing monitoring tools and also for collected data to be nicely correlated with os and network stats.

Sun's SNMP guide to enabling a port for monitoring is straight forward, but I found learning about snmp from google painful, and the best bits were hard to find.

If you've got/done the following then it will be easy to get started:

1. net-snmp installed and read some doco on SNMP and MIBS, the net-snmp doco is better than most but has alot of fluff.
2. tiger
3. have added followed the snmp monitoring setup so you have a listenter and configure acl.
4. and some kind of server to run

If any of the acl settings are wrong or file permissions aren't set correctly, the jvm won't start.

Once it is going you could double check, using netstat.

Now to use the snmp tools you will need to load in Sun's JVM-MANAGEMENT-MIB

It's good practice to use snmptranslate first to see if there are any problems with the MIB definition, in particular any missing dependencies.

snmptranslate -M .:/usr/share/snmp/mibs -m JVM-MANAGEMENT-MIB -IR -Tp jvmMgtMIB

Assuming you are running this from the current directory where the new MIB is that command should work. -M is giving a search path for mibs and -m is telling is specifying which mibs we want to load. The final parameter, jvmMgtMIB is the module identity, you can see that inside the mib file.

You should see a tree view of the mib starting like this:
+--jvmMgtMIB(1)
|
+--jvmMgtMIBObjects(1)
| |
| +--jvmClassLoading(1)
| | |
| | +-- -R-- Gauge jvmClassesLoadedCount(1)
| | +-- -R-- Counter64 jvmClassesTotalLoadedCount(2)
| | +-- -R-- Counter64 jvmClassesUnloadedCount(3)
| | +-- -RW- EnumVal jvmClassesVerboseLevel(4)
...

And finally we can peek at some jvm stats, just plugin your port:

snmpwalk -v 2c -M .:/usr/share/snmp/mibs -m JVM-MANAGEMENT-MIB -c public localhost:port jvmMgtMIB

So in this snippet we are using snmp version 2c and still loading the sun mib.

You should see some results like this:
...
JVM-MANAGEMENT-MIB::jvmMemoryHeapInitSize.0 = Counter64: 838860800 bytes
JVM-MANAGEMENT-MIB::jvmMemoryHeapUsed.0 = Counter64: 39430568 bytes
JVM-MANAGEMENT-MIB::jvmMemoryHeapCommitted.0 = Counter64: 832438272 bytes
JVM-MANAGEMENT-MIB::jvmMemoryHeapMaxSize.0 = Counter64: 1664876544 bytes
JVM-MANAGEMENT-MIB::jvmMemoryNonHeapInitSize.0 = Counter64: 8552448 bytes
JVM-MANAGEMENT-MIB::jvmMemoryNonHeapUsed.0 = Counter64: 9260472 bytes
JVM-MANAGEMENT-MIB::jvmMemoryNonHeapCommitted.0 = Counter64: 9863168 bytes
JVM-MANAGEMENT-MIB::jvmMemoryNonHeapMaxSize.0 = Counter64: 100663296 bytes
...

To make this more useful you should install the mib properly, put it in with the others and update your ~/.snmp/snmp.conf file to have "mibs +JVM-MANAGEMENT-MIB".

One of the things I hated about getting started with SNMP were the misleading errors.
Such as

• snmpwalk: Timeout - can be caused by a bunch of things, like not specifying the correct version of snmp that you are trying to query!
• SNMPv2-SMI::mib-2 = No Such Object available on this agent at this OID - can be caused when you don't specify the OID, eg. jvmMgtMIB

Well writing this out takes too long, I'll have to figure out the jffnms integration later.

Usually extra logging is turned on or added, but if your running in production in the heat of the moment that may not be an option.

Also you may not have convenient access to monitor other servers involved.

I think socketcounting your server process is an under estimated technique for quickly seeing what your server process is connected too, or what is connecting to it.

socketcount <server.pid>

to sum connections per server/ip address from/to your server process.


file:/usr/local/bin/socketcount
#!/bin/bash

echo server pid = $REGEX, using temp file $SNAPSHOT
#check all tcp connections except listeners if using local box
#export NETSTAT_CMD="netstat -tnpa | grep ' $REGEX' | egrep -v '(LISTEN||^Proto||^\(Not)'"
#echo "executing $NETSTAT_CMD"
netstat -tpa | grep $REGEX | egrep -v 'LISTEN' > $SNAPSHOT
echo "------------------------------------"
echo "Connections - this counts connections both ways, so running locally may appear weird."
#echo "ServerConnections can be identified, by host and port."
cat $SNAPSHOT | awk '{print $5}' | sort | awk -f /usr/local/bin/count-dups.awk


The socketcount file relies on the following awk script that sums up by hostname, it would be nice to keep it in one file, I'm sure there is an easier way to do it, but atleast it's readable.


file:/usr/local/bin/count-dups.awk
# expect single field for aggregation
BEGIN{
name=0;
count=0;
total=0}
{
if (name != $1) {
if (name != 0) {
total += count;
printf("%s = %d\n", name, count);
count = 1
}
name = $1
count = 1
}
else {
count++
}
}
END {
printf("%s = %d\n",name,count)
total += count
printf("Total = %d\n",total)
}