One aspect of my described System Management Patterns includes gathering correlated metrics through the various layers of your system. Since SNMP is the defacto standard in monitoring. I used the SNMP aspects of the JVM.
I never really got into SNMP before and since Tiger directly supports it, the time had come.
The aim was to add our JVM to an existing monitoring system, jffnms.
Being a newbie to SNMP I had a few setbacks than anticipated.
While the jconsole is cool and useful, I wanted application/JVM monitoring to fit in with our existing monitoring tools and also for collected data to be nicely correlated with os and network stats.
Sun's SNMP guide to enabling a port for monitoring is straight forward, but I found learning about snmp from google painful, and the best bits were hard to find.
If you've got/done the following then it will be easy to get started:
- net-snmp installed and read some doco on SNMP and MIBS, the net-snmp doco is better than most but has alot of fluff.
- tiger
- have added followed the snmp monitoring setup so you have a listenter and configure acl.
- and some kind of server to run
If any of the acl settings are wrong or file permissions aren't set correctly, the jvm won't start.
Once it is going you could double check, using netstat.
Now to use the snmp tools you will need to load in Sun's JVM-MANAGEMENT-MIB
It's good practice to use snmptranslate first to see if there are any problems with the MIB definition, in particular any missing dependencies.
snmptranslate -M .:/usr/share/snmp/mibs -m JVM-MANAGEMENT-MIB -IR -Tp jvmMgtMIB
Assuming you are running this from the current directory where the new MIB is that command should work. -M is giving a search path for mibs and -m is telling is specifying which mibs we want to load. The final parameter, jvmMgtMIB is the module identity, you can see that inside the mib file.
You should see a tree view of the mib starting like this:
+--jvmMgtMIB(1)
|
+--jvmMgtMIBObjects(1)
| |
| +--jvmClassLoading(1)
| | |
| | +-- -R-- Gauge jvmClassesLoadedCount(1)
| | +-- -R-- Counter64 jvmClassesTotalLoadedCount(2)
| | +-- -R-- Counter64 jvmClassesUnloadedCount(3)
| | +-- -RW- EnumVal jvmClassesVerboseLevel(4)
...
And finally we can peek at some jvm stats, just plugin your port:
snmpwalk -v 2c -M .:/usr/share/snmp/mibs -m JVM-MANAGEMENT-MIB -c public localhost:port jvmMgtMIB
So in this snippet we are using snmp version 2c and still loading the sun mib.
You should see some results like this:
...
JVM-MANAGEMENT-MIB::jvmMemoryHeapInitSize.0 = Counter64: 838860800 bytes
JVM-MANAGEMENT-MIB::jvmMemoryHeapUsed.0 = Counter64: 39430568 bytes
JVM-MANAGEMENT-MIB::jvmMemoryHeapCommitted.0 = Counter64: 832438272 bytes
JVM-MANAGEMENT-MIB::jvmMemoryHeapMaxSize.0 = Counter64: 1664876544 bytes
JVM-MANAGEMENT-MIB::jvmMemoryNonHeapInitSize.0 = Counter64: 8552448 bytes
JVM-MANAGEMENT-MIB::jvmMemoryNonHeapUsed.0 = Counter64: 9260472 bytes
JVM-MANAGEMENT-MIB::jvmMemoryNonHeapCommitted.0 = Counter64: 9863168 bytes
JVM-MANAGEMENT-MIB::jvmMemoryNonHeapMaxSize.0 = Counter64: 100663296 bytes
...
To make this more useful you should install the mib properly, put it in with the others and update your ~/.snmp/snmp.conf file to have "mibs +JVM-MANAGEMENT-MIB".
One of the things I hated about getting started with SNMP were the misleading errors.
Such as
- snmpwalk: Timeout - can be caused by a bunch of things, like not specifying the correct version of snmp that you are trying to query!
- SNMPv2-SMI::mib-2 = No Such Object available on this agent at this OID - can be caused when you don't specify the OID, eg. jvmMgtMIB
Well writing this out takes too long, I'll have to figure out the jffnms integration later.
6 Responses
Stay in touch with the conversation, subscribe to the RSS feed for comments on this post.
Hi,
Have you integrated the JVM monitoring to JFFNMS yet??
Thanks!!! Precise and well written. This is excellent. I tried this for Windows NET-SNMP 5.4.2.1 with a Tomcat 5.5.27 JVM and it works! I have been looking for this for a month!
I just want to point out one thing, in your snmptranslate above,
“snmptranslate -M .:/usr/share/snmp/mibs -m JVM-MANAGEMENT-MIB -IR-Tp jvmMgtMIB”
there should be a space between “-IR-Tp”, so it should be
“snmptranslate -M .:/usr/share/snmp/mibs -m JVM-MANAGEMENT-MIB -IR -Tp jvmMgtMIB”
This is working out beautifully. Thanks again.
Just want to add that I also found this article helpful:
Open NMS – JVM Monitoring using SNMP
http://www.opennms.org/index.php/JVM_Monitoring_using_SNMP
In my snmpd.conf, I added this line:
“proxy -v 2c -c public 127.0.0.1:1666 .1.3.6.1.4.1.42″. So if I simply do “snmpwalk -v 2c -c public 127.0.0.1 .1″, it will include my Tomcat’s JVM info.
Not it leaves me wondering if I can include multiple JVMs into NET-SNMP… I can specify different port value (“-Dcom.sun.management.snmp.port=xxxx”) for each JVM but I am thinking the JVM-MANAGEMENT-MIB’s OIDs will collide. May be I can tweak the (top) OID inside JVM-MANAGEMENT-MIB so each JVM has it’s own MIB with different OIDs. Just a theory.
I also want to add that I tried to write (more like hack) the same SNMP JVM proxy using SNMP4J (snmp4j.org), I had some success but using the NET-SNMP with JVM-MANAGEMENT-MIB is definitely cleaner and require less future maintenance.
In addition to monitoring JVM, I wonder if I can monitor web server’s (say WebLogic\Tomcat) MBeans via SNMP. Anyone done this? I imagine I would need a tool to generate MBean-MIB, but how do I hook NET-SNMP, MBean-MIB and the MBean together?
Judging by the below link you cannot poll a MBean using SNMP.
http://java.sun.com/developer/technicalArticles/J2SE/jconsole.html
We have ditched snmp monitoring altogether, it wasn’t worth the hassle for our situation. Which was monitoring only, polling every 2-5mins.
Besides jvm stats, we had custom application stats that we wanted monitored. These stats were originally available for query by snmp.
The snmp manager tools are too much trouble to import a custom mib. Custom Mibs are too much trouble to manage, change etc. (devs need to know too much about it).
The snmp managers mostly just shielded rrdtool anyway and forced you to use their crappy UI to edit rrdtool properties. So we bypassed them altogether.
Now we have also removed snmp, so all stats are available over http. (Not quite RESTful, but a decent start).
When we finally got rid of snmp, it made the adding new metrics really easy, many more devs have been adding them now. Before it was just too costly in time/effort for ppl new to it to work it all out to add new ones.