check_jmx4perl: New Nagios configuration style

Posted on August 23rd, 2010 by roland

Since version 0.70, check_jmx4perl has support for configuration files. JMX Nagios checks are now considerably simpler to configure and multi checks add even more performance and flexibility.

Gerhard presented in a previous article a well crafted setup of Nagios commands and service definitions for use with check_jmx4perl. Although these suggestions nicely decouple the checks into an service hierarchy and provide a service dependency to avoid superflous notifications, starting with 0.70 things can still be improved further.

First, the amount of command definitions can be reduced to a single one. Here is the command to rule them all:

define command {
   command_name j4p_cmd   
   command_line $USER5$/check_jmx4perl \
                --config $USER5$/jmx4perl.cfg \
                --server $HOSTNAME$ \
                --check  $ARG1$ $ARG2$ $ARG3$ $ARG4$
}

$USER5$ is the top-level directory for jmx4perl specific Nagios stuff. It is recommended to put it below the Nagios etc/ directory (e.g. /usr/local/nagios/etc/jmxperl). $USER5$ needs to be declared in Nagios’ resources.cfg as usual. As it can be seen from the example above, check_jmx4perl is called with three arguments only:

--config to point to check_jmxperls configuration file. More on this later.
--server for the server to monitor. The trick here is, that $HOSTNAME$ is used for pointing to the server definition (which contains e.g. the URL for accessing the jmx4perl agent) within the configuration file. So, each Nagios host which is configured for monitoring with check_jmx4perl should have a <Server> section in the configuration with the same name (case sensitive).
--check finally specifies the <Check> or <MultiCheck> defined in the configuration file. $ARG1$ specifies the check name and must be provied in the service definition using this command. $ARG2$ (and the others) is an optional argument used for providing check parameters.

Since jmx4perl comes with some standard checks, which can be customized to individual needs, it is a good idea, to not mix these with your customizations within the same file. Further jmx4perl updates will add to these standard checks, so the recommendation for the directory layout is as follows:

Extract each jmx4perl distribution in a dedictated jmx4perl source directory (e.g. /usr/local/src/jmx4perl). Use a symlink to point to the latest jmx4perl distribution in use:

 $ mkdir /usr/local/src/jmx4perl
 $ cd /usr/local/src/jmx4perl
 $ tar zxvf /tmp/jmx4perl-0.71.tar.gz
 $ ln -s -f jmx4perl-0.71 jmx4perl

Install jmx4perl as usual (the perl Build.PL; ./Build install combo),
Within Nagios’ jmx4perl directory, create symlinks to check_jmx4perl and the default configuration:

 $ cd /usr/local/nagios/etc
 $ mkdir jmx4perl
 $ cd jmx4perl
 $ ln -s /usr/bin/check_jmx4perl .
 $ ln -s /usr/local/src/jmx4perl/jmx4perl/config default

After these preparations, the individual site configuraton for check_jmx4perl can be added. In order to separate server and check declarations, two files are recommended: servers.cfg for holding <Server> definitions only and jmx4perl.cfg which holds the checks and which is referenced from within the Nagios command definition shown above.

The servers.cfg looks like:

  # Server definitions.
  # Names must reflect nagos host names 
  # (case sensitive)
 
  <Server webshop>
    Url http://webshop:8080/j4p
    User j4p
    Password consol
  </Server>
 
  <Server cms>
    Url http://cms:8081/j4p
  </Server>

In this example, two servers are defined. webshop (using a user/password for authentication) and cms. Note, that there should be Nagios host definitions with the same name.

Finally, the main configuration file jmx4perl.cfg:

# Hosts
include servers.cfg
 
# Default definitions
include default/memory.cfg
include default/tomcat.cfg
include default/threads.cfg
 
# ====================================
# Check definitions
 
<Check j4p_memory_heap>
  Use memory_heap
  Critical 90
  Warning 80
</Check>
 
<Check j4p_thread_count>
  Use thread_count
  Critical 1000
  Warning 800
</Check>
 
# Check for uptime, used as kind of 'ping' for 
# service dependencies
<Check j4p_uptime>
  MBean java.lang:type=Runtime
  Attribute Uptime
  Warning 120:
  Critical 60:
</Check>
 
# A multi check combining two checks
<MultiCheck j4p_jvm>
  Check j4p_memory_heap
  Check j4p_thread_count
</Check>
 
# ===========================================
# Tomcat Apps
# -----------
 
<Check app_sessions>
  Use tc_session_active($0)
  Critical 1000
  Warning 800
</Check>
 
<Check app_servlet_requests>
  Use tc_servlet_requests($0)
  Critical 6000
  Warning 5000
</Check>
 
# Multicheck for a webapp
# $0: Web-Application name (e.g. "j4p")
# $1: Servlet-Name (e.g. "j4p-agent")
<MultiCheck app_all>
  Check app_sessions($0)
  Check app_servlet_requests($1)
</MultiCheck>
 
# Connector related multicheck
<MultiCheck connector_all>
  Check tc_connector_threads($0)
  Check tc_connector_received_rate($0)
  Check tc_connector_sent_rate($0)
  Check tc_connector_processing_time($0)
  Check tc_connector_requests($0)
  Check tc_connector_error_count($0)
</MultiCheck>

servers.cfg is included for refering to the server definitions and a set of default definitions is included with predefined checks which are references later on in this file via the Use directive.

After this prelude, the real checks are defined. These are a mixture of plain checks (<Check>) and multi checks (<MultiCheck>). Each of these checks can be referenced from a Nagios service definition via its name.

For the Nagios service definitions, there are essentially two styles (which can be matched and mixed):

Use multiple, single checks, with a proper service dependency (as described here in detail).
Use a multi check in order to combine several checks.

Both approaches have advantages and disadvantages:

Multichecks are faster since they combine multiple checks in a single request, resulting in a single server turnaround
Multichecks don’t need a service dependency, except you want to use multiple multi-checks on a single host in which case a service dependency of the multi checks on an ‘uptime’ check make sense.
You can only attach a single notification group to a multi check, single checks are more flexible here.
Pnp4Nagios graphing can handle both modes.

Finally here are the service definitions:

define service {
    use                   generic-service,srv-pnp
    service_description   jvm_memory_heap
    display_name          Heap Memory
    hostgroup_name        tomcat,jetty
    check_command         j4p_cmd!j4p_memory_heap
}
 
# Multicheck combining multiple JVM checks
define service {
    use                   generic-service,srv-pnp
    service_description   jvm_all
    display_name          All JVM Params
    hostgroup_name        tomcat,jetty
    check_command         j4p_cmd!j4p_jvm
}
 
# Tomcat connector checks
# -----------------------
define service {
    use                   generic-service,srv-pnp
    service_description   tc_connector_8080
    display_name          Connector 8080 - Checks
    hostgroup_name        tomcat
    check_command         j4p_cmd!connector_all!http-8080
}
 
# Tomcat web-app checks 
# ---------------------
define service {
    use                   generic-service,srv-pnp
    service_description   tc_webshop_sessions
    display_name          Webshop Number sessions
    host_name             webshop
    check_command         j4p_cmd!app_sessions!shopApp
}

Multiple single checks and multi-checks are combined in this example. Also note, that for generic JVM checks and connector checks, hostgroup_name is used in order to define the services for similar hosts only once. Two hostgroups are used here: tomcat for Tomcat servers and jetty for Jetty servers. Both are associated with the same generic JVM checks, in addition the host group tomcat is associated with tomcat specific connector checks. The most specific check, which checks a servlet is directly associated with a host (since a webapplication only runs on a single host only in non-clustered environments).

So, what is the take away of this post ? (and BTW, congratulations that you made it until the end ;-)

You only need a single Nagios command definition.
Switching over check definitions into check_jmx4perl configuration has serveral advantages:
- Advanced features like inheritance, multi-checks and parameterization.
- Changes are applied hot during the next check without restarting Nagios.
- Using check_jmx4perl configuration files keeps Nagios configuration tidy.
You have the choice: Multiple single checks or multi-check (or a wild mixture :).
Syncing Nagios host names and server names in the check_jmxperl configuration files simplifies quite some things.
Don’t edit the default checks, use includes instead and use an intelligent directory layout.

For more information, please consult the man page of check_jmx4perl. Looking back, writing these 30+ pages seemed to be the hardest part for version 0.70. Writing documentation (man pages or books) is tough business. Am I right, Gerhard ? ;-)

Tags: check_jmx4perl, jmxp4erl, Nagios
Filed under Jmx4Perl, Nagios |

12 Responses to “check_jmx4perl: New Nagios configuration style”

Christian Ruschke Says:
September 1st, 2010 at 11:54
Hi Roland,

I am a happy jmx4perl user since half a year and monitor roughly 30 Glassfish deployments with it. However with updating to 0.71 I think some problem has been introduced which I would like to share with you (but it can also be my mistake as well :-).

Here is the problem… We have to use the proxy mode for given reason, and this command

jmx4perl http://localhost:8180/j4p –target service:jmx:rmi:///jndi/rmi://jeeserver:8686/jmxrmi –target-user admin –target-password mypassword

works absolutely OK.

However the Nagios check command with similar parameters

check_jmx4perl –url http://localhost:8180/j4p –target service:jmx:rmi:///jndi/rmi://jeeserver:8686/jmxrmi –warning :92 –critical :95 –target-user admin –target-password mypassword –alias MEMORY_HEAP_USED –base MEMORY_HEAP_MAX –unit=B

gives following error message UNKNOWN – Error: Error searching for :j2eeType=J2EEServer,: java.lang.Exception: Invalid credentials specified for administrative user at /usr/local/share/perl/5.10.1/JMX/Jmx4Perl.pm line 486.

In my proxy server logfile there is after enabling j4pdebugging there is a

[#|2010-09-01T10:16:43.683+0100|SEVERE|sun-appserver2.1|javax.enterprise.system.container.web|_ThreadID=24;_ThreadName=httpSSLWorkerThread-8180-1;_RequestID=1d0cfe90-0c57- 4380-9aab-2ece0623f491;|WebModule[/j4p]j4p-agent: Error 403 java.lang.Exception: Invalid credentials specified for administrative user at org.jmx4perl.http.HttpRequestHandler.handleThrowable(HttpRequestHandler.java:162) … My impression is that the check_jmx4perl somehow loses the target username/password.

Please let me know your thoughts.

Cheers Christian
[Reply]
roland Reply:
September 1st, 2010 at 18:11
Hi Christian,

Thanks for your Bug report. I will look at it immediately after my holidays. Unfortunately I only have limited internet access at the moment. …roland
[Reply]
Christian Ruschke Reply:
September 2nd, 2010 at 11:28
@roland, Hi Roland, thanks, you can safely enjoy your vacation as I was able to fix that one temporarily by downgrading back to v0.65 :-). Made my Nagios looking much healthier immediately…

Cheers Christian
[Reply]
giuseppe Says:
September 2nd, 2010 at 18:15
Hi Roland,

maybe I’ve got the same issue with v0.71 in proxy mode.

This command is ok :

jmx4perl http://192.168.30.33:7001/j4p –target service:jmx:rmi:///jndi/rmi://192.168.30.33:7091/jmxrmi read java.lang:type=Memory HeapMemoryUsage used –product weblogic –target-user myuser –target-password mypassword

but Nagios check failes :

./check_jmx4perl –url http://192.168.30.33:7001/j4p –name memory_used –mbean java.lang:type=Memory –attribute HeapMemoryUsage –path used –critical 10000000 –warning 5000000 –target service:jmx:rmi:///jndi/rmi://192.168.30.33:7091/jmxrmi –target-user myuser –target-password mypassword

UNKNOWN – Error: 403 java.lang.Exception: Authentication failed! Credentials required

Not sure if it’s a bug or my miss configuration because I’ve been using Jmx4perl only for few days.

As a temporary workaround I’ve disabled authentication so all users are allowed all access.

Cheers Giuseppe
[Reply]
roland Says:
September 2nd, 2010 at 18:37
@giuseppe, this probably is the same issue as Christian just reported. Give me a week, I will tackle this problem as soon as my vacation is finished (or at least when I have a better internet connection ;-)

Thanx for the report.

…roland
[Reply]
roland Says:
September 5th, 2010 at 22:51
Christian & Giuseppe,

I nailed down the bug and fixed it in the current developer release 0.72_1 just uploaded to CPAN. Thanks for spotting this one, it slipped in during the refactorings for the extended configuration syntax. In fact, –target-user and –target-password were always ignored (same BTW for –proxy-user and –proxy-password). This should work now.

BTW, if you want to use the proxy-mode together with server definitions in a configuration file, you can define your servers like
```
 <Server tc-proxy>
   Url http://localhost:9090/j4p
   <Target>
     Url service:jmx:rmi:///jndi/rmi://localhost:8686/jmxrmi
     User admin
     Password admin
   </Target>
 </Server>
```
As I know now, that there are really users of the proxy-mode out there ;-), I will try to polish it a bit for the next versions:
- Integrate it into my integration test suite
- Add some connection pooling for reusing JSR-160 connections within the proxy
No promises, though ;-) …

If you give 0.72_1 a try, please let me know, whether it works for you.

…roland
[Reply]
giuseppe Reply:
September 6th, 2010 at 13:54
@roland, Hi, V0.72_1 is working for me :) I’ve tested authentication and server definitions in a config file and it’s working now. The new release has been uploaded on http://cpan.panu.it/authors/id/R/RO/ROLAND/ but http://search.cpan.org/~roland/jmx4perl/scripts/check_jmx4perl points to V0.71

Thanks for your support
[Reply]
roland Reply:
September 6th, 2010 at 14:19
@giuseppe, fine to hear that it works for you.

Since V0.72_1 is a developer version, you won’t get it by default. By default, only the last, stable version is offered without version number (0.71). On search.cpan.org you can go to http://search.cpan.org/~roland/jmx4perl-0.72_1/ for the 0.72_1 release.
[Reply]
roland Reply:
September 6th, 2010 at 14:20
Ah, I forgot: You can expect a final 0.72 version within the next two weeks.
[Reply]
Christian Ruschke Reply:
September 7th, 2010 at 10:46
@giuseppe, v0.72_1 works OK for me either, thanks for this fix.

Cheers Christian
[Reply]
giuseppe Says:
September 27th, 2010 at 11:59
Hi Roland,

sorry if I’m stressing you a little :) I didn’t get a reply and I’m wondering if you have got my Bug Report last friday.

Cheers Giuseppe
[Reply]
roland Reply:
September 27th, 2010 at 12:52
Sorry, no. Where did you post it ? BTW, the prefered way for opening bug in rt.cpan.org (https://rt.cpan.org/Public/Dist/Display.html?Name=jmx4perl)
[Reply]

check_jmx4perl: New Nagios configuration style

12 Responses to “check_jmx4perl: New Nagios configuration style”

Leave a Reply

Projects

Forum

Tags