[Xymon] jboss monitorin

12 Sep 2012


      hi
you can also use the jvm check done by olivier (sent yesterday)
it uses jps/jinfo and jstat.
maybe you have to adapt the jps grep but it is minor.
#!/usr/bin/perl
use Data::Dumper;
my @jps = split /\n/, sudo  /usr/java/jdk1.6/bin/jps  -v | grep Bootstrap;
my %res;
foreach my $jvm (@jps) {
   my @t = split /\s+/, $jvm;
   my $pid = shift @t;
   if ( my @tt = grep /-Dcatalina.base/, @t) {
           my $tid = shift @tt;
           my @ttid = split /\//, $tid;
           my $id = pop @ttid;
           if ( `sudo /usr/java/jdk1.6/bin/jinfo $pid 2>&1 ` =~ /java.specification.version/ ) {
                   $res{"$id"}{"version"}  = "jdk1.6";
                   $res{"$id"}{"pid"}      = $pid;
           } elsif ( `sudo /usr/java/jdk1.5/bin/jinfo $pid 2>&1 ` =~ /java.specification.version/ ) {
                   $res{"$id"}{"version"} = "jdk1.5";
                   $res{"$id"}{"pid"}    = $pid;
           } else { print "Cannot identify jvm version for $pid\n"; }


   }
}
foreach my $id (keys %res) {
   my $pid = $res{"$id"}{"pid"};
   my $bin = $res{"$id"}{"version"};

   $str = `sudo /usr/java/$bin/bin/jstat -gccapacity -t $pid 1 1 | tail -n 1`;
   ($t,$timestamp,$NGCMN,$NGCMX,$NGC,$S0C,$S1C,$EC,$OGCMN,$OGCMX,$OGC,$OC,$PGCMN,$PGCMX,$PGC,$PC,$YGC,$FGC) = split /\s+/, $str;
   $str = `sudo /usr/java/$bin/bin/jstat -gc -t $pid 1 1 | tail -n 1`;
   ($t,$timestamp,$s0c,$s1c,$s0u,$s1u,$ec,$eu,$oc,$ou,$pc,$pused,$ygc,$ygct,$fgc,$fgct,$gct) = split /\s+/, $str;

   my $perused = sprintf ("%.2f",( $pused * 100 ) / $PGCMX );
   if ($perused > 99) {
           $color = "red";
   } elsif ( $perused > 98 ) {
           $color = "yellow";
   } else { $color = "green"; }


   $res{"$id"}{"gccap"}{"label"} = "Gc Capacity";
   $res{"$id"}{"gccap"}{"value"} = $perused;
   $res{"$id"}{"gccap"}{"alert"} = 0;

   $str = `sudo /usr/java/$bin/bin/jstat -gc -t $pid 1 1 | tail -n 1`;
   ($t,$timestamp,$s0c,$s1c,$s0u,$s1u,$ec,$eu,$oc,$ou,$pc,$pu,$ygc,$ygct,$fgc,$fgct,$gct) = split /\s+/, $str;

   $s0c *= 1024; $s1c *= 1024; $s0u *= 1024; $s1u *= 1024; $ec  *= 1024; $eu  *= 1024; $oc  *= 1024; $ou  *= 1024;
   $pc  *= 1024; $pu  *= 1024;
   $res{"$id"}{"gccap"}{"trends"} = "[jvm.$id.rrd]
DS:SOC:GAUGE:600:0:U $s0c
DS:S1C:GAUGE:600:0:U $s1c
DS:S0U:GAUGE:600:0:U $s0u
DS:S1U:GAUGE:600:0:U $s1u
DS:EC:GAUGE:600:0:U $ec
DS:EU:GAUGE:600:0:U $eu
DS:OC:GAUGE:600:0:U $oc
DS:OU:GAUGE:600:0:U $ou
DS:PC:GAUGE:600:0:U $pc
DS:PU:GAUGE:600:0:U $pu
DS:GCAP:GAUGE:600:0:U $perused";
}
$line = "Gc Capacity and VM memory reporting\n";
#$line = "$ENV{'BB'} $ENV{'BBDISP'} \"status $ENV{'MACHINE'}.jvm";
$mcolor = 0;
@colors = ("green","yellow","red");
$trends = "";
foreach my $id (sort keys %res) {
   $line .= "&". at colors[$res{"$id"}{"gccap"}{"alert"}]." $id : ".$res{"$id"}{"gccap"}{"value"}."%\n";
   if ( $res{"$id"}{"gccap"}{"alert"} > $mcolor ) {
           $mcolor = $res{"$id"}{"gccap"}{"alert"} ;
   }
   $trends .= $res{"$id"}{"gccap"}{"trends"}."\n";
}
$date = /bin/date;
$color = $colors[$mcolor];
$line .= "\n<!-- linecount=6 -->";
system qq/ $ENV{'BB'} $ENV{'BBDISP'} "status $ENV{'MACHINE'}.jvm $color $line $date <!-- linecount=6 --> "/;
system qq/ $ENV{'BB'} $ENV{'BBDISP'} "data $ENV{'MACHINE'}.trends\n$trends"/;
graphs.cfg
[jvm]
FNPATTERN jvm.(.*).rrd
TITLE Garbage Collection
YAXIS Bytes
DEF:EC at RRDIDX@=@RRDFN@:EC:AVERAGE
DEF:EU at RRDIDX@=@RRDFN@:EU:AVERAGE
DEF:OC at RRDIDX@=@RRDFN@:OC:AVERAGE
DEF:OU at RRDIDX@=@RRDFN@:OU:AVERAGE
DEF:PC at RRDIDX@=@RRDFN@:PC:AVERAGE
DEF:PU at RRDIDX@=@RRDFN@:PU:AVERAGE
DEF:GCAP at RRDIDX@=@RRDFN@:GCAP:AVERAGE
LINE1:EC at RRDIDX@#@COLOR@:@RRDPARAM@ Current eden space capacity.\n
GPRINT:EC at RRDIDX@:LAST: \: %5.1lf (cur)
GPRINT:EC at RRDIDX@:MAX: \: %5.1lf (max)
GPRINT:EC at RRDIDX@:MIN: \: %5.1lf (min)
GPRINT:EC at RRDIDX@:AVERAGE: \: %5.1lf (avg)\n
LINE1:EU at RRDIDX@#@COLOR@:@RRDPARAM@ Eden space utilization.\n
GPRINT:EU at RRDIDX@:LAST: \: %5.1lf (cur)
GPRINT:EU at RRDIDX@:MAX: \: %5.1lf (max)
GPRINT:EU at RRDIDX@:MIN: \: %5.1lf (min)
GPRINT:EU at RRDIDX@:AVERAGE: \: %5.1lf (avg)\n
LINE1:OC at RRDIDX@#@COLOR@:@RRDPARAM@ Current old space capacity.\n
GPRINT:OC at RRDIDX@:LAST: \: %5.1lf (cur)
GPRINT:OC at RRDIDX@:MAX: \: %5.1lf (max)
GPRINT:OC at RRDIDX@:MIN: \: %5.1lf (min)
GPRINT:OC at RRDIDX@:AVERAGE: \: %5.1lf (avg)\n
LINE1:OU at RRDIDX@#@COLOR@:@RRDPARAM@ Old space utilization.\n
GPRINT:OU at RRDIDX@:LAST: \: %5.1lf (cur)
GPRINT:OU at RRDIDX@:MAX: \: %5.1lf (max)
GPRINT:OU at RRDIDX@:MIN: \: %5.1lf (min)
GPRINT:OU at RRDIDX@:AVERAGE: \: %5.1lf (avg)\n
LINE1:PC at RRDIDX@#@COLOR@:@RRDPARAM@ Current permanent space capacity.\n
GPRINT:PC at RRDIDX@:LAST: \: %5.1lf (cur)
GPRINT:PC at RRDIDX@:MAX: \: %5.1lf (max)
GPRINT:PC at RRDIDX@:MIN: \: %5.1lf (min)
GPRINT:PC at RRDIDX@:AVERAGE: \: %5.1lf (avg)\n
LINE1:PU at RRDIDX@#@COLOR@:@RRDPARAM@ Permanent space utilization.\n
GPRINT:PU at RRDIDX@:LAST: \: %5.1lf (cur)
GPRINT:PU at RRDIDX@:MAX: \: %5.1lf (max)
GPRINT:PU at RRDIDX@:MIN: \: %5.1lf (min)
GPRINT:PU at RRDIDX@:AVERAGE: \: %5.1lf (avg)\n
cheers
nico
Le 12 sept. 2012 à 17:25, Scot Kreienkamp a écrit :
...
I didn't think it was really appropriate to post because it's heavily customized for my different environments and such.  You'll have to do some work before it's suitable for yours.
Basically I pull a list of queues using curl then loop back through each queue with curl to grab the queue stats.  It sends those in a status message then sends the stats in a second data message so they can be graphed.  It will also clear any queue with DLQ in the name once it gets above 25.  Otherwise the DLQ's never clear.  I started doing that because the DLQ's got so big they brought the server to almost a standstill after several months of nobody paying attention to it.
The entries in your hosts file are like: jbossjms:8080
(jbossjms being the name of the test, and 8080 being the port the jmx console is on.  This assumes no auth on the jmx web console, which there isn't by default.)
Graphs.cfg entry:
[jboss]
FNPATTERN ^jboss.(.+).rrd
TITLE Queue sizes
YAXIS Queue size
DEF:p at RRDIDX@=@RRDFN@:lambda:MAX
LINE2:p at RRDIDX@#@COLOR@:@RRDPARAM@
GPRINT:p at RRDIDX@:LAST: \: %5.0lf (cur)
GPRINT:p at RRDIDX@:MAX: \: %5.0lf (max)
GPRINT:p at RRDIDX@:MIN: \: %5.0lf (min)
GPRINT:p at RRDIDX@:AVERAGE: \: %5.0lf (avg)\n
I have mine setup to graph the max value, you probably need to change that to average.
You need to use splitncv in the xymonserver.cfg also.  Otherwise anytime you add or remove a queue the graph won't show up or the RRD updater will error out as the new data point won't exist in the RRD.
Works very well for me.  Here it is, hope this helps.
#!/bin/bash
function GETSTATS ()
{
COLOR=green
RETURNQUEUEMESSAGE="Queue_Name Message_Count Consumer_Count"
RETURNQUEUEMONITOR=""
ENVIRONMENTPREFIX=echo $1 | cut -c 1-4
RETURNMESSAGE=""
CONSUMERCOUNTER=""
case $1 in
retv3040.na.lzb.hq)
ENVIRONMENTPREFIX=train
;;
retv3041.na.lzb.hq)
ENVIRONMENTPREFIX=train
;;
retv3042.na.lzb.hq)
ENVIRONMENTPREFIX=train
;;
retv3043.na.lzb.hq)
ENVIRONMENTPREFIX=train
;;
retv3044.na.lzb.hq)
ENVIRONMENTPREFIX=train
;;
retv3045.na.lzb.hq)
ENVIRONMENTPREFIX=train
;;
esac
while read QUEUENAME ; do
curl -s -m 10 "http://$1:$2/jmx-console/HtmlAdaptor?action=inspectMBean&name=jboss.messagin..." > /tmp/$$
MESSAGECOUNT=cat /tmp/$$ |grep -A 3 "The number of messages in the queue"|tail -1 | awk '{print $1}'
            #setup some environment variables for prod VS dev and for specific queues
            if [ "$ENVIRONMENTPREFIX" = "retv" ] ; then
                            case $QUEUENAME in
                                            PricingUploadResponseQueue)
                                                            MSGCOUNTRED=5 #the number of messages in the queue to turn the test red
                                                            MSGCOUNTYELLOW=3 # the number of messages in the queue to turn the test yellow
                                                            MINCONSUMERCOUNT=12 # the expected number of consumers for the PricingUploadResponseQueue and UnMeteredReportResponseQueue
                                            ;;
                                            *)
                                                            MSGCOUNTRED=500
                                                            MSGCOUNTYELLOW=250
                                                            MINCONSUMERCOUNT=0
                                            ;;            
                            esac
            else
                            case $QUEUENAME in
                                            PricingUploadResponseQueue)
                                                            MSGCOUNTRED=10
                                                            MSGCOUNTYELLOW=5
                                                            MINCONSUMERCOUNT=0
                                            ;;
                                            *)
                                                            MSGCOUNTRED=500
                                                            MSGCOUNTYELLOW=250
                                                            MINCONSUMERCOUNT=0
                                            ;;
                            esac
            fi
           

            if [ "$MESSAGECOUNT" -gt "$MSGCOUNTRED" ] ; then
                            COLOR=red
                            RETURNSTATUS="Not OK"
                            RETURNMESSAGE="$RETURNMESSAGE\n$QUEUENAME is very high!"
            elif [ "$MESSAGECOUNT" -gt "$MSGCOUNTYELLOW" ] ; then
                            RETURNMESSAGE="$RETURNMESSAGE\n$QUEUENAME is higher than normal"
                            RETURNSTATUS="Caution"
                            if [ ! "$COLOR" = "red" ] ; then
                                            COLOR=yellow
                            fi
            fi
            CONSUMERCOUNT=`cat /tmp/$$ |grep -A 3 "The number of consumers on the queue"|tail -1 | awk '{print $1}'`
            if [ "$CONSUMERCOUNT" -lt "$MINCONSUMERCOUNT" ] ; then
                            COLOR=yellow
                            RETURNSTATUS="Not OK"
                            RETURNMESSAGE="$RETURNMESSAGE\n$QUEUENAME has less than expected $MINCONSUMERCOUNT queue consumers!"
            fi
            RETURNQUEUEMESSAGE="$RETURNQUEUEMESSAGE\n$QUEUENAME $MESSAGECOUNT $CONSUMERCOUNT"
            RETURNQUEUEMONITOR="$RETURNQUEUEMONITOR\n$QUEUENAME : $MESSAGECOUNT"
            if [ "$MESSAGECOUNT" -gt "25" ] ; then
                            if echo $QUEUENAME | grep "DLQ" ; then
                                            curl  -m 10 -s "http://$1:$2/jmx-console/HtmlAdaptor?action=invokeOp&methodIndex=5&name=jboss.messaging.destination%3Aservice%3DQueue%2Cname%3D$QUEUENAME" >/dev/null 2>&1
                                            RETURNMESSAGE="$RETURNMESSAGE\n$QUEUENAME was cleared"
                                            COLOR=yellow
                            fi
            fi
done < <(curl -s -m 10 "http://$1:$2/jmx-console/HtmlAdaptor?action=displayMBeans&filter=jboss.messa..." |grep -o ">name=.*,"|sed -e 's/>name=//g' -e 's/,$//g')
#set how many queue consumers we should expect
if [ "$ENVIRONMENTPREFIX" = "retv" ] ; then
MINQUEUECONSUMERSCOUNT=7 # the minimum number of queue consumers registered with JBoss before the test turns red for prod
else
MINQUEUECONSUMERSCOUNT=1 # the minimum number of queue consumers registered with JBoss before the test turns red for dev
fi
#get list of registered queue consumers
LISTEDCONSUMERS="Queue Consumers:<table>"
while read CONSUMERIPADDRESS ; do
let CONSUMERCOUNTER+=1
HOSTNAME=host $CONSUMERIPADDRESS | awk '{print $5}'
LISTEDCONSUMERS="$LISTEDCONSUMERS<tr><td>$HOSTNAME</td><td>$CONSUMERIPADDRESS</td></tr>"
done < <(curl -s "http://$1:$2/jmx-console/HtmlAdaptor?action=invokeOp&methodIndex=19&name=jbo..." |grep '<td>10' |sed -e 's/<td>//g' -e 's/<\/td>//g' |sort |uniq)
LISTEDCONSUMERS="$LISTEDCONSUMERS </table>"
if [ "$CONSUMERCOUNTER" -lt "$MINQUEUECONSUMERSCOUNT" ] ; then
RETURNMESSAGE="$RETURNMESSAGE\nSome queue consumers are missing!"
COLOR=red
elif [ "$CONSUMERCOUNTER" -eq "$MINQUEUECONSUMERSCOUNT" ] ; then
LISTEDCONSUMERS=""
fi
RETURNMESSAGE="$RETURNMESSAGE\nExpected $MINQUEUECONSUMERSCOUNT, found $CONSUMERCOUNTER queue consumers currently registered."
if [ "$RETURNQUEUEMONITOR" = "" ] ; then
RETURNQUEUEMESSAGE="JBoss is down or unreachable!!!"
COLOR=red
fi
if [ "$RETURNSTATUS" = "" ] ; then
RETURNSTATUS="OK"
fi
if [ "$RETURNMESSAGE" = "" ] ; then
RETURNMESSAGE="OK"
fi
/home/hobbit/client/bin/xymon retv6100.na.lzb.hq "status $1.jboss $COLOR date $RETURNSTATUS
echo -e "Status message:\n $RETURNMESSAGE"
$LISTEDCONSUMERS
Queue List:
echo -e $RETURNQUEUEMESSAGE|column -t
"
/home/hobbit/client/bin/xymon retv6100.na.lzb.hq "data $1.jboss green date OK
echo -e $RETURNQUEUEMONITOR|sed -e 1d -e 's/\(.*\)/\U\1/' -e 's/QUEUE//g' -e 's/ //g' -e 's/INVENTORY/INVTRY/g' -e 's/RETAIL/RETL/g' -e 's/PRICE/PRC/g' -e 's/RESPONSE/RESPNS/g' -e 's/STATUS/STS/g' -e 's/FREIGHT/FRGT/g' -e 's/REQUEST/REQST/g' | awk -F ":" '{print substr($1,1,18)" : "$2}'
"
rm -f /tmp/$$
}
while read HOSTLINE ; do
SERVERNAME=echo $HOSTLINE | awk '{print $2}'
SERVERPORT=echo $HOSTLINE | awk -F ":" '{print $2}'
GETSTATS $SERVERNAME $SERVERPORT
done < <(/home/hobbit/server/bin/xymongrep jbossjms:*)
Scot Kreienkamp | Senior Systems Engineer | La-Z-Boy Incorporated
1284 N. Telegraph Rd. | Monroe, MI 48162 | scot.kreienkamp at la-z-boy.com | www.la-z-boy.com
From: Larry Barber [mailto:lebarber at gmail.com]
Sent: Wednesday, September 12, 2012 10:59 AM
To: Scot Kreienkamp
Cc: xymon at xymon.com
Subject: Re: [Xymon] jboss monitorin
Could you send me a copy, or post it on Xymonton?
Thanks,
Larry Barber
On Wed, Sep 12, 2012 at 9:50 AM, Scot Kreienkamp <SKreien at la-z-boy.com> wrote:
I wrote a jboss client that pulls message and consumer counts from the jmx web console.  That way I can tell if it's up, how full the queues are, and whether or not consumers are registered.  That's about all the useful info I could find in the web console.
Scot Kreienkamp | Senior Systems Engineer | La-Z-Boy Incorporated
1284 N. Telegraph Rd. | Monroe, MI 48162 | scot.kreienkamp at la-z-boy.com | www.la-z-boy.com
From: xymon-bounces at xymon.com [mailto:xymon-bounces at xymon.com] On Behalf Of Larry Barber
Sent: Wednesday, September 12, 2012 10:43 AM
To: xymon at xymon.com
Subject: [Xymon] jboss monitorin
Has anybody found a good way to monitor jboss servers? I have a bunch of jboss installations and need some way to monitor their performance. I checked Xymonton but couldn't find anything useful (the jmxstat project appears to be dead, at least the link leading to the download is dead). Any help would be much appreciated.
Thanks,
Larry Barber
This message is intended only for the individual or entity to which it is addressed. It may contain privileged, confidential information which is exempt from disclosure under applicable laws. If you are not the intended recipient, please note that you are strictly prohibited from disseminating or distributing this information (other than to the intended recipient) or copying this information. If you have received this communication in error, please notify us immediately by e-mail or by telephone at the above number. Thank you.
This message is intended only for the individual or entity to which it is addressed. It may contain privileged, confidential information which is exempt from disclosure under applicable laws. If you are not the intended recipient, please note that you are strictly prohibited from disseminating or distributing this information (other than to the intended recipient) or copying this information. If you have received this communication in error, please notify us immediately by e-mail or by telephone at the above number. Thank you._______________________________________________
Xymon mailing list
Xymon at xymon.com
http://lists.xymon.com/mailman/listinfo/xymon

[Xymon] jboss monitorin

nicolas＠lienard.name