[hobbit] Ack events sent to multiple BBDISP (issue with bback.cgi and cookies)
Hi,
A co-worker worked on the multiple BBDISP ack problem i posted here in February, and i'd like to share his work :-)
Here his investigation results and the fix :
Hobbit does not allow ack's without the correct acknowledgement number as it is this acknum that is used to look up the {HOST}.{TEST} to be acknowledged. So there was no way that the simple trick of using the main/ area section in hobbitserver.cfg was going to work. A side note.... -{ACKNUM} will acknowledge all events for {HOST}
The code only ever forwards to the first IP listed in BBDISPLAYS if the messages type is not one of those as defined in lib/sendmsg.c:multircptcmds.
static char *multircptcmds[] = { "status", "combo", "meta", "data", "notify", "enable", "disable", "drop", "rename", "client", NULL };
The ordering of the parameters in hobbitcfg.conf is critical. The --area must be prior to the --env file parameter so that the correct environment is selected.
We need to use --nopin
We can keep the cookies enabled
The Fix :D
The new bb-ack.cgi now does the following:
Grab a list of all acknowledgable events from the list of Hobbit BBDISPLAY servers.
Present the normal looking "Acknowledge Alert" page, showing the list of events and entry fields to send acknowledgements.
Send the acknowledgements to the relevant BBDISPLAY servers.
Provide feedback as normal for the acknowledgements sent.
The Configuration:
- Change the CGI_ACK_OPTS setting in /opt/hobbit/server/etc/hobbitcgi.conf to:-
CGI_ACK_OPTS="--area=maint --env=/opt/hobbit/server/etc/hobbitserver.cfg --no-pin"
- Add to ~hobbit/server/etc/hobbitserver.conf
acks for others hobbit servers
maint/BBDISP="0.0.0.0" maint/BBDISPLAYS="$BBSERVERIP 10.0.0.2 10.0.0.3"
acks end
Known Issues:
Not fully tested =-o
Could be issues with performance if the BBDISPLAY list is long.
If a BBDISPLAY does not respond (IP is unbound, etc.) then there is a delay while the TCP connect timesout [order of a few seconds].
Theoretical possibility of timing issue between alerts being 'raised' and acknowledged while the alerts are being propogated between BBDISPLAYS.
CGI code is not fantastically 'safe' but I think that's constant with the rest of the code (i.e. I haven't made it any worse!) :/.
Find the new bb-ack.c in attachment. The modifications done are commented between /* Begin Changes for multiBBDISPLAYS for acks */ tag.
Henrik, if this fix is ok for you, it could be good to add it into xymon.
Regards,
Nico
On 11/02/09 20:19, "nico" <nico at crysto.org> wrote:
Hi, Sorry, i'm sending again this email because when i wrote the last, i didn't see it has been merged in another thread :-( I'd like to send ack events to multiple BBDISPLAY.
To do that, i did a change on hobbitserver.cfg with this configuration:
http://www.hswn.dk/hobbiton/2005/10/msg00203.html
- In hobbitserver.cfg on both display servers, after the BBDISPLAY setting add # For the enable/disable and acknowledge scripts. maint/BBDISP="0.0.0.0" maint/BBDISPLAYS="10.0.0.1 172.16.12.10" (replace with your display-server IP-adresses)
- In hobbitcgi.cfg, change the line for the bb-ack.cgi tool: # bb-ack.cgi options CGI_ACK_OPTS="--area=maint --env=/usr/lib/hobbit/server/etc/hobbitserver.cfg"
The first problem is that it seems to not send acks events to the multiples IP but only to local.
The second problem is that when i modified BBDISP with the IP of another hobbit server, we got an error:
2009-02-11 17:02:52 Cookie 338777 not found, dropping ack 2009-02-11 17:06:34 Cookie 338777 not found, dropping ack
The reason is that the cookie is different for the same host/test:
serverA$ ${BB} ${BBSERVERIP}:${BBPORT} "hobbitdboard color=red,yellow fields=hostname,testname,cookie" cusaa-ws02|ftp|640796
serverB$ ${BB} ${BBSERVERIP}:${BBPORT} "hobbitdboard color=red,yellow fields=hostname,testname,cookie" cusaa-ws02|ftp|338777
We tried to disable cookie by adding "--no-cookies" into CGI_ACK_OPTS variable of hobbitcgi.cfg but the problem still happens.
Is there any way to fix it please ? Thanks. Regards, nicolas
Hi all, I¹ve been trying to create graphs with Devmon, and a few internet howtos out there, and for the likes of me, I can¹t seem to figure out how to get them to show up.
Maybe someone can point out to me what I¹m doing wrong?
I followed the advice here:
http://permalink.gmane.org/gmane.comp.monitoring.hobbit/10245
I¹m not to sure what they mean about #4
With putting an entry in the hobbitgraph.cfg: [if_load]
is there only 1 entry, or 1 entry for every test? one entry for [if_load}, one entry for [if_error] etc etc...
Also with regards to #6, what is this patch they are talking about?
and I got the extra-rrd.pl script from here:
http://www.hswn.dk/hobbiton/2007/09/msg00148.html
any one?
thanks
William
-- This message has been scanned for dangerous content by MailScanner, and is believed to be clean.
On Wednesday 01 April 2009 21:42:16 William Ottley wrote:
Hi all, I¹ve been trying to create graphs with Devmon, and a few internet howtos out there,
Why do you consult out-of-date howtos, instead of the documentation supplied with the software (see docs/GRAPHING, or http://devmon.svn.sourceforge.net/viewvc/devmon/trunk/docs/GRAPHING)
(this documentation originated shortly after the mailing list entry you refer to below, and hasn't been updated that much since the devmon rrd collector was merged into Xymon, so I am updating it a bit to reflect that, and reduce the amount of detail on the implementation).
If you believe there are errors with that document, please feel free to supply a patch to improve it.
Please supply the URLs of the howtos, so I can try and have the author note that it is outdated and refer to the authoritative documentation.
and for the likes of me, I can¹t seem to figure out how to get them to show up.
Maybe someone can point out to me what I¹m doing wrong?
Since you haven't listed what you have done, or what the results are, no.
None of your mails to the mailing list have actually stated which versions of anything you are running, so maybe: 1)Are you using Hobbit 4.2.0, or Xymon 4.2.2 or later ? 2)What version of devmon are you running?
The advice may differ based on the answers (hint, it's much easier using the latest software).
I followed the advice here:
http://permalink.gmane.org/gmane.comp.monitoring.hobbit/10245
This advice is outdated (which is to be expected, considering it is over 18 months old), and applies to software for which there have been two subsequent releases (each).
I¹m not to sure what they mean about #4
With putting an entry in the hobbitgraph.cfg: [if_load]
is there only 1 entry, or 1 entry for every test? one entry for [if_load}, one entry for [if_error] etc etc...
I am not aware of any devmon templates currently enabling RRD graphs for if_error (although, it can be done quite easily, but maybe you should stick to what more or less works out-the-box, if_load). Current devmon (0.3.0 final or later) ships with the graph definitions in docs/devmon-graph.cfg.
Also with regards to #6, what is this patch they are talking about?
An outdated one.
and I got the extra-rrd.pl script from here:
http://www.hswn.dk/hobbiton/2007/09/msg00148.html
any one?
If you are running Xymon 4.2.2 or later, this is what you need to do to get graphs (let's concentrate on if_load first) working:
1)Ensure you have 'if_load=devmon' in the TEST2RRD variable in hobbitserver.cfg (should be there by default AFAIK) 2)Ensure you have "if_load::1" and "devmon" in the GRAPHS variable in hobbitserver.cfg (should be there by default AFAIK)
At this point, you should have RRD files for any hosts that have devmon if_load tests, e.g. $BBVAR/rrd/<hostname>/if_load.<interface_name>.rrd
3)Make sure Hobbit/Xymon knows how to draw the graph, by ensuring that there is a graph definition for if_load. The easiest way (IMHO), is to add "directory /etc/xymon/hobbitgraph.d" to hobbitgraph.cfg, and drop the devmon-graph.cfg from Devmon's extras directory in /etc/xymon/hobbitgraph.d.
Hi again Buchan,
Your instructions say:
2)Ensure you have "if_load::1" and "devmon" in the GRAPHS variable in hobbitserver.cfg (should be there by default AFAIK)
What I have is: devmon::1,if_load::1
Slightly different. Is this ok?
With regards to the rrd files, All I have is one file for each of the cisco's I'm monitoring:
$BBVAR/rrd/<hostname>/tcp.conn.rrd
I have a few servers that are being monitored, and there are several different .rrd files.....
With regards to your #3. I'm a bit confused. My structure is different. I have everything in:
/home/xymon/ data /home/xymon/ cgi-secure /home/xymon/ cgi-bin /home/xymon/ client /home/xymon/ server /etc... /home/xymon/ devmon
I don't see the hobbitgraph.d I do see the devmon-graph.cfg in /home/xymon/devmon/extras/ So I just simply did a cat devmon-graph.cfg >> hobbitgraph.cfg
I restarted everything, and I still only get the one graph: tcp.conn.rrd
So maybe I should show you how my web page looks like, and an example of my bb-hosts file?
My Web page has:
Switches / Firewall conn info trends firewall conn:green:2d12h51m info:green:10.136.209.150 trends:green: walmart conn:green:2d15h22m info:green:161.174.32.2 trends:green:
Each of these sections only has 1 graph.
Bb-hosts:
page switches Switches / Firewall group-compress <H3><I>Switches / Firewall</I></H3> 10.136.209.150 firewall # DEVMON:model(cisco;asa),cid(cmisnmp),tests(memory,if_stat,if_load,if_err,if_ dsc,cpu,cluster,serial,specs,connects) 161.174.32.2 walmart # DEVMON:model(cisco;2600),cid(cmisnmp)
Does this help?
Will
-- This message has been scanned for dangerous content by MailScanner, and is believed to be clean.
On Thursday 02 April 2009 16:21:50 William Ottley wrote:
Hi again Buchan,
Your instructions say:
2)Ensure you have "if_load::1" and "devmon" in the GRAPHS variable in hobbitserver.cfg (should be there by default AFAIK)
What I have is: devmon::1,if_load::1
Slightly different. Is this ok?
With regards to the rrd files, All I have is one file for each of the cisco's I'm monitoring:
$BBVAR/rrd/<hostname>/tcp.conn.rrd
I have a few servers that are being monitored, and there are several different .rrd files.....
With regards to your #3. I'm a bit confused. My structure is different. I have everything in:
/home/xymon/ data /home/xymon/ cgi-secure /home/xymon/ cgi-bin /home/xymon/ client /home/xymon/ server /etc... /home/xymon/ devmon
I don't see the hobbitgraph.d I do see the devmon-graph.cfg in /home/xymon/devmon/extras/ So I just simply did a cat devmon-graph.cfg >> hobbitgraph.cfg
I restarted everything, and I still only get the one graph: tcp.conn.rrd
So maybe I should show you how my web page looks like, and an example of my bb-hosts file?
My Web page has:
Switches / Firewall conn
info trends firewall conn:green:2d12h51m info:green:10.136.209.150 trends:green: walmart conn:green:2d15h22m info:green:161.174.32.2
trends:green:
OK, so devmon isn't actually polling these devices. Having (e.g.) an if_load test displayed on the hobbit server is the first step (you won't get rrd files for the if_load test if Hobbit isn't getting any if_load data) ....
Each of these sections only has 1 graph.
Bb-hosts:
page switches Switches / Firewall group-compress <H3><I>Switches / Firewall</I></H3> 10.136.209.150 firewall # DEVMON:model(cisco;asa),cid(cmisnmp),tests(memory,if_stat,if_load,if_err,if _ dsc,cpu,cluster,serial,specs,connects) 161.174.32.2 walmart # DEVMON:model(cisco;2600),cid(cmisnmp)
What do you get from running devmon with the -readbbhosts option? Have these hosts (walmart, firewall) been added to the devmon hosts.db file (you can use 'strings /path/to/hosts.db' to see what text is in the db file)? Are there any relevant errors in the devmon log file?
Hi Buchan, I guess what you say makes sense. I only get the "default" xymon page.
Now you say I have to put in the tests in bb-hosts, but how is this done? When you say "an if_load test displayed on the hobbit server"
To me I would put that test in the bb-hosts file like this:??
HOW I HAVE IT NOW: 161.174.32.2 walmart # DEVMON:model(cisco;2600),cid(cmisnmp)
HOW I SHOULD HAVE IT??: 161.174.32.2 walmart # if_load DEVMON:model(cisco;2600),cid(cmisnmp) {I tried it this way, and nothing comes up}
Output from /home/xymon/devmon/hosts.db:
firewall^[10.136.209.150^[cisco^[asa^[memory,if_stat,if_load,if_err,if_dsc,c pu,cluster,serial,specs,connects^[cmisnmp^[^[ router-main^[10.136.209.1^[cisco^[2600^[all^[cmisnmp^[^[ switch-233^[10.136.209.233^[cisco^[2960^[all^[public^[^[ walmart^[161.174.32.2^[cisco^[2600^[all^[cmisnmp^[^[
When I do devmon readbbhosts
Nothing omes up, I need to use the vv, and this comes up:
[xymon at centos devmon]$ ./devmon --readbbhosts -vv [09-04-02 at 07:11:22] Option 'bblocation' defaulting to: [09-04-02 at 07:11:23] SNMP querying all hosts in bb-hosts file, please wait... [09-04-02 at 07:11:23] Checking if # bbd http://centos/ apache matches NET:. [09-04-02 at 07:11:23] Checking if # matches NET:. [09-04-02 at 07:11:23] Checking if # http://10.136.209.209/ matches NET:. [09-04-02 at 07:11:23] Checking if # matches NET:. [09-04-02 at 07:11:23] Checking if # DEVMON:model(cisco;2960) matches NET:. [09-04-02 at 07:11:23] Checking if # DEVMON:model(cisco;2600),cid(cmisnmp) matches NET:. [09-04-02 at 07:11:23] Checking if # DEVMON:model(cisco;asa),cid(cmisnmp),tests(memory,if_stat,if_load,if_err,if_ dsc,cpu,cluster,serial,specs,connects) matches NET:. [09-04-02 at 07:11:23] Checking if # if_load if_dsc connects temp DEVMON:model(cisco;2600),cid(cmisnmp) matches NET:. [09-04-02 at 07:11:23] Option 'bbdateformat' defaulting to: . [09-04-02 at 07:11:23] Option 'dispport' defaulting to: 1984. [09-04-02 at 07:11:23] Option 'dispserv' defaulting to: localhost. [09-04-02 at 07:11:23] Querying pre-existing hosts [09-04-02 at 07:11:23] Discovered switch-233 as a cisco 2960 [09-04-02 at 07:11:23] Discovered walmart as a cisco 2600 [09-04-02 at 07:11:23] Discovered router-main as a cisco 2600 [09-04-02 at 07:11:23] Discovered firewall as a cisco asa
-- This message has been scanned for dangerous content by MailScanner, and is believed to be clean.
On Thursday 02 April 2009 18:11:32 William Ottley wrote:
Hi Buchan, I guess what you say makes sense. I only get the "default" xymon page.
Now you say I have to put in the tests in bb-hosts, but how is this done?
Devmon must send it a status report, for which devmon needs to be running (nothing else is required by default).
[...]
Output from /home/xymon/devmon/hosts.db:
firewall^[10.136.209.150^[cisco^[asa^[memory,if_stat,if_load,if_err,if_dsc, c pu,cluster,serial,specs,connects^[cmisnmp^[^[ router-main^[10.136.209.1^[cisco^[2600^[all^[cmisnmp^[^[ switch-233^[10.136.209.233^[cisco^[2960^[all^[public^[^[ walmart^[161.174.32.2^[cisco^[2600^[all^[cmisnmp^[^[
So, devmon has discovered them, good.
Nothing omes up, I need to use the vv, and this comes up:
[xymon at centos devmon]$ ./devmon --readbbhosts -vv [09-04-02 at 07:11:22] Option 'bblocation' defaulting to: [09-04-02 at 07:11:23] SNMP querying all hosts in bb-hosts file, please wait... [09-04-02 at 07:11:23] Checking if # bbd http://centos/ apache matches NET:. [09-04-02 at 07:11:23] Checking if # matches NET:. [09-04-02 at 07:11:23] Checking if # http://10.136.209.209/ matches NET:. [09-04-02 at 07:11:23] Checking if # matches NET:. [09-04-02 at 07:11:23] Checking if # DEVMON:model(cisco;2960) matches NET:. [09-04-02 at 07:11:23] Checking if # DEVMON:model(cisco;2600),cid(cmisnmp) matches NET:. [09-04-02 at 07:11:23] Checking if # DEVMON:model(cisco;asa),cid(cmisnmp),tests(memory,if_stat,if_load,if_err,if _ dsc,cpu,cluster,serial,specs,connects) matches NET:. [09-04-02 at 07:11:23] Checking if # if_load if_dsc connects temp DEVMON:model(cisco;2600),cid(cmisnmp) matches NET:. [09-04-02 at 07:11:23] Option 'bbdateformat' defaulting to: . [09-04-02 at 07:11:23] Option 'dispport' defaulting to: 1984. [09-04-02 at 07:11:23] Option 'dispserv' defaulting to: localhost. [09-04-02 at 07:11:23] Querying pre-existing hosts [09-04-02 at 07:11:23] Discovered switch-233 as a cisco 2960 [09-04-02 at 07:11:23] Discovered walmart as a cisco 2600 [09-04-02 at 07:11:23] Discovered router-main as a cisco 2600 [09-04-02 at 07:11:23] Discovered firewall as a cisco asa
Ok, but have you actually started devmon ? As in (in your case). Running it with --readbbhosts doesn't start devmon, it just updates the hosts.db file ...
$ ./devmon
or
$ ./devmon -vv
BTW, there are RPMS of xymon and devmon for RHEL/Centos, which have init scripts and everyhing ....
If devmon is running, it should be logging to it's log file (even with no verbosity, it logs that it starting (IIRC).
Regards, Buchan
participants (3)
-
bgmilne＠staff.telkomsa.net
-
nico＠crysto.org
-
wottley＠cmicanada.com