Email alert based on specific process
Hi All
I am using bb-proctab to monitor a list of processes and if any process a whole group get alert. Is there a way I can setup the alert with process specific. So if proc 1 out the list dies only user 1 get alert and so on. I know I could use the SCRIPT feature but like to keep that as the last resort. I am using hobbit 4.1.2 on Solaris 8. The process list I am concern about is running bb1.9c client on Solaris 8 as well.
Sorry if it were already discussed in the mailing list.
Thanks
-- Asif Iqbal PGP Key: 0xE62693C5 KeyServer: pgp.mit.edu "..there are two kinds of people: those who work and those who take the credit...try to be in the first group;...less competition there." - Indira Gandhi
On Fri, Nov 11, 2005 at 04:22:32PM -0500, Asif Iqbal wrote:
I am using bb-proctab to monitor a list of processes and if any process a whole group get alert. Is there a way I can setup the alert with process specific. So if proc 1 out the list dies only user 1 get alert and so on.
Short answer: No.
Longer answer: Alerts in both BB and Hobbit trigger on a status column. So you can define an alert that triggers when the "procs" column goes red, but you cannot define alerts that distinguish between the different processes that cause the procs column to go red.
I've been thinking about implementing some way of doing this with the Hobbit client - it would be useful not only for the process-checks, but also for disk-checks and (once that gets implemented) for the logfile checks. My idea is that you can associate some random "label" with the rules you define in the hobbit-clients.cfg file. E.g. if you have two process checks for a host, the check for the "httpd" process can be associated with a "web" label, and the check for the "mysqld" process gets a "database" label. Then you can define alerts that use these labels just like we currently use e.g. the service names, i.e.
HOST=* MAIL dbadmin at foo.com LABEL=database MAIL webadmin at foo.com LABEL=web
Regards, Henrik
On Fri, 11 Nov 2005, (Henrik Stoerner) wrote:
the rules you define in the hobbit-clients.cfg file. E.g. if you have two process checks for a host, the check for the "httpd" process can be associated with a "web" label, and the check for the "mysqld" process gets a "database" label. Then you can define alerts that use these labels just like we currently use e.g. the service names, i.e.
HOST=* MAIL dbadmin at foo.com LABEL=database MAIL webadmin at foo.com LABEL=web
Beautiful. And it would be ideal if those 'labels' also ended up as columns in the display, not just for notifications.
So for the the 'database' label, you would like to see mysql up, nothing bad in the log file, and the /mysql partition is OK.
-- Scott Walters -PacketPusher
It would be nice if Hobbit came with a tool, that you could run on your bb servers rrd directory, or a copy of it, and it would automagically copy the data to the host-based subdirs that hobbit uses.
I have a BB server that has been running for about 1.5 years. I have a Hobbit server I have been running in unison for several months, (via the BBRELAY option on the bb server). Soon I am going to totally decommission the bb server and use only the Hobbit server.
However, I do not wish to lose the trends history that the bb server has. Is there any sort of tool to migrate the rrd data from bb to hobbit? I started doing it manually, and it wasn't very fun.
The method I used to move the memory rrd data over is:
First off, I noticed that the directory structure of the ~hobbit/data/rrd directories and files is different from the ~bb/bbvar/rrd directories and files:
[*hobbit*]$ ls data/rrd/host.my-domain.net #Note that hobbit stores the rrds in subdirs **disk,bigdisk.rrd *memory.physical.rrd* tcp.http.host.my-domain.net,.rrd disk,boot.rrd *memory.real.rrd* tcp.http.host.my-domain.net.rrd disk,root.rrd *memory.swap.rrd* tcp.ssh.rrd la.rrd procs.rrd users.rrd *memory.actual.rrd* tcp.conn.rrd (Hmm it looks like there is a duplicate tcp.http.host.my-domain.net,|.rrd? No idea where that came from, but I digress.)
[*bbhost*]$ ls bbvar/rrd |grep host #Note that bb stores all of the rrds in one dir host.my-domain.net.disk,bigdisk.rrd host.my-domain.net.disk,boot.rrd host.my-domain.net.disk,root.rrd host.my-domain.net.la.rrd *host,my-domain,net.memory.actual.rrd host,my-domain,net.memory.physical.rrd host,my-domain,net.memory.real.rrd host,my-domain,net.memory.swap.rrd* host.my-domain.net.procs.rrd host.my-domain.net.tcp.conn.rrd host.my-domain.net.tcp.http.host.my-domain.net.rrd host.my-domain.net.tcp.ssh.rrd host.my-domain.net.users.rrd
So it looks like on the bbhost, for some reason, the host name is comma delimited....but only for the memory rrd files? This is strange. Ah well I just need to get them to the hobbit server, and with the proper filenames, and in the proper subdirs. So I scp'd all of the memory rrd files to the hobbit server into /tmp/mem, and then:
[hobbit] $ cd ~hobbit/data/rrd; for hostdir in ls; do BBHOST=echo $hostdir|sed 's/\./,/g'; cp -vf /tmp/mem/$BBHOST.memory.real.rrd
$hostdir/memory.real.rrd; done
I repeated this command for the "swap", "actual", and "real" files. The sed command was to generate the comma-delimited filenames I needed to find the files. This command copied all of the memory rrd files into the proper subdirs.
I'm not sure why only the memory rrd files on my bb server are comma-delimited. Maybe some misconfiguration I did in the past? The memory graphs look normal in the trends column though...ah well.
It would be nice if Hobbit came with a tool, that you could run on your bb servers rrd directory, or a copy of it, and it would automagically copy the data to the host-based subdirs that hobbit uses.
I have a BB server that has been running for about 1.5 years. I have a Hobbit server I have been running in unison for several months, (via the BBRELAY option on the bb server). Soon I am going to totally decommission the bb server and use only the Hobbit server.
However, I do not wish to lose the trends history that the bb server has. Is there any sort of tool to migrate the rrd data from bb to hobbit? I started doing it manually, and it wasn't very fun.
The method I used to move the memory rrd data over is:
First off, I noticed that the directory structure of the ~hobbit/data/rrd directories and files is different from the ~bb/bbvar/rrd directories and files:
[*hobbit*]$ ls data/rrd/host.my-domain.net #Note that hobbit stores the rrds in subdirs **disk,bigdisk.rrd *memory.physical.rrd* tcp.http.host.my-domain.net,.rrd disk,boot.rrd *memory.real.rrd* tcp.http.host.my-domain.net.rrd disk,root.rrd *memory.swap.rrd* tcp.ssh.rrd la.rrd procs.rrd users.rrd *memory.actual.rrd* tcp.conn.rrd (Hmm it looks like there is a duplicate tcp.http.host.my-domain.net,|.rrd? No idea where that came from, but I digress.)
[*bbhost*]$ ls bbvar/rrd |grep host #Note that bb stores all of the rrds in one dir host.my-domain.net.disk,bigdisk.rrd host.my-domain.net.disk,boot.rrd host.my-domain.net.disk,root.rrd host.my-domain.net.la.rrd *host,my-domain,net.memory.actual.rrd host,my-domain,net.memory.physical.rrd host,my-domain,net.memory.real.rrd host,my-domain,net.memory.swap.rrd* host.my-domain.net.procs.rrd host.my-domain.net.tcp.conn.rrd host.my-domain.net.tcp.http.host.my-domain.net.rrd host.my-domain.net.tcp.ssh.rrd host.my-domain.net.users.rrd
So it looks like on the bbhost, for some reason, the host name is comma delimited....but only for the memory rrd files? This is strange. Ah well I just need to get them to the hobbit server, and with the proper filenames, and in the proper subdirs. So I scp'd all of the memory rrd files to the hobbit server into /tmp/mem, and then:
[hobbit] $ cd ~hobbit/data/rrd; for hostdir in ls; do BBHOST=echo $hostdir|sed 's/\./,/g'; cp -vf /tmp/mem/$BBHOST.memory.real.rrd
$hostdir/memory.real.rrd; done
I repeated this command for the "swap", "actual", and "real" files. The sed command was to generate the comma-delimited filenames I needed to find the files. This command copied all of the memory rrd files into the proper subdirs.
I'm not sure why only the memory rrd files on my bb server are comma-delimited. Maybe some misconfiguration I did in the past? The memory graphs look normal in the trends column though...ah well.
Sounds like you did not read the migration guide ? There is a moverrd.sh script in the source hobbitd dir that will move all your old RRD files to the hobbit installation and reorder them for you.
Regards, Thomas
Charles Jones wrote:
It would be nice if Hobbit came with a tool, that you could run on your bb servers rrd directory, or a copy of it, and it would automagically copy the data to the host-based subdirs that hobbit uses.
I have a BB server that has been running for about 1.5 years. I have a Hobbit server I have been running in unison for several months, (via the BBRELAY option on the bb server). Soon I am going to totally decommission the bb server and use only the Hobbit server.
However, I do not wish to lose the trends history that the bb server has. Is there any sort of tool to migrate the rrd data from bb to hobbit? I started doing it manually, and it wasn't very fun.
The method I used to move the memory rrd data over is:
First off, I noticed that the directory structure of the ~hobbit/data/rrd directories and files is different from the ~bb/bbvar/rrd directories and files:
[*hobbit*]$ ls data/rrd/host.my-domain.net #Note that hobbit stores the rrds in subdirs disk,bigdisk.rrd *memory.physical.rrd* tcp.http.host.my-domain.net,.rrd disk,boot.rrd *memory.real.rrd* tcp.http.host.my-domain.net.rrd disk,root.rrd *memory.swap.rrd* tcp.ssh.rrd la.rrd procs.rrd users.rrd *memory.actual.rrd* tcp.conn.rrd (Hmm it looks like there is a duplicate tcp.http.host.my-domain.net,|.rrd? No idea where that came from, but I digress.)
[*bbhost*]$ ls bbvar/rrd |grep host #Note that bb stores all of the rrds in one dir host.my-domain.net.disk,bigdisk.rrd host.my-domain.net.disk,boot.rrd host.my-domain.net.disk,root.rrd host.my-domain.net.la.rrd *host,my-domain,net.memory.actual.rrd host,my-domain,net.memory.physical.rrd host,my-domain,net.memory.real.rrd host,my-domain,net.memory.swap.rrd* host.my-domain.net.procs.rrd host.my-domain.net.tcp.conn.rrd host.my-domain.net.tcp.http.host.my-domain.net.rrd host.my-domain.net.tcp.ssh.rrd host.my-domain.net.users.rrd
So it looks like on the bbhost, for some reason, the host name is comma delimited....but only for the memory rrd files? This is strange. Ah well I just need to get them to the hobbit server, and with the proper filenames, and in the proper subdirs. So I scp'd all of the memory rrd files to the hobbit server into /tmp/mem, and then:
[hobbit] $ cd ~hobbit/data/rrd; for hostdir in
ls; do BBHOST=echo $hostdir|sed 's/\./,/g'; cp -vf /tmp/mem/$BBHOST.memory.real.rrd $hostdir/memory.real.rrd; doneI repeated this command for the "swap", "actual", and "real" files. The sed command was to generate the comma-delimited filenames I needed to find the files. This command copied all of the memory rrd files into the proper subdirs.
I'm not sure why only the memory rrd files on my bb server are comma-delimited. Maybe some misconfiguration I did in the past? The memory graphs look normal in the trends column though...ah well.
Thomas,
Thanks for pointing out what should have been obvious to me. Man, I gotta start reading more thoroughly before I ask questions :)
-Charles
Thomas wrote:
Sounds like you did not read the migration guide ? There is a moverrd.sh script in the source hobbitd dir that will move all your old RRD files to the hobbit installation and reorder them for you.
Hi all,
Since 4.2.0 Hobbit officially supports the GROUP-label mechanism for distinguishing processes, disks, files, etc. When I quote Henrik correctly "alerts in both BB and Hobbit trigger on a status column.", this gives me the following problem.
Let's say we have the following config:
hobbit-clients.cfg:
SERVER=name DISK / 90 95 GROUP=unix DISK /database 90 95 GROUP=dba
hobbit-alerts.cfg:
GROUP=dba MAIL dba at somewhere.com bla bla
GROUP=unix MAIL unix at somewhere.com bla bla
The threshold for /database is exceeded, status is yellow, mail is sent to the dba. Great! Now the /-partition starts to grow, the status remains yellow because the dba's are investigating the problem but have not solved the issue, but the unix-team is not alerted because no status change has occured.
Any hints???
Below I found a discussion from last year, when I think the idea was born and the term LABEL was introduced but is called GROUP now.
Thanks, Peter
2005/11/13, Scott Walters <scott at packetpushers.com>:
On Fri, 11 Nov 2005, (Henrik Stoerner) wrote:
the rules you define in the hobbit-clients.cfg file. E.g. if you have two process checks for a host, the check for the "httpd" process can be associated with a "web" label, and the check for the "mysqld" process gets a "database" label. Then you can define alerts that use these labels just like we currently use e.g. the service names, i.e.
HOST=* MAIL dbadmin at foo.com LABEL=database MAIL webadmin at foo.com LABEL=web
Beautiful. And it would be ideal if those 'labels' also ended up as columns in the display, not just for notifications.
So for the the 'database' label, you would like to see mysql up, nothing bad in the log file, and the /mysql partition is OK.
-- Scott Walters -PacketPusher
participants (6)
-
henrik@hswn.dk
-
iqbala-hobbit@qwestip.net
-
jonescr@cisco.com
-
peter.welter@gmail.com
-
scott@PacketPushers.com
-
tlp-hobbit@holme-pedersen.dk