Missing events on server's history page?
Running xymon 4.3.17 , on linux, wondering if anyone's seen anything like this? for example our "edapps2-cf2" just alerted on cpu.
The non-green view shows these most recent event
Wed Apr 22 09:33:25 2015 edapps2-cf2.example.com cpu (yellow->green) Wed Apr 22 09:28:24 2015 edapps2-cf2.example.com cpu (red->yellow) Wed Apr 22 09:18:22 2015 edapps2-cf2.example.com cpu (yellow->red) Wed Apr 22 09:08:19 2015 edapps2-cf2.example.com cpu (green->yellow)
When we go to the host page then cpu dot,and look at the full history we don't see them
All log entries Date Status Duration Mon Apr 13 17:00:44 2015 8 days 16:40:47 (green) Mon Apr 13 16:45:41 2015 0:15:03 (yellow) Tue Mar 24 19:11:14 2015 19 days 21:34:27 (green)
When we look at Reports/Event log they show up
Wed Apr 22 09:43:45 2015 edapps2-cf2.example.com http green
From -> To yellow Wed Apr 22 09:33:25 2015 edapps2-cf2.example.com cpu yellow From -> To green Wed Apr 22 09:28:24 2015 edapps2-cf2.example.com cpu red From -> To yellow Wed Apr 22 09:18:22 2015 edapps2-cf2.example.com cpu yellow From -> To red Wed Apr 22 09:08:19 2015 edapps2-cf2.example.com cpu green From -> To yellow
why aren't they on the host/cpu history page? thanks for any clues/explanation. We do see other events with shorter duration on that history page
Hi,
I've saw something like this quite a few times at my previous employer (This was with an ancient 4.2.x installation). The problem in that case was that the specific history event file for the test had a negative timed duration entry (normally duration was -1 second ... ). Looking at the entry after, calculate and insert a correct duration and you should get your complete history log for the specific test in the web gui.
In your case I wold check out: $XYMONHISTDIR/edapps2-cf2,example,com.cpu
regards Johan Björklund
On Wed, Apr 22, 2015 at 3:59 PM, Betsy Schwartz <betsy.schwartz at gmail.com> wrote:
Running xymon 4.3.17 , on linux, wondering if anyone's seen anything like this? for example our "edapps2-cf2" just alerted on cpu.
The non-green view shows these most recent event
Wed Apr 22 09:33:25 2015 edapps2-cf2.example.com cpu (yellow->green) Wed Apr 22 09:28:24 2015 edapps2-cf2.example.com cpu (red->yellow) Wed Apr 22 09:18:22 2015 edapps2-cf2.example.com cpu (yellow->red) Wed Apr 22 09:08:19 2015 edapps2-cf2.example.com cpu (green->yellow)
When we go to the host page then cpu dot,and look at the full history we don't see them
All log entries Date Status Duration Mon Apr 13 17:00:44 2015 8 days 16:40:47 (green) Mon Apr 13 16:45:41 2015 0:15:03 (yellow) Tue Mar 24 19:11:14 2015 19 days 21:34:27 (green)
When we look at Reports/Event log they show up
Wed Apr 22 09:43:45 2015 edapps2-cf2.example.com http green From -> To yellow Wed Apr 22 09:33:25 2015 edapps2-cf2.example.com cpu yellow From -> To green Wed Apr 22 09:28:24 2015 edapps2-cf2.example.com cpu red From -> To yellow Wed Apr 22 09:18:22 2015 edapps2-cf2.example.com cpu yellow From -> To red Wed Apr 22 09:08:19 2015 edapps2-cf2.example.com cpu green From -> To yellow
why aren't they on the host/cpu history page? thanks for any clues/explanation. We do see other events with shorter duration on that history page
Xymon mailing list Xymon at xymon.com http://lists.xymon.com/mailman/listinfo/xymon
Thank you! those events aren't actually in the file. If they're not in the file how does the event report see them, which file does that look at?
[root at xymon1 hist]# pwd /usr/local/xymon/data/hist [root at xymon1 hist]# tail edapps2-cf2,exampe,com.cpu Thu Mar 5 19:19:02 2015 green 1425601142 1395994 Sun Mar 22 00:05:36 2015 red 1426997136 1514 Sun Mar 22 00:30:50 2015 yellow 1426998650 301 Sun Mar 22 00:35:51 2015 green 1426998951 1210 Sun Mar 22 00:56:01 2015 yellow 1427000161 2099 Sun Mar 22 01:31:00 2015 green 1427002260 236015 Tue Mar 24 19:04:35 2015 blue 1427238275 399 Tue Mar 24 19:11:14 2015 green 1427238674 1719267 Mon Apr 13 16:45:41 2015 yellow 1428957941 903 Mon Apr 13 17:00:44 2015 green 1428958844[root at xymon1 hist]#
this is getting to be a huge issue. We have huge gaps in what is showing up on an individual server's event history everything shows up in the event log report
On Wed, Apr 22, 2015 at 11:06 AM, Betsy Schwartz <betsy.schwartz at gmail.com> wrote:
Thank you! those events aren't actually in the file. If they're not in the file how does the event report see them, which file does that look at?
[root at xymon1 hist]# pwd /usr/local/xymon/data/hist [root at xymon1 hist]# tail edapps2-cf2,exampe,com.cpu Thu Mar 5 19:19:02 2015 green 1425601142 1395994 Sun Mar 22 00:05:36 2015 red 1426997136 1514 Sun Mar 22 00:30:50 2015 yellow 1426998650 301 Sun Mar 22 00:35:51 2015 green 1426998951 1210 Sun Mar 22 00:56:01 2015 yellow 1427000161 2099 Sun Mar 22 01:31:00 2015 green 1427002260 236015 Tue Mar 24 19:04:35 2015 blue 1427238275 399 Tue Mar 24 19:11:14 2015 green 1427238674 1719267 Mon Apr 13 16:45:41 2015 yellow 1428957941 903 Mon Apr 13 17:00:44 2015 green 1428958844[root at xymon1 hist]#
On Mon, May 4, 2015 8:41 am, Betsy Schwartz wrote:
this is getting to be a huge issue. We have huge gaps in what is showing up on an individual server's event history everything shows up in the event log report
On Wed, Apr 22, 2015 at 11:06 AM, Betsy Schwartz <betsy.schwartz at gmail.com> wrote:
Thank you! those events aren't actually in the file. If they're not in the file how does the event report see them, which file does that look at?
[root at xymon1 hist]# pwd /usr/local/xymon/data/hist [root at xymon1 hist]# tail edapps2-cf2,exampe,com.cpu Thu Mar 5 19:19:02 2015 green 1425601142 1395994 Sun Mar 22 00:05:36 2015 red 1426997136 1514 Sun Mar 22 00:30:50 2015 yellow 1426998650 301 Sun Mar 22 00:35:51 2015 green 1426998951 1210 Sun Mar 22 00:56:01 2015 yellow 1427000161 2099 Sun Mar 22 01:31:00 2015 green 1427002260 236015 Tue Mar 24 19:04:35 2015 blue 1427238275 399 Tue Mar 24 19:11:14 2015 green 1427238674 1719267 Mon Apr 13 16:45:41 2015 yellow 1428957941 903 Mon Apr 13 17:00:44 2015 green 1428958844[root at xymon1 hist]#
Hi Betsy,
History lines will be present in three distinct files (assuming you've enabled the various options for saving events):
In your case, you should see, for each 'cpu' event on this host, entries in:
- /usr/local/xymon/data/hist/edapps2-cf2,exampe,com.cpu
- /usr/local/xymon/data/hist/edapps2-cf2.exampe.com
- /usr/local/xymon/data/hist/allevents
It should look something like this: [root at rhel6-x86-64 hist]# tail -n 3 f21.build f21,build.conn ==> f21.build <== conn 1430851017 1430828798 22219 gr re 1 conn 1430851048 1430828798 22250 re gr 2 conn 1430851081 1430828798 22283 gr re 1
==> f21,build.conn <== Tue May 5 04:38:48 2015 red 1430825928 2870 Tue May 5 05:26:38 2015 green 1430828798 22283 Tue May 5 11:38:01 2015 green 1430851081[root at rhel6-x86-64 hist]#
[root at rhel6-x86-64 hist]# grep f21 allevents | grep conn | tail -3 f21.build conn 1430828802 1430828798 4 re gr 2 f21.build conn 1430851048 1430828798 22250 re gr 2 f21.build conn 1430851081 1430828798 22283 gr re 1
Do you notice any unusual log lines for the relevant tasks? Alternatively, are you low on disk space on the volume xymond_history is running on? By default, it's configured to not save full snapshots of status logs when less than 5% is remaining.
On any restart of xymond, xymond will send a current status report through xymond_history, which will open up each host+svc event file and validate the last color matches what's currently set (updating if needed -- if the color changed while xymond was turned off), which *should* fix a missing event record.
It might also help to have xymond_history in --debug mode, to see if there's a potential logic flaw in how it's writing out.
HTH,
-jc
participants (3)
-
betsy.schwartz＠gmail.com
-
cleaver＠terabithia.org
-
the＠whero.net