Preventing Purple - Periodical Checks
I have a custom service check that is only done during certain times of the day and only on certain days of the week.
We use crontab and a shell script to generate the output and pipe it into the front-end using the xymon executable.
xymon 127.0.0.1 "status $server.$service $green|$red|$yellow $timestamp,$description"
This check works terrific when it runs, but in between checks it goes purple.
Without creating a fake service check in between each scheduled cron entry, what's the quickest way to configure that this check goes purple less often? (or if that's not possible, never go purple?) How are others handling occasional or daily service checks? Are you hard-coding the times into the script itself?
-Tres
- Tres.Finocchiaro at gmail.com
I think the "right" way would be to put the scheduling logic in the script, to have it always report green off-hours, rather than try to tell Xymon to ignore missing reports at certain times.
Xymon is specifically designed (AFAIK) to receive regular data feeds, not intermittent ones.
--
*Steve Coile*Senior Network and Systems Engineer, McClatchy Interactive <http://www.mcclatchyinteractive.com/> Office: 919-861-1247 | Mobile: 919-622-5369 | Fax: 919-861-1300
On Mon, Sep 22, 2014 at 3:55 PM, Tres Finocchiaro < tres.finocchiaro at gmail.com> wrote:
I have a custom service check that is only done during certain times of the day and only on certain days of the week.
We use crontab and a shell script to generate the output and pipe it into the front-end using the xymon executable.
xymon 127.0.0.1 "status $server.$service $green|$red|$yellow $timestamp,$description"
This check works terrific when it runs, but in between checks it goes purple.
Without creating a fake service check in between each scheduled cron entry, what's the quickest way to configure that this check goes purple less often? (or if that's not possible, never go purple?) How are others handling occasional or daily service checks? Are you hard-coding the times into the script itself?
-Tres
- Tres.Finocchiaro at gmail.com
Xymon mailing list Xymon at xymon.com http://lists.xymon.com/mailman/listinfo/xymon
Under BB, I would put in a "status+$delay" where the $delay was just a bit longer than needed to get me to the next check time. For example, if a check was to run once a day, I'd put in a delay of 25 hrs. this way, I would at least get a Purple light if it didn't run as often as I expected.
From: Xymon [mailto:xymon-bounces at xymon.com] On Behalf Of Steve Coile Sent: Monday, September 22, 2014 1:02 PM To: Tres Finocchiaro Cc: xymon at xymon.com Subject: Re: [Xymon] Preventing Purple - Periodical Checks
I think the "right" way would be to put the scheduling logic in the script, to have it always report green off-hours, rather than try to tell Xymon to ignore missing reports at certain times.
Xymon is specifically designed (AFAIK) to receive regular data feeds, not intermittent ones.
-- Steve Coile Senior Network and Systems Engineer, McClatchy Interactive<http://www.mcclatchyinteractive.com/> Office: 919-861-1247 | Mobile: 919-622-5369 | Fax: 919-861-1300
On Mon, Sep 22, 2014 at 3:55 PM, Tres Finocchiaro <tres.finocchiaro at gmail.com<mailto:tres.finocchiaro at gmail.com>> wrote: I have a custom service check that is only done during certain times of the day and only on certain days of the week.
We use crontab and a shell script to generate the output and pipe it into the front-end using the xymon executable.
xymon 127.0.0.1 "status $server.$service $green|$red|$yellow $timestamp,$description"
This check works terrific when it runs, but in between checks it goes purple.
Without creating a fake service check in between each scheduled cron entry, what's the quickest way to configure that this check goes purple less often? (or if that's not possible, never go purple?) How are others handling occasional or daily service checks? Are you hard-coding the times into the script itself?
-Tres
- Tres.Finocchiaro at gmail.com<mailto:Tres.Finocchiaro at gmail.com>
Xymon mailing list Xymon at xymon.com<mailto:Xymon at xymon.com> http://lists.xymon.com/mailman/listinfo/xymon
This message may contain confidential or proprietary information intended only for the use of the
addressee(s) named above or may contain information that is legally privileged. If you are
not the intended addressee, or the person responsible for delivering it to the intended addressee,
you are hereby notified that reading, disseminating, distributing or copying this message is strictly
prohibited. If you have received this message by mistake, please immediately notify us by
replying to the message and delete the original message and any copies immediately thereafter.
Thank you.
CLLD
@Paul/Mike,
Top notch. I decided to put the entire delay value in the string to prevent a syntax error if it's blank. In our case, we check a few times from noon until 6PM, so I chose 20 hours.
Delay for Xymon to prevent purple
DELAY="+20h";
Friday, add two more days
if [[ $(date +%u) == 5 ]] ; then DELAY="+68h"; fi
[...]
xymon 127.0.0.1 "status${DELAY} $server.$service $color $msg"
P.S. I don't know how we ever survived on a Windows platform. This product was meant for Linux and it's tremendously useful scripting capabilities. :)
Thanks again.
-Tres
I got a slightly different approach that seems to work fairly well for several same type tests. I got several things I run the same way, periodically so xymon would go purple. Usually it is cronjobs that runs doing whatever, but we added some xymon functionality to them. They output xymon data to a couple of files that xymon then checks through a standard xymon script. DATAFILE that shows whatever data from the cronjob. COLOR that just shows the color. HEADER just a one liner test heading in HTML. I then have xymon checking the age of the color file and just report on that color. For one test that is a binary we just run a wrapper cronscript that makes the xymon data and runs the binary. The application team has all control over what is put in the xymon files without the need to become root or the xymon user.
TEST=MYrpt
MYRPTDIR=/var/tmp
MYRPTBASE=MY-reports-xym
MYRPTDATAFILE=${MYRPTDIR}/${MYRPTBASE}.data
MYRPTCOLOR=${MYRPTDIR}/${MYRPTBASE}.color
MYRPTHEADER=${MYRPTDIR}/${MYRPTBASE}.header
ERRORTEXT="<H3>ERROR: the ${MYRPTCOLOR} file is older than one day.</H3><BR>
That probably means that the cronjob for checking the MY reports hasn't run.<BR>
Please check this in hostname<BR>
ls -l ${MYRPTCOLOR}"
Check that the color file is less than 1 day old, if not the cronjob haven't run.
AGE=find ${MYRPTDIR} -name ${MYRPTBASE}.color -a -mtime +1
if [ -n "$AGE" ]
then
HEADER=cat ${MYRPTHEADER}
COLOR=red
DATA="$ERRORTEXT"
else
HEADER=cat ${MYRPTHEADER}
COLOR=cat ${MYRPTCOLOR}
DATA=cat ${MYRPTDATAFILE}
fi
LINE="status $MACHINE.$TEST $COLOR date
${HEADER}
${DATA}
"
Send to Xymon Server
$XYMON $XYMSRV "$LINE"
From: Xymon [mailto:xymon-bounces at xymon.com] On Behalf Of Tres Finocchiaro Sent: Tuesday, 23 September 2014 6:28 AM To: Michael Short Cc: xymon at xymon.com Subject: Re: [Xymon] Preventing Purple - Periodical ChecksRoland
@Paul/Mike,
Top notch. I decided to put the entire delay value in the string to prevent a syntax error if it's blank. In our case, we check a few times from noon until 6PM, so I chose 20 hours.
Delay for Xymon to prevent purple
DELAY="+20h";
Friday, add two more days
if [[ $(date +%u) == 5 ]] ; then DELAY="+68h"; fi
[...]
xymon 127.0.0.1 "status${DELAY} $server.$service $color $msg"
P.S. I don't know how we ever survived on a Windows platform. This product was meant for Linux and it's tremendously useful scripting capabilities. :)
Thanks again.
-Tres
Actually, you’d want it to report what the test is already set as.
From: Xymon [mailto:xymon-bounces at xymon.com] On Behalf Of Steve Coile Sent: Monday, September 22, 2014 3:01 PM To: Tres Finocchiaro Cc: xymon at xymon.com Subject: Re: [Xymon] Preventing Purple - Periodical Checks
I think the "right" way would be to put the scheduling logic in the script, to have it always report green off-hours, rather than try to tell Xymon to ignore missing reports at certain times.
Xymon is specifically designed (AFAIK) to receive regular data feeds, not intermittent ones.
-- Steve Coile Senior Network and Systems Engineer, McClatchy Interactive<http://www.mcclatchyinteractive.com/> Office: 919-861-1247 | Mobile: 919-622-5369 | Fax: 919-861-1300
On Mon, Sep 22, 2014 at 3:55 PM, Tres Finocchiaro <tres.finocchiaro at gmail.com<mailto:tres.finocchiaro at gmail.com>> wrote: I have a custom service check that is only done during certain times of the day and only on certain days of the week.
We use crontab and a shell script to generate the output and pipe it into the front-end using the xymon executable.
xymon 127.0.0.1 "status $server.$service $green|$red|$yellow $timestamp,$description"
This check works terrific when it runs, but in between checks it goes purple.
Without creating a fake service check in between each scheduled cron entry, what's the quickest way to configure that this check goes purple less often? (or if that's not possible, never go purple?) How are others handling occasional or daily service checks? Are you hard-coding the times into the script itself?
-Tres
- Tres.Finocchiaro at gmail.com<mailto:Tres.Finocchiaro at gmail.com>
Xymon mailing list Xymon at xymon.com<mailto:Xymon at xymon.com> http://lists.xymon.com/mailman/listinfo/xymon
Set the time the status is good for
Xymon 127.0.0.1 “status+1d blah blah blah”
Day, hour, minute, week, I believe are all valid. Check the man page of xymon.
From: Xymon [mailto:xymon-bounces at xymon.com] On Behalf Of Tres Finocchiaro Sent: Monday, September 22, 2014 2:55 PM To: xymon at xymon.com Subject: [Xymon] Preventing Purple - Periodical Checks
I have a custom service check that is only done during certain times of the day and only on certain days of the week.
We use crontab and a shell script to generate the output and pipe it into the front-end using the xymon executable.
xymon 127.0.0.1 "status $server.$service $green|$red|$yellow $timestamp,$description"
This check works terrific when it runs, but in between checks it goes purple.
Without creating a fake service check in between each scheduled cron entry, what's the quickest way to configure that this check goes purple less often? (or if that's not possible, never go purple?) How are others handling occasional or daily service checks? Are you hard-coding the times into the script itself?
-Tres
- Tres.Finocchiaro at gmail.com<mailto:Tres.Finocchiaro at gmail.com>
What we do, and it has proven to be very helpful, at least for purple indications of Windows clients.
We send email alerts when “disk” has been purple for over 30 minutes.
Why, you may ask. There are specific instances when tests go purple and there is no need to alert on all tests. If disk is purple, most likely all the rest are purple except for connection.
- The number one cause of purple alerts is that the BBWin service is not running. Please check this service FIRST and start/restart if necessary.
- The client server is hung, sometimes it will respond to a ping and not otherwise indicate that it is down.
- The C drive is full. 99% of the time the purple is not indicative of a server issue, but when it is, sometimes this is the only way to catch it early.
Thanks, John Upcoming PTO: (none)
John Rothlisberger IT Strategy, Infrastructure & Security - Technology Growth Platform TGP for Business Process Outsourcing Accenture 312.693.3136 office
From: Xymon [mailto:xymon-bounces at xymon.com] On Behalf Of Tres Finocchiaro Sent: Monday, September 22, 2014 2:55 PM To: xymon at xymon.com Subject: [Xymon] Preventing Purple - Periodical Checks
I have a custom service check that is only done during certain times of the day and only on certain days of the week.
We use crontab and a shell script to generate the output and pipe it into the front-end using the xymon executable.
xymon 127.0.0.1 "status $server.$service $green|$red|$yellow $timestamp,$description"
This check works terrific when it runs, but in between checks it goes purple.
Without creating a fake service check in between each scheduled cron entry, what's the quickest way to configure that this check goes purple less often? (or if that's not possible, never go purple?) How are others handling occasional or daily service checks? Are you hard-coding the times into the script itself?
-Tres
- Tres.Finocchiaro at gmail.com<mailto:Tres.Finocchiaro at gmail.com>
This message is for the designated recipient only and may contain privileged, proprietary, or otherwise confidential information. If you have received it in error, please notify the sender immediately and delete the original. Any other use of the e-mail by you is prohibited. Where allowed by local law, electronic communications with Accenture and its affiliates, including e-mail and instant messaging (including content), may be scanned by our systems for the purposes of information security and assessment of internal compliance with Accenture policy.
www.accenture.com
We have a bunch of feeds that happen on odd schedules, and they go red or yellow on various problems, but are usually green. On days or times that there is no feed to process, the test goes clear. We usually set the lifetime to at least two iterations of the test, so there are no purples unless there is a different problem.
-Tracy
On Sep 23, 2014, at 8:48 AM, <john.r.rothlisberger at accenture.com> <john.r.rothlisberger at accenture.com> wrote:
What we do, and it has proven to be very helpful, at least for purple indications of Windows clients.
We send email alerts when “disk” has been purple for over 30 minutes.
Why, you may ask. There are specific instances when tests go purple and there is no need to alert on all tests. If disk is purple, most likely all the rest are purple except for connection. The number one cause of purple alerts is that the BBWin service is not running. Please check this service FIRST and start/restart if necessary. The client server is hung, sometimes it will respond to a ping and not otherwise indicate that it is down. The C drive is full. 99% of the time the purple is not indicative of a server issue, but when it is, sometimes this is the only way to catch it early.
Thanks, John Upcoming PTO: (none)
John Rothlisberger IT Strategy, Infrastructure & Security - Technology Growth Platform TGP for Business Process Outsourcing Accenture 312.693.3136 office
From: Xymon [mailto:xymon-bounces at xymon.com] On Behalf Of Tres Finocchiaro Sent: Monday, September 22, 2014 2:55 PM To: xymon at xymon.com Subject: [Xymon] Preventing Purple - Periodical Checks
I have a custom service check that is only done during certain times of the day and only on certain days of the week.
We use crontab and a shell script to generate the output and pipe it into the front-end using the xymon executable.
xymon 127.0.0.1 "status $server.$service $green|$red|$yellow $timestamp,$description"
This check works terrific when it runs, but in between checks it goes purple.
Without creating a fake service check in between each scheduled cron entry, what's the quickest way to configure that this check goes purple less often? (or if that's not possible, never go purple?) How are others handling occasional or daily service checks? Are you hard-coding the times into the script itself?
-Tres
- Tres.Finocchiaro at gmail.com
This message is for the designated recipient only and may contain privileged, proprietary, or otherwise confidential information. If you have received it in error, please notify the sender immediately and delete the original. Any other use of the e-mail by you is prohibited. Where allowed by local law, electronic communications with Accenture and its affiliates, including e-mail and instant messaging (including content), may be scanned by our systems for the purposes of information security and assessment of internal compliance with Accenture policy.
www.accenture.com
Xymon mailing list Xymon at xymon.com http://lists.xymon.com/mailman/listinfo/xymon
participants (7)
-
gendalia@iastate.edu
-
john.r.rothlisberger@accenture.com
-
mshort@corelogic.com
-
Paul.Root@CenturyLink.com
-
Rolands@logicaltech.com.au
-
scoile@mcclatchyinteractive.com
-
tres.finocchiaro@gmail.com