So this one has me scratching my head... Noticing that ntpd wasn't running on a secondary server and that xymon didn't notice. Looking at the xymon procs page for the host, I notice that the entire contents of root's crontab is somehow on the ps list! Here's what the offending line in the output looks like.
19130 19120 562 01:46:38 S 19 0.0 00:00:00 0.0 600 8360
../bin/xymon 173.166.236.101 status nye,schoolpathways,com.crontab green Thu
Feb 5 01:46:39 UTC 2015??# min hour dofmonth month dow?1 * * * * wget --quiet
http://alpha.schoolpathways.com/.hidden/croncheck.php?hostname=`hostname` -O
/dev/null?1 1 * * * /root/bin/std/checkyum.php -e?2 3 * * *
/root/bin/std/ntpdate.update.sh?1 10 * * * /root/bin/std/fscheck.php
check???#SPFS ?1 2 * * * /sis/lr444/bin/scrublogs.php COMMIT?2 3 * * *
/sis/lr444/bin/clean_data.php?3 4 * * * /sis/lr444/bin/spfs_clean_temp.sh
delete?1 2 * * * /sis/mr-refactor-333/bin/scrublogs.php COMMIT?2 3 * * *
/sis/mr-refactor-333/bin/clean_data.php?3 4 * * * /sis/mr-
refactor-333/bin/spfs_clean_temp.sh delete???# zfs ?# */5 * * * *
/root/zfs.usage.php?1 1 * * 1 /sbin/zpool scrub spfstank?1 5 * * *
/sis/lr444/bin/zfs.snapshot.php nightly?1 10 * * * /sis/mr-
refactor-333/bin/zfs.snapshot.php nightly??# cat crontab files to
/tmp/cron_list.log for xymon crontab.sh?0 * * * *
/root/bin/std/show_crontab.sh??#XYMON?#?30 * * * * /root/bin/std/smart_chk.sh
2>/dev/null
Yes, that's the entirety of root's crontab, your standard assortment of admin and maintenance scripts... I have no idea how this is even possible, as xymon has no read access to root's cron script.
I've tried running ~xymonclient/client/bin/xymonclient-linux.sh (as user xymonclient) and it doesn't put out any line like above. Also, I notice that 562 is the "user" even though /etc/passwd has an entry for 562 being xymonclient.
Environment: CentOS 6 xymon-4.3.7
What else can I provide to help diagnose?
Ben