March 2016 - Xymon - Xymon mailman web

error running report.sh
by jlaidman＠rebel-it.com.au 16 Mar '16

16 Mar '16

Jason On Wed, Mar 16, 2016 at 12:09 PM Jason Brockdorf via Xymon <xymon at xymon.com> wrote: > I'm getting an error message: Cannot create output directory > > here's what I've done so far to try to fix it: > > 1. going to the /var/cache/xymon/ directory and creating a rep directory > You haven't set what your XYMONREPDIR is set to. On my system it's /usr/lib/xymon/server/www/rep/. On yours, it's probably ~xymon/server/www/rep/. 2. giving rwxrwxrwx permissions to that directory > That's a common solution to a permissions problem, but is not recommended. I'm hoping you are planning on removing the permissions once you get the problem sorted. When making world-writeable directories, it's best to set the sticky bit to avoid some potential security risks. Do this with "chmod +t /path/to/dir". The permissions should look the same as /tmp: $ ls -ld /tmp drwxrwxrwt 30 root root 344064 2016-03-16 14:33 /tmp But when you think about it, there are only two possible users. One is the xymon user and the other is the apache user. On my system, there's a www group, to which the apache user belongs. My "rep" directory is writeable by both the xymon user and the apache group, so that it looks like this: $ ls -ld ~xymon/server/www/rep drwxrwxr-x 3 xymon www 4096 2016-02-23 14:17 /usr/lib/xymon/server/www/rep The XYMONREPDIR environment variable is used in creating the directory > where the report will go, but I don't know: > > 1. How to check which process is running the file that's encountering the > problem (httpd or some xymon process) > It's the web server, so "apache" or "wwwrun" or whatever your web server is running as. > 2. How to check for the presence or value of this variable in the > environment that process is using > The relevant variables are set by CGI shell scripts. These scripts import variables from cgioptions.cfg, which in turn pulls in variables from xymonserver.cfg, and in particular, XYMONREPDIR. You can probably simulate the environment that the CGI process runs as, fairly accurately, by doing: $ sudo -u www ~xymon/server/bin/xymoncmd --env=`echo ~xymon/server/etc/cgioptions.cfg` --env=`echo ~xymon/server/etc/xymonserver.cfg` $ echo $XYMONREPDIR /usr/lib/xymon/server/www/rep Thus, it would seem that I am unable to reproduce the problem with "stock" > xymon, and I'm apparently not bright enough to fix it with terabithia srpms > either. At this point, I feel like I've put in enough effort to warrant > asking for help. Anyone, please help? > I'm not aware of a fault with the Terabithia (S)RPMs. At this stage I'm not sure you've proven that it's a problem with the package and not a problem with a config file or directory permission. I don't think you mentioned the OS you're using, and so it's possible that the Terabithia packages are not quite in alignment with your OS. I had a look through your patch, and I can't see any reason why it would not work. What I would do is create a copy of the report.sh script, eg report-test.sh and update it to run "strace" against the binary, so it looks like this: exec strace -f /usr/lib/xymon/server/bin/report.cgi $CGI_REP_OPTS $XYMONGENREPOPTS 2>/tmp/report.stderr Then run a report in your browser by adjusting the URL, and check the output file /tmp/report.stderr. This will not only show you where it's trying to create a report file, but the ownership of the file will tell you what user needs to own the rep directory. Cheers Jeremy

1 0

error running report.sh
by apocalysque＠yahoo.com 16 Mar '16

16 Mar '16

I'm still working on setting up Xymon (from the terabithia RPMs) and my manager tells me today that there's an error message when trying to run reports, like the availability report, snapshot report, etc. I'm getting an error message: Cannot create output directory When examining the logs for more details I don't get a whole lot to go on. I checked with both httpd and xymon logs and I didn't see anything relevant to this problem. I'm really trying to help myself and not make my problems everyone else's problems, and I'd really like to contribute to the community in some way, so here's what I've done so far to try to fix it: 1. going to the /var/cache/xymon/ directory and creating a rep directory2. giving rwxrwxrwx permissions to that directory I then managed to download the srpms from terabithia, find the offending file and examine the source to see what happens. The XYMONREPDIR environment variable is used in creating the directory where the report will go, but I don't know: 1. How to check which process is running the file that's encountering the problem (httpd or some xymon process)2. How to check for the presence or value of this variable in the environment that process is using So I decided to see if I could modify the source and make xymon give me a little more information. The resultant patch file is attached, though be forewarned, my C experience is limited to a primer class I took as a freshman in highschool and on top of not having that much knowledge in the first place, I'm very rusty. >From what I understand in the source there's not much error checking or information provided if it fails. I do see: envcheck(reqenv); But this by itself doesn't prevent continued execution if say my $XYMONREPDIR is not found. Well, I added a line to report which directory xymon is attempting to create, and another which should report the error message if the attempt was unsuccessful. I did manage to create the attached patch file, but I was unable to recompile using the srpm *sniffle* <-- I'm crying from frustration at this point So, I downloaded the source from sourceforge, replaced the report.c file with the one I made, compiled using that, copied the report.sh file to a working xymon server to try to test it... and I get internal server error :( I then went back to test it on xymon compiled from sourceforge source, and I get an error that I don't have permission to access that directory. *rips hair out* Thus, it would seem that I am unable to reproduce the problem with "stock" xymon, and I'm apparently not bright enough to fix it with terabithia srpms either. At this point, I feel like I've put in enough effort to warrant asking for help. Anyone, please help?

1 0

Monitoring large file
by neilsimmonds1808＠gmail.com 14 Mar '16

14 Mar '16

Hi all, I've got a requirement to monitor for the existence of some quite large files but I'm having a problem. I've updated the client-local.cfg file and the relevant files are showing green on the Xymon page, however even if i code the full name of one of the files in analysis.cfg it's not turning yellow. The only thing I can find that might be the problem is if I look at the client data from the link on the Xymon page, all the file entries look like this, [file:/veeam1/SERVER1_PROD/SERVER1_PROD2016-03-11T173239.vbk] ERROR: Value too large to be stored in data type These files are all quite large (this particular one is 691Gb) Is this a limitation in Xymon that I'm being affected by? Bizarrely if I code the following in my analysis.cfg file, FILE "/veeam1/SERVER1_PROD/SERVER1_PROD2016-03-12T170312.vbk" yellow exist It then reports yellow and says the file is missing despite the fact that it clearly isn't. Thanks, Neil.

2 2

Splitting acks in one page
by tomas.wachtarczyk＠gmail.com 14 Mar '16

14 Mar '16

Hello, Is it possible to multi-acknowledge problems from one page ? Example /usr 95% /application/whatever 100% Usr will go to unix admins /application will go to application team. I understand GROUP on DISK in analysis.cfg, but it does not seem to be right thing for me. Because it will be 2 different incidents. Can i acknowledge disk.usr with comment "some incident number" and disk.application with "different incident number" when both are under disk page ? Thank you for hint, W.

1 0

strange behavior
by mailinglists＠websitemanagers.com.au 13 Mar '16

13 Mar '16

On 13/03/16 17:41, Jason Brockdorf via Xymon wrote: > Well, I figured it out. > > I uninstalled and reinstalled xymon server to see if using all stock > would help and it didn't. All I added back into hosts.cfg was ONLY > the IPs and hostnames with no page definitions. > > It was then either one of two things that I noticed (I didn't wait > long enough between doing both of them to know which one fixed it...) > > First: I commented out the default #directory/etc/xymon/hosts.d line > (because I noticed everything above was working and everything below > was not) and then I noticed something else while waiting for it to update: > > In my history.log there was a bunch of lines: 2016-03-13 > 00:15:10.061160 Cannot create > /var/lib/xymon/histlogs/someservername/cpu: File exists > (someservername was one of the offenders) > Well, this was certainly one of the problems... > So I went and deleted everything in /var/lib/xymon/histlogs > > After that everything magically started to appear on the main page. I > believe it was deleting the directories in histlogs that fixed it > because they didn't all come back at once, they came back a few at a > time (just as my winscp was deleting the directories serially instead > of all at once) but that could be the client updates were coming in at > different times as well *shrug*. > > I'm sure it was something I did wrong but it was extremely frustrating > to say the least. Perhaps Xymon could try to append (is that > appropriate for those files?) in such a situation instead of just > logging an error and giving up? > The problem is that it was a file, and it should be a directory. Xymon doesn't delete the file (because it might contain valuable information), but it needs a directory to store it's information in multiple files. So the best it can do is log a message to "alert" you. > Thanks for all your help everyone that tried. Glad it is working for you now. Regards, Adam -- Adam Goryachev Website Managers www.websitemanagers.com.au

1 0

strange behavior
by apocalysque＠yahoo.com 13 Mar '16

13 Mar '16

I finished my masterpiece hosts.cfg and have had my xymon server up and running for a while but now that I've got client data coming in, I've noticed some very strange behavior. When I have the host on the front page, all tests show up fine. When I move the host to a page (first level, using the page command) the tests that depend on client data go "purple" and the data doesn't update. Also: hosts on pages cannot be seen when I click the "info" column and they're not seen in enable/disable tests either, even when I click on that option from the page they show up on. I've tried explicitly setting the groups they're in to display the "missing" data but it doesn't help. I have made some changes... but none that I think would/should be affecting it like this. I've changed the xymonclient-linux.sh script to use different commands, like adding the -h switch to df, or changing MACHINEDOTS="`hostname -s | tr [:upper:] [:lower:]`" (always a short name, always lower case, to address inconsistencies) but as soon as I move them back to the front page, the data shows up again and they're "green". The data from when they were "missing" still shows as missing in the history... I'm stumped on this one. I'm relatively certain the only changes I've made server side are only cosmetic changes to the underlying HTML and CSS files. I'm using the terabitha RPMs for my server and built from source for my client:xymon.x86_64 4.3.26-1.el7 @xymon-testing I DID change the config so that xymon is at the / of my web server, I'm not hosting anything else here. I'm relatively certain any client-side changes I've made aren't the problem since it works fine when on the main page. Any ideas how to fix this? I've spent a lot of time on this already and I just want to be done with it. :( Thanks in advance for any help, Signed,Extremely Frustrated (Jason Brockdorf)

3 6

Always purple history after time shift on server - how to fix
by A.Chervonets＠cominder.eu 10 Mar '16

10 Mar '16

I would like to share some hints in resolving history reporting problem after big time shift on monitoring server - about 4 hours. May be it will help anyone else. It was some month ago, but I have found time to fix it only today. What happened: 1. Time on monitoring host increased for 4 hours. 2. As result - all metrics reported Purple status (it is intended functionality, but would be nice XyMon detect big time shift and adopt reporting in some way) 3. It was problem at virtual host provider, I had reported the problem and time was fixed back to correct value 4. To fix current reporting I had cleaned some files under xymon/logs or acks (really I do not remember which ones right now) - this has reset last status duration information, but current values for all metrics become correct 5. Everythig become OK, except that when I check history for metric ( ...xymon-cgi/history.sh? ...) for some metrics. XyMon always reported Purple for last event (since that incident time). It was just for some metrics (not all) and I had second monitoring server with the same information (not having time shift incident) and I was able to live with it some month. Solution: Today I have fixed that reporting problem with the following steps, which should be executed for every host-metric pair having the problem We should operate with 2 files: 1) host history file like hist/HOSTNAME # here we should find records with negative duration values like: svcs 1435410898 1435426055 -15157 gr pu 1 who 1435410899 1435426055 -15156 gr pu 1 msgs 1435410899 1435426055 -15156 gr pu 1 netstat 1435410899 1435426055 -15156 gr pu 1 memory 1435411034 1435426055 -15021 ye pu 2 uptime 1435411140 1435426055 -14915 gr pu 1 procs 1435411145 1435426055 -14910 gr pu 1 disk 1435411150 1435426055 -14905 ye pu 2 cpu 1435411222 1435426055 -14833 gr pu 1 # and drop them 2) service history file like hist/HOSTNAME.svc # again - find records with negative duration values like: Sat Jun 27 20:27:35 2015 purple 1435426055 -15157 # and drop record(s) - really should be just one Really to fix just one service reporting - it is enough to drop negative duration records from service history file only (tested). But I do not see any reason to have such records in host history file, so I delete from that file too. How to automate the process: # find hist files for # step 1: find hist/ -print0 -name "*.*" | xargs -0 grep " -" | awk '{print $1" :"$4}' | grep ":-" #output like: ... hist/idc-oracle03.msc-sh.local:ssh :-14862 hist/idc-oracle03.msc-sh.local:dblock :-15012 hist/idc-oracle03.msc-sh.local:dbrec :-15012 hist/idc-oracle03.msc-sh.local:dbup :-15011 hist/idc-oracle03.msc-sh.local:dbext :-14989 ... # step 2: find hist/ -print0 -name "*.*" | xargs -0 grep " -" | awk '{print $1" :"$8}' | grep ":-" # output like: .. hist/idc-oracle03,domain.com.dbrec:Sat :-15012 hist/gdc-oracle03,domain.com.dbup:Sat :-15136 hist/idc-oracle01,domain.com.disk:Sat :-14961 hist/gdc-oracle01,domain.com.dbaud:Thu :-26793 hist/gdc-oracle01,domain.com.dbaud:Sat :-14940 .. Then can automate the records removal too. Best regards, Andrey Chervonets ---------------------- SIA CoMinder http://www.cominder.eu/

1 0

Xymon column headers in the Dashboard.
by cs＠agege.com 10 Mar '16

10 Mar '16

Greetings, I am new to Xymon and I need your help in the following areas. 1. How can I modify Xymon column headers in the Dashboard. ( i.e ssh cpu men). 2. I am looking for ways to get alerts when ever ipv6 is enable on the servers. Thank you for your considerations. Best regards, Agege

2 2

jmxstat
by abs＠shadymint.com 09 Mar '16

09 Mar '16

---------- Forwarded message ---------- > From: Galen Johnson <Galen.Johnson at sas.com> > Date: 9 March 2016 at 00:26 > Subject: [Possible Spam] [Xymon] jmxstat > To: "xymon at xymon.com" <xymon at xymon.com> > > Hey, > > I don't think this is specific to jmxstat but I'm trying to implement. I'm getting the > following in apache log when I hit the page: > > Setup error: Service GCInfo has a graph GCInfo, but no graph-definition,... > > However, the definition does exist and I can view the graphs if I select a different > > service and change the name in the URL. RRDs are being created. > > Anyone else run into this and overcome it? > > thanks Not seen that before, but just check for me please, is GCInfo mentioned in both TEST2RRD and GRAPHS in xymonserver.cfg? -- Andy

2 1

How to change the refresh time for acknowledgements
by novosirj＠rutgers.edu 09 Mar '16

09 Mar '16

John, Please keep replies “on list.” I don’t know that there’s much that you can do about that as there is a sweep that goes on, each run of the xymonnet process that runs at whatever interval you’ve selected (the default being 5 mins). I think if you acknowledge and catch that process in the middle, you’re going to get the messages from that run. They might also even have been received already by your e-mail system, before you acknowledged it. Someone else will know more than I do about the particulars here. > On Mar 9, 2016, at 1:23 PM, john boris <jborissr at gmail.com> wrote: > > Ryan, > I understand the web page takes some time to get changed but I have acknowledge issues and still receive notifications for a a few minutes after I acknowledge it. I will have to check the time the next time I do this just to be sure of the lag time. > > On Wed, Mar 9, 2016 at 1:15 PM, Ryan Novosielski <novosirj at rutgers.edu> wrote: > > On Mar 9, 2016, at 1:03 PM, john boris <jborissr at gmail.com> wrote: > > > > How can you change the response time when you acknowledge an issue so that it shows up as a check and you don't get pinged repeatedly. It looks like it takes about 5 minutes for the acknowledgement to take place. > > I’m pretty sure that acknowledgement is more-or-less immediate, and what you’re talking about is the delay before the display is updated. -- ____ *Note: UMDNJ is now Rutgers-Biomedical and Health Sciences* || \\UTGERS |---------------------*O*--------------------- ||_// Biomedical | Ryan Novosielski - Senior Technologist || \\ and Health | novosirj at rutgers.edu - 973/972.0922 (2x0922) || \\ Sciences | OIRT/High Perf & Res Comp - MSB C630, Newark `'

2 1