October 2019 - Xymon - Xymon mailman web

xymond_rrd - Program crashed after fresh install of Xymon 4.3.30 and data from Xymon 4.3.17
by A.Chervonets＠cominder.eu 18 Oct '19

18 Oct '19

To get more information I have enabled "--debug" to both channels (status and data). Then we see a bit more information in rrd-status.log: .... 2019-10-17 13:40:02.376153 Host 'synologyhost.domain.eu' reports netstat for an unknown OS 408 2019-10-17 13:40:02.376181 Flush, but xymonmsg is empty 408 2019-10-17 13:40:02.376185 0 status messages merged into 1 transmissions 408 2019-10-17 13:40:02.376203 xymond_rrd: Got message 612 @@status#612/synologyhost.domain.eu|1571308802.357389|83.99.221.6||synologyhost.domain.eu|procs|1571326802|green||green|1570620002|0||0||1571051696||p_cominder|0| 408 2019-10-17 13:40:02.376210 startpos 95710, fillpos 99309, endpos 97006 408 2019-10-17 13:40:02.376227 Flush, but xymonmsg is empty 408 2019-10-17 13:40:02.376233 0 status messages merged into 1 transmissions 408 2019-10-17 13:40:02.376244 xymond_rrd: Got message 613 @@status#613/synologyhost.domain.eu|1571308802.357673|83.99.221.6||synologyhost.domain.eu|raid|1571326802|green||green|1570620002|0||0||1571051696||p_cominder|0| 408 2019-10-17 13:40:02.376251 startpos 97010, fillpos 99309, endpos 97945 408 2019-10-17 13:40:02.376269 Flush, but xymonmsg is empty 408 2019-10-17 13:40:02.376276 0 status messages merged into 1 transmissions 408 2019-10-17 13:40:02.376288 xymond_rrd: Got message 614 @@status#614/synologyhost.domain.eu|1571308802.368308|83.99.221.6||synologyhost.domain.eu|temperature|1571326802|green||green|1570620002|0||0||1571051696||p_cominder|0| 408 2019-10-17 13:40:02.376294 startpos 97949, fillpos 99309, endpos 98645 2019-10-17 13:40:02.381339 Child process 408 died: Signal 6 2019-10-17 13:40:04.432302 Peer at 0.0.0.0:0 failed: Broken pipe 2019-10-17 13:40:04.452708 Peer not up, flushing message queue 13920 2019-10-17 13:40:04.557656 setup_feedback_queue: got ID -1 for key 0xA03EB91 13920 2019-10-17 13:40:04.558141 Opening file /u01/app/xymon/product/xymon4.3.30/server/etc/rrddefinitions.cfg 13920 2019-10-17 13:40:04.558326 Want msg 1, startpos 0, fillpos 0, endpos -1, usedbytes=0, bufleft=1052671 13920 2019-10-17 13:40:04.558359 Got 6716 bytes ... Here we can see processing of data from our Synology NAS with Synology Monitoring Tool 1.4.8, http://www.sysco.ch/synomon/ enabled. Make note - despite RRD crash we can see good status and text of "temperature" metric status like: -- Device Temp(C) Temp(F) --------------------------------------- green system 52 125 green /dev/sda 36 96 green /dev/sdb 38 100 green /dev/sdd 36 96 --------------------------------------- Synology Monitoring Tool 1.4.8, http://www.sysco.ch/synomon/ Model: RS812+ (synologyhost,domain.eu) Processor: Intel(R) Atom(TM) CPU D2701 @ 2.13GHz System temperature: 52?C Serial number: serialnumberdata-replaced Firmware: 6.2-24922 MAC address(s): number-replaced, number-replaced Linux version 3.10.105 (root at build10) (gcc version 4.9.3 20150311 (prerelease) (crosstool-NG 1.20.0) ) #24922 SMP Fri May 10 02:51:01 CST 2019 -- After stopping the plugin on Synology we have got no more data from it and no more xymond_rrd crash (red changed to purple, as expected). I am note sure where is the problem/bug. So I have added the Synology Monitoring Tool developers e-mail to our communictaion. Please, review and give the hint how can we fix the problem - our NAS state monitoring is quite critical thing we need. The suspection has been also proved by GDC info (as instructed at: http://www.robertandrobert.com/xymon/help/known-issues.html ): -- [xymon at synologyhost server]$ /bin/gdb /u01/app/xymon/product/xymon4.3.30/server/bin/xymond_rrd tmp/core.408 GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-115.el7 ... copyright... ... Reading symbols from /u01/app/xymon/product/xymon4.3.30/server/bin/xymond_rrd...done. [New LWP 408] [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/libthread_db.so.1". Core was generated by `xymond_rrd --rrddir=/u01/app/xymon/product/xymon4.3.30/data/rrd --debug'. Program terminated with signal 6, Aborted. #0 0x00007f62fcd85337 in raise () from /lib64/libc.so.6 Missing separate debuginfos, use: debuginfo-install bzip2-libs-1.0.6-13.el7.x86_64 cairo-1.15.12-4.el7.x86_64 expat-2.1.0-10.el7_3.x86_64 fontconfig-2.13.0-4.3.el7.x86_64 freetype-2.8-14.el7.x86_64 fribidi-1.0.2-1.el7.x86_64 glib2-2.56.1-5.el7.x86_64 glibc-2.17-292.el7.x86_64 graphite2-1.3.10-1.el7_3.x86_64 harfbuzz-1.7.5-2.el7.x86_64 keyutils-libs-1.5.8-3.el7.x86_64 krb5-libs-1.15.1-37.el7_7.2.x86_64 libX11-1.6.7-2.el7.x86_64 libXau-1.0.8-2.1.el7.x86_64 libXext-1.3.3-3.el7.x86_64 libXrender-0.9.10-1.el7.x86_64 libcom_err-1.42.9-16.el7.x86_64 libffi-3.0.13-18.el7.x86_64 libgcc-4.8.5-39.el7.x86_64 libglvnd-1.0.1-0.8.git5baa1e5.el7.x86_64 libglvnd-egl-1.0.1-0.8.git5baa1e5.el7.x86_64 libglvnd-glx-1.0.1-0.8.git5baa1e5.el7.x86_64 libpng-1.5.13-7.el7_2.x86_64 libselinux-2.5-14.1.el7.x86_64 libthai-0.1.14-9.el7.x86_64 libtirpc-0.2.4-0.16.el7.x86_64 libuuid-2.23.2-61.el7.x86_64 libxcb-1.13-1.el7.x86_64 libxml2-2.9.1-6.el7_2.3.x86_64 openssl-libs-1.0.2k-19.el7.x86_64 pango-1.42.4-4.el7_7.x86_64 pcre-8.32-17.el7.x86_64 pixman-0.34.0-1.el7.x86_64 rrdtool-1.4.8-9.el7.x86_64 xz-libs-5.2.2-1.el7.x86_64 zlib-1.2.7-18.el7.x86_64 (gdb) (gdb) (gdb) bt #0 0x00007f62fcd85337 in raise () at /lib64/libc.so.6 #1 0x00007f62fcd86a28 in abort () at /lib64/libc.so.6 #2 0x0000000000428e63 in sigsegv_handler (signum=<optimized out>) at sig.c:57 #3 0x00007f62fcd853b0 in <signal handler called> () at /lib64/libc.so.6 #4 0x00007f62fcd89f97 in ____strtoll_l_internal () at /lib64/libc.so.6 #5 0x000000000040f9c2 in do_temperature_rrd (__nptr=0x0) at /usr/include/stdlib.h:280 #6 0x000000000040f9c2 in do_temperature_rrd (hostname=hostname at entry=0x7f62fdfceb43 "synologyhost.domain.eu", testname=testname at entry=0x7f62fdfceb58 "temperature", classname=classname at entry=0x7f62fdfceb99 "p_cominder", pagepaths=pagepaths at entry=0x7f62fdfceba4 "0", msg=msg at entry=0x7f62fdfceba7 "status+300 synologyhost,domain.eu.temperature green 2019-10-17 13:40:01 [synologyhost.domain.eu] - temperature\nDevice", ' ' <repeats 13 times>, "Temp(C) Temp(F)\n", '-' <repeats 39 times>, "\n&green system"..., tstamp=tstamp at entry=1571308802) at rrd/do_temperature.c:100 #7 0x000000000041316b in update_rrd (hostname=hostname at entry=0x7f62fdfceb43 "synologyhost.domain.eu", testname=<optimized out>, testname at entry=0x7f62fdfceb58 "temperature", msg=msg at entry=0x7f62fdfceba7 "status+300 synologyhost,domain.eu.temperature green 2019-10-17 13:40:01 [synologyhost.domain.eu] - temperature\nDevice", ' ' <repeats 13 times>, "Temp(C) Temp(F)\n", '-' <repeats 39 times>, "\n&green system"..., tstamp=tstamp at entry=1571308802, sender=sender at entry=0x7f62fdfceb36 "83.99.221.6", ldef=<optimized out>, classname=classname at entry=0x7f62fdfceb99 "p_cominder", pagepaths=pagepaths at entry=0x7f62fdfceba4 "0") at do_rrd.c:714 #8 0x0000000000403434 in main (argc=<optimized out>, argv=0x7ffffb4bd4b8) at xymond_rrd.c:391 (gdb) -- So, we know which metric cause RRD crash, we have workaround (to make RRD working to generate other metrics graphs), but we need better solution to make all that working as expected. P.S. Note: real hostname is replaced in all outputs submitted in e-mail (just if there are some checksums are used). Best regards, Andrey Chervonets ---------------------- CoMinder Support http://www.cominder.eu/ mobile: +371 26517848 "Xymon" <xymon-bounces at xymon.com> wrote on 15.10.2019 13:00:01: > From: xymon-request at xymon.com > To: xymon at xymon.com > Date: 15.10.2019 13:00 > Subject: Xymon Digest, Vol 105, Issue 9 > Sent by: "Xymon" <xymon-bounces at xymon.com> > > ---------------------------------------------------------------------- > > Message: 1 > Date: Mon, 14 Oct 2019 15:09:53 +0300 > From: Andrey Chervonets <A.Chervonets at cominder.eu> > To: xymon at xymon.com > Subject: [Xymon] xymond_rrd - Program crashed after fresh install of > Xymon 4.3.30 and data from Xymon 4.3.17 > Message-ID: > <OFD5D1CD2D.3E1D4B14-ONC2258493.00408D6C-C2258493.0042D300 at cominder.eu> > > Content-Type: text/plain; charset="us-ascii" > > Good day! > > Recently we have installed Xymon 4.3.30 on new VM (CentOS Linux release > 7.7.1908 (Core) - guest under KVM > Guest Kernel: 3.10.0-1062.1.1.el7.x86_64 #1 SMP Fri Sep 13 22:55:44 UTC > 2019 x86_64 x86_64 x86_64 GNU/Linux > > All OK, except xymond_rrd is crashing frequently - the "xymond_rrd" metric > is always red (was never green) with message: > - Program crashed > Fatal signal caught! > > In rrd-status.log we can find frequent messages like: > > 2019-10-14 14:35:03.609265 Child process 2997 died: Signal 6 > 2019-10-14 14:35:04.239677 Peer at 0.0.0.0:0 failed: Broken pipe > 2019-10-14 14:35:08.886124 Peer not up, flushing message queue > 2019-10-14 14:36:45.883398 Host 'synologyhost.domain.eu' reports netstat > for an unknown OS > 2019-10-14 14:36:45.888875 Child process 21622 died: Signal 6 > 2019-10-14 14:36:52.510319 Peer at 0.0.0.0:0 failed: Broken pipe > 2019-10-14 14:36:52.510720 Peer not up, flushing message queue > 2019-10-14 14:40:02.689062 Host 'synologyhost.domain.eu' reports netstat > for an unknown OS > 2019-10-14 14:40:02.694320 Child process 28158 died: Signal 6 > 2019-10-14 14:40:05.119354 Peer at 0.0.0.0:0 failed: Broken pipe > 2019-10-14 14:40:05.250422 Peer not up, flushing message queue > > Note: lines like "Host 'synologyhost.domain.eu' reports netstat for an > unknown OS" are comining from Synonlogy NAS with Monitoring package > installed. > I am sure it is not related - it was working on old Xymon 4.3.17 (CentOS > 6.6) > > After fresh installation we just remapped (with symbolic link) the data > directory to continue employ old data logs and rra. > > There is plenty of core files under server/tmp/ > srw-rw-rw- 1 xymon monitor 0 Oct 14 14:40 rrdctl.572 > -rw------- 1 xymon monitor 3252224 Oct 14 14:45 core.572 > srw-rw-rw- 1 xymon monitor 0 Oct 14 14:45 rrdctl.17027 > -rw------- 1 xymon monitor 3248128 Oct 14 14:50 core.17027 > srw-rw-rw- 1 xymon monitor 0 Oct 14 14:50 rrdctl.30574 > -rw------- 1 xymon monitor 3248128 Oct 14 14:55 core.30574 > srw-rw-rw- 1 xymon monitor 0 Oct 14 14:55 rrdctl.13275 > -rw------- 1 xymon monitor 3239936 Oct 14 15:00 core.13275 > -rw-r--r-- 1 xymon monitor 1887355 Oct 14 15:02 xymond.chk > -rw-r--r-- 1 xymon monitor 0 Oct 14 15:02 alert.chk.sub > -rw-r--r-- 1 xymon monitor 70921 Oct 14 15:02 alert.chk > srw-rw-rw- 1 xymon monitor 0 Oct 14 15:02 rrdctl.5887 > srw-rw-rw- 1 xymon monitor 0 Oct 14 15:02 rrdctl.5954 > -rw------- 1 xymon monitor 3764224 Oct 14 15:05 core.5887 > srw-rw-rw- 1 xymon monitor 0 Oct 14 15:05 rrdctl.10234 > > > Question: How can we diagnose what is the cause of the problem? > > > > Best regards, > > Andrey Chervonets > ---------------------- > SIA CoMinder > http://www.cominder.eu/ > mobile: +371 26517848 >

2 1

[PATCH] - xymond readers grammar
by john.horne＠plymouth.ac.uk 17 Oct '19

17 Oct '19

Hello, Using the Terabithia 4.3.30 RPMs, and while I was in the xymond code, we noticed the xymond web page statistics showed the number of readers per channel. However, it was showing them as (for example): (1 readers) This is a trivial patch of the 'xymond/xymond.c' file just to correct the grammar so that a value of 1 is treated as singular (so '1 reader'), and anything else is given the plural form (e.g. '2 readers'). John. -- John Horne | Senior Operations Analyst | Technology and Information Services University of Plymouth | Drake Circus | Plymouth | Devon | PL4 8AA | UK ________________________________ [http://www.plymouth.ac.uk/images/email_footer.gif]<http://www.plymouth.ac.uk/worldclass> This email and any files with it are confidential and intended solely for the use of the recipient to whom it is addressed. If you are not the intended recipient then copying, distribution or other use of the information contained is strictly prohibited and you should not rely on it. If you have received this email in error please let the sender know immediately and delete it from your system(s). Internet emails are not necessarily secure. While we take every care, University of Plymouth accepts no responsibility for viruses and it is your responsibility to scan emails and their attachments. University of Plymouth does not accept responsibility for any changes made after it was sent. Nothing in this email or its attachments constitutes an order for goods or services unless accompanied by an official order form.

1 0

[PATCH] xymond - usermsg stats
by john.horne＠plymouth.ac.uk 17 Oct '19

17 Oct '19

Hello, Using the Terabithia 4.3.30 RPMs, we noticed that on the 'xymond' web page the incoming messages stats showed a lot of messages counted as 'Bogus/Unknown'. Several thousand of them (although the number depends on how long Xymon has been up and running). It turned out that these were 'usermsg' messages. We run a lot of client tests which report back their results over the user channel, so that explained that. Looking at the xymond source we could see that the statistics did not cater for 'usermsg' messages. This patch corrects that. It is literally a one-line patch of the 'xymond/xymond.c' file. Note that the location of the 'usermsg' entry in the file was chosen simply to avoid a clash with the 'xymon.timeoutstats.patch' already provided with the RPMs. John. -- John Horne | Senior Operations Analyst | Technology and Information Services University of Plymouth | Drake Circus | Plymouth | Devon | PL4 8AA | UK ________________________________ [http://www.plymouth.ac.uk/images/email_footer.gif]<http://www.plymouth.ac.uk/worldclass> This email and any files with it are confidential and intended solely for the use of the recipient to whom it is addressed. If you are not the intended recipient then copying, distribution or other use of the information contained is strictly prohibited and you should not rely on it. If you have received this email in error please let the sender know immediately and delete it from your system(s). Internet emails are not necessarily secure. While we take every care, University of Plymouth accepts no responsibility for viruses and it is your responsibility to scan emails and their attachments. University of Plymouth does not accept responsibility for any changes made after it was sent. Nothing in this email or its attachments constitutes an order for goods or services unless accompanied by an official order form.

1 0

Disabled in script alerts?
by nuitari＠nuitari.net 17 Oct '19

17 Oct '19

In SCRIPTS for alerts, how can we tell that something got disabled?

2 1

Enable/disable Issue
by scott.birl＠temple.edu 15 Oct '19

15 Oct '19

Instead of "-1" trying using "1m" (for 1 minute). Ive always had problems with spaces with the "why", so where you have powered down for now I had to change to powered_down_for_now ... so just in case you get another error. -----Original Message----- From: Xymon <xymon-bounces at xymon.com> On Behalf Of Uddin, Enaze via Xymon Sent: Thursday, October 10, 2019 02:53 PM To: Root, Paul T <Paul.Root at CenturyLink.com>; xymon at xymon.com Subject: Re: [Xymon] Enable/disable Issue Ok I tried again and I get this: $ ./xymon 127.0.0.1 "disable xymon.comm -1 powered down for now" Unknown option -1 Xymon version 4.3.28 Usage: ./xymon [--debug] [--merge] [--proxy=http://ip.of.the.proxy:port/] RECIPIENT DATA RECIPIENT: IP-address, hostname or URL DATA: Message to send, or "-" to read from stdin Is my syntax incorrect?

4 12

xymond_rrd - Program crashed after fresh install of Xymon 4.3.30 and data from Xymon 4.3.17
by A.Chervonets＠cominder.eu 14 Oct '19

14 Oct '19

Good day! Recently we have installed Xymon 4.3.30 on new VM (CentOS Linux release 7.7.1908 (Core) - guest under KVM Guest Kernel: 3.10.0-1062.1.1.el7.x86_64 #1 SMP Fri Sep 13 22:55:44 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux All OK, except xymond_rrd is crashing frequently - the "xymond_rrd" metric is always red (was never green) with message: - Program crashed Fatal signal caught! In rrd-status.log we can find frequent messages like: 2019-10-14 14:35:03.609265 Child process 2997 died: Signal 6 2019-10-14 14:35:04.239677 Peer at 0.0.0.0:0 failed: Broken pipe 2019-10-14 14:35:08.886124 Peer not up, flushing message queue 2019-10-14 14:36:45.883398 Host 'synologyhost.domain.eu' reports netstat for an unknown OS 2019-10-14 14:36:45.888875 Child process 21622 died: Signal 6 2019-10-14 14:36:52.510319 Peer at 0.0.0.0:0 failed: Broken pipe 2019-10-14 14:36:52.510720 Peer not up, flushing message queue 2019-10-14 14:40:02.689062 Host 'synologyhost.domain.eu' reports netstat for an unknown OS 2019-10-14 14:40:02.694320 Child process 28158 died: Signal 6 2019-10-14 14:40:05.119354 Peer at 0.0.0.0:0 failed: Broken pipe 2019-10-14 14:40:05.250422 Peer not up, flushing message queue Note: lines like "Host 'synologyhost.domain.eu' reports netstat for an unknown OS" are comining from Synonlogy NAS with Monitoring package installed. I am sure it is not related - it was working on old Xymon 4.3.17 (CentOS 6.6) After fresh installation we just remapped (with symbolic link) the data directory to continue employ old data logs and rra. There is plenty of core files under server/tmp/ srw-rw-rw- 1 xymon monitor 0 Oct 14 14:40 rrdctl.572 -rw------- 1 xymon monitor 3252224 Oct 14 14:45 core.572 srw-rw-rw- 1 xymon monitor 0 Oct 14 14:45 rrdctl.17027 -rw------- 1 xymon monitor 3248128 Oct 14 14:50 core.17027 srw-rw-rw- 1 xymon monitor 0 Oct 14 14:50 rrdctl.30574 -rw------- 1 xymon monitor 3248128 Oct 14 14:55 core.30574 srw-rw-rw- 1 xymon monitor 0 Oct 14 14:55 rrdctl.13275 -rw------- 1 xymon monitor 3239936 Oct 14 15:00 core.13275 -rw-r--r-- 1 xymon monitor 1887355 Oct 14 15:02 xymond.chk -rw-r--r-- 1 xymon monitor 0 Oct 14 15:02 alert.chk.sub -rw-r--r-- 1 xymon monitor 70921 Oct 14 15:02 alert.chk srw-rw-rw- 1 xymon monitor 0 Oct 14 15:02 rrdctl.5887 srw-rw-rw- 1 xymon monitor 0 Oct 14 15:02 rrdctl.5954 -rw------- 1 xymon monitor 3764224 Oct 14 15:05 core.5887 srw-rw-rw- 1 xymon monitor 0 Oct 14 15:05 rrdctl.10234 Question: How can we diagnose what is the cause of the problem? Best regards, Andrey Chervonets ---------------------- SIA CoMinder http://www.cominder.eu/ mobile: +371 26517848

1 0

Unsubscribe
by slambert＠lambertsolutions.com 11 Oct '19

11 Oct '19

Unsubscribe [photo] Stew Lambert President, Lambert Technology Solutions 630-357-2047<ccom:6303572047 at null> | slambert at lambertsoltuions.com<mailto:slambert at lambertsoltuions.com> www.lambertsolutions.com<http://www.lambertsolutions.com/> 23866 West Industrial Dr. North Plainfield, IL 60585 [https://dn3tzca2xtljm.cloudfront.net/social_icons/16px/facebook.png]<http://www.facebook.com/LambertTech/> [https://dn3tzca2xtljm.cloudfront.net/social_icons/16px/linkedin.png]<http://www.linkedin.com/in/stew-lambert-6b681111/>

2 1

DISPLAYGROUP/SUBPARENT not grouping/working
by xymon＠shared-files.de 11 Oct '19

11 Oct '19

Hello ? Theres an old thread "[Xymon] DISPLAYGROUP fails in analysis.cfg"[https://lists.xymon.com/archive/2019-February/046037.html] regarding same topic but problem not solved with 4.3.30 ? I configured this setup: ? hosts.cfg: ? page Company some fantasy company ?include /tmp/room1.cfg? include /tmp/room2.cfg? ? group house ?include /tmp/room1.cfg ? ? analysis.cfg: ? DISPLAYGROUP=house ? PROC %eecd 1 1 ? But it doesn?t matter what I do, all hosts are messured with ?%eecd 1 1?, not just group ?house?, also the hosts in /tmp/room2.cfg are affected. ? So I can say DISPLAYGROUP is not working at all in analysis.cfg In reality 300 hosts are affected. Btw. same thing with subparent. ? Also I know in version 4.3.18 and 4.3.10 in the Changelog they write something is solved with DISPLAYGROUP and analysed..but no clue if it is related to this issue: "- xymond_client: Fix DISPLAYGROUP handling in analysis.cfg rules" ? ? We are currently on version 4.3.30 ? ? Anyone knows about this ? ? ? Tom ? ?

1 0

DISPLAYGROUP/SUBPARENT not grouping/working
by xymon＠shared-files.de 11 Oct '19

11 Oct '19

An HTML attachment was scrubbed... URL: <http://lists.xymon.com/pipermail/xymon/attachments/20191011/f69e6ac2/attach…>

1 0

DISPLAYGROUP fails in analysis.cfg
by xymon＠shared-files.de 11 Oct '19

11 Oct '19

An HTML attachment was scrubbed... URL: <http://lists.xymon.com/pipermail/xymon/attachments/20191011/a187e247/attach…>

1 0