xymond not accepting connections
Hi, all!
We have a rather murky situation. A colleague accidentally completely removed the Xymon (4.3.3 / Solaris) server home directory recently. It was restored from backups, but since then that server has not been completely functioning. ('Don't know if our symptoms are related to the home dir "zap" or what...)
We periodically run the ghostlist.cgi app from cron and now these instances sometimes don't exit. When I run truss on them, I see they are almost continuously calling brk(): allocating anonymous memory for that instance's heap. It's gotten so bad that we've had this server's resources completely depleted and now have had to turn off the cron jobs
The xymond daemon is no longer accepting connections, despite the fact that this server has been stable for years.
The system was rebooted last night and seemed to be functioning throughout the night but stopped updating around 7:30 AM
Confirmed xymond is no longer accepting connections via:
17:03:49 pwsu020:/var/log/xymon> telnet 127.0.0.1 1984
Trying 127.0.0.1...
telnet: Unable to connect to remote host: Connection refused
17:02:53 pwsu020:/var/log/xymon> ps -u hobbit -f
UID PID PPID C STIME TTY TIME CMD
hobbit 4132 4131 1 17:02:36 ? 0:26 xymond --pidfile=/var/log/xymon/xymond.pid --restart=/export/xymon/server/tmp/x
hobbit 4288 4279 0 17:03:01 ? 0:01 /usr/bin/perl -w /usr/local/devmon/devmon
hobbit 12895 12867 0 10:54:01 pts/5 0:00 -bash
hobbit 4278 1908 0 17:03:00 ? 0:00 sh -c /usr/local/devmon/bin/restart.devmon> /dev/null 2>&1
hobbit 12491 5466 0 03:10:04 ? 0:00 /usr/local/apache2/bin/httpd -k start
hobbit 12490 5466 0 03:10:04 ? 0:00 /usr/local/apache2/bin/httpd -k start
hobbit 12487 5466 0 03:10:04 ? 0:00 /usr/local/apache2/bin/httpd -k start
hobbit 15958 5466 0 11:20:04 ? 0:00 /usr/local/apache2/bin/httpd -k start
hobbit 15612 5466 0 11:17:01 ? 0:00 /usr/local/apache2/bin/httpd -k start
hobbit 17158 5466 0 03:45:03 ? 0:00 /usr/local/apache2/bin/httpd -k start
hobbit 12488 5466 0 03:10:04 ? 0:00 /usr/local/apache2/bin/httpd -k start
hobbit 12489 5466 0 03:10:04 ? 0:00 /usr/local/apache2/bin/httpd -k start
hobbit 4290 4289 0 17:03:01 ? 0:01 /usr/bin/perl -w /usr/local/devmon/devmon
hobbit 4257 4143 0 17:02:46 ? 0:00 /home/hobbit/xymon/client/bin/xymon 0.0.0.0 @
hobbit 15776 5466 0 11:18:51 ? 0:00 /usr/local/apache2/bin/httpd -k start
hobbit 4148 4141 1 17:02:41 ? 0:21 /export/xymon/server/bin/xymonnet --ping --checkresponse --timeout=10 --dns-tim
hobbit 4279 4278 0 17:03:01 ? 0:00 /bin/ksh /usr/local/devmon/bin/restart.devmon
hobbit 4135 4131 0 17:02:41 ? 0:00 xymond_channel --channel=client --log=/var/log/xymon/clientdata.log xymond_clie
hobbit 4140 4131 1 17:02:41 ? 0:21 xymonnet --report --ping --checkresponse --timeout=10 --dns-timeout=2 --dnslog=
hobbit 4141 4131 0 17:02:41 ? 0:00 /bin/sh /export/xymon/server/ext/xymonnet-again.sh
hobbit 4137 4131 0 17:02:41 ? 0:00 xymond_channel --channel=data --log=/var/log/xymon/rrd-data.log xymond_rrd --rr
hobbit 4144 4131 0 17:02:41 ? 0:00 xymond_channel --channel=data --log=/var/log/xymon/data.log xymond_filestore --
hobbit 4131 1 0 17:02:36 ? 0:00 /export/xymon-4.3.3/server/bin/xymonlaunch --config=/export/xymon-4.3.3/server/
hobbit 4136 4131 0 17:02:41 ? 0:00 xymond_channel --channel=status --log=/var/log/xymon/rrd-status.log xymond_rrd
hobbit 4133 4131 0 17:02:41 ? 0:00 xymond_channel --channel=stachg --log=/var/log/xymon/history.log xymond_history
hobbit 12912 12885 0 10:54:09 pts/7 0:00 -bash
hobbit 4143 4131 0 17:02:41 ? 0:00 /bin/sh /export/xymon-4.3.3/client/bin/xymonclient.sh
hobbit 4289 4288 0 17:03:01 ? 0:00 /usr/bin/perl -w /usr/local/devmon/devmon
hobbit 4139 4131 1 17:02:41 ? 0:21 xymongen --recentgifs --subpagecolumns=2 --ignorecolumns=files --tooltips=never
hobbit 4134 4131 0 17:02:41 ? 0:00 xymond_channel --channel=page --log=/var/log/xymon/alert.log xymond_alert --che
hobbit 4138 4131 0 17:02:41 ? 0:00 xymond_channel --channel=clichg --log=/var/log/xymon/hostdata.log xymond_hostdaThe only other clue I've been able to find is this note in the xymonlaunch.log file:
15:29:24 pwsu020:/var/log/xymon> tail -50f xymonlaunch.log
...
2016-11-17 13:54:36 xymonlaunch starting
2016-11-17 13:54:36 Loading tasklist configuration from /export/xymon-4.3.3/server/etc/tasks.cfg
2016-11-17 13:54:36 Loading hostnames
2016-11-17 13:54:41 xgetenv: Cannot find value for variable HOME
2016-11-17 13:57:44 xgetenv: Cannot find value for variable HOME
2016-11-17 14:00:46 xgetenv: Cannot find value for variable HOME
Yet, when I tried this, as well as grep'ing through xymonlaunch "truss" output for HOME, I see valid home directory values:
13:58:35 pwsu020:~> echo 'echo HOME=$HOME XYMSRV=$XYMSRV XYMSERVERS=$XYMSERVERS XYMONDPORT=$XYMONDPORT' | /home/hobbit/xymon/client/bin/xymoncmd
2016-11-17 13:58:38 Using default environment file /export/xymon-4.3.3/client/etc/xymonclient.cfg
HOME=/home/hobbit XYMSRV=0.0.0.0 XYMSERVERS=10.235.57.11 10.235.157.56 XYMONDPORT=1984Help!
david
~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~ David Mills Systems Administrator Northrop Grumman (512) 595-1238 (mobile)
OK... This is new: more details from xymonlaunch.log file:
...
2016-11-18 07:13:34 Loading saved state
2016-11-18 07:13:59 Setting up network listener on 0.0.0.0:1984
2016-11-18 07:13:59 Setting up signal handlers
2016-11-18 07:13:59 Setting up xymond channels
2016-11-18 07:13:59 FATAL: xymond sees clientcount 2, should be 0
Check for hanging xymond_channel processes or stale semaphores
2016-11-18 07:13:59 Cannot setup data channel
2016-11-18 07:13:59 Task xymond terminated, status 1
2016-11-18 07:13:59 Task xymongen terminated by signal 15
2016-11-18 07:13:59 Task xymonnet terminated by signal 15
2016-11-18 07:13:59 Loading hostnames
2016-11-18 07:14:39 xgetenv: Cannot find value for variable HOME
2016-11-18 07:17:41 xgetenv: Cannot find value for variable HOME
2016-11-18 07:20:40 xgetenv: Cannot find value for variable HOME
2016-11-18 07:23:30 Task xymonnetagain terminated, status 208
...The above lines pretty much cycle endlessly.
From: Xymon [mailto:xymon-bounces at xymon.com] On Behalf Of Mills,David (HHSC Contractor) Sent: Thursday, November 17, 2016 5:17 PM To: 'xymon at xymon.com' Subject: [Xymon] xymond not accepting connections
Hi, all!
We have a rather murky situation. A colleague accidentally completely removed the Xymon (4.3.3 / Solaris) server home directory recently. It was restored from backups, but since then that server has not been completely functioning. ('Don't know if our symptoms are related to the home dir "zap" or what...)
We periodically run the ghostlist.cgi app from cron and now these instances sometimes don't exit. When I run truss on them, I see they are almost continuously calling brk(): allocating anonymous memory for that instance's heap. It's gotten so bad that we've had this server's resources completely depleted and now have had to turn off the cron jobs
The xymond daemon is no longer accepting connections, despite the fact that this server has been stable for years.
The system was rebooted last night and seemed to be functioning throughout the night but stopped updating around 7:30 AM
Confirmed xymond is no longer accepting connections via:
17:03:49 pwsu020:/var/log/xymon> telnet 127.0.0.1 1984
Trying 127.0.0.1...
telnet: Unable to connect to remote host: Connection refused
17:02:53 pwsu020:/var/log/xymon> ps -u hobbit -f
UID PID PPID C STIME TTY TIME CMD
hobbit 4132 4131 1 17:02:36 ? 0:26 xymond --pidfile=/var/log/xymon/xymond.pid --restart=/export/xymon/server/tmp/x
hobbit 4288 4279 0 17:03:01 ? 0:01 /usr/bin/perl -w /usr/local/devmon/devmon
hobbit 12895 12867 0 10:54:01 pts/5 0:00 -bash
hobbit 4278 1908 0 17:03:00 ? 0:00 sh -c /usr/local/devmon/bin/restart.devmon> /dev/null 2>&1
hobbit 12491 5466 0 03:10:04 ? 0:00 /usr/local/apache2/bin/httpd -k start
hobbit 12490 5466 0 03:10:04 ? 0:00 /usr/local/apache2/bin/httpd -k start
hobbit 12487 5466 0 03:10:04 ? 0:00 /usr/local/apache2/bin/httpd -k start
hobbit 15958 5466 0 11:20:04 ? 0:00 /usr/local/apache2/bin/httpd -k start
hobbit 15612 5466 0 11:17:01 ? 0:00 /usr/local/apache2/bin/httpd -k start
hobbit 17158 5466 0 03:45:03 ? 0:00 /usr/local/apache2/bin/httpd -k start
hobbit 12488 5466 0 03:10:04 ? 0:00 /usr/local/apache2/bin/httpd -k start
hobbit 12489 5466 0 03:10:04 ? 0:00 /usr/local/apache2/bin/httpd -k start
hobbit 4290 4289 0 17:03:01 ? 0:01 /usr/bin/perl -w /usr/local/devmon/devmon
hobbit 4257 4143 0 17:02:46 ? 0:00 /home/hobbit/xymon/client/bin/xymon 0.0.0.0 @
hobbit 15776 5466 0 11:18:51 ? 0:00 /usr/local/apache2/bin/httpd -k start
hobbit 4148 4141 1 17:02:41 ? 0:21 /export/xymon/server/bin/xymonnet --ping --checkresponse --timeout=10 --dns-tim
hobbit 4279 4278 0 17:03:01 ? 0:00 /bin/ksh /usr/local/devmon/bin/restart.devmon
hobbit 4135 4131 0 17:02:41 ? 0:00 xymond_channel --channel=client --log=/var/log/xymon/clientdata.log xymond_clie
hobbit 4140 4131 1 17:02:41 ? 0:21 xymonnet --report --ping --checkresponse --timeout=10 --dns-timeout=2 --dnslog=
hobbit 4141 4131 0 17:02:41 ? 0:00 /bin/sh /export/xymon/server/ext/xymonnet-again.sh
hobbit 4137 4131 0 17:02:41 ? 0:00 xymond_channel --channel=data --log=/var/log/xymon/rrd-data.log xymond_rrd --rr
hobbit 4144 4131 0 17:02:41 ? 0:00 xymond_channel --channel=data --log=/var/log/xymon/data.log xymond_filestore --
hobbit 4131 1 0 17:02:36 ? 0:00 /export/xymon-4.3.3/server/bin/xymonlaunch --config=/export/xymon-4.3.3/server/
hobbit 4136 4131 0 17:02:41 ? 0:00 xymond_channel --channel=status --log=/var/log/xymon/rrd-status.log xymond_rrd
hobbit 4133 4131 0 17:02:41 ? 0:00 xymond_channel --channel=stachg --log=/var/log/xymon/history.log xymond_history
hobbit 12912 12885 0 10:54:09 pts/7 0:00 -bash
hobbit 4143 4131 0 17:02:41 ? 0:00 /bin/sh /export/xymon-4.3.3/client/bin/xymonclient.sh
hobbit 4289 4288 0 17:03:01 ? 0:00 /usr/bin/perl -w /usr/local/devmon/devmon
hobbit 4139 4131 1 17:02:41 ? 0:21 xymongen --recentgifs --subpagecolumns=2 --ignorecolumns=files --tooltips=never
hobbit 4134 4131 0 17:02:41 ? 0:00 xymond_channel --channel=page --log=/var/log/xymon/alert.log xymond_alert --che
hobbit 4138 4131 0 17:02:41 ? 0:00 xymond_channel --channel=clichg --log=/var/log/xymon/hostdata.log xymond_hostdaThe only other clue I've been able to find is this note in the xymonlaunch.log file:
15:29:24 pwsu020:/var/log/xymon> tail -50f xymonlaunch.log
...
2016-11-17 13:54:36 xymonlaunch starting
2016-11-17 13:54:36 Loading tasklist configuration from /export/xymon-4.3.3/server/etc/tasks.cfg
2016-11-17 13:54:36 Loading hostnames
2016-11-17 13:54:41 xgetenv: Cannot find value for variable HOME
2016-11-17 13:57:44 xgetenv: Cannot find value for variable HOME
2016-11-17 14:00:46 xgetenv: Cannot find value for variable HOME
Yet, when I tried this, as well as grep'ing through xymonlaunch "truss" output for HOME, I see valid home directory values:
13:58:35 pwsu020:~> echo 'echo HOME=$HOME XYMSRV=$XYMSRV XYMSERVERS=$XYMSERVERS XYMONDPORT=$XYMONDPORT' | /home/hobbit/xymon/client/bin/xymoncmd
2016-11-17 13:58:38 Using default environment file /export/xymon-4.3.3/client/etc/xymonclient.cfg
HOME=/home/hobbit XYMSRV=0.0.0.0 XYMSERVERS=10.235.57.11 10.235.157.56 XYMONDPORT=1984Help!
david
~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~ David Mills Systems Administrator Northrop Grumman (512) 595-1238 (mobile)
This is (probably) a sign that you have stuck SysV IPC semaphores, probably from the previous crash.
The fix is to stop all xymon/hobbit processes, and then remove the hobbit-owned IPC stuff manually. On Linux, you'd run ipcs -a to find any segments (also, queues and arrays) owned by the xymon/hobbit user and use ipcrm to remove them.
ipcs output on a running system will look something like this:
------ Shared Memory Segments -------- key shmid owner perms bytes nattch status 0x010045d6 0 xymon 600 262144 4 0x020045d6 32769 xymon 600 262144 2 0x030045d6 65538 xymon 600 262144 2 0x040045d6 98307 xymon 600 262144 3 0x050045d6 131076 xymon 600 262144 2 0x060045d6 163845 xymon 600 32768 2 0x070045d6 196614 xymon 600 26214400 3 0x080045d6 229383 xymon 600 26214400 2 0x090045d6 262152 xymon 600 131072 1
I'm not sure if the commands are the same on Solaris (don't have a Sun box handy at the moment), but once those are gone things should start back up.
This is reduced to an error in 4.x instead of an abort, but the behavior is still undefined since it's easy for xymond to get into a deadlock with pre-existing semaphores set, while we wait for a message to be picked up by a process that may not exist.
HTH, -jc
On 11/18/2016 8:53 AM, Mills,David (HHSC Contractor) wrote:
OK… This is new: more details from xymonlaunch.log file:
…
2016-11-18 07:13:34 Loading saved state 2016-11-18 07:13:59 Setting up network listener on 0.0.0.0:1984 2016-11-18 07:13:59 Setting up signal handlers 2016-11-18 07:13:59 Setting up xymond channels 2016-11-18 07:13:59 FATAL: xymond sees clientcount 2,should be 0
Check for hanging xymond_channel processes or stale semaphores 2016-11-18 07:13:59 Cannot setup data channel 2016-11-18 07:13:59 Task xymond terminated, status 1 2016-11-18 07:13:59 Task xymongen terminated by signal 15 2016-11-18 07:13:59 Task xymonnet terminated by signal 15 2016-11-18 07:13:59 Loading hostnames 2016-11-18 07:14:39 xgetenv: Cannot find value for variableHOME
2016-11-18 07:17:41 xgetenv: Cannot find value for variableHOME
2016-11-18 07:20:40 xgetenv: Cannot find value for variableHOME
2016-11-18 07:23:30 Task xymonnetagain terminated, status 208 …The above lines pretty much cycle endlessly.
*From:*Xymon [mailto:xymon-bounces at xymon.com] *On Behalf Of *Mills,David (HHSC Contractor) *Sent:* Thursday, November 17, 2016 5:17 PM *To:* 'xymon at xymon.com' *Subject:* [Xymon] xymond not accepting connections
Hi, all!
We have a rather murky situation. A colleague accidentally completely removed the Xymon (4.3.3 / Solaris) server home directory recently. It was restored from backups, but since then that server has not been completely functioning. (‘Don’t know if our symptoms are related to the home dir “zap” or what…)
We periodically run the ghostlist.cgi app from cron and now these instances sometimes don’t exit. When I run truss on them, I see they are almost continuously calling brk(): allocating anonymous memory for that instance’s heap. It’s gotten so bad that we’ve had this server’s resources completely depleted and now have had to turn off the cron jobs
The xymond daemon is no longer accepting connections, despite the fact that this server has been stable for years.
The system was rebooted last night and seemed to befunctioning throughout the night but stopped updating around 7:30 AM
Confirmed xymond is no longer accepting connections via:17:03:49 pwsu020:/var/log/xymon> telnet 127.0.0.1 1984
Trying 127.0.0.1...
telnet: Unable to connect to remote host: Connection refused
17:02:53 pwsu020:/var/log/xymon> ps -u hobbit -f
UID PID PPID C STIME TTY TIME CMD
hobbit 4132 4131 1 17:02:36 ? 0:26 xymond --pidfile=/var/log/xymon/xymond.pid --restart=/export/xymon/server/tmp/x
hobbit 4288 4279 0 17:03:01 ? 0:01 /usr/bin/perl -w /usr/local/devmon/devmon
hobbit 12895 12867 0 10:54:01 pts/5 0:00 -bash
hobbit 4278 1908 0 17:03:00 ? 0:00 sh -c /usr/local/devmon/bin/restart.devmon> /dev/null 2>&1
hobbit 12491 5466 0 03:10:04 ? 0:00 /usr/local/apache2/bin/httpd -k start
hobbit 12490 5466 0 03:10:04 ? 0:00 /usr/local/apache2/bin/httpd -k start
hobbit 12487 5466 0 03:10:04 ? 0:00 /usr/local/apache2/bin/httpd -k start
hobbit 15958 5466 0 11:20:04 ? 0:00 /usr/local/apache2/bin/httpd -k start
hobbit 15612 5466 0 11:17:01 ? 0:00 /usr/local/apache2/bin/httpd -k start
hobbit 17158 5466 0 03:45:03 ? 0:00 /usr/local/apache2/bin/httpd -k start
hobbit 12488 5466 0 03:10:04 ? 0:00 /usr/local/apache2/bin/httpd -k start
hobbit 12489 5466 0 03:10:04 ? 0:00 /usr/local/apache2/bin/httpd -k start
hobbit 4290 4289 0 17:03:01 ? 0:01 /usr/bin/perl -w /usr/local/devmon/devmon
hobbit 4257 4143 0 17:02:46 ? 0:00 /home/hobbit/xymon/client/bin/xymon 0.0.0.0 @
hobbit 15776 5466 0 11:18:51 ? 0:00 /usr/local/apache2/bin/httpd -k start
hobbit 4148 4141 1 17:02:41 ? 0:21 /export/xymon/server/bin/xymonnet --ping --checkresponse --timeout=10 --dns-tim
hobbit 4279 4278 0 17:03:01 ? 0:00 /bin/ksh /usr/local/devmon/bin/restart.devmon
hobbit 4135 4131 0 17:02:41 ? 0:00 xymond_channel --channel=client --log=/var/log/xymon/clientdata.log xymond_clie
hobbit 4140 4131 1 17:02:41 ? 0:21 xymonnet --report --ping --checkresponse --timeout=10 --dns-timeout=2 --dnslog=
hobbit 4141 4131 0 17:02:41 ? 0:00 /bin/sh /export/xymon/server/ext/xymonnet-again.sh
hobbit 4137 4131 0 17:02:41 ? 0:00 xymond_channel --channel=data --log=/var/log/xymon/rrd-data.log xymond_rrd --rr
hobbit 4144 4131 0 17:02:41 ? 0:00 xymond_channel --channel=data --log=/var/log/xymon/data.log xymond_filestore --
hobbit 4131 1 0 17:02:36 ? 0:00 /export/xymon-4.3.3/server/bin/xymonlaunch --config=/export/xymon-4.3.3/server/
hobbit 4136 4131 0 17:02:41 ? 0:00 xymond_channel --channel=status --log=/var/log/xymon/rrd-status.log xymond_rrd
hobbit 4133 4131 0 17:02:41 ? 0:00 xymond_channel --channel=stachg --log=/var/log/xymon/history.log xymond_history
hobbit 12912 12885 0 10:54:09 pts/7 0:00 -bash
hobbit 4143 4131 0 17:02:41 ? 0:00 /bin/sh /export/xymon-4.3.3/client/bin/xymonclient.sh
hobbit 4289 4288 0 17:03:01 ? 0:00 /usr/bin/perl -w /usr/local/devmon/devmon
hobbit 4139 4131 1 17:02:41 ? 0:21 xymongen --recentgifs --subpagecolumns=2 --ignorecolumns=files --tooltips=never
hobbit 4134 4131 0 17:02:41 ? 0:00 xymond_channel --channel=page --log=/var/log/xymon/alert.log xymond_alert --che
hobbit 4138 4131 0 17:02:41 ? 0:00 xymond_channel --channel=clichg --log=/var/log/xymon/hostdata.log xymond_hostda
The only other clue I’ve been able to find is this note in the xymonlaunch.log file:
15:29:24 pwsu020:/var/log/xymon> tail -50f xymonlaunch.log
...
2016-11-17 13:54:36 xymonlaunch starting
2016-11-17 13:54:36 Loading tasklist configuration from /export/xymon-4.3.3/server/etc/tasks.cfg
2016-11-17 13:54:36 Loading hostnames
2016-11-17 13:54:41 xgetenv: Cannot find value for variable HOME
2016-11-17 13:57:44 xgetenv: Cannot find value for variable HOME
2016-11-17 14:00:46 xgetenv: Cannot find value for variable HOME
Yet, when I tried this, as well as grep’ing throughxymonlaunch “truss” output for HOME, I see valid home directory values:
13:58:35 pwsu020:~> echo 'echo HOME=$HOME XYMSRV=$XYMSRV XYMSERVERS=$XYMSERVERS XYMONDPORT=$XYMONDPORT' | /home/hobbit/xymon/client/bin/xymoncmd
2016-11-17 13:58:38 Using default environment file /export/xymon-4.3.3/client/etc/xymonclient.cfg
HOME=/home/hobbit XYMSRV=0.0.0.0 XYMSERVERS=10.235.57.11 10.235.157.56 XYMONDPORT=1984
Help!
david
~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~
David Mills Systems Administrator */Northrop Grumman/* (512) 595-1238 (mobile)
Xymon mailing list Xymon at xymon.com http://lists.xymon.com/mailman/listinfo/xymon
Thanks, Japheph!
In fact, I did have a number of semaphores that remained in memory after all Xymon processes had exited… (BTW: same “ipcs -a” to view in Solaris). I don’t know if it was coincidence or causal, but w/in a couple of hours of removing, the system was locked up and I had to reboot.
Now I can start up Xymon and exit without leaving semaphores littering the landscape. However, I have two new symptoms:
The xymonnet and xymonetagain daemons refuse to exit when I run the “xymon.sh stop” script. They just continue to run, growing in memory constantly (I ran “truss” against xymonnet and it just kept calling brk(…) – successfully – again and again, etc.The xymongen daemon does not regenerate a new $HOME/xymon/server/www/ directory if I remove the old one. There is no new updates to the old directory that I can see either.
I’m going to have Oracle take a look at the host to make sure there isn’t something wrong with the OS. I’m kind of stuck and would be grateful for any debugging suggestions…
TIA!
david
From: Japheth Cleaver [mailto:cleaver at terabithia.org] Sent: Friday, November 18, 2016 11:53 AM To: Mills,David (HHSC Contractor); 'xymon at xymon.com' Subject: Re: [Xymon] xymond not accepting connections
This is (probably) a sign that you have stuck SysV IPC semaphores, probably from the previous crash.
The fix is to stop all xymon/hobbit processes, and then remove the hobbit-owned IPC stuff manually. On Linux, you'd run ipcs -a to find any segments (also, queues and arrays) owned by the xymon/hobbit user and use ipcrm to remove them.
ipcs output on a running system will look something like this:
------ Shared Memory Segments --------
key shmid owner perms bytes nattch status
0x010045d6 0 xymon 600 262144 4
0x020045d6 32769 xymon 600 262144 2
0x030045d6 65538 xymon 600 262144 2
0x040045d6 98307 xymon 600 262144 3
0x050045d6 131076 xymon 600 262144 2
0x060045d6 163845 xymon 600 32768 2
0x070045d6 196614 xymon 600 26214400 3
0x080045d6 229383 xymon 600 26214400 2
0x090045d6 262152 xymon 600 131072 1 I'm not sure if the commands are the same on Solaris (don't have a Sun box handy at the moment), but once those are gone things should start back up.
This is reduced to an error in 4.x instead of an abort, but the behavior is still undefined since it's easy for xymond to get into a deadlock with pre-existing semaphores set, while we wait for a message to be picked up by a process that may not exist.
HTH, -jc
On 11/18/2016 8:53 AM, Mills,David (HHSC Contractor) wrote: OK… This is new: more details from xymonlaunch.log file:
…
2016-11-18 07:13:34 Loading saved state
2016-11-18 07:13:59 Setting up network listener on 0.0.0.0:1984
2016-11-18 07:13:59 Setting up signal handlers
2016-11-18 07:13:59 Setting up xymond channels
2016-11-18 07:13:59 FATAL: xymond sees clientcount 2, should be 0
Check for hanging xymond_channel processes or stale semaphores
2016-11-18 07:13:59 Cannot setup data channel
2016-11-18 07:13:59 Task xymond terminated, status 1
2016-11-18 07:13:59 Task xymongen terminated by signal 15
2016-11-18 07:13:59 Task xymonnet terminated by signal 15
2016-11-18 07:13:59 Loading hostnames
2016-11-18 07:14:39 xgetenv: Cannot find value for variable HOME
2016-11-18 07:17:41 xgetenv: Cannot find value for variable HOME
2016-11-18 07:20:40 xgetenv: Cannot find value for variable HOME
2016-11-18 07:23:30 Task xymonnetagain terminated, status 208
…The above lines pretty much cycle endlessly.
From: Xymon [mailto:xymon-bounces at xymon.com] On Behalf Of Mills,David (HHSC Contractor) Sent: Thursday, November 17, 2016 5:17 PM To: 'xymon at xymon.com<mailto:xymon at xymon.com>' Subject: [Xymon] xymond not accepting connections
Hi, all!
We have a rather murky situation. A colleague accidentally completely removed the Xymon (4.3.3 / Solaris) server home directory recently. It was restored from backups, but since then that server has not been completely functioning. (‘Don’t know if our symptoms are related to the home dir “zap” or what…)
We periodically run the ghostlist.cgi app from cron and now these instances sometimes don’t exit. When I run truss on them, I see they are almost continuously calling brk(): allocating anonymous memory for that instance’s heap. It’s gotten so bad that we’ve had this server’s resources completely depleted and now have had to turn off the cron jobs
The xymond daemon is no longer accepting connections, despite the fact that this server has been stable for years.
The system was rebooted last night and seemed to be functioning throughout the night but stopped updating around 7:30 AM
Confirmed xymond is no longer accepting connections via:
17:03:49 pwsu020:/var/log/xymon> telnet 127.0.0.1 1984
Trying 127.0.0.1...
telnet: Unable to connect to remote host: Connection refused
17:02:53 pwsu020:/var/log/xymon> ps -u hobbit -f
UID PID PPID C STIME TTY TIME CMD
hobbit 4132 4131 1 17:02:36 ? 0:26 xymond --pidfile=/var/log/xymon/xymond.pid --restart=/export/xymon/server/tmp/x
hobbit 4288 4279 0 17:03:01 ? 0:01 /usr/bin/perl -w /usr/local/devmon/devmon
hobbit 12895 12867 0 10:54:01 pts/5 0:00 -bash
hobbit 4278 1908 0 17:03:00 ? 0:00 sh -c /usr/local/devmon/bin/restart.devmon> /dev/null 2>&1
hobbit 12491 5466 0 03:10:04 ? 0:00 /usr/local/apache2/bin/httpd -k start
hobbit 12490 5466 0 03:10:04 ? 0:00 /usr/local/apache2/bin/httpd -k start
hobbit 12487 5466 0 03:10:04 ? 0:00 /usr/local/apache2/bin/httpd -k start
hobbit 15958 5466 0 11:20:04 ? 0:00 /usr/local/apache2/bin/httpd -k start
hobbit 15612 5466 0 11:17:01 ? 0:00 /usr/local/apache2/bin/httpd -k start
hobbit 17158 5466 0 03:45:03 ? 0:00 /usr/local/apache2/bin/httpd -k start
hobbit 12488 5466 0 03:10:04 ? 0:00 /usr/local/apache2/bin/httpd -k start
hobbit 12489 5466 0 03:10:04 ? 0:00 /usr/local/apache2/bin/httpd -k start
hobbit 4290 4289 0 17:03:01 ? 0:01 /usr/bin/perl -w /usr/local/devmon/devmon
hobbit 4257 4143 0 17:02:46 ? 0:00 /home/hobbit/xymon/client/bin/xymon 0.0.0.0 @
hobbit 15776 5466 0 11:18:51 ? 0:00 /usr/local/apache2/bin/httpd -k start
hobbit 4148 4141 1 17:02:41 ? 0:21 /export/xymon/server/bin/xymonnet --ping --checkresponse --timeout=10 --dns-tim
hobbit 4279 4278 0 17:03:01 ? 0:00 /bin/ksh /usr/local/devmon/bin/restart.devmon
hobbit 4135 4131 0 17:02:41 ? 0:00 xymond_channel --channel=client --log=/var/log/xymon/clientdata.log xymond_clie
hobbit 4140 4131 1 17:02:41 ? 0:21 xymonnet --report --ping --checkresponse --timeout=10 --dns-timeout=2 --dnslog=
hobbit 4141 4131 0 17:02:41 ? 0:00 /bin/sh /export/xymon/server/ext/xymonnet-again.sh
hobbit 4137 4131 0 17:02:41 ? 0:00 xymond_channel --channel=data --log=/var/log/xymon/rrd-data.log xymond_rrd --rr
hobbit 4144 4131 0 17:02:41 ? 0:00 xymond_channel --channel=data --log=/var/log/xymon/data.log xymond_filestore --
hobbit 4131 1 0 17:02:36 ? 0:00 /export/xymon-4.3.3/server/bin/xymonlaunch --config=/export/xymon-4.3.3/server/
hobbit 4136 4131 0 17:02:41 ? 0:00 xymond_channel --channel=status --log=/var/log/xymon/rrd-status.log xymond_rrd
hobbit 4133 4131 0 17:02:41 ? 0:00 xymond_channel --channel=stachg --log=/var/log/xymon/history.log xymond_history
hobbit 12912 12885 0 10:54:09 pts/7 0:00 -bash
hobbit 4143 4131 0 17:02:41 ? 0:00 /bin/sh /export/xymon-4.3.3/client/bin/xymonclient.sh
hobbit 4289 4288 0 17:03:01 ? 0:00 /usr/bin/perl -w /usr/local/devmon/devmon
hobbit 4139 4131 1 17:02:41 ? 0:21 xymongen --recentgifs --subpagecolumns=2 --ignorecolumns=files --tooltips=never
hobbit 4134 4131 0 17:02:41 ? 0:00 xymond_channel --channel=page --log=/var/log/xymon/alert.log xymond_alert --che
hobbit 4138 4131 0 17:02:41 ? 0:00 xymond_channel --channel=clichg --log=/var/log/xymon/hostdata.log xymond_hostdaThe only other clue I’ve been able to find is this note in the xymonlaunch.log file:
15:29:24 pwsu020:/var/log/xymon> tail -50f xymonlaunch.log
...
2016-11-17 13:54:36 xymonlaunch starting
2016-11-17 13:54:36 Loading tasklist configuration from /export/xymon-4.3.3/server/etc/tasks.cfg
2016-11-17 13:54:36 Loading hostnames
2016-11-17 13:54:41 xgetenv: Cannot find value for variable HOME
2016-11-17 13:57:44 xgetenv: Cannot find value for variable HOME
2016-11-17 14:00:46 xgetenv: Cannot find value for variable HOME
Yet, when I tried this, as well as grep’ing through xymonlaunch “truss” output for HOME, I see valid home directory values:
13:58:35 pwsu020:~> echo 'echo HOME=$HOME XYMSRV=$XYMSRV XYMSERVERS=$XYMSERVERS XYMONDPORT=$XYMONDPORT' | /home/hobbit/xymon/client/bin/xymoncmd
2016-11-17 13:58:38 Using default environment file /export/xymon-4.3.3/client/etc/xymonclient.cfg
HOME=/home/hobbit XYMSRV=0.0.0.0 XYMSERVERS=10.235.57.11 10.235.157.56 XYMONDPORT=1984Help!
david
~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~ David Mills Systems Administrator Northrop Grumman (512) 595-1238 (mobile)
Xymon mailing list
Xymon at xymon.com<mailto:Xymon at xymon.com>
participants (2)
-
cleaver＠terabithia.org
-
David.Mills＠hhsc.state.tx.us