Hey, all...
Have a puzzle to solve: we have a host (Solaris 10) running our tried-and-true version of Xymon (4.3.2) which is functioning just fine except for one test: the standard "memory" test.
It has worked fine in the past (I'm pretty sure!) but at some point we noticed Xymon isn't posting the "status" results for this test on the web page (it's a '-').
Debugging I've done so far:
On the client side, found the output file corresponding to "$MSGTMPFILE" / "$MSGFILE" used in the "xymonclient.sh" script. It _does_ contain valid data for the "[memory]" stanza:
[memory]
0 8 0 39549560 48447184 1039 3853 3934 14 14 0 0 0 0 0 0 20276 71323 29065 57 16 28
From this client, manually ran the command to post the data to the server:
$ ~xymon/client/bin/xymon <server IP> "@" < msg.hostname.txt log:/var/adm/messages:10240
ignore %nxge0|nxge1|nxge4|nxge5|nxge0|107833
On the server, ran:
$ ~xymon/server/bin/xymon 127.0.0.1 "xymondboard host=hostname " hostname|trends|green||0|0|0|0|0||0| hostname|info|green||0|0|0|0|0||0| hostname|files|clear||1434028386|1446841258|1446843058|0|0|10.235.130.17|-1|clear Files status at Fri Nov 6 14:20:43 CST 2015 hostname|msgs|green||1434028386|1446841258|1446843058|0|0|10.235.130.17|-1|green System logs at Fri Nov 6 14:20:43 CST 2015 hostname|ports|red||1446840042|1446841258|1446843058|0|0|10.235.130.17|27225|red Fri Nov 6 14:20:43 CST 2015 - Ports NOT ok ...
(The timestamps are current, matching the most recent version of the client file sent)
However, the "memory" test status is simply missing! For example, on the server:
$ ~xymon/server/bin/xymon 127.0.0.1 "xymondboard host=hostname fields=msg" | perl -pe 's!\\n!\n!gm' > out $ grep '^status' out status hostname.files clear Files status at Fri Nov 6 14:25:46 CST 2015 status hostname.msgs green System logs at Fri Nov 6 14:25:46 CST 2015 status/group:unix-sadm hostname.ports red Fri Nov 6 14:25:46 CST 2015 - Ports NOT ok status/group:unix-sadm hostname.procs red Fri Nov 6 14:25:46 CST 2015 - Processes NOT ok status hostname.disk green Fri Nov 6 14:25:46 CST 2015 - Filesystems OK status hostname.cpu green Fri Nov 6 14:25:46 CST 2015 up: 249 days, 10 users, 956 procs, load=21.63 status hostname.prtdiag green Fri Nov 6 14:25:48 CST 2015 status hostname.raid green Fri Nov 6 14:25:46 CST 2015 Veritas Volume Manager OK status hostname.meta green Fri Nov 6 14:25:47 CST 2015 status hostname.zones green Fri Nov 6 14:25:46 CST 2015 status+30 hostname.conn green <!-- [flags:OrdAstLe] --> Fri Nov 6 14:23:37 2015 conn ok status hostname.memUse GREEN status hostname.iostat GREEN status hostname.sar green Fri Nov 6 14:25:50 CST 2015 status+30 hostname.ssh green <!-- [flags:OrdastLe] --> Fri Nov 6 14:19:58 2015 ssh ok status+30 hostname.ntp green <!-- [flags:OrdastLe] --> Fri Nov 6 14:19:58 2015 ntp ok
Note: no "memory" status line!
Help?
Thanks!
david
~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~ David Mills Systems Administrator Northrop Grumman (512) 873-6665
However, the “memory” test status is simply missing! For example, on the server: ... status hostname.memUse GREEN ...
It does not look to be missing it looks to be misconfigured/munged somewhere. Is the field in the mem|memory display green with that single '-' ?
Do you have in the equiv of clients.cfg : MEMPHYS 100 101 MEMSWAP 87 90 MEMACT 90 97 values may not match, but do you have these set anywhere in your server config?
Just curious, this may not be relevant but is the client running on the Global Zone, or on a Child Zone?
Unrelated, but if it ever hits you there is likely somthing on the board you can search for:
The only issue I ever had with the memory test on Solaris 10 (x86) related to really odd reports similar to the following. Memory Used Total Percentage red Physical 4294966985M 30720M 4294967295% green Swap 173M 29153M 0%
Tim
From: Xymon [xymon-bounces at xymon.com] on behalf of Mills,David (HHSC Contractor) [David.Mills at hhsc.state.tx.us] Sent: Friday, November 06, 2015 12:36 PM To: xymon at xymon.com Subject: [Xymon] Missing client memory test!?
Hey, all…
Have a puzzle to solve: we have a host (Solaris 10) running our tried-and-true version of Xymon (4.3.2) which is functioning just fine except for one test: the standard “memory” test.
It has worked fine in the past (I’m pretty sure!) but at some point we noticed Xymon isn’t posting the “status” results for this test on the web page (it’s a ‘-‘).
Debugging I’ve done so far:
· On the client side, found the output file corresponding to “$MSGTMPFILE” / “$MSGFILE” used in the “xymonclient.sh” script. It _does_ contain valid data for the “[memory]” stanza:
[memory]
0 8 0 39549560 48447184 1039 3853 3934 14 14 0 0 0 0 0 0 20276 71323 29065 57 16 28
· From this client, manually ran the command to post the data to the server:
$ ~xymon/client/bin/xymon <server IP> "@" < msg.hostname.txt log:/var/adm/messages:10240
ignore %nxge0|nxge1|nxge4|nxge5|nxge0|107833
· On the server, ran:
$ ~xymon/server/bin/xymon 127.0.0.1 "xymondboard host=hostname " hostname|trends|green||0|0|0|0|0||0| hostname|info|green||0|0|0|0|0||0| hostname|files|clear||1434028386|1446841258|1446843058|0|0|10.235.130.17|-1|clear Files status at Fri Nov 6 14:20:43 CST 2015 hostname|msgs|green||1434028386|1446841258|1446843058|0|0|10.235.130.17|-1|green System logs at Fri Nov 6 14:20:43 CST 2015 hostname|ports|red||1446840042|1446841258|1446843058|0|0|10.235.130.17|27225|red Fri Nov 6 14:20:43 CST 2015 - Ports NOT ok …
(The timestamps are current, matching the most recent version of the client file sent)
· However, the “memory” test status is simply missing! For example, on the server:
$ ~xymon/server/bin/xymon 127.0.0.1 "xymondboard host=hostname fields=msg" | perl -pe 's!\\n!\n!gm' > out $ grep '^status' out status hostname.files clear Files status at Fri Nov 6 14:25:46 CST 2015 status hostname.msgs green System logs at Fri Nov 6 14:25:46 CST 2015 status/group:unix-sadm hostname.ports red Fri Nov 6 14:25:46 CST 2015 - Ports NOT ok status/group:unix-sadm hostname.procs red Fri Nov 6 14:25:46 CST 2015 - Processes NOT ok status hostname.disk green Fri Nov 6 14:25:46 CST 2015 - Filesystems OK status hostname.cpu green Fri Nov 6 14:25:46 CST 2015 up: 249 days, 10 users, 956 procs, load=21.63 status hostname.prtdiag green Fri Nov 6 14:25:48 CST 2015 status hostname.raid green Fri Nov 6 14:25:46 CST 2015 Veritas Volume Manager OK status hostname.meta green Fri Nov 6 14:25:47 CST 2015 status hostname.zones green Fri Nov 6 14:25:46 CST 2015 status+30 hostname.conn green <!-- [flags:OrdAstLe] --> Fri Nov 6 14:23:37 2015 conn ok status hostname.memUse GREEN status hostname.iostat GREEN status hostname.sar green Fri Nov 6 14:25:50 CST 2015 status+30 hostname.ssh green <!-- [flags:OrdastLe] --> Fri Nov 6 14:19:58 2015 ssh ok status+30 hostname.ntp green <!-- [flags:OrdastLe] --> Fri Nov 6 14:19:58 2015 ntp ok
Note: no “memory” status line!
Help?
Thanks!
david
~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~ David Mills Systems Administrator Northrop Grumman (512) 873-6665
Hey, Tim...
Thx for responding!
Re: "... the field in the mem|memory...": sorry -- not quite sure what you mean... If you're referring to the "memUse" test, that's a customized "ext" program I wrote, independent of the standard "memory"
Re: the MEM* limits: In analysis.cfg we do have those set for some hosts, but not for this one in particular. The class=... does have default values though.
Re: zones: this host only has the global zone...
;-)
-----Original Message----- From: Tim McCloskey [mailto:tm at freedom.com] Sent: Friday, November 06, 2015 3:56 PM To: Mills,David (HHSC Contractor); xymon at xymon.com Subject: RE: Missing client memory test!?
However, the "memory" test status is simply missing! For example, on the server: ... status hostname.memUse GREEN ...
It does not look to be missing it looks to be misconfigured/munged somewhere. Is the field in the mem|memory display green with that single '-' ?
Do you have in the equiv of clients.cfg : MEMPHYS 100 101 MEMSWAP 87 90 MEMACT 90 97 values may not match, but do you have these set anywhere in your server config?
Just curious, this may not be relevant but is the client running on the Global Zone, or on a Child Zone?
Unrelated, but if it ever hits you there is likely somthing on the board you can search for:
The only issue I ever had with the memory test on Solaris 10 (x86) related to really odd reports similar to the following. Memory Used Total Percentage red Physical 4294966985M 30720M 4294967295% green Swap 173M 29153M 0%
Tim
From: Xymon [xymon-bounces at xymon.com] on behalf of Mills,David (HHSC Contractor) [David.Mills at hhsc.state.tx.us] Sent: Friday, November 06, 2015 12:36 PM To: xymon at xymon.com Subject: [Xymon] Missing client memory test!?
Hey, all...
Have a puzzle to solve: we have a host (Solaris 10) running our tried-and-true version of Xymon (4.3.2) which is functioning just fine except for one test: the standard "memory" test.
It has worked fine in the past (I'm pretty sure!) but at some point we noticed Xymon isn't posting the "status" results for this test on the web page (it's a '-').
Debugging I've done so far:
On the client side, found the output file corresponding to "$MSGTMPFILE" / "$MSGFILE" used in the "xymonclient.sh" script. It _does_ contain valid data for the "[memory]" stanza:
[memory]
0 8 0 39549560 48447184 1039 3853 3934 14 14 0 0 0 0 0 0 20276 71323 29065 57 16 28
From this client, manually ran the command to post the data to the server:
$ ~xymon/client/bin/xymon <server IP> "@" < msg.hostname.txt log:/var/adm/messages:10240
ignore %nxge0|nxge1|nxge4|nxge5|nxge0|107833
On the server, ran:
$ ~xymon/server/bin/xymon 127.0.0.1 "xymondboard host=hostname " hostname|trends|green||0|0|0|0|0||0| hostname|info|green||0|0|0|0|0||0| hostname|files|clear||1434028386|1446841258|1446843058|0|0|10.235.130.17 hostname|files|clear||1434028386|1446841258|1446843058|0|0||-1|clear hostname|files|clear||1434028386|1446841258|1446843058|0|0|Files status hostname|files|clear||1434028386|1446841258|1446843058|0|0|at Fri Nov 6 hostname|files|clear||1434028386|1446841258|1446843058|0|0|14:20:43 CST hostname|files|clear||1434028386|1446841258|1446843058|0|0|2015 hostname|msgs|green||1434028386|1446841258|1446843058|0|0|10.235.130.17| hostname|msgs|green||1434028386|1446841258|1446843058|0|0|-1|green hostname|msgs|green||1434028386|1446841258|1446843058|0|0|System logs at hostname|msgs|green||1434028386|1446841258|1446843058|0|0|Fri Nov 6 hostname|msgs|green||1434028386|1446841258|1446843058|0|0|14:20:43 CST hostname|msgs|green||1434028386|1446841258|1446843058|0|0|2015 hostname|ports|red||1446840042|1446841258|1446843058|0|0|10.235.130.17|2 hostname|ports|red||1446840042|1446841258|1446843058|0|0|7225|red Fri hostname|ports|red||1446840042|1446841258|1446843058|0|0|Nov 6 14:20:43 hostname|ports|red||1446840042|1446841258|1446843058|0|0|CST 2015 - hostname|ports|red||1446840042|1446841258|1446843058|0|0|Ports NOT ok ...
(The timestamps are current, matching the most recent version of the client file sent)
However, the "memory" test status is simply missing! For example, on the server:
$ ~xymon/server/bin/xymon 127.0.0.1 "xymondboard host=hostname fields=msg" | perl -pe 's!\\n!\n!gm' > out $ grep '^status' out status hostname.files clear Files status at Fri Nov 6 14:25:46 CST 2015 status hostname.msgs green System logs at Fri Nov 6 14:25:46 CST 2015 status/group:unix-sadm hostname.ports red Fri Nov 6 14:25:46 CST 2015 - Ports NOT ok status/group:unix-sadm hostname.procs red Fri Nov 6 14:25:46 CST 2015 - Processes NOT ok status hostname.disk green Fri Nov 6 14:25:46 CST 2015 - Filesystems OK status hostname.cpu green Fri Nov 6 14:25:46 CST 2015 up: 249 days, 10 users, 956 procs, load=21.63 status hostname.prtdiag green Fri Nov 6 14:25:48 CST 2015 status hostname.raid green Fri Nov 6 14:25:46 CST 2015 Veritas Volume Manager OK status hostname.meta green Fri Nov 6 14:25:47 CST 2015 status hostname.zones green Fri Nov 6 14:25:46 CST 2015 status+30 hostname.conn green <!-- [flags:OrdAstLe] --> Fri Nov 6 status+14:23:37 2015 conn ok status hostname.memUse GREEN status hostname.iostat GREEN status hostname.sar green Fri Nov 6 14:25:50 CST 2015 status+30 hostname.ssh green <!-- [flags:OrdastLe] --> Fri Nov 6 status+14:19:58 2015 ssh ok status+30 hostname.ntp green <!-- [flags:OrdastLe] --> Fri Nov 6 status+14:19:58 2015 ntp ok
Note: no "memory" status line!
Help?
Thanks!
david
~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~ David Mills Systems Administrator Northrop Grumman (512) 873-6665
Yup, the memUse was what looked odd, it's an ext so clearly irrelevant. I'm guessing you've already checked to see if the similar hosts in the class= are working fine. I don't see how defining the MEM$foo values in analysis.cfg for that host would hurt. If it's anything like older versions you also need to make sure your default stanza is parsed last.
Since I still work with an older version day to day (pre analysys.cfg) I'm a bit rusty on the new config files. I'd need to go poke at one of the new versions we have to refresh the brain on that. If I get a chance I will, but I defer to others who have the newer versions memorized.
Good Luck, Tim
From: Mills,David (HHSC Contractor) [David.Mills at hhsc.state.tx.us] Sent: Friday, November 06, 2015 3:15 PM To: Tim McCloskey; xymon at xymon.com Subject: RE: Missing client memory test!?
Hey, Tim...
Thx for responding!
Re: "... the field in the mem|memory...": sorry -- not quite sure what you mean... If you're referring to the "memUse" test, that's a customized "ext" program I wrote, independent of the standard "memory"
Re: the MEM* limits: In analysis.cfg we do have those set for some hosts, but not for this one in particular. The class=... does have default values though.
Re: zones: this host only has the global zone...
;-)
-----Original Message----- From: Tim McCloskey [mailto:tm at freedom.com] Sent: Friday, November 06, 2015 3:56 PM To: Mills,David (HHSC Contractor); xymon at xymon.com Subject: RE: Missing client memory test!?
However, the "memory" test status is simply missing! For example, on the server: ... status hostname.memUse GREEN ...
It does not look to be missing it looks to be misconfigured/munged somewhere. Is the field in the mem|memory display green with that single '-' ?
Do you have in the equiv of clients.cfg : MEMPHYS 100 101 MEMSWAP 87 90 MEMACT 90 97 values may not match, but do you have these set anywhere in your server config?
Just curious, this may not be relevant but is the client running on the Global Zone, or on a Child Zone?
Unrelated, but if it ever hits you there is likely somthing on the board you can search for:
The only issue I ever had with the memory test on Solaris 10 (x86) related to really odd reports similar to the following. Memory Used Total Percentage red Physical 4294966985M 30720M 4294967295% green Swap 173M 29153M 0%
Tim
From: Xymon [xymon-bounces at xymon.com] on behalf of Mills,David (HHSC Contractor) [David.Mills at hhsc.state.tx.us] Sent: Friday, November 06, 2015 12:36 PM To: xymon at xymon.com Subject: [Xymon] Missing client memory test!?
Hey, all...
Have a puzzle to solve: we have a host (Solaris 10) running our tried-and-true version of Xymon (4.3.2) which is functioning just fine except for one test: the standard "memory" test.
It has worked fine in the past (I'm pretty sure!) but at some point we noticed Xymon isn't posting the "status" results for this test on the web page (it's a '-').
Debugging I've done so far:
On the client side, found the output file corresponding to "$MSGTMPFILE" / "$MSGFILE" used in the "xymonclient.sh" script. It _does_ contain valid data for the "[memory]" stanza:
[memory]
0 8 0 39549560 48447184 1039 3853 3934 14 14 0 0 0 0 0 0 20276 71323 29065 57 16 28
From this client, manually ran the command to post the data to the server:
$ ~xymon/client/bin/xymon <server IP> "@" < msg.hostname.txt log:/var/adm/messages:10240
ignore %nxge0|nxge1|nxge4|nxge5|nxge0|107833
On the server, ran:
$ ~xymon/server/bin/xymon 127.0.0.1 "xymondboard host=hostname " hostname|trends|green||0|0|0|0|0||0| hostname|info|green||0|0|0|0|0||0| hostname|files|clear||1434028386|1446841258|1446843058|0|0|10.235.130.17 hostname|files|clear||1434028386|1446841258|1446843058|0|0||-1|clear hostname|files|clear||1434028386|1446841258|1446843058|0|0|Files status hostname|files|clear||1434028386|1446841258|1446843058|0|0|at Fri Nov 6 hostname|files|clear||1434028386|1446841258|1446843058|0|0|14:20:43 CST hostname|files|clear||1434028386|1446841258|1446843058|0|0|2015 hostname|msgs|green||1434028386|1446841258|1446843058|0|0|10.235.130.17| hostname|msgs|green||1434028386|1446841258|1446843058|0|0|-1|green hostname|msgs|green||1434028386|1446841258|1446843058|0|0|System logs at hostname|msgs|green||1434028386|1446841258|1446843058|0|0|Fri Nov 6 hostname|msgs|green||1434028386|1446841258|1446843058|0|0|14:20:43 CST hostname|msgs|green||1434028386|1446841258|1446843058|0|0|2015 hostname|ports|red||1446840042|1446841258|1446843058|0|0|10.235.130.17|2 hostname|ports|red||1446840042|1446841258|1446843058|0|0|7225|red Fri hostname|ports|red||1446840042|1446841258|1446843058|0|0|Nov 6 14:20:43 hostname|ports|red||1446840042|1446841258|1446843058|0|0|CST 2015 - hostname|ports|red||1446840042|1446841258|1446843058|0|0|Ports NOT ok ...
(The timestamps are current, matching the most recent version of the client file sent)
However, the "memory" test status is simply missing! For example, on the server:
$ ~xymon/server/bin/xymon 127.0.0.1 "xymondboard host=hostname fields=msg" | perl -pe 's!\\n!\n!gm' > out $ grep '^status' out status hostname.files clear Files status at Fri Nov 6 14:25:46 CST 2015 status hostname.msgs green System logs at Fri Nov 6 14:25:46 CST 2015 status/group:unix-sadm hostname.ports red Fri Nov 6 14:25:46 CST 2015 - Ports NOT ok status/group:unix-sadm hostname.procs red Fri Nov 6 14:25:46 CST 2015 - Processes NOT ok status hostname.disk green Fri Nov 6 14:25:46 CST 2015 - Filesystems OK status hostname.cpu green Fri Nov 6 14:25:46 CST 2015 up: 249 days, 10 users, 956 procs, load=21.63 status hostname.prtdiag green Fri Nov 6 14:25:48 CST 2015 status hostname.raid green Fri Nov 6 14:25:46 CST 2015 Veritas Volume Manager OK status hostname.meta green Fri Nov 6 14:25:47 CST 2015 status hostname.zones green Fri Nov 6 14:25:46 CST 2015 status+30 hostname.conn green <!-- [flags:OrdAstLe] --> Fri Nov 6 status+14:23:37 2015 conn ok status hostname.memUse GREEN status hostname.iostat GREEN status hostname.sar green Fri Nov 6 14:25:50 CST 2015 status+30 hostname.ssh green <!-- [flags:OrdastLe] --> Fri Nov 6 status+14:19:58 2015 ssh ok status+30 hostname.ntp green <!-- [flags:OrdastLe] --> Fri Nov 6 status+14:19:58 2015 ntp ok
Note: no "memory" status line!
Help?
Thanks!
david
~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~ David Mills Systems Administrator Northrop Grumman (512) 873-6665
David
On 7 November 2015 at 07:36, Mills,David (HHSC Contractor) < David.Mills at hhsc.state.tx.us> wrote:
It has worked fine in the past (I’m pretty sure!) but at some point we noticed Xymon isn’t posting the “status” results for this test on the web page (it’s a ‘-‘).
The server-side processing requires that the client data contain both [memory] and [prtconf] sections. It also requires either a [swap] or a [swaplist] section (but both may be present on Solaris). Can you please check your client data display (eg go to CPU and click on the "client data available" message just above the graph) to see which of these are present, and if they look normal?
participants (3)
-
David.Mills＠hhsc.state.tx.us
-
jlaidman＠rebel-it.com.au
-
tm＠freedom.com