xymond_channel consuming a lot of CPU?
Hi,
I saw a little bit about this back in 8/31/12 (http://lists.xymon.com/archive/2012-August/035381.html), but it was never resolved? Has anyone else seen xymond_channel consuming so much CPU time? I can't imagine this being normal.
top - 14:07:56 up 6 days, 2:52, 1 user, load average: 0.97, 0.63, 0.58 Tasks: 117 total, 2 running, 115 sleeping, 0 stopped, 0 zombie Cpu(s): 22.7%us, 26.7%sy, 0.0%ni, 49.5%id, 0.3%wa, 0.2%hi, 0.0%si, 0.7%st Mem: 245792k total, 235308k used, 10484k free, 9248k buffers Swap: 1304648k total, 96604k used, 1208044k free, 44164k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 24914 xymon 20 0 7080 1388 1148 R 98.7 0.6 22:59.70 xymond_channel 1 root 20 0 3232 504 316 S 0.0 0.2 0:01.75 init 2 root 20 0 0 0 0 S 0.0 0.0 0:00.02 kthreadd 3 root RT 0 0 0 0 S 0.0 0.0 0:00.22 migration/0 4 root 20 0 0 0 0 S 0.0 0.0 0:02.03 ksoftirqd/0 5 root RT 0 0 0 0 S 0.0 0.0 0:00.00 migration/0 6 root RT 0 0 0 0 S 0.0 0.0 0:00.22 migration/1 7 root RT 0 0 0 0 S 0.0 0.0 0:00.00 migration/1 8 root 20 0 0 0 0 S 0.0 0.0 0:01.88 ksoftirqd/1 9 root 20 0 0 0 0 S 0.0 0.0 0:48.12 events/0 10 root 20 0 0 0 0 S 0.0 0.0 0:58.04 events/1 11 root 20 0 0 0 0 S 0.0 0.0 0:00.00 cpuset 12 root 20 0 0 0 0 S 0.0 0.0 0:00.00 khelper 13 root 20 0 0 0 0 S 0.0 0.0 0:00.00 netns 14 root 20 0 0 0 0 S 0.0 0.0 0:00.00 async/mgr 15 root 20 0 0 0 0 S 0.0 0.0 0:00.00 pm 16 root 20 0 0 0 0 S 0.0 0.0 0:01.54 sync_supers 17 root 20 0 0 0 0 S 0.0 0.0 0:02.81 bdi-default 18 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kintegrityd/0 19 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kintegrityd/1 20 root 20 0 0 0 0 S 0.0 0.0 0:00.32 kblockd/0 21 root 20 0 0 0 0 S 0.0 0.0 0:00.26 kblockd/1 22 root 20 0 0 0 0 S 0.0 0.0 0:00.00 md/0 23 root 20 0 0 0 0 S 0.0 0.0 0:00.00 md/1 24 root 20 0 0 0 0 S 0.0 0.0 0:00.00 md_misc/0 25 root 20 0 0 0 0 S 0.0 0.0 0:00.00 md_misc/1 26 root 20 0 0 0 0 S 0.0 0.0 0:00.10 cio 27 root 20 0 0 0 0 S 0.0 0.0 0:00.00 cio_chp 28 root 20 0 0 0 0 S 0.0 0.0 0:01.56 appldata 29 root 20 0 0 0 0 S 0.0 0.0 0:00.38 khungtaskd 30 root 20 0 0 0 0 S 0.0 0.0 0:04.48 kswapd0 31 root 25 5 0 0 0 S 0.0 0.0 0:00.00 ksmd
David Diep RHEL 6.2 for IBM zSeries Kernel: 2.6.32-358.23.2.el6.s390x
Look out for those in need this winter. When the temperature or wind chill is 32?F or below, the District issues a Hypothermia Alert. For assistance during an Alert, call the Shelter Hotline<http://dhs.dc.gov/page/hypothermia-alert-plan> at 1-800-535-7252 or 311.
On Mon, December 23, 2013 11:26 am, Diep, David (OCTO-Contractor) wrote:
Hi,
I saw a little bit about this back in 8/31/12 (http://lists.xymon.com/archive/2012-August/035381.html), but it was never resolved? Has anyone else seen xymond_channel consuming so much CPU time? I can't imagine this being normal.
top - 14:07:56 up 6 days, 2:52, 1 user, load average: 0.97, 0.63, 0.58 Tasks: 117 total, 2 running, 115 sleeping, 0 stopped, 0 zombie Cpu(s): 22.7%us, 26.7%sy, 0.0%ni, 49.5%id, 0.3%wa, 0.2%hi, 0.0%si, 0.7%st Mem: 245792k total, 235308k used, 10484k free, 9248k buffers Swap: 1304648k total, 96604k used, 1208044k free, 44164k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 24914 xymon 20 0 7080 1388 1148 R 98.7 0.6 22:59.70 xymond_channel
Hi David,
That much CPU time definitely isn't normal... Were you able to get an strace or see anything in the logs from around when that was happening.
The only thing that immediately comes to mind might be some sort of semaphore/IPC issue causing it to spin on incoming channel messages from xymond.
Regards,
-jc
Hi JC,
I don't see a lot of messages in /var/log/xymon, but I do see some messages in the messages log:
Dec 30 10:00:39 vipservd kernel: type=1400 audit(1388415639.785:228): avc: denied { read } for pid=22630 comm="httpd" name="xymon.html" dev=dm-0 ino=156411 scontext=system_u:system_r:httpd_t:s0 tcontext=unconfined_u:object_r:user_home_t:s0 tclass=file
30 10:00:39 vipservd kernel: __ratelimit: 39 callbacks suppressed
Dec 30 10:00:39 vipservd kernel: type=1400 audit(1388415639.768:226): avc: denied { search } for pid=22630 comm="httpd" name="xymon" dev=dm-0 ino=162241 scontext=system_u:system_r:httpd_t:s0 tcontext=unconfined_u:object_r:user_home_dir_t:s0 tclass=dir
Dec 30 10:00:39 vipservd kernel: type=1400 audit(1388415639.785:227): avc: denied { search } for pid=22630 comm="httpd" name="server" dev=dm-0 ino=162242 scontext=system_u:system_r:httpd_t:s0 tcontext=unconfined_u:object_r:user_home_t:s0 tclass=dir
Dec 30 10:00:39 vipservd kernel: type=1400 audit(1388415639.785:228): avc: denied { read } for pid=22630 comm="httpd" name="xymon.html" dev=dm-0 ino=156411 scontext=system_u:system_r:httpd_t:s0 tcontext=unconfined_u:object_r:user_home_t:s0 tclass=file
Dec 30 10:00:39 vipservd kernel: type=1400 audit(1388415639.785:229): avc: denied { open } for pid=22630 comm="httpd" name="xymon.html" dev=dm-0 ino=156411 scontext=system_u:system_r:httpd_t:s0 tcontext=unconfined_u:object_r:user_home_t:s0 tclass=file
Would I just execute the strace like so:
strace ./xymon.sh
David Diep
Look out for those in need this winter. When the temperature or wind chill is 32?F or below, the District issues a Hypothermia Alert. For assistance during an Alert, call the Shelter Hotline<http://dhs.dc.gov/page/hypothermia-alert-plan> at 1-800-535-7252 or 311.
-----Original Message----- From: J.C. Cleaver [mailto:cleaver at terabithia.org] Sent: Thursday, December 26, 2013 5:11 PM To: Diep, David (OCTO-Contractor) Cc: xymon at xymon.com Subject: Re: [Xymon] xymond_channel consuming a lot of CPU?
On Mon, December 23, 2013 11:26 am, Diep, David (OCTO-Contractor) wrote:
Hi,
I saw a little bit about this back in 8/31/12
(http://lists.xymon.com/archive/2012-August/035381.html), but it was
never resolved? Has anyone else seen xymond_channel consuming so much CPU time?
I can't imagine this being normal.
top - 14:07:56 up 6 days, 2:52, 1 user, load average: 0.97, 0.63, 0.58
Tasks: 117 total, 2 running, 115 sleeping, 0 stopped, 0 zombie
Cpu(s): 22.7%us, 26.7%sy, 0.0%ni, 49.5%id, 0.3%wa, 0.2%hi, 0.0%si,
0.7%st
Mem: 245792k total, 235308k used, 10484k free, 9248k buffers
Swap: 1304648k total, 96604k used, 1208044k free, 44164k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
24914 xymon 20 0 7080 1388 1148 R 98.7 0.6 22:59.70
xymond_channel
Hi David,
That much CPU time definitely isn't normal... Were you able to get an strace or see anything in the logs from around when that was happening.
The only thing that immediately comes to mind might be some sort of semaphore/IPC issue causing it to spin on incoming channel messages from xymond.
Regards,
-jc
Den 23.12.2013 20:26, Diep, David (OCTO-Contractor) skrev:
I saw a little bit about this back in 8/31/12 (http://lists.xymon.com/archive/2012-August/035381.html [1]), but it was never resolved? Has anyone else seen xymond_channel consuming so much CPU time? I can't imagine this being normal.
It isn't. I know of a couple of weird cases where this has happened, but not enough to pin-point what is causing it.
An immediate workaround is to just kill that process - it will restart automatically and should then work as expected.
Regards,
Henrik
Links:
participants (4)
-
cleaver＠terabithia.org
-
David.Diep＠dc.gov
-
henrik＠hswn.dk
-
jlaidman＠rebel-it.com.au