If it helps, here's the same core, with pstack.
pstack ./core_smcconsole_xymond_rrd_61_61_1426033626_20899
core './core_smcconsole_xymond_rrd_61_61_1426033626_20899' of 20899: xymond_rrd --debug --rrddir=/opt/local/xymon/data/rrd --no-cache feebebc4 _lwp_kill (6, 0, 0, 3a9a0, ffffffff, 6) + 8 fee329b0 abort (0, 1, feebafec, ffb3c, fef35518, 0) + 110 0003a9a0 sigsegv_handler (b, 0, ffbfa160, 0, fe8a2a00, ffbfa160) + 30 feebafec __sighndlr (b, 0, ffbfa160, 3a970, 0, 1) + c feeaf69c call_user_handler (b, 0, 0, 0, fe8a2a00, ffbfa160) + 3b8 feeaf884 sigacthandler (b, 0, ffbfa160, ffbfb290, 0, fee91924) + 60 --- called from signal handler with signal 11 (SIGSEGV) --- fee22d10 strlen (514cf, ffbfb3ec, ffbfa9d1, 0, 0, 0) + 50 fee91924 vfprintf (6c3d0, 514c0, ffbfb3e8, 0, a0ba4, 33e1c) + ec 0002fb4c dbgprintf (514c0, 0, 51400, 6c3f0, bf, 2ab388) + a4 00033e1c dump_tcp_services (a0, 1c00, fef37940, 0, 51400, 51400) + 74 000347f0 init_tcp_services (6a400, 51400, 51400, 6c3f0, bf, 23) + 91c 00030188 rrd_setup (93314, ffbfcfc4, 63c00, ffbfcf50, 6a400, 6a400) + 15c 00030518 find_xymon_rrd (93314, 511e8, 54ff8bda, 0, 932f3, 2e) + 4 00016614 main (93314, ffbfcfc4, 63c00, ffbfcf50, 54ff8bda, 93374) + 948 00015a40 _start (0, 0, 0, 0, 0, 0) + 5c
On 11 March 2015 at 08:37, Vernon Everett <everett.vernon at gmail.com> wrote:
Hi all
And even with --no-cache, I am still getting these corrupted rrd files. I tried again with --debug (and --no-cache) and it core dumps.
Here's the backtrace.
adb /zones/smcconsole/root/opt/local/xymon/server/bin/xymond_rrd
./core_smcconsole_xymond_rrd_61_61_1426033626_20899 core file = ./core_smcconsole_xymond_rrd_61_61_1426033626_20899 -- program `` /zones/smcconsole/root/opt/local/xymon/server/bin/xymond_rrd'' on platform SUNW,SPARC-Enterprise-T2000 SIGABRT: Abort $c libc.so.1
_lwp_kill+8(6, 0, 0, 3a9a0, ffffffff, 6) libc.so.1abort+0x110(0, 1, feebafec, ffb3c, fef35518, 0) sigsegv_handler+0x30(b, 0, ffbfa160, 0, fe8a2a00, ffbfa160) libc.so.1__sighndlr+0xc(b, 0, ffbfa160, 3a970, 0, 1) libc.so.1call_user_handler+0x3b8(b, 0, 0, 0, fe8a2a00, ffbfa160) libc.so.1sigacthandler+0x60(b, 0, ffbfa160, ffbfb290, 0, fee91924) libc.so.1strlen+0x50(514cf, ffbfb3ec, ffbfa9d1, 0, 0, 0) libc.so.1`vfprintf+0xec(6c3d0, 514c0, ffbfb3e8, 0, a0ba4, 33e1c) dbgprintf+0xa4(514c0, 0, 51400, 6c3f0, bf, 2ab388) dump_tcp_services+0x74(a0, 1c00, fef37940, 0, 51400, 51400) init_tcp_services+0x91c(6a400, 51400, 51400, 6c3f0, bf, 23) rrd_setup+0x15c(93314, ffbfcfc4, 63c00, ffbfcf50, 6a400, 6a400) find_xymon_rrd+4(93314, 511e8, 54ff8bda, 0, 932f3, 2e) main+0x948(93314, ffbfcfc4, 63c00, ffbfcf50, 54ff8bda, 93374) _start+0x5c(0, 0, 0, 0, 0, 0)Anything else I can offer that will assist?
Regards Vernon
On 5 March 2015 at 09:11, J.C. Cleaver <cleaver at terabithia.org> wrote:
On Wed, March 4, 2015 2:52 pm, Jeremy Laidman wrote:
On 04/03/2015 6:02 PM, "Vernon Everett" <everett.vernon at gmail.com> wrote:
Looks like we might need to check with JC for more on that GOCLIENT thing. I just find it odd that it happened about the same time as the corruption. I haven't seen it again today, and haven't seen any other corruption either.
If there's a correlation it might help us work out where the fault is. But it might be only a symptom.
As for the --debug option, it caused xymond_rrd to crash and burn, dumping cores as we go.
Could be that thensame bug causing the crash during debug is also causing the corrupt filename. Have you analyzed the core dumps?
GOCLIENT is indeed the means by which xymond_channel listeners communicate with xymond for the picking up of messages over SysV IPC. I believe the messages there are just a side effect of it re-launching the channel listener pipe to xymond_rrd.
The cache routines in xymond_rrd should be stable at this point. Can you send a backtrace in from one of the cores? I'm curious where things could be acting up here.
Regards,
-jc
-- "Accept the challenges so that you can feel the exhilaration of victory"
- General George Patton
-- "Accept the challenges so that you can feel the exhilaration of victory"
- General George Patton