This may not help, but....

I've had core dumps from xymonnet that I *think* are related to running LDAP checks against an Active Directory that won't talk back. Or possibly it gets junk back that it can't understand, I don't know. Some times xymonnet core-dumps, other times it hangs up. Once it hangs, no more network tests happen - I guess the scheduler sees xymonnet already running and won't start another. I've had to install a cron job that looks for xymonnet running longer than 30 minutes and kills it. This is with xymon-4.3.12, RHEL5, compiled from source.

Can you separate out some of your network tests? Maybe spin up another copy of xymon and offload either your ping or ssh or http(s) tests to it?

Ralph Mitchell

On Mon, Dec 14, 2020 at 9:53 AM Carl Melgaard <Carl.Melgaard at stab.rm.dk> wrote:

Hi,

After running for 5 hrs on my new installation on a RH 7.9, xymond has already allocated 11.5GB of memory? Last night it core-dumped multiple times, and threw ?Cannot allocate memory? in multiple xymon logfiles, ala ?newstrbuffer: Attempt to allocate failed (initialsize=1009956863): Cannot allocate memory?. Monitoring 1900 hosts currently ? on my primary system I do this with only 4 GB of memory with no issues.

Any idea where I should start to look ? it?s a terabithia installation.

Heres a couple of the core-dumps gdb?ed:

Reading symbols from /usr/libexec/xymon/xymongen...Reading symbols from /usr/lib/debug/usr/libexec/xymon/xymongen.debug...done.

done.

[New LWP 10035]

[Thread debugging using libthread_db enabled]

Using host libthread_db library "/lib64/libthread_db.so.1".

Core was generated by `xymongen --recentgifs --subpagecolumns=4 --report --max-eventtime=1440 --max-ac'.

Program terminated with signal 6, Aborted.

#0 0x00007f8bb64aa387 in __GI_raise (sig=sig at entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:55

55 return INLINE_SYSCALL (tgkill, 3, pid, selftid, sig);

(gdb) bt

#0 0x00007f8bb64aa387 in __GI_raise (sig=sig at entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:55

#1 0x00007f8bb64aba78 in __GI_abort () at abort.c:90

#2 0x0000561f05bf6115 in sigsegv_handler (signum=<optimized out>) at sig.c:57

#3 <signal handler called>

#4 strbuf_addtobuffer (buf=0x0, newtext=0x561f0701db60 "extcombo", ' ' <repeats 192 times>..., newlen=2000) at strfunc.c:115

#5 0x0000561f05bf79b5 in addtobufferraw (buf=<optimized out>, newdata=<optimized out>, bytes=<optimized out>) at strfunc.c:184

#6 0x0000561f05c00d32 in combo_start () at sendmsg.c:908

#7 0x0000561f05bd7ccb in main (argc=6, argv=0x7ffe092839a8) at xymongen.c:706

Reading symbols from /usr/libexec/xymon/xymonnet...Reading symbols from /usr/lib/debug/usr/libexec/xymon/xymonnet.debug...done.

done.

[New LWP 15437]

[Thread debugging using libthread_db enabled]

Using host libthread_db library "/lib64/libthread_db.so.1".

Core was generated by `xymonnet --report --ping --checkresponse --dns-timeout=3 --dnslog=/var/log/xymo'.

Program terminated with signal 6, Aborted.

#0 0x00007f96383f0387 in __GI_raise (sig=sig at entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:55

55 return INLINE_SYSCALL (tgkill, 3, pid, selftid, sig);

(gdb) bt

#0 0x00007f96383f0387 in __GI_raise (sig=sig at entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:55

#1 0x00007f96383f1a78 in __GI_abort () at abort.c:90

#2 0x0000000000422d95 in sigsegv_handler (signum=<optimized out>) at sig.c:57

#3 <signal handler called>

#4 strbuf_addtobuffer (buf=0x0, newtext=0x2a99910 "extcombo", ' ' <repeats 192 times>..., newlen=2000) at strfunc.c:115

#5 0x0000000000424635 in addtobufferraw (buf=<optimized out>, newdata=<optimized out>, bytes=<optimized out>) at strfunc.c:184

#6 0x000000000042d9b2 in combo_start () at sendmsg.c:908

#7 0x00000000004064dc in main (argc=6, argv=0x7ffc4e0055d8) at xymonnet.c:2554

Seems like all core-dumps are from xymonnet and xymongen?

Where do I start?

Regards,

Carl Melgaard

---

Xymon mailing list Xymon at xymon.com http://lists.xymon.com/mailman/listinfo/xymon

Hi Carl

On Tue, 15 Dec 2020 at 01:53, Carl Melgaard <Carl.Melgaard at stab.rm.dk> wrote:

Hi,

After running for 5 hrs on my new installation on a RH 7.9, xymond has already allocated 11.5GB of memory?

xymond using a lot of RAM could be something different from the core dumps. But I suspect they're related. For instance, if it's having to keep lots of large combo messages in RAM while other modules send or receive them, but the other modules keep crashing. It's not clear if the xymonnet and xymongen crashes are causing the high RAM usage, or the other way around. It might be worth checking log timestamps to work out what happened first.

Last night it core-dumped multiple times, and threw ?Cannot allocate

memory? in multiple xymon logfiles, ala ?newstrbuffer: Attempt to allocate failed (initialsize=1009956863): Cannot allocate memory?.

"Cannot allocate memory" - do you have swap space? Is it being used?

Monitoring 1900 hosts currently ? on my primary system I do this with only 4 GB of memory with no issues.

What version of Xymon are you running on the primary system? Similar OS?

Any idea where I should start to look ? it?s a terabithia installation.

Heres a couple of the core-dumps gdb?ed:

The two core dumps suggest the same cause.

#2 0x0000561f05bf6115 in sigsegv_handler (signum=<optimized out>) at sig.c:57

#3 <signal handler called>

The sigsegv handler was called, which probably means there was a memory segment violation - typically using memory that hasn't been allocated.

I'm not a C programmer, but I'm guessing from this:

#4 strbuf_addtobuffer (buf=0x0, newtext=0x561f0701db60 "extcombo", ' '

<repeats 192 times>..., newlen=2000) at strfunc.c:115

that the code responsible is (in strfunc.c):

void strbuf_addtobuffer(strbuffer_t *buf, char *newtext, size_t newlen) { if (buf->s == NULL) { buf->used = 0; buf->sz = newlen + BUFSZINCREMENT; buf->s = (char *) malloc(buf->sz); *(buf->s) = '\0'; }

The "malloc()" operation may have failed due to running out of memory. Then the next line tries to store a "0" byte into unallocated RAM. I'd guess this would cause a sigsegv.

In other parts of the same file, malloc() is followed by a check for failure, before the memory is used:

For instance:

    newbuf->s = (char *)malloc(initialsize);
    if (newbuf->s == NULL) {
            errprintf("newstrbuffer: Attempt to allocate failed

(initialsize=%d): %s\n", initialsize, strerror(errno)); xfree(newbuf); return NULL; } *(newbuf->s) = '\0';

The above error checking has been added to some of the code, but perhaps there are places it still needs to be added.

This appears to have happened during the addition of a combo message string to allocated memory, while creating a message to send to xymond (sendmsg.c):

#5 0x0000561f05bf79b5 in addtobufferraw (buf=<optimized out>,

newdata=<optimized out>, bytes=<optimized out>) at strfunc.c:184

#6 0x0000561f05c00d32 in combo_start () at sendmsg.c:908

Combo messages can be large, and this could a) cause increased RAM usage, or b) be affected by it. Again, it's not clear if the behaviour of xymongen and xymonnet are the cause of your problems or the result of them.

It looks like the call to combo_start() in xymongen is in code that runs as a result of the "--report" switch. In xymonnet, it's in common code, but there seems to be a modified code path available if you were to add "--bfq" (backfeed queue). I know nothing about the backfeed queue feature, but there's a little about it in the README.backfeed file.

So some things to consider, mostly just work-arounds and troubleshooting:

1. Make sure you have swap enabled, and monitor swap-in/swap-out.
2. See if anything else is using excessive RAM.
3. Play with combo message sizes. Perhaps a smaller size would help. You can set MAXMSGSPERCOMBO in xymonserver.cfg.
4. Run an older version of Xymon on your new installation, perhaps the same as your current installation. Or perhaps just copy the binaries for xymonnet and/or xymongen to the new server?
5. Patch the strfunc.c file to include the malloc error checking. You'd need to get the SRPM from Terabithia and build it yourself. Only the xymonnet and xymongen binaries would need to be replaced.
6. Profile the xymond process's memory usage. I'm not sure how to do this. Perhaps you can get it to dump core, then analyse the core (perhaps just run "strings" over it) to see what's using up all the memory. Perhaps there's some gdb techniques for this.
7. Try running xymongen without "--report", and xymonnet with "--bfq" or "--no-bfq".

Hope that helps.

Cheers Jeremy