It appears I have answered my own question
In the /var/lib/Xymon/rrd directory (I’m using Terabithia RPM version of Xymon) there were a large number of files with names similar to
disk,tmp,veeam,{f8dc516e-d17d-4a7e-b473-fd40e2d1a531}.rrd
these, as far as I can tell are veeam snapshots that are obviously being seen by the disk check and rrd file created but when there’s too many it crashed the trends column.
There were only two servers that had these excess files. I have now deleted them and all is working again.
Neil Simmonds Senior Platform & Middleware Engineer (Unix) |
 |
PHONE | 0344 245 9200 | ADDRESS | Unit A Brook Park East, NG20 8RY |
Credit is provided by Frasers Group Financial Services Limited (Registered Company no. 00718151), which is authorised and regulated by the Financial Conduct Authority (FRN 311908) for consumer credit and general insurance and a member of the Finance and Leasing Association. Frasers Group Financial Services Limited is registered in England and its registered office is: Express House, Petre Road, Clayton Business Park, Accrington, Lancashire, BB5 5JB. Studio is a trading name used by both Frasers Group Financial Services Limited and Studio Retail Trading Limited. For regulated payment services, Frasers Group Financial Services Limited is a payment agent of Transact Payments Limited, a company authorised and regulated by the Gibraltar Financial Services Commission as an electronic money institution. Transact Payments Limited, a company incorporated in Gibraltar (No 108217). Registered address 6.20 World Trade Center, 6 Bayside Road, Gibraltar, GX11 1AA. |
From: Neil
Simmonds via Xymon <xymon@xymon.com>
Sent: 18 February 2025 14:05
To: xymon@xymon.com
Cc: Neil Simmonds <Neil.Simmonds@frasers.group>
Subject: [Xymon] Weird issue with Trands on a small amount of servers
Hi all,
On a couple of servers (maybe more) I get the following when trying to access the “trends” column
Internal Server Error
The server encountered an internal error or misconfiguration and was unable to complete your request.
Please contact the server administrator at root@localhost to inform them of the time this error occurred, and the actions you performed just before this error.
More information about this error may be available in the server error log.
When I look in messages.log on the server I can see this
Feb 18 13:04:25 XymonServer kernel: svcstatus.cgi[3993679]: segfault at 1fe5965 ip 00007faddb3f79ad sp 00007ffd48ba9c60 error 6 in libc-2.28.so[7faddb369000+1cd000]
Feb 18 13:04:25 XymonServer kernel: Code: ba ff ff ff ff be 00 80 00 00 e8 8e 53 00 00 48 89 df 48 89 e9 4c 89 e2 48 8d 05 fe e6 33 00 48 89 ee 48 89 84 24 d8 00 00 00 <c6> 45 00 00 e8 3a 68 00 00 48 89 df 4c
89 f2 4c 89 ee e8 6c 7a fd
Feb 18 13:04:25 XymonServer systemd[1]: Started Process Core Dump (PID 3993684/UID 0).
Feb 18 13:04:25 XymonServer systemd-coredump[3993685]: Resource limits disable core dumping for process 3993679 (svcstatus.cgi).
Feb 18 13:04:25 XymonServer systemd-coredump[3993685]: Process 3993679 (svcstatus.cgi) of user 48 dumped core.
Feb 18 13:04:25 XymonServer systemd[1]:
systemd-coredump@109-3993684-0.service: Succeeded.
I’m unable to find an actual core dump file to help any further so am struggling to diagnose this one. Can anyone help?
Kind regards,
Neil.
|
Senior Platform & Middleware Engineer (Unix) |
|
|
|
PHONE |
0344 245 9200 |
ADDRESS |
Unit A Brook Park East, NG20 8RY |
|
Credit is provided by Frasers Group Financial Services Limited (Registered Company no. 00718151), which is authorised and regulated by the Financial Conduct Authority (FRN 311908) for
consumer credit and general insurance and a member of the Finance and Leasing Association. Frasers Group Financial Services Limited is registered in England and its registered office is: Express House, Petre Road, Clayton Business Park, Accrington, Lancashire,
BB5 5JB. Studio is a trading name used by both Frasers Group Financial Services Limited and Studio Retail Trading Limited. For regulated payment services, Frasers Group Financial Services Limited is a payment agent of Transact Payments Limited, a company authorised
and regulated by the Gibraltar Financial Services Commission as an electronic money institution. Transact Payments Limited, a company incorporated in Gibraltar (No 108217). Registered address 6.20 World Trade Center, 6 Bayside Road, Gibraltar, GX11 1AA. |