/var/lib/xymon/rrd folders growing larger than anticipated
Morning all,
We are currently running Xymon 4.3.28-1.el7.terabithia (on CentOS 7.x) and I have been struggling with this since we switched to the RPMs from an older version of xymon built from source about a year ago. We've gone from a few GB on the old server to now almost 40GB just for this directory.
I posed the question a little while back to the group and thought I had found the cause in the rrddefinitions.cfg file under the line:
RRA:HWPREDICT:8064:0.1:0.0035:2016
I tried a few different things but ended up deciding to comment out this line to see if it would slow the growth, but it does not appear to be working. Is there any reason these files would be increasing in number or size? Is there something I can do with rrdtool to clean up data? Is it normal that an rrd directory for a single host is upwards of 60M on the newer versions of Xymon?
Here is an example of the files in a folder for a host:
[root at xymon]# du -h --max-depth=1 /var/lib/xymon/rrd/server.example/* 216K /var/lib/xymon/rrd/server.example/clock.rrd 424K /var/lib/xymon/rrd/server.example/disk,boot.rrd 424K /var/lib/xymon/rrd/server.example/disk,dev,shm.rrd 424K /var/lib/xymon/rrd/server.example/disk,elasticsearch.rrd 424K /var/lib/xymon/rrd/server.example/disk,home.rrd 424K /var/lib/xymon/rrd/server.example/disk,root.rrd 424K /var/lib/xymon/rrd/server.example/disk,run.rrd 424K /var/lib/xymon/rrd/server.example/disk,run,user,0.rrd 424K /var/lib/xymon/rrd/server.example/disk,run,user,1000.rrd 424K /var/lib/xymon/rrd/server.example/disk,run,user,1001.rrd 424K /var/lib/xymon/rrd/server.example/disk,run,user,1002.rrd 424K /var/lib/xymon/rrd/server.example/disk,run,user,1003.rrd 424K /var/lib/xymon/rrd/server.example/disk,run,user,1004.rrd 424K /var/lib/xymon/rrd/server.example/disk,run,user,1014.rrd 424K /var/lib/xymon/rrd/server.example/disk,run,user,1016.rrd 424K /var/lib/xymon/rrd/server.example/disk,sys,fs,cgroup.rrd 424K /var/lib/xymon/rrd/server.example/disk,tmpfs.rrd 424K /var/lib/xymon/rrd/server.example/disk,tmp.rrd 424K /var/lib/xymon/rrd/server.example/disk,var.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.docker0.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.eno16780032.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.veth0506ddf.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.veth08022d9.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.veth081fb4b.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.veth0828eba.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.veth0ed858f.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.veth164c573.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.veth1740a33.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.veth189a45e.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.veth197189c.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.veth19d7cca.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.veth1dfb69f.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.veth2090f52.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.veth2282b2c.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.veth258f7aa.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.veth273f639.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.veth28aa403.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.veth29630d9.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.veth2c50b73.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.veth2ccd63f.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.veth38ba0c9.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.veth38c413a.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.veth3a47851.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.veth3c5e33e.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.veth3cb1195.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.veth3e20beb.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.veth42dcce0.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.veth4311e0a.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.veth461d830.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.veth4855b73.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.veth5074706.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.veth51e84f6.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.veth525c545.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.veth569cd53.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.veth59a3d88.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.veth5c4439e.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.veth5f09582.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.veth5fa1dd2.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.veth67d3cbf.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.veth67dbb70.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.veth682379a.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.veth684bcf3.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.veth69ab431.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.veth6ad9701.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.veth6c80cac.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.veth6ce4f39.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.veth6e24fbf.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.veth6e59721.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.veth6fb0766.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.veth71f97e2.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.veth741d0d5.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.veth7c752f1.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.veth7f93b1f.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.veth8386dfb.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.veth8e3b4e4.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.veth8f68e08.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.veth9063983.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.veth91bc3c6.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.veth96c1fac.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.veth9a73f58.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.veth9ed52d1.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.vetha09504c.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.vetha184ef3.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.vetha410e0c.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.vetha460481.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.vetha5a56d6.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.vetha6cec13.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.vethad8eda4.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.vethad8faa1.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.vethaffd7df.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.vethb3c04b7.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.vethb426b52.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.vethb52aae4.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.vethb6937d3.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.vethba647c0.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.vethbd846e1.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.vethc01c9dd.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.vethc81bb06.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.vethc8ea13a.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.vethcc1a6d6.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.vethcc51afb.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.vethd01eb33.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.vethd105c62.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.vethd1b1789.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.vethd676e0b.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.vethd947e9e.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.vethdcb0028.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.vethed73659.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.vethf05d463.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.vethf4659bf.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.vethf84f9fc.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.vethf8efbee.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.vethfb569a5.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.vethfb5988e.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.vethfeec0dd.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.vethffc6c75.rrd 424K /var/lib/xymon/rrd/server.example/inode,boot.rrd 424K /var/lib/xymon/rrd/server.example/inode,dev,shm.rrd 424K /var/lib/xymon/rrd/server.example/inode,elasticsearch.rrd 424K /var/lib/xymon/rrd/server.example/inode,home.rrd 424K /var/lib/xymon/rrd/server.example/inode,root.rrd 424K /var/lib/xymon/rrd/server.example/inode,run.rrd 424K /var/lib/xymon/rrd/server.example/inode,run,user,0.rrd 424K /var/lib/xymon/rrd/server.example/inode,run,user,1000.rrd 424K /var/lib/xymon/rrd/server.example/inode,run,user,1001.rrd 424K /var/lib/xymon/rrd/server.example/inode,run,user,1002.rrd 424K /var/lib/xymon/rrd/server.example/inode,run,user,1003.rrd 424K /var/lib/xymon/rrd/server.example/inode,run,user,1004.rrd 424K /var/lib/xymon/rrd/server.example/inode,run,user,1014.rrd 424K /var/lib/xymon/rrd/server.example/inode,run,user,1016.rrd 424K /var/lib/xymon/rrd/server.example/inode,sys,fs,cgroup.rrd 424K /var/lib/xymon/rrd/server.example/inode,tmpfs.rrd 424K /var/lib/xymon/rrd/server.example/inode,tmp.rrd 424K /var/lib/xymon/rrd/server.example/inode,var.rrd 216K /var/lib/xymon/rrd/server.example/la.rrd 216K /var/lib/xymon/rrd/server.example/memory.actual.rrd 216K /var/lib/xymon/rrd/server.example/memory.real.rrd 216K /var/lib/xymon/rrd/server.example/memory.swap.rrd 3.3M /var/lib/xymon/rrd/server.example/netstat.rrd 216K /var/lib/xymon/rrd/server.example/procs.rrd 216K /var/lib/xymon/rrd/server.example/tcp.conn.rrd 216K /var/lib/xymon/rrd/server.example/users.rrd 3.5M /var/lib/xymon/rrd/server.example/vmstat.rrd
Do these look like reasonable file sizes/number of files compared to everyone else? How much space should we reasonably expect to use for this given we have roughly 1k hosts being monitored (mix of Windows and Linux).
Thanks, *Eliza Danyi* Operating Systems Analyst VCU Computer Center Phone: 804-827-1132 E-mail: danyiem at vcu.edu
*Don't be a phishing victim -- VCU and other reputable organizations will never use email to request that you reply with your password, social security number or confidential personal information. For more details, visit **https://phishing.vcu.edu/* <https://phishing.vcu.edu/>
On Fri, September 20, 2019 5:42 am, Eliza Danyi wrote:
Morning all,
We are currently running Xymon 4.3.28-1.el7.terabithia (on CentOS 7.x) and I have been struggling with this since we switched to the RPMs from an older version of xymon built from source about a year ago. We've gone from a few GB on the old server to now almost 40GB just for this directory.
I posed the question a little while back to the group and thought I had found the cause in the rrddefinitions.cfg file under the line:
RRA:HWPREDICT:8064:0.1:0.0035:2016
I tried a few different things but ended up deciding to comment out this line to see if it would slow the growth, but it does not appear to be working. Is there any reason these files would be increasing in number or size? Is there something I can do with rrdtool to clean up data? Is it normal that an rrd directory for a single host is upwards of 60M on the newer versions of Xymon?
Here is an example of the files in a folder for a host:
Hi,
RRD files by themselves do not change size (unless intentionally manipulated), however new RRAs would have been added in I believe when it was first run.
The ones you're seeing below, though, are a result of two things; one easier to fix than the other. Systemd creates "/run/user/xxxx" partitions on user login, and if the client is running when a new UID is logged in, you'll get a new RRD file dropped in. This was changed in 4.3.29+ I believe, but you should be able to exclude them by adding this in to your analysis.cfg file:
CLASS=linux # Ignore some usually uninteresting tmpfs mounts. DISK /dev IGNORE DISK /lib/init/rw IGNORE DISK /sys/fs/cgroup IGNORE DISK %^/run/user/\d+$ IGNORE INODE /dev IGNORE INODE /lib/init/rw IGNORE INODE /sys/fs/cgroup IGNORE INODE %^/run/user/\d+$ IGNORE
After that's in there, you should be able to confirm that the /run/user/* ones aren't being updated any more and you can purge them.
The second issue, however, is with your network interfaces: ifstat.veth08022d9.rrd, etc... It looks like something is giving you highly variable transient interface names?
Xymon does not currently have a mechanism to exclude interfaces by name like this, since they just look like new things to track. Are these changing during known-events that might be able to trigger a cleanup?
As a workaround, you should be able to purge ones that have not been updated after a certain length of time.
HTH, -jc
[root at xymon]# du -h --max-depth=1 /var/lib/xymon/rrd/server.example/* 216K /var/lib/xymon/rrd/server.example/clock.rrd 424K /var/lib/xymon/rrd/server.example/disk,boot.rrd 424K /var/lib/xymon/rrd/server.example/disk,dev,shm.rrd 424K /var/lib/xymon/rrd/server.example/disk,elasticsearch.rrd 424K /var/lib/xymon/rrd/server.example/disk,home.rrd 424K /var/lib/xymon/rrd/server.example/disk,root.rrd 424K /var/lib/xymon/rrd/server.example/disk,run.rrd 424K /var/lib/xymon/rrd/server.example/disk,run,user,0.rrd 424K /var/lib/xymon/rrd/server.example/disk,run,user,1000.rrd 424K /var/lib/xymon/rrd/server.example/disk,run,user,1001.rrd 424K /var/lib/xymon/rrd/server.example/disk,run,user,1002.rrd 424K /var/lib/xymon/rrd/server.example/disk,run,user,1003.rrd 424K /var/lib/xymon/rrd/server.example/disk,run,user,1004.rrd 424K /var/lib/xymon/rrd/server.example/disk,run,user,1014.rrd 424K /var/lib/xymon/rrd/server.example/disk,run,user,1016.rrd 424K /var/lib/xymon/rrd/server.example/disk,sys,fs,cgroup.rrd 424K /var/lib/xymon/rrd/server.example/disk,tmpfs.rrd 424K /var/lib/xymon/rrd/server.example/disk,tmp.rrd 424K /var/lib/xymon/rrd/server.example/disk,var.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.docker0.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.eno16780032.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.veth0506ddf.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.veth08022d9.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.veth081fb4b.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.veth0828eba.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.veth0ed858f.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.veth164c573.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.veth1740a33.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.veth189a45e.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.veth197189c.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.veth19d7cca.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.veth1dfb69f.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.veth2090f52.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.veth2282b2c.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.veth258f7aa.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.veth273f639.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.veth28aa403.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.veth29630d9.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.veth2c50b73.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.veth2ccd63f.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.veth38ba0c9.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.veth38c413a.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.veth3a47851.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.veth3c5e33e.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.veth3cb1195.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.veth3e20beb.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.veth42dcce0.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.veth4311e0a.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.veth461d830.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.veth4855b73.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.veth5074706.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.veth51e84f6.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.veth525c545.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.veth569cd53.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.veth59a3d88.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.veth5c4439e.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.veth5f09582.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.veth5fa1dd2.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.veth67d3cbf.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.veth67dbb70.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.veth682379a.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.veth684bcf3.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.veth69ab431.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.veth6ad9701.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.veth6c80cac.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.veth6ce4f39.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.veth6e24fbf.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.veth6e59721.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.veth6fb0766.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.veth71f97e2.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.veth741d0d5.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.veth7c752f1.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.veth7f93b1f.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.veth8386dfb.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.veth8e3b4e4.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.veth8f68e08.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.veth9063983.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.veth91bc3c6.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.veth96c1fac.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.veth9a73f58.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.veth9ed52d1.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.vetha09504c.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.vetha184ef3.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.vetha410e0c.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.vetha460481.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.vetha5a56d6.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.vetha6cec13.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.vethad8eda4.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.vethad8faa1.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.vethaffd7df.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.vethb3c04b7.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.vethb426b52.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.vethb52aae4.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.vethb6937d3.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.vethba647c0.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.vethbd846e1.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.vethc01c9dd.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.vethc81bb06.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.vethc8ea13a.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.vethcc1a6d6.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.vethcc51afb.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.vethd01eb33.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.vethd105c62.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.vethd1b1789.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.vethd676e0b.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.vethd947e9e.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.vethdcb0028.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.vethed73659.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.vethf05d463.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.vethf4659bf.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.vethf84f9fc.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.vethf8efbee.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.vethfb569a5.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.vethfb5988e.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.vethfeec0dd.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.vethffc6c75.rrd 424K /var/lib/xymon/rrd/server.example/inode,boot.rrd 424K /var/lib/xymon/rrd/server.example/inode,dev,shm.rrd 424K /var/lib/xymon/rrd/server.example/inode,elasticsearch.rrd 424K /var/lib/xymon/rrd/server.example/inode,home.rrd 424K /var/lib/xymon/rrd/server.example/inode,root.rrd 424K /var/lib/xymon/rrd/server.example/inode,run.rrd 424K /var/lib/xymon/rrd/server.example/inode,run,user,0.rrd 424K /var/lib/xymon/rrd/server.example/inode,run,user,1000.rrd 424K /var/lib/xymon/rrd/server.example/inode,run,user,1001.rrd 424K /var/lib/xymon/rrd/server.example/inode,run,user,1002.rrd 424K /var/lib/xymon/rrd/server.example/inode,run,user,1003.rrd 424K /var/lib/xymon/rrd/server.example/inode,run,user,1004.rrd 424K /var/lib/xymon/rrd/server.example/inode,run,user,1014.rrd 424K /var/lib/xymon/rrd/server.example/inode,run,user,1016.rrd 424K /var/lib/xymon/rrd/server.example/inode,sys,fs,cgroup.rrd 424K /var/lib/xymon/rrd/server.example/inode,tmpfs.rrd 424K /var/lib/xymon/rrd/server.example/inode,tmp.rrd 424K /var/lib/xymon/rrd/server.example/inode,var.rrd 216K /var/lib/xymon/rrd/server.example/la.rrd 216K /var/lib/xymon/rrd/server.example/memory.actual.rrd 216K /var/lib/xymon/rrd/server.example/memory.real.rrd 216K /var/lib/xymon/rrd/server.example/memory.swap.rrd 3.3M /var/lib/xymon/rrd/server.example/netstat.rrd 216K /var/lib/xymon/rrd/server.example/procs.rrd 216K /var/lib/xymon/rrd/server.example/tcp.conn.rrd 216K /var/lib/xymon/rrd/server.example/users.rrd 3.5M /var/lib/xymon/rrd/server.example/vmstat.rrd
Do these look like reasonable file sizes/number of files compared to everyone else? How much space should we reasonably expect to use for this given we have roughly 1k hosts being monitored (mix of Windows and Linux).
Thanks, *Eliza Danyi* Operating Systems Analyst VCU Computer Center Phone: 804-827-1132 E-mail: danyiem at vcu.edu
*Don't be a phishing victim -- VCU and other reputable organizations will never use email to request that you reply with your password, social security number or confidential personal information. For more details, visit **https://phishing.vcu.edu/* <https://phishing.vcu.edu/>
Xymon mailing list Xymon at xymon.com http://lists.xymon.com/mailman/listinfo/xymon
I often see those kind of interface RRD files on systems that have Docker containers running on them. Every time a new container is started, a random interface like that is created on the host for the container network (and thus an RRD file is created). Every time a container is restarted, a new random interface is created for it and the old interface isn't used anymore, but the RRD file remains for it. For me at least, these are on systems that we don't admin but still monitor with Xymon, so I never know when the containers will be created/restarted/etc.
Depending on how often the containers are restarted/created/etc., the number of random veth* interface RRD files per host can be A LOT. Normally, they aren't very large by themselves, but I can see how combined it might take up a considerable amount of space.
I run https://wiki.xymonton.org/doku.php/addons:finit periodically to find inactive RRD files and remove the ones that I know won't be updated again. The script will output rm commands you can use to remove the inactive RRD files. You just need to change the HomeDir and Threshold (in seconds) variables to what you want to use. This script works really well to find and clean up the inactive random veth* interfaces (and other inactive RRD files). It can also help to find hosts that can just be removed (dropped) completely from Xymon instead ('xymon drop X').
Alternatively, if you don't care about these network interfaces, and know the specific interfaces you do care about, you could use the INTERFACES option on the host entries in hosts.cfg:
INTERFACES:REGEXP
On systems with multiple network interfaces, the operatingsystem may report a number of net? work interface where the statistics are of no interest. By default Xymon tracks and graphs the traffic on all network interfaces. This option defines a regular expression, and only those interfaces whose name matches the expression are tracked.
Hope this helps!
-- Matt Vander Werf HPC System Administrator University of Notre Dame Center for Research Computing - Union Station 506 W. South Street South Bend, IN 46601 Phone: (574) 631-0692
On Fri, Sep 20, 2019 at 8:22 PM J.C. Cleaver <cleaver at terabithia.org> wrote:
On Fri, September 20, 2019 5:42 am, Eliza Danyi wrote:
Morning all,
We are currently running Xymon 4.3.28-1.el7.terabithia (on CentOS 7.x) and I have been struggling with this since we switched to the RPMs from an older version of xymon built from source about a year ago. We've gone from a few GB on the old server to now almost 40GB just for this directory.
I posed the question a little while back to the group and thought I had found the cause in the rrddefinitions.cfg file under the line:
RRA:HWPREDICT:8064:0.1:0.0035:2016
I tried a few different things but ended up deciding to comment out this line to see if it would slow the growth, but it does not appear to be working. Is there any reason these files would be increasing in number or size? Is there something I can do with rrdtool to clean up data? Is it normal that an rrd directory for a single host is upwards of 60M on the newer versions of Xymon?
Here is an example of the files in a folder for a host:
Hi,
RRD files by themselves do not change size (unless intentionally manipulated), however new RRAs would have been added in I believe when it was first run.
The ones you're seeing below, though, are a result of two things; one easier to fix than the other. Systemd creates "/run/user/xxxx" partitions on user login, and if the client is running when a new UID is logged in, you'll get a new RRD file dropped in. This was changed in 4.3.29+ I believe, but you should be able to exclude them by adding this in to your analysis.cfg file:
CLASS=linux # Ignore some usually uninteresting tmpfs mounts. DISK /dev IGNORE DISK /lib/init/rw IGNORE DISK /sys/fs/cgroup IGNORE DISK %^/run/user/\d+$ IGNORE INODE /dev IGNORE INODE /lib/init/rw IGNORE INODE /sys/fs/cgroup IGNORE INODE %^/run/user/\d+$ IGNORE
After that's in there, you should be able to confirm that the /run/user/* ones aren't being updated any more and you can purge them.
The second issue, however, is with your network interfaces: ifstat.veth08022d9.rrd, etc... It looks like something is giving you highly variable transient interface names?
Xymon does not currently have a mechanism to exclude interfaces by name like this, since they just look like new things to track. Are these changing during known-events that might be able to trigger a cleanup?
As a workaround, you should be able to purge ones that have not been updated after a certain length of time.
HTH, -jc
[root at xymon]# du -h --max-depth=1 /var/lib/xymon/rrd/server.example/* 216K /var/lib/xymon/rrd/server.example/clock.rrd 424K /var/lib/xymon/rrd/server.example/disk,boot.rrd 424K /var/lib/xymon/rrd/server.example/disk,dev,shm.rrd 424K /var/lib/xymon/rrd/server.example/disk,elasticsearch.rrd 424K /var/lib/xymon/rrd/server.example/disk,home.rrd 424K /var/lib/xymon/rrd/server.example/disk,root.rrd 424K /var/lib/xymon/rrd/server.example/disk,run.rrd 424K /var/lib/xymon/rrd/server.example/disk,run,user,0.rrd 424K /var/lib/xymon/rrd/server.example/disk,run,user,1000.rrd 424K /var/lib/xymon/rrd/server.example/disk,run,user,1001.rrd 424K /var/lib/xymon/rrd/server.example/disk,run,user,1002.rrd 424K /var/lib/xymon/rrd/server.example/disk,run,user,1003.rrd 424K /var/lib/xymon/rrd/server.example/disk,run,user,1004.rrd 424K /var/lib/xymon/rrd/server.example/disk,run,user,1014.rrd 424K /var/lib/xymon/rrd/server.example/disk,run,user,1016.rrd 424K /var/lib/xymon/rrd/server.example/disk,sys,fs,cgroup.rrd 424K /var/lib/xymon/rrd/server.example/disk,tmpfs.rrd 424K /var/lib/xymon/rrd/server.example/disk,tmp.rrd 424K /var/lib/xymon/rrd/server.example/disk,var.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.docker0.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.eno16780032.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.veth0506ddf.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.veth08022d9.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.veth081fb4b.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.veth0828eba.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.veth0ed858f.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.veth164c573.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.veth1740a33.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.veth189a45e.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.veth197189c.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.veth19d7cca.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.veth1dfb69f.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.veth2090f52.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.veth2282b2c.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.veth258f7aa.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.veth273f639.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.veth28aa403.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.veth29630d9.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.veth2c50b73.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.veth2ccd63f.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.veth38ba0c9.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.veth38c413a.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.veth3a47851.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.veth3c5e33e.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.veth3cb1195.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.veth3e20beb.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.veth42dcce0.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.veth4311e0a.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.veth461d830.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.veth4855b73.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.veth5074706.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.veth51e84f6.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.veth525c545.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.veth569cd53.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.veth59a3d88.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.veth5c4439e.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.veth5f09582.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.veth5fa1dd2.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.veth67d3cbf.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.veth67dbb70.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.veth682379a.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.veth684bcf3.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.veth69ab431.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.veth6ad9701.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.veth6c80cac.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.veth6ce4f39.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.veth6e24fbf.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.veth6e59721.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.veth6fb0766.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.veth71f97e2.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.veth741d0d5.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.veth7c752f1.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.veth7f93b1f.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.veth8386dfb.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.veth8e3b4e4.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.veth8f68e08.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.veth9063983.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.veth91bc3c6.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.veth96c1fac.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.veth9a73f58.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.veth9ed52d1.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.vetha09504c.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.vetha184ef3.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.vetha410e0c.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.vetha460481.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.vetha5a56d6.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.vetha6cec13.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.vethad8eda4.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.vethad8faa1.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.vethaffd7df.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.vethb3c04b7.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.vethb426b52.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.vethb52aae4.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.vethb6937d3.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.vethba647c0.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.vethbd846e1.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.vethc01c9dd.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.vethc81bb06.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.vethc8ea13a.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.vethcc1a6d6.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.vethcc51afb.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.vethd01eb33.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.vethd105c62.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.vethd1b1789.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.vethd676e0b.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.vethd947e9e.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.vethdcb0028.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.vethed73659.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.vethf05d463.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.vethf4659bf.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.vethf84f9fc.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.vethf8efbee.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.vethfb569a5.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.vethfb5988e.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.vethfeec0dd.rrd 424K /var/lib/xymon/rrd/server.example/ifstat.vethffc6c75.rrd 424K /var/lib/xymon/rrd/server.example/inode,boot.rrd 424K /var/lib/xymon/rrd/server.example/inode,dev,shm.rrd 424K /var/lib/xymon/rrd/server.example/inode,elasticsearch.rrd 424K /var/lib/xymon/rrd/server.example/inode,home.rrd 424K /var/lib/xymon/rrd/server.example/inode,root.rrd 424K /var/lib/xymon/rrd/server.example/inode,run.rrd 424K /var/lib/xymon/rrd/server.example/inode,run,user,0.rrd 424K /var/lib/xymon/rrd/server.example/inode,run,user,1000.rrd 424K /var/lib/xymon/rrd/server.example/inode,run,user,1001.rrd 424K /var/lib/xymon/rrd/server.example/inode,run,user,1002.rrd 424K /var/lib/xymon/rrd/server.example/inode,run,user,1003.rrd 424K /var/lib/xymon/rrd/server.example/inode,run,user,1004.rrd 424K /var/lib/xymon/rrd/server.example/inode,run,user,1014.rrd 424K /var/lib/xymon/rrd/server.example/inode,run,user,1016.rrd 424K /var/lib/xymon/rrd/server.example/inode,sys,fs,cgroup.rrd 424K /var/lib/xymon/rrd/server.example/inode,tmpfs.rrd 424K /var/lib/xymon/rrd/server.example/inode,tmp.rrd 424K /var/lib/xymon/rrd/server.example/inode,var.rrd 216K /var/lib/xymon/rrd/server.example/la.rrd 216K /var/lib/xymon/rrd/server.example/memory.actual.rrd 216K /var/lib/xymon/rrd/server.example/memory.real.rrd 216K /var/lib/xymon/rrd/server.example/memory.swap.rrd 3.3M /var/lib/xymon/rrd/server.example/netstat.rrd 216K /var/lib/xymon/rrd/server.example/procs.rrd 216K /var/lib/xymon/rrd/server.example/tcp.conn.rrd 216K /var/lib/xymon/rrd/server.example/users.rrd 3.5M /var/lib/xymon/rrd/server.example/vmstat.rrd
Do these look like reasonable file sizes/number of files compared to everyone else? How much space should we reasonably expect to use for this given we have roughly 1k hosts being monitored (mix of Windows and
Linux).
Thanks, *Eliza Danyi* Operating Systems Analyst VCU Computer Center Phone: 804-827-1132 E-mail: danyiem at vcu.edu
*Don't be a phishing victim -- VCU and other reputable organizations will never use email to request that you reply with your password, social security number or confidential personal information. For more details, visit **https://phishing.vcu.edu/* <https://phishing.vcu.edu/>
Xymon mailing list Xymon at xymon.com http://lists.xymon.com/mailman/listinfo/xymon
Xymon mailing list Xymon at xymon.com http://lists.xymon.com/mailman/listinfo/xymon
participants (3)
-
cleaver＠terabithia.org
-
danyiem＠vcu.edu
-
mvanderw＠nd.edu