Hi,

On 12/10/2026 15:38, Tom Schmidt via Xymon wrote:

Christoph, You can easily make each GPU report to its own RRD file. Then a single xymon command can upload the rrd data to the server.  Here is some pseudo code using bash.

Tom

RRDDATA="" NL=" "  # newline for GPU in $GPUS; do value1=$(command to get GPU value for $GPU) RRDDATA="$RRDDATA$NL[nvidia.gpu$GPU.rrd]${NL} DS:gpuval1:GAUGE:600:0:U $value1" # repeat for each value you want to record done $XYMON $XYMSRV "data $HOSTNAME.trends$RRDDATA"

okay, this looks promising, I have to try that, thanks!

Christoph

On Mon, Oct 12, 2026 at 7:07 AM Bruno Manzoni via Xymon <xymon@xymon.com <mailto:xymon@xymon.com>> wrote:

Hi Christophe,

I would recommend having one test per graph (so 5 different metrics = 5
tests).

If I remember correctly, an RRD file can contain multiple instances of
the same measurement. For example, if you have 6 GPUs and want to
monitor the usage of each one, you can store all 6 measurements in a
single RRD file.

Then, I think you can use the NCV or Devmon-style approach (also built
into Xymon) to pass the parameters. (devmon way seems less documented
but more powerfull)

Also just that you know: We've fixed some issues in recent Xymon
versions related to "grouping" graph data (multiple graphs in the same
image). So it's important to know whether you're running the latest
version.

-> The tests you are doing is also on my todo list. (as well as 2
features: 1. restructuring the way to pass data to xymon and to 2.
stack
multiple test into one)

Let me know if this help and do not hesitate to share some part of the
script.

Bruno


Le 12.10.2026 à 14:23, Christoph Zechner via Xymon a écrit :
 > Hi,
 >
 > I've stumbled upon a problem I cannot seem to solve myself and hope
 > someone here can help me out...
 >
 > A custom check I wrote is collecting data from Nvidia GPUs via the
 > nvidia-smi command, the check works great and displays all the
 > relevant data in its message, but I want to create graphs from
certain
 > data points too.
 >
 > Here's my problem: in a server with 8 GPUs, I am collecting 5
 > different things from each GPU (like Power Draw, Utilisation and so
 > on), these values get saved in a RRD file as well, but only the
values
 > from the first GPU.
 >
 > I want to save a RRD file for each GPU and its values, so multiple
 > data points in a RRD, but 8 different files (one per GPU) and I
do not
 > get how I can do this in a xymon check (bash script).
 >
 > Can someone point me in the right direction here? What am I missing?
 >
 > Thanks in advance!
 >
 > Best regards
 > Christoph
 > _______________________________________________
 > Xymon mailing list -- xymon@xymon.com <mailto:xymon@xymon.com>
 > To unsubscribe send an email to xymon-leave@xymon.com
<mailto:xymon-leave@xymon.com>
_______________________________________________
Xymon mailing list -- xymon@xymon.com <mailto:xymon@xymon.com>
To unsubscribe send an email to xymon-leave@xymon.com <mailto:xymon-
leave@xymon.com>

---

Xymon mailing list -- xymon@xymon.com To unsubscribe send an email to xymon-leave@xymon.com

Hi Christoph,

Thanks for the details, 4.3.30 is fine for this.

Yes, I'd make one test per metric. Putting several metrics in one test is possible (the RRD side works either way, and GRAPHS_<test> can show several graphs on one test), but everything Xymon does with a status works per test, not per metric:

• One color. A test has one color. If power is red, you can't see from the overview that memory is also yellow, and when power goes back to green the column stays yellow.
• One alert. alerts.cfg selects alerts by test (SERVICE=). You can't send power alerts to one team and memory alerts to another, or use a different delay or repeat for each.
• One ack and one disable. Acking or disabling the test silences every metric in it. If you ack a known power issue, a new memory problem on the same test goes unnoticed until the ack expires.
• One history. The history and availability reports record one color change per test. Afterwards you can't tell which metric caused an outage.

Inode is likewise its own column, separate from disk. Several instances of the same metric in one test is fine, which is what disk does with all its filesystems, and what the script below does with your GPUs.

Several metrics in one test are also fine when they always go together, for example traffic in/out on an interface, or memory used/total below. A good test is whether you would ever want a different alert or ack for one of them. If you would, split them.

The reason only GPU 0 was stored is SPLITNCV. Every GPU's "Power Draw" line goes to the same file, nvidia-gpu,PowerDraw.rrd, and xymond_rrd keeps only the first update to a file from each message.

The devmon format, which xymond_rrd also handles, fixes that. Each test adds a block inside HTML comments, so it doesn't show on the status page:

<!-- linecount=0 --> <!--DEVMON RRD: nvidia-mem 0 0 DS:memory_used:GAUGE:600:0:U DS:memory_total:GAUGE:600:0:U 0 4096:81920 1 2048:81920 -->

• The DS: line defines the data sources.
• Each line after it is one GPU: an id with no spaces, then the values separated by ":", in DS order.
• xymond_rrd writes one RRD file per line: nvidia-mem.0.rrd, nvidia-mem.1.rrd, and so on.
• The <!-- linecount=... --> line is needed for the status page to show the graph. Without it, 4.3.30 shows no graph for a devmon test whose name isn't in its built-in list (disk, inode, if_load, ...). Its value decides how the status page groups the GPUs, see the two options below.

This also corrects what I wrote before: it is not one RRD file holding all GPUs, but one file per GPU, the same way "disk" keeps one file per filesystem. The graph then puts them together.

Two options for the status page

The trends page shows each graph with at most 5 GPUs per image in both cases, on 4.3.30 and on the development version. The difference is the status page of each test:

┌──────────────────────────────┬─────────────────────────┬──────────────────────────┐ │  SPLITGRAPHS in the script   │         4.3.30          │  development version    │ ├──────────────────────────────┼─────────────────────────┼──────────────────────────┤ │ 0 (linecount=0)              │ one image with all GPUs │ one image with all GPUs  │ ├──────────────────────────────┼─────────────────────────┼──────────────────────────┤ │ 1 (linecount=number of GPUs) │ one image per GPU       │ at most 5 GPUs per image │ └──────────────────────────────┴─────────────────────────┴──────────────────────────┘

Option 0 looks the same on both versions. Option 1 gives the nicer layout once you upgrade, but one image per GPU on 4.3.30. Both keep the shipped "devmon::1" setting, so Devmon's own graphs are not affected.

1. Client script (one ext script, it sends one status per test):

#!/bin/sh

One Xymon test per metric (like disk/inode), one RRD per GPU per test

(<test>.<index>.rrd), stored by xymond_rrd's devmon handler.

Server: TEST2RRD "<test>=devmon" for each test below.

<test>=<nvidia-smi fields>: the fields become the DS names ("." -> "_").

TESTS="nvidia-power=power.draw nvidia-util=utilization.gpu nvidia-memutil=utilization.memory nvidia-mem=memory.used,memory.total"

Status page: 0 = one graph with every GPU; 1 = split the GPUs into graphs

(one per GPU on 4.3.30, at most "<test>::N" per graph on newer versions).

SPLITGRAPHS=0

for t in $TESTS; do TEST=${t%%=*} FIELDS=${t#*=} COLOR=green RAW=$(nvidia-smi --format=csv,noheader,nounits --query-gpu=index,name,$FIELDS 2>&1) [ $? -eq 0 ] && [ -n "$RAW" ] || COLOR=red

        # Set COLOR here from $RAW to apply per-test thresholds.

        BODY=$(printf '%s\n' "$RAW" | awk -F', *' -v ds="$FIELDS" -v test="$TEST" -v splitgraphs="$SPLITGRAPHS" ' BEGIN { n = split(ds, d, ","); for (i = 1; i <= n; i++) gsub(/\./, "_", d[i]) } NF >= n + 2 { txt = txt sprintf("GPU %s (%s):", $1, $2) v = "" for (i = 1; i <= n; i++) { x = $(i + 2) if (x !~ /^-?[0-9]+(\.[0-9]+)?$/) x = "U"     # "[N/A]" -> unknown txt = txt " " d[i] "=" x v = v (i > 1 ? ":" : "") x } txt = txt "\n" rrd = rrd $1 " " v "\n" gpus++ } END { # The status page needs a linecount to show the graph: 0 draws every # GPU in one graph, the GPU count lets the status page split them. printf "%s\n<!-- linecount=%d -->\n", txt, (splitgraphs ? gpus : 0) printf "<!--DEVMON RRD: %s 0 0\n", test for (i = 1; i <= n; i++) printf "DS:%s:GAUGE:600:0:U ", d[i] printf "\n%s-->\n", rrd }') [ "$COLOR" = red ] && BODY="nvidia-smi failed: $RAW"

        $XYMON $XYMSRV "status $MACHINE.$TEST $COLOR $(date) $BODY" done

To add a metric, add one line to TESTS: the test name, then the nvidia-smi field(s) to store. DS names are limited to 19 characters. If you add a field to an existing test, delete its RRD files so they are recreated; otherwise the updates are rejected.

If you prefer to keep your own script: your current MSG loop can stay as the human-readable part. Send one status per test, and add to each the linecount line (0, or the number of GPUs) and the <!--DEVMON RRD: ... block with one line per GPU. In each block, use the index as the id and only that test's values.

2. xymonserver.cfg on the server:

TEST2RRD="...existing entries...,nvidia-power=devmon,nvidia-util=devmon,nvidia-memutil=devmon,nvidia-mem=devmon" GRAPHS_nvidia-power="nvidia-power" GRAPHS_nvidia-util="nvidia-util" GRAPHS_nvidia-memutil="nvidia-memutil" GRAPHS_nvidia-mem="nvidia-mem"

and add the four graphs at the end of the existing GRAPHS line, leaving "devmon::1" as it is:

GRAPHS="...existing entries...,nvidia-power::5,nvidia-util::5,nvidia-memutil::5,nvidia-mem::5"

• <test>=devmon routes each column to the devmon handler.
• GRAPHS_<test> tells the status page which graph section to use.
• <test>::5 is the maximum number of GPUs per image. It puts the graphs on the trends page, 5 per image. With option 1, the development version also uses it on the status pages; 4.3.30 does not, so there you get one image per GPU.
• Nothing needs to change when you upgrade.
• Remove your NCV_/SPLITNCV_ settings for the old test. You can delete the old nvidia-gpu,*.rrd files.

3. graphs.cfg: one section per test, each image showing its GPUs as lines:

[nvidia-power] FNPATTERN ^nvidia-power\.(.+)\.rrd TITLE GPU Power Draw YAXIS Watt DEF:p@RRDIDX@=@RRDFN@:power_draw:AVERAGE LINE2:p@RRDIDX@#@COLOR@:GPU @RRDPARAM@ GPRINT:p@RRDIDX@:LAST: \: %6.1lf (cur) GPRINT:p@RRDIDX@:MAX: \: %6.1lf (max) GPRINT:p@RRDIDX@:AVERAGE: \: %6.1lf (avg)\n

[nvidia-util] FNPATTERN ^nvidia-util\.(.+)\.rrd TITLE GPU Utilization YAXIS % -l 0 -u 100 DEF:p@RRDIDX@=@RRDFN@:utilization_gpu:AVERAGE LINE2:p@RRDIDX@#@COLOR@:GPU @RRDPARAM@ GPRINT:p@RRDIDX@:LAST: \: %5.1lf (cur) GPRINT:p@RRDIDX@:MAX: \: %5.1lf (max) GPRINT:p@RRDIDX@:AVERAGE: \: %5.1lf (avg)\n

[nvidia-memutil] FNPATTERN ^nvidia-memutil\.(.+)\.rrd TITLE GPU Memory Bus Utilization YAXIS % -l 0 -u 100 DEF:p@RRDIDX@=@RRDFN@:utilization_memory:AVERAGE LINE2:p@RRDIDX@#@COLOR@:GPU @RRDPARAM@ GPRINT:p@RRDIDX@:LAST: \: %5.1lf (cur) GPRINT:p@RRDIDX@:MAX: \: %5.1lf (max) GPRINT:p@RRDIDX@:AVERAGE: \: %5.1lf (avg)\n

[nvidia-mem] FNPATTERN ^nvidia-mem\.(.+)\.rrd TITLE GPU Memory Used YAXIS % -l 0 -u 100 DEF:u@RRDIDX@=@RRDFN@:memory_used:AVERAGE DEF:t@RRDIDX@=@RRDFN@:memory_total:AVERAGE CDEF:p@RRDIDX@=u@RRDIDX@,t@RRDIDX@,/,100,* LINE2:p@RRDIDX@#@COLOR@:GPU @RRDPARAM@ GPRINT:p@RRDIDX@:LAST: \: %5.1lf (cur) GPRINT:p@RRDIDX@:MAX: \: %5.1lf (max) GPRINT:p@RRDIDX@:AVERAGE: \: %5.1lf (avg)\n

The nvidia-mem test stores both used and total, so its graph can show memory used as a percentage.

Restart Xymon after editing xymonserver.cfg (graphs.cfg is read on each request).

Note: I tested both options on a Xymon 4.3.30 server built from the release and on the development version, with a simulated nvidia-smi for 8 GPUs (including "[N/A]" values, which become "U", i.e. unknown, and show as a gap in the graph). Both produced one RRD file per GPU per test and the layouts in the table above. On 4.3.30 the plotted values matched what was sent. I haven't run it on a real GPU server, so please let me know how it goes.

Bruno

Le 12.10.2026 à 15:53, Christoph Zechner via Xymon a écrit :

Hi Bruno,

On 12/10/2026 15:06, Bruno Manzoni via Xymon wrote:

...

Hi Christophe,

I would recommend having one test per graph (so 5 different metrics = 5 tests).

You mean instead of cramming five checks into one, I should separate those into individual tests?

...

If I remember correctly, an RRD file can contain multiple instances of the same measurement. For example, if you have 6 GPUs and want to monitor the usage of each one, you can store all 6 measurements in a single RRD file.

But how do I tell xymon I want six values per GPU per server? I wanted a behaviour similar to the "disk" check for example, where multiple disks show up in one graph per host.

...

Then, I think you can use the NCV or Devmon-style approach (also built into Xymon) to pass the parameters. (devmon way seems less documented but more powerfull)

I tried NCV and also SPLITNCV (which gave me different files for each value, but also only for the first GPU).

...

Also just that you know: We've fixed some issues in recent Xymon versions related to "grouping" graph data (multiple graphs in the same image). So it's important to know whether you're running the latest version.

oh, sorry, I forgot to add that information: Xymon 4.3.30 (running on Debian trixie).

...

-> The tests you are doing is also on my todo list. (as well as 2 features: 1. restructuring the way to pass data to xymon and to 2. stack multiple test into one)

Good to know, looking forward to the next version!

...

Let me know if this help and do not hesitate to share some part of the script.

This is the part in the for loop which "builds" the message for the xymon server:

MSG="${MSG} &${COLOR} Index: ${GPUINDEX} Name: ${GPUNAME} Power Draw: ${GPUPOWERDRAW} Watt Utilization (GPU): ${GPUUTILGPU} % Utilization (Memory bus): ${GPUUTILMEM} % GPU Memory Total: ${GPUMEMTOTAL} MB GPU Memory Used: ${GPUMEMUSED} MB " done

This part is working great, but the RRD files (named after the test [nvidia-gpu] and its value, e.g. "nvidia-gpu,PowerDraw.rrd") only contain the value for the first GPU in the check.

Thanks again!

Christoph

...

Bruno

Le 12.10.2026 à 14:23, Christoph Zechner via Xymon a écrit :

...

Hi,

I've stumbled upon a problem I cannot seem to solve myself and hope someone here can help me out...

A custom check I wrote is collecting data from Nvidia GPUs via the nvidia-smi command, the check works great and displays all the relevant data in its message, but I want to create graphs from certain data points too.

Here's my problem: in a server with 8 GPUs, I am collecting 5 different things from each GPU (like Power Draw, Utilisation and so on), these values get saved in a RRD file as well, but only the values from the first GPU.

I want to save a RRD file for each GPU and its values, so multiple data points in a RRD, but 8 different files (one per GPU) and I do not get how I can do this in a xymon check (bash script).

Can someone point me in the right direction here? What am I missing?

Thanks in advance!

Best regards Christoph

---

Xymon mailing list -- xymon@xymon.com To unsubscribe send an email to xymon-leave@xymon.com

---

Xymon mailing list -- xymon@xymon.com To unsubscribe send an email to xymon-leave@xymon.com

---

Xymon mailing list -- xymon@xymon.com To unsubscribe send an email to xymon-leave@xymon.com

Hi,

On 12/10/2026 17:49, Stef Coene via Xymon wrote:

Hi,

I recommend taking a look at the data/trends message.

In short, you make a message with 1 rrd per GPU: [GPU-<number>.rrd] DS:counter1:GAUGE:600:0:U $value1 DS:counter2:GAUGE:600:0:U $value2

You send this is a data message: xymon $XYMSRV "data hostname.trends\n<message>"

This will create the .rrd dilws.

thank you very much, this part got solved with the help of another person on here as well.

And then define a graph like:

[counter] FNPATTERN ^GPU-(.*).rrd

   DEF:counter1@RRDIDX@=@RRDFN@:counter1:AVERAGE TITLE Counter 1 YAXIS Counter LINE1:counter1@RRDIDX@#@COLOR@:@RRDPARAM@: GPRINT:counter1@RRDIDX@:LAST: cur\:%8.4lf GPRINT:counter1@RRDIDX@:MAX: max\:%8.4lf GPRINT:counter1@RRDIDX@:MIN: min\:%8.4lf GPRINT:counter1@RRDIDX@:AVERAGE: avg\:%8.4lf \n

This is the part not working for me at the moment, I have all the RRD files and my graph definition, but xymon won't produce any graphs...

my graphs file: [nvidia-gpu] FNPATTERN ^nvidia-gpu(.*).rrd TITLE GPU stats YAXIS Watt DEF:powerdraw@RRDIDX@=@RRDFN@:powerdraw:AVERAGE LINE1:powerdraw@RRDIDX@#@COLOR@:@RRDPARAM@: COMMENT:\n GPRINT:powerdraw@RRDIDX@:LAST: cur\:%8.4lf GPRINT:powerdraw@RRDIDX@:MAX: max\:%8.4lf GPRINT:powerdraw@RRDIDX@:MIN: min\:%8.4lf GPRINT:powerdraw@RRDIDX@:AVERAGE: avg\:%8.4lf \n

The DS in the RRD files created for "Power draw" are called "powerdraw", so this should work.

Is it a regex problem?

My files are called "nvidia-gpu_0_powerdraw.rrd", "nvidia-gpu_1_powerdraw.rrd" and so on, so the regex has to match...

Cheers Christoph

Stef

On 2026-10-12 14:23, Christoph Zechner via Xymon wrote:

...

Hi,

I've stumbled upon a problem I cannot seem to solve myself and hope someone here can help me out...

A custom check I wrote is collecting data from Nvidia GPUs via the nvidia-smi command, the check works great and displays all the relevant data in its message, but I want to create graphs from certain data points too.

Here's my problem: in a server with 8 GPUs, I am collecting 5 different things from each GPU (like Power Draw, Utilisation and so on), these values get saved in a RRD file as well, but only the values from the first GPU.

I want to save a RRD file for each GPU and its values, so multiple data points in a RRD, but 8 different files (one per GPU) and I do not get how I can do this in a xymon check (bash script).

Can someone point me in the right direction here? What am I missing?

Thanks in advance!

Best regards Christoph

---

Xymon mailing list -- xymon@xymon.com To unsubscribe send an email to xymon-leave@xymon.com

---

Xymon mailing list -- xymon@xymon.com To unsubscribe send an email to xymon-leave@xymon.com

You can also make the graphs by hand. Something like this should give you a PNG file you can load in your browser. Tweak the graph settings until it looks right, then update the graphs.cfg for Xymon.

NOW=date +%s START=expr $NOW - 604800 - 604800 # this sets the start to NOW-14 days UPDATE=date +'%F %T' | sed -e 's/:/\\\:/g' RRD=/var/lib/xymon/rrd/somehostname/something.rrd

rrdtool graph /var/www/html/somehostname.png
"DEF:thing=$RRD:thingvariable:AVERAGE"
"LINE2:thing#$COLOR:Total Thing Status"
"GPRINT:thing:LAST:\: %5.1lf %s (cur)"
"GPRINT:thing:MAX: %5.1lf %s (max)"
"GPRINT:thing:MIN: %5.1lf %s (min)"
"GPRINT:thing:AVERAGE: %5.1lf %s (avg)\n"
"COMMENT: Updated\: $UPDATE\n"
--title "Thing Status - 14 days"
--vertical-label "Thing"
--width 864
--height 120
--start $START
--end $NOW

Ralph Mitchell

On Fri, Oct 16, 2026 at 8:35 AM Stef Coene via Xymon <xymon@xymon.com> wrote:

...

This is the part not working for me at the moment, I have all the RRD files and my graph definition, but xymon won't produce any graphs... If you check the page for the graphs, you have it with an URL like service=nvidia-gpu ?

If the graphs are not shown, open the page source, copy the graph URL and download that URL with wget or curl. It will not be a graph but text and it will most likely be the rrd error.

Stef

---

Xymon mailing list -- xymon@xymon.com To unsubscribe send an email to xymon-leave@xymon.com