Hi Christoph,

Thanks for the details, 4.3.30 is fine for this.

Yes, I'd make one test per metric. Putting several metrics in one test is possible (the RRD side works either way, and GRAPHS_<test> can show several graphs on one test), but everything Xymon does with a
status works per test, not per metric:

- One color. A test has one color. If power is red, you can't see from the overview that memory is also yellow, and when power goes back to green the column stays yellow.
- One alert. alerts.cfg selects alerts by test (SERVICE=). You can't send power alerts to one team and memory alerts to another, or use a different delay or repeat for each.
- One ack and one disable. Acking or disabling the test silences every metric in it. If you ack a known power issue, a new memory problem on the same test goes unnoticed until the ack expires.
- One history. The history and availability reports record one color change per test. Afterwards you can't tell which metric caused an outage.

Inode is likewise its own column, separate from disk. Several instances of the same metric in one test is fine, which is what disk does with all its filesystems, and what the script below does with your GPUs.

Several metrics in one test are also fine when they always go together, for example traffic in/out on an interface, or memory used/total below. A good test is whether you would ever want a different alert or
ack for one of them. If you would, split them.

The reason only GPU 0 was stored is SPLITNCV. Every GPU's "Power Draw" line goes to the same file, nvidia-gpu,PowerDraw.rrd, and xymond_rrd keeps only the first update to a file from each message.

The devmon format, which xymond_rrd also handles, fixes that. Each test adds a block inside HTML comments, so it doesn't show on the status page:

<!-- linecount=0 -->
<!--DEVMON RRD: nvidia-mem 0 0
DS:memory_used:GAUGE:600:0:U DS:memory_total:GAUGE:600:0:U
0 4096:81920
1 2048:81920
-->

- The DS: line defines the data sources.
- Each line after it is one GPU: an id with no spaces, then the values separated by ":", in DS order.
- xymond_rrd writes one RRD file per line: nvidia-mem.0.rrd, nvidia-mem.1.rrd, and so on.
- The <!-- linecount=... --> line is needed for the status page to show the graph. Without it, 4.3.30 shows no graph for a devmon test whose name isn't in its built-in list (disk, inode, if_load, ...). Its
  value decides how the status page groups the GPUs, see the two options below.

This also corrects what I wrote before: it is not one RRD file holding all GPUs, but one file per GPU, the same way "disk" keeps one file per filesystem. The graph then puts them together.

Two options for the status page

The trends page shows each graph with at most 5 GPUs per image in both cases, on 4.3.30 and on the development version. The difference is the status page of each test:

┌──────────────────────────────┬─────────────────────────┬──────────────────────────┐
│  SPLITGRAPHS in the script   │         4.3.30          │   development version    │
├──────────────────────────────┼─────────────────────────┼──────────────────────────┤
│ 0 (linecount=0)              │ one image with all GPUs │ one image with all GPUs  │
├──────────────────────────────┼─────────────────────────┼──────────────────────────┤
│ 1 (linecount=number of GPUs) │ one image per GPU       │ at most 5 GPUs per image │
└──────────────────────────────┴─────────────────────────┴──────────────────────────┘

Option 0 looks the same on both versions. Option 1 gives the nicer layout once you upgrade, but one image per GPU on 4.3.30. Both keep the shipped "devmon::1" setting, so Devmon's own graphs are not affected.

1. Client script (one ext script, it sends one status per test):

#!/bin/sh
# One Xymon test per metric (like disk/inode), one RRD per GPU per test
# (<test>.<index>.rrd), stored by xymond_rrd's devmon handler.
# Server: TEST2RRD "<test>=devmon" for each test below.

# <test>=<nvidia-smi fields>: the fields become the DS names ("." -> "_").
TESTS="nvidia-power=power.draw
nvidia-util=utilization.gpu
nvidia-memutil=utilization.memory
nvidia-mem=memory.used,memory.total"

# Status page: 0 = one graph with every GPU; 1 = split the GPUs into graphs
# (one per GPU on 4.3.30, at most "<test>::N" per graph on newer versions).
SPLITGRAPHS=0

for t in $TESTS; do
        TEST=${t%%=*}
        FIELDS=${t#*=}
        COLOR=green
        RAW=$(nvidia-smi --format=csv,noheader,nounits --query-gpu=index,name,$FIELDS 2>&1)
        [ $? -eq 0 ] && [ -n "$RAW" ] || COLOR=red

        # Set COLOR here from $RAW to apply per-test thresholds.

        BODY=$(printf '%s\n' "$RAW" | awk -F', *' -v ds="$FIELDS" -v test="$TEST" -v splitgraphs="$SPLITGRAPHS" '
        BEGIN { n = split(ds, d, ","); for (i = 1; i <= n; i++) gsub(/\./, "_", d[i]) }
        NF >= n + 2 {
                txt = txt sprintf("GPU %s (%s):", $1, $2)
                v = ""
                for (i = 1; i <= n; i++) {
                        x = $(i + 2)
                        if (x !~ /^-?[0-9]+(\.[0-9]+)?$/) x = "U"       # "[N/A]" -> unknown
                        txt = txt " " d[i] "=" x
                        v = v (i > 1 ? ":" : "") x
                }
                txt = txt "\n"
                rrd = rrd $1 " " v "\n"
                gpus++
        }
        END {
                # The status page needs a linecount to show the graph: 0 draws every
                # GPU in one graph, the GPU count lets the status page split them.
                printf "%s\n<!-- linecount=%d -->\n", txt, (splitgraphs ? gpus : 0)
                printf "<!--DEVMON RRD: %s 0 0\n", test
                for (i = 1; i <= n; i++) printf "DS:%s:GAUGE:600:0:U ", d[i]
                printf "\n%s-->\n", rrd
        }')
        [ "$COLOR" = red ] && BODY="nvidia-smi failed:
$RAW"

        $XYMON $XYMSRV "status $MACHINE.$TEST $COLOR $(date)
$BODY"
done

To add a metric, add one line to TESTS: the test name, then the nvidia-smi field(s) to store. DS names are limited to 19 characters. If you add a field to an existing test, delete its RRD files so they are
recreated; otherwise the updates are rejected.

If you prefer to keep your own script: your current MSG loop can stay as the human-readable part. Send one status per test, and add to each the linecount line (0, or the number of GPUs) and the <!--DEVMON
RRD: ... block with one line per GPU. In each block, use the index as the id and only that test's values.

2. xymonserver.cfg on the server:

TEST2RRD="...existing entries...,nvidia-power=devmon,nvidia-util=devmon,nvidia-memutil=devmon,nvidia-mem=devmon"
GRAPHS_nvidia-power="nvidia-power"
GRAPHS_nvidia-util="nvidia-util"
GRAPHS_nvidia-memutil="nvidia-memutil"
GRAPHS_nvidia-mem="nvidia-mem"

and add the four graphs at the end of the existing GRAPHS line, leaving "devmon::1" as it is:

GRAPHS="...existing entries...,nvidia-power::5,nvidia-util::5,nvidia-memutil::5,nvidia-mem::5"

- <test>=devmon routes each column to the devmon handler.
- GRAPHS_<test> tells the status page which graph section to use.
- <test>::5 is the maximum number of GPUs per image. It puts the graphs on the trends page, 5 per image. With option 1, the development version also uses it on the status pages; 4.3.30 does not, so there you
  get one image per GPU.
- Nothing needs to change when you upgrade.
- Remove your NCV_/SPLITNCV_ settings for the old test. You can delete the old nvidia-gpu,*.rrd files.

3. graphs.cfg: one section per test, each image showing its GPUs as lines:

[nvidia-power]
        FNPATTERN ^nvidia-power\.(.+)\.rrd
        TITLE GPU Power Draw
        YAXIS Watt
        DEF:p@RRDIDX@=@RRDFN@:power_draw:AVERAGE
        LINE2:p@RRDIDX@#@COLOR@:GPU @RRDPARAM@
        GPRINT:p@RRDIDX@:LAST: \: %6.1lf (cur)
        GPRINT:p@RRDIDX@:MAX: \: %6.1lf (max)
        GPRINT:p@RRDIDX@:AVERAGE: \: %6.1lf (avg)\n

[nvidia-util]
        FNPATTERN ^nvidia-util\.(.+)\.rrd
        TITLE GPU Utilization
        YAXIS %
        -l 0
        -u 100
        DEF:p@RRDIDX@=@RRDFN@:utilization_gpu:AVERAGE
        LINE2:p@RRDIDX@#@COLOR@:GPU @RRDPARAM@
        GPRINT:p@RRDIDX@:LAST: \: %5.1lf (cur)
        GPRINT:p@RRDIDX@:MAX: \: %5.1lf (max)
        GPRINT:p@RRDIDX@:AVERAGE: \: %5.1lf (avg)\n

[nvidia-memutil]
        FNPATTERN ^nvidia-memutil\.(.+)\.rrd
        TITLE GPU Memory Bus Utilization
        YAXIS %
        -l 0
        -u 100
        DEF:p@RRDIDX@=@RRDFN@:utilization_memory:AVERAGE
        LINE2:p@RRDIDX@#@COLOR@:GPU @RRDPARAM@
        GPRINT:p@RRDIDX@:LAST: \: %5.1lf (cur)
        GPRINT:p@RRDIDX@:MAX: \: %5.1lf (max)
        GPRINT:p@RRDIDX@:AVERAGE: \: %5.1lf (avg)\n

[nvidia-mem]
        FNPATTERN ^nvidia-mem\.(.+)\.rrd
        TITLE GPU Memory Used
        YAXIS %
        -l 0
        -u 100
        DEF:u@RRDIDX@=@RRDFN@:memory_used:AVERAGE
        DEF:t@RRDIDX@=@RRDFN@:memory_total:AVERAGE
        CDEF:p@RRDIDX@=u@RRDIDX@,t@RRDIDX@,/,100,*
        LINE2:p@RRDIDX@#@COLOR@:GPU @RRDPARAM@
        GPRINT:p@RRDIDX@:LAST: \: %5.1lf (cur)
        GPRINT:p@RRDIDX@:MAX: \: %5.1lf (max)
        GPRINT:p@RRDIDX@:AVERAGE: \: %5.1lf (avg)\n

The nvidia-mem test stores both used and total, so its graph can show memory used as a percentage.

Restart Xymon after editing xymonserver.cfg (graphs.cfg is read on each request).

Note: I tested both options on a Xymon 4.3.30 server built from the release and on the development version, with a simulated nvidia-smi for 8 GPUs (including "[N/A]" values, which become "U", i.e. unknown,
and show as a gap in the graph). Both produced one RRD file per GPU per test and the layouts in the table above. On 4.3.30 the plotted values matched what was sent. I haven't run it on a real GPU server, so
please let me know how it goes.

Bruno

Hi Bruno,

On 12/10/2026 15:06, Bruno Manzoni via Xymon wrote:

Hi Christophe,

I would recommend having one test per graph (so 5 different metrics = 5 tests).

You mean instead of cramming five checks into one, I should separate those into individual tests?

If I remember correctly, an RRD file can contain multiple instances of the same measurement. For example, if you have 6 GPUs and want to monitor the usage of each one, you can store all 6 measurements in a single RRD file.

But how do I tell xymon I want six values per GPU per server? I wanted a behaviour similar to the "disk" check for example, where multiple disks show up in one graph per host.

Then, I think you can use the NCV or Devmon-style approach (also built into Xymon) to pass the parameters. (devmon way seems less documented but more powerfull)

I tried NCV and also SPLITNCV (which gave me different files for each value, but also only for the first GPU).

Also just that you know: We've fixed some issues in recent Xymon versions related to "grouping" graph data (multiple graphs in the same image). So it's important to know whether you're running the latest version.

oh, sorry, I forgot to add that information: Xymon 4.3.30 (running on Debian trixie).

-> The tests you are doing is also on my todo list. (as well as 2 features: 1. restructuring the way to pass data to xymon and to 2. stack multiple test into one)

Good to know, looking forward to the next version!

Let me know if this help and do not hesitate to share some part of the script.

This is the part in the for loop which "builds" the message for the xymon server:

MSG="${MSG}
&${COLOR} Index: ${GPUINDEX}
Name: ${GPUNAME}
Power Draw: ${GPUPOWERDRAW} Watt
Utilization (GPU): ${GPUUTILGPU} %
Utilization (Memory bus): ${GPUUTILMEM} %
GPU Memory Total: ${GPUMEMTOTAL} MB
GPU Memory Used: ${GPUMEMUSED} MB
"
done

This part is working great, but the RRD files (named after the test [nvidia-gpu] and its value, e.g. "nvidia-gpu,PowerDraw.rrd") only contain the value for the first GPU in the check.

Thanks again!

Christoph

Bruno


Le 12.10.2026 à 14:23, Christoph Zechner via Xymon a écrit :

Hi,

I've stumbled upon a problem I cannot seem to solve myself and hope someone here can help me out...

A custom check I wrote is collecting data from Nvidia GPUs via the nvidia-smi command, the check works great and displays all the relevant data in its message, but I want to create graphs from certain data points too.

Here's my problem: in a server with 8 GPUs, I am collecting 5 different things from each GPU (like Power Draw, Utilisation and so on), these values get saved in a RRD file as well, but only the values from the first GPU.

I want to save a RRD file for each GPU and its values, so multiple data points in a RRD, but 8 different files (one per GPU) and I do not get how I can do this in a xymon check (bash script).

Can someone point me in the right direction here? What am I missing?

Thanks in advance!

Best regards
Christoph
_______________________________________________
Xymon mailing list -- xymon@xymon.com
To unsubscribe send an email to xymon-leave@xymon.com

_______________________________________________
Xymon mailing list -- xymon@xymon.com
To unsubscribe send an email to xymon-leave@xymon.com

_______________________________________________
Xymon mailing list -- xymon@xymon.com
To unsubscribe send an email to xymon-leave@xymon.com