newer
how to ignore or hide MEMPHYS from...

Monitoring network traffic

older
Bug? procs test going red green...

R.Schrittenlocher＠ub.uni-frankfurt.de

4 Apr 2024 4 Apr '24

7:45 a.m.

Hi,

first thanks to all contributing to xymon and the mailing list - we profit from your work for many years up to now!

Our challenge at moment is how to monitor traffic quantity in/out in order to detect suspicious activities on Solaris 10. Is there are way to do this with xymon?

greetings and thanks

Rolf

Rolf Schrittenlocher

Bibliotheksmanagementsystem IT | IT-Services (ITS)

Universit?tsbibliothek Johann Christian Senckenberg

Goethe-Universit?t Frankfurt | Campus Bockenheim

Zentralbibliothek | Freimannplatz 1

60325 Frankfurt am Main | GERMANY

Telefon Sammelnummer +49 (0)69 798 28830

Telefon pers?nlich +49 (0)69 798 28908

E-Mail: lbs-it at ub.uni-frankfurt.de

E-Mail (pers?nlich) r.schrittenlocher at ub.uni-frankfurt.de

Website: https://www.ub.uni-frankfurt.de

Show replies by date

abe＠deuxchevaux.org

4 Apr 4 Apr

8:17 a.m.

Hi Rolf,

Schrittenlocher, Rolf schrieb am Thu, Apr 04, 2024 at 07:45:58AM +0000:

...

Our challenge at moment is how to monitor traffic quantity in/out in order to detect suspicious activities on Solaris 10. Is there are way to do this with xymon?

Definitely. ;-)

For our own use (in a university, too :-) and published via Debian's hobbit-plugins package, I've written a plugin simply called "net" which can check many network interface characteristics including monitoring network traffic (calculating bytes/second average from the rx/tx difference of 10 seconds), but so far it's just for Linux and uses common Linux commandline tools and /proc/ links:

https://salsa.debian.org/debian/hobbit-plugins/-/blob/master/src/usr/lib/xym...

(It also uses the Hobbit.pm Perl module from the same package: https://salsa.debian.org/debian/hobbit-plugins/-/blob/master/src/usr/share/p...)

It though shouldn't be too hard to adapt it to some Solaris commandline tools and their output. I'm just not sure how to convert the /proc/ stuff. Maybe there's a Linux compat mode like in FreeBSD? (Haven't touched any Solaris for like 20 years or so, back when I was a student.)

	Regards, Axel

-- PGP: 2FF9CD59612616B5 /~\ Plain Text Ribbon Campaign, http://arc.pasp.de/ Mail: abe at deuxchevaux.org \ / Gegen HTML in E-Mails und Usenet Mail+Jabber: abe at noone.org X https://axel.beckert.ch/ / \ I love long mails: https://email.is-not-s.ms/

R.Schrittenlocher＠ub.uni-frankfurt.de

8:29 a.m.

Hi,

thanks Axel. I just saw that "trends" shows network traffic. So the data is already collected and available on the server. xymon server is Linux, only the clients are Solaris. So someone can tell me how I can access the data either with a client script or on server side?

kind regards

Rolf

Rolf Schrittenlocher

Bibliotheksmanagementsystem IT | IT-Services (ITS)

Universit?tsbibliothek Johann Christian Senckenberg

Goethe-Universit?t Frankfurt | Campus Bockenheim

Zentralbibliothek | Freimannplatz 1

60325 Frankfurt am Main | GERMANY

Telefon Sammelnummer +49 (0)69 798 28830

Telefon pers?nlich +49 (0)69 798 28908

E-Mail: lbs-it at ub.uni-frankfurt.de

E-Mail (pers?nlich) r.schrittenlocher at ub.uni-frankfurt.de

Website: https://www.ub.uni-frankfurt.de

Von: Axel Beckert <abe at deuxchevaux.org> Gesendet: Donnerstag, 4. April 2024 10:17 An: Schrittenlocher, Rolf Cc: Xymon at xymon.com Betreff: Re: [Xymon] Monitoring network traffic

Hi Rolf,

Schrittenlocher, Rolf schrieb am Thu, Apr 04, 2024 at 07:45:58AM +0000:

...

Our challenge at moment is how to monitor traffic quantity in/out in order to detect suspicious activities on Solaris 10. Is there are way to do this with xymon?

Definitely. ;-)

https://salsa.debian.org/debian/hobbit-plugins/-/blob/master/src/usr/lib/xym...

(It also uses the Hobbit.pm Perl module from the same package: https://salsa.debian.org/debian/hobbit-plugins/-/blob/master/src/usr/share/p...)

            Regards, Axel

abe＠deuxchevaux.org

9:45 a.m.

Hi Rolf,

Schrittenlocher, Rolf schrieb am Thu, Apr 04, 2024 at 08:29:54AM +0000:

...

I just saw that "trends" shows network traffic. So the data is already collected and available on the server.

Yes, that data comes from the generic data collection (like process list, load, uptime, etc.) each client sends.

There's just no alerting on traffic thresholds possible. That's one of the metrics for which my plugin can warn or alert (with measurements and comparisons done on the client side, though).

...

So someone can tell me how I can access the data either with a client script or on server side?

Sorry, not out of my mind. I mostly know how to parse hosts.cfg and extract parameters and flags from there.

The man page xymon(1) shows quite some ways to extract data from the server, except that I was not able to extract anything useful related to trends, netstat or ifstat.

An example of how to work with server data might be our ircbot plugin at https://salsa.debian.org/debian/hobbit-plugins/-/blob/master/src/usr/lib/xym...

But it also just uses "xymoncmd xymon xymondboard" and "xymoncmd xymon query" to fetch data from the server and that one doesn't seem to work with data or trends.

The only way I currently see is to use the "xymoncmd xymon clientlog $hostname" command which fetches the latest raw client message including e.g. the "ifconfig" output. It also has a "netstat" section which e.g. looks like this:

---8<--- ? [netstat] Ip: Forwarding: 1 867497 total packets received 0 forwarded 0 incoming packets discarded 853230 incoming packets delivered 835141 requests sent out 225 outgoing packets dropped Icmp: 21635 ICMP messages received 2 input ICMP message failed ICMP input histogram: destination unreachable: 650 echo requests: 20985 92359 ICMP messages sent 0 ICMP messages failed ICMP output histogram: destination unreachable: 651 echo requests: 70723 echo replies: 20985 IcmpMsg: InType3: 650 InType8: 20985 OutType0: 20985 OutType3: 651 OutType8: 70723 Tcp: 23811 active connection openings 2102 passive connection openings 4 failed connection attempts 1911 connection resets received 21 connections established 1007911 segments received 1268414 segments sent out 176 segments retransmitted 0 bad segments received 853 resets sent Udp: 53387 packets received 649 packets to unknown port received 0 packet receive errors 53372 packets sent 0 receive buffer errors 0 send buffer errors IgnoredMulti: 5414 UdpLite: TcpExt: 1 resets received for embryonic SYN_RECV sockets 12258 TCP sockets finished time wait in fast timer 17779 delayed acks sent 16 delayed acks further delayed because of locked socket Quick ack mode was activated 96 times 71063 packet headers predicted 142582 acknowledgments not containing data payload received 546613 predicted acknowledgments TCPSackRecovery: 43 Detected reordering 2106 times using SACK Detected reordering 36 times using time stamp 2 congestion windows fully recovered without slow start 35 congestion windows partially recovered using Hoe heuristic TCPDSACKUndo: 4 1 congestion windows recovered without slow start after partial ack TCPLostRetransmit: 69 67 fast retransmits 1 retransmits in slow start TCPTimeouts: 87 TCPLossProbes: 26 TCPLossProbeRecovery: 4 TCPBacklogCoalesce: 2432 TCPDSACKOldSent: 96 TCPDSACKRecv: 53 120 connections reset due to unexpected data 13 connections reset due to early user close 7 connections aborted due to timeout TCPDSACKIgnoredNoUndo: 43 TCPSackShifted: 194 TCPSackMerged: 34 TCPSackShiftFallback: 4424 TCPRcvCoalesce: 66342 TCPOFOQueue: 352 TCPChallengeACK: 1 TCPAutoCorking: 28376 TCPFromZeroWindowAdv: 32 TCPToZeroWindowAdv: 32 TCPWantZeroWindowAdv: 323 TCPSynRetrans: 21 TCPOrigDataSent: 1019410 TCPHystartTrainDetect: 656 TCPHystartTrainCwnd: 50284 TCPACKSkippedSynRecv: 11 TCPWinProbe: 1 TCPKeepAlive: 26 TCPDelivered: 1042042 TCPAckCompressed: 109 TcpTimeoutRehash: 80 TcpDuplicateDataRehash: 15 TCPDSACKRecvSegs: 63 IpExt: InMcastPkts: 1579 OutMcastPkts: 4 InBcastPkts: 5414 InOctets: 217038821 OutOctets: 653115273 InMcastOctets: 50528 OutMcastOctets: 160 InBcastOctets: 1360064 InNoECTPkts: 909738 MPTcpExt: Sctp: 0 Current Associations 0 Active Associations 0 Passive Associations 0 Number of Aborteds 0 Number of Graceful Terminations 0 Number of Out of Blue packets 0 Number of Packets with invalid Checksum 0 Number of control chunks sent 0 Number of ordered chunks sent 0 Number of Unordered chunks sent 0 Number of control chunks received 0 Number of ordered chunks received 0 Number of Unordered chunks received 0 Number of messages fragmented 0 Number of messages reassembled 0 Number of SCTP packets sent 0 Number of SCTP packets received [?] ? --->8---

But you would need to parse the data interesting for you out of this yourself. Hope this helps nevertheless.

            Regards, Axel

josh＠imaginenetworksllc.com

3:19 p.m.

The clientlog includes [netstat] which has a snapshot of activity in text

The trends puts it in a pretty graph stored in rrd.

On Thu, Apr 4, 2024 at 4:30?AM Schrittenlocher, Rolf < R.Schrittenlocher at ub.uni-frankfurt.de> wrote:

...

Hi,

thanks Axel. I just saw that "trends" shows network traffic. So the data is already collected and available on the server. xymon server is Linux, only the clients are Solaris. So someone can tell me how I can access the data either with a client script or on server side?

kind regards

Rolf

Rolf Schrittenlocher

Bibliotheksmanagementsystem IT | IT-Services (ITS)

Universit?tsbibliothek Johann Christian Senckenberg

Goethe-Universit?t Frankfurt | Campus Bockenheim

Zentralbibliothek | Freimannplatz 1

60325 Frankfurt am Main | GERMANY

Telefon Sammelnummer +49 (0)69 798 28830

Telefon pers?nlich +49 (0)69 798 28908

E-Mail: lbs-it at ub.uni-frankfurt.de

E-Mail (pers?nlich) r.schrittenlocher at ub.uni-frankfurt.de

Website: https://www.ub.uni-frankfurt.de

*Von:* Axel Beckert <abe at deuxchevaux.org> *Gesendet:* Donnerstag, 4. April 2024 10:17 *An:* Schrittenlocher, Rolf *Cc:* Xymon at xymon.com *Betreff:* Re: [Xymon] Monitoring network traffic

Hi Rolf,

Schrittenlocher, Rolf schrieb am Thu, Apr 04, 2024 at 07:45:58AM +0000:

...
Our challenge at moment is how to monitor traffic quantity in/out in order to detect suspicious activities on Solaris 10. Is there are way to do this with xymon?

Definitely. ;-)

For our own use (in a university, too :-) and published via Debian's hobbit-plugins package, I've written a plugin simply called "net" which can check many network interface characteristics including monitoring network traffic (calculating bytes/second average from the rx/tx difference of 10 seconds), but so far it's just for Linux and uses common Linux commandline tools and /proc/ links:

https://salsa.debian.org/debian/hobbit-plugins/-/blob/master/src/usr/lib/xym...

(It also uses the Hobbit.pm Perl module from the same package:

https://salsa.debian.org/debian/hobbit-plugins/-/blob/master/src/usr/share/p... )

It though shouldn't be too hard to adapt it to some Solaris commandline tools and their output. I'm just not sure how to convert the /proc/ stuff. Maybe there's a Linux compat mode like in FreeBSD? (Haven't touched any Solaris for like 20 years or so, back when I was a student.)
            Regards, Axel
-- PGP: 2FF9CD59612616B5 /~\ Plain Text Ribbon Campaign, http://arc.pasp.de/ Mail: abe at deuxchevaux.org \ / Gegen HTML in E-Mails und Usenet Mail+Jabber: abe at noone.org X https://axel.beckert.ch/ / \ I love long mails: https://email.is-not-s.ms/

Xymon mailing list Xymon at xymon.com http://lists.xymon.com/mailman/listinfo/xymon

norkrie＠gmail.com

9:27 p.m.

Hi,

I created a server side script for all the *nix servers where I extract the network info from the clientlog. The script identifies all server with a ssh column (this is clearly a *nix server) and then loops over all these targets to create a "nic" column with interface info. Nothing to configure especially, a new *nic server will be automatically identified and get the column with detailed info and some graphs.

Some snippets to get the idea:

grab all client info

get_all_info(){

$XYMONBIN localhost "clientlog $TARGET"

}

ALLINFO=get_all_info

##################################################

grab the nic details

get_nic_info(){

echo "$ALLINFO" | \

$NAWK '/^\[ifconfig/,/^\[route/' | \

$GREP -v "^\["

}

##################################################

grab the route

get_route_info(){

echo "$ALLINFO" | \

$NAWK '/^\[route/,/^\[netstat/' | \

$GREP -v "^\["

}

##################################################

grab the ports

get_ports_info(){

ALLPORTS=`echo "$ALLINFO" | \

$NAWK '/^\[ports/,/^\[ifstat/' | \

$GREP -v "^\["`

PORTSTATUS=`echo "$ALLPORTS" | \

$NAWK '/^tcp/{print $NF}' | \

$SORT -u`

for stat in $PORTSTATUS

NUM=`echo "$ALLPORTS" | \

     $NAWK 'BEGIN{i=0}

            /'$stat'/{i++};BEGIN{i=0}

            END{print i}'`

echo "tcp ports in status $stat: $NUM"

done

}

create the output to send to xymon

echo "&lt;h4>interface info&lt;/h4>"

get_nic_info



echo "&lt;h4>route info&lt;/h4>"

get_route_info



echo "&lt;h4>active tcp connections&lt;/h4>"

get_ports_info


showgraph ifstat_kB

All these data are then send to the xymon server daemon and create a nic column.

A complete run over 500 servers will take approx. 60 secs (but you can run more scripts in parallel if needed).

HTH

Norbert

Am Do., 4. Apr. 2024 um 19:21 Uhr schrieb Josh Luthman < josh at imaginenetworksllc.com>:

...

The clientlog includes [netstat] which has a snapshot of activity in text

The trends puts it in a pretty graph stored in rrd.

On Thu, Apr 4, 2024 at 4:30?AM Schrittenlocher, Rolf < R.Schrittenlocher at ub.uni-frankfurt.de> wrote:

...
Hi,

thanks Axel. I just saw that "trends" shows network traffic. So the data is already collected and available on the server. xymon server is Linux, only the clients are Solaris. So someone can tell me how I can access the data either with a client script or on server side?

kind regards

Rolf

Rolf Schrittenlocher

Bibliotheksmanagementsystem IT | IT-Services (ITS)

Universit?tsbibliothek Johann Christian Senckenberg

Goethe-Universit?t Frankfurt | Campus Bockenheim

Zentralbibliothek | Freimannplatz 1

60325 Frankfurt am Main | GERMANY

Telefon Sammelnummer +49 (0)69 798 28830

Telefon pers?nlich +49 (0)69 798 28908

E-Mail: lbs-it at ub.uni-frankfurt.de

E-Mail (pers?nlich) r.schrittenlocher at ub.uni-frankfurt.de

Website: https://www.ub.uni-frankfurt.de

*Von:* Axel Beckert <abe at deuxchevaux.org> *Gesendet:* Donnerstag, 4. April 2024 10:17 *An:* Schrittenlocher, Rolf *Cc:* Xymon at xymon.com *Betreff:* Re: [Xymon] Monitoring network traffic

Hi Rolf,

Schrittenlocher, Rolf schrieb am Thu, Apr 04, 2024 at 07:45:58AM +0000:

...
Our challenge at moment is how to monitor traffic quantity in/out in order to detect suspicious activities on Solaris 10. Is there are way to do this with xymon?

Definitely. ;-)

For our own use (in a university, too :-) and published via Debian's hobbit-plugins package, I've written a plugin simply called "net" which can check many network interface characteristics including monitoring network traffic (calculating bytes/second average from the rx/tx difference of 10 seconds), but so far it's just for Linux and uses common Linux commandline tools and /proc/ links:

https://salsa.debian.org/debian/hobbit-plugins/-/blob/master/src/usr/lib/xym...

(It also uses the Hobbit.pm Perl module from the same package:

https://salsa.debian.org/debian/hobbit-plugins/-/blob/master/src/usr/share/p... )

It though shouldn't be too hard to adapt it to some Solaris commandline tools and their output. I'm just not sure how to convert the /proc/ stuff. Maybe there's a Linux compat mode like in FreeBSD? (Haven't touched any Solaris for like 20 years or so, back when I was a student.)
            Regards, Axel
-- PGP: 2FF9CD59612616B5 /~\ Plain Text Ribbon Campaign, http://arc.pasp.de/ Mail: abe at deuxchevaux.org \ / Gegen HTML in E-Mails und Usenet Mail+Jabber: abe at noone.org X https://axel.beckert.ch/ / \ I love long mails: https://email.is-not-s.ms/

Xymon mailing list Xymon at xymon.com http://lists.xymon.com/mailman/listinfo/xymon
Xymon mailing list Xymon at xymon.com http://lists.xymon.com/mailman/listinfo/xymon

R.Schrittenlocher＠ub.uni-frankfurt.de

5 Apr 5 Apr

4:32 a.m.

Hi,

@Josh : Yes I saw it, I hoped there's an easy way to reuse the data used for the trends presentation

@Norbert : Thanks's a lot that helps a lot. I'll adapt it to our needs

Kind regards

Rolf

Rolf Schrittenlocher

Bibliotheksmanagementsystem IT | IT-Services (ITS)

Universit?tsbibliothek Johann Christian Senckenberg

Goethe-Universit?t Frankfurt | Campus Bockenheim

Zentralbibliothek | Freimannplatz 1

60325 Frankfurt am Main | GERMANY

Telefon Sammelnummer +49 (0)69 798 28830

Telefon pers?nlich +49 (0)69 798 28908

E-Mail: lbs-it at ub.uni-frankfurt.de

E-Mail (pers?nlich) r.schrittenlocher at ub.uni-frankfurt.de

Website: https://www.ub.uni-frankfurt.de

Von: nor krie <norkrie at gmail.com> Gesendet: Donnerstag, 4. April 2024 23:27 An: Josh Luthman Cc: Schrittenlocher, Rolf; Xymon at xymon.com Betreff: Re: [Xymon] Monitoring network traffic

Hi,

Some snippets to get the idea:

grab all client info

get_all_info(){

$XYMONBIN localhost "clientlog $TARGET"

}

ALLINFO=get_all_info

##################################################

grab the nic details

get_nic_info(){

echo "$ALLINFO" | \

$NAWK '/^\[ifconfig/,/^\[route/' | \

$GREP -v "^\["

}

##################################################

grab the route

get_route_info(){

echo "$ALLINFO" | \

$NAWK '/^\[route/,/^\[netstat/' | \

$GREP -v "^\["

}

##################################################

grab the ports

get_ports_info(){

ALLPORTS=`echo "$ALLINFO" | \

$NAWK '/^\[ports/,/^\[ifstat/' | \

$GREP -v "^\["`

PORTSTATUS=`echo "$ALLPORTS" | \

$NAWK '/^tcp/{print $NF}' | \

$SORT -u`

for stat in $PORTSTATUS

NUM=`echo "$ALLPORTS" | \

     $NAWK 'BEGIN{i=0}

            /'$stat'/{i++};BEGIN{i=0}

            END{print i}'`

echo "tcp ports in status $stat: $NUM"

done

}

create the output to send to xymon

echo "&lt;h4>interface info&lt;/h4>"

get_nic_info



echo "&lt;h4>route info&lt;/h4>"

get_route_info



echo "&lt;h4>active tcp connections&lt;/h4>"

get_ports_info


showgraph ifstat_kB

All these data are then send to the xymon server daemon and create a nic column.

A complete run over 500 servers will take approx. 60 secs (but you can run more scripts in parallel if needed).

HTH

Norbert

Am Do., 4. Apr. 2024 um 19:21 Uhr schrieb Josh Luthman <josh at imaginenetworksllc.com<mailto:josh at imaginenetworksllc.com>>: The clientlog includes [netstat] which has a snapshot of activity in text

The trends puts it in a pretty graph stored in rrd.

On Thu, Apr 4, 2024 at 4:30?AM Schrittenlocher, Rolf <R.Schrittenlocher at ub.uni-frankfurt.de<mailto:R.Schrittenlocher at ub.uni-frankfurt.de>> wrote:

Hi,

kind regards

Rolf

Rolf Schrittenlocher

Bibliotheksmanagementsystem IT | IT-Services (ITS)

Universit?tsbibliothek Johann Christian Senckenberg

Goethe-Universit?t Frankfurt | Campus Bockenheim

Zentralbibliothek | Freimannplatz 1

60325 Frankfurt am Main | GERMANY

Telefon Sammelnummer +49 (0)69 798 28830

Telefon pers?nlich +49 (0)69 798 28908

E-Mail: lbs-it at ub.uni-frankfurt.de<mailto:lbs-it at ub.uni-frankfurt.de>

E-Mail (pers?nlich) r.schrittenlocher at ub.uni-frankfurt.de<mailto:r.schrittenlocher at ub.uni-frankfurt.de>

Website: https://www.ub.uni-frankfurt.de

Von: Axel Beckert <abe at deuxchevaux.org<mailto:abe at deuxchevaux.org>> Gesendet: Donnerstag, 4. April 2024 10:17 An: Schrittenlocher, Rolf Cc: Xymon at xymon.com<mailto:Xymon at xymon.com> Betreff: Re: [Xymon] Monitoring network traffic

Hi Rolf,

Schrittenlocher, Rolf schrieb am Thu, Apr 04, 2024 at 07:45:58AM +0000:

...

Our challenge at moment is how to monitor traffic quantity in/out in order to detect suspicious activities on Solaris 10. Is there are way to do this with xymon?

Definitely. ;-)

https://salsa.debian.org/debian/hobbit-plugins/-/blob/master/src/usr/lib/xym...

(It also uses the Hobbit.pm Perl module from the same package: https://salsa.debian.org/debian/hobbit-plugins/-/blob/master/src/usr/share/p...)

            Regards, Axel

-- PGP: 2FF9CD59612616B5 /~\ Plain Text Ribbon Campaign, http://arc.pasp.de/ Mail: abe at deuxchevaux.org<mailto:abe at deuxchevaux.org> \ / Gegen HTML in E-Mails und Usenet Mail+Jabber: abe at noone.org<mailto:abe at noone.org> X https://axel.beckert.ch/ / \ I love long mails: https://email.is-not-s.ms/

Xymon mailing list Xymon at xymon.com<mailto:Xymon at xymon.com> http://lists.xymon.com/mailman/listinfo/xymon

jeremy.ruffer＠gmail.com

8:45 a.m.

Hi Rolf,

You could try using rrdfetch to get the data that Trends uses.

HTH

Jeremy

------ Original Message ------ From: "Schrittenlocher, Rolf" <R.Schrittenlocher at ub.uni-frankfurt.de> To: "nor krie" <norkrie at gmail.com>; "Josh Luthman" <josh at imaginenetworksllc.com> Cc: "Xymon at xymon.com" <Xymon at xymon.com> Sent: 05/04/2024 05:32:01 Subject: Re: [Xymon] Monitoring network traffic

...

Hi,

@Josh : Yes I saw it, I hoped there's an easy way to reuse the data used for the trends presentation

@Norbert : Thanks's a lot that helps a lot. I'll adapt it to our needs

Kind regards

Rolf

Rolf Schrittenlocher

Bibliotheksmanagementsystem IT | IT-Services (ITS)

Universit?tsbibliothek Johann Christian Senckenberg

Goethe-Universit?t Frankfurt | Campus Bockenheim

Zentralbibliothek | Freimannplatz 1

60325 Frankfurt am Main | GERMANY

Telefon Sammelnummer +49 (0)69 798 28830

Telefon pers?nlich +49 (0)69 798 28908

E-Mail: lbs-it at ub.uni-frankfurt.de

E-Mail (pers?nlich) r.schrittenlocher at ub.uni-frankfurt.de

Website: https://www.ub.uni-frankfurt.de

Von: nor krie <norkrie at gmail.com> Gesendet: Donnerstag, 4. April 2024 23:27 An: Josh Luthman Cc: Schrittenlocher, Rolf; Xymon at xymon.com Betreff: Re: [Xymon] Monitoring network traffic

Hi,

I created a server side script for all the *nix servers where I extract the network info from the clientlog. The script identifies all server with a ssh column (this is clearly a *nix server) and then loops over all these targets to create a "nic" column with interface info. Nothing to configure especially, a new *nic server will be automatically identified and get the column with detailed info and some graphs.

Some snippets to get the idea:

grab all client info

get_all_info(){

$XYMONBIN localhost "clientlog $TARGET"

}

ALLINFO=get_all_info

##################################################

grab the nic details

get_nic_info(){

echo "$ALLINFO" | \
$NAWK '/^\[ifconfig/,/^\[route/' | \

$GREP -v "^\["
}

##################################################

grab the route

get_route_info(){

echo "$ALLINFO" | \
$NAWK '/^\[route/,/^\[netstat/' | \

$GREP -v "^\["
}

##################################################

grab the ports

get_ports_info(){

ALLPORTS=`echo "$ALLINFO" | \
$NAWK '/^\[ports/,/^\[ifstat/' | \

$GREP -v "^\["`
PORTSTATUS=`echo "$ALLPORTS" | \
$NAWK '/^tcp/{print $NF}' | \

$SORT -u`
for stat in $PORTSTATUS

do
NUM=`echo "$ALLPORTS" | \

     $NAWK 'BEGIN{i=0}

            /'$stat'/{i++};BEGIN{i=0}

            END{print i}'`

echo "tcp ports in status $stat: $NUM"
done

}

create the output to send to xymon
echo "&lt;h4>interface info&lt;/h4>"

get_nic_info



echo "&lt;h4>route info&lt;/h4>"

get_route_info



echo "&lt;h4>active tcp connections&lt;/h4>"

get_ports_info



showgraph ifstat_kB
All these data are then send to the xymon server daemon and create a nic column.

A complete run over 500 servers will take approx. 60 secs (but you can run more scripts in parallel if needed).

HTH

Norbert

Am Do., 4. Apr. 2024 um 19:21 Uhr schrieb Josh Luthman <josh at imaginenetworksllc.com>:

...
The clientlog includes [netstat] which has a snapshot of activity in text

The trends puts it in a pretty graph stored in rrd.

On Thu, Apr 4, 2024 at 4:30?AM Schrittenlocher, Rolf <R.Schrittenlocher at ub.uni-frankfurt.de> wrote:

...
Hi,

thanks Axel. I just saw that "trends" shows network traffic. So the data is already collected and available on the server. xymon server is Linux, only the clients are Solaris. So someone can tell me how I can access the data either with a client script or on server side?

kind regards

Rolf

Rolf Schrittenlocher

Bibliotheksmanagementsystem IT | IT-Services (ITS)

Universit?tsbibliothek Johann Christian Senckenberg

Goethe-Universit?t Frankfurt | Campus Bockenheim

Zentralbibliothek | Freimannplatz 1

60325 Frankfurt am Main | GERMANY

Telefon Sammelnummer +49 (0)69 798 28830

Telefon pers?nlich +49 (0)69 798 28908

E-Mail: lbs-it at ub.uni-frankfurt.de

E-Mail (pers?nlich) r.schrittenlocher at ub.uni-frankfurt.de

Website: https://www.ub.uni-frankfurt.de

Von: Axel Beckert <abe at deuxchevaux.org> Gesendet: Donnerstag, 4. April 2024 10:17 An: Schrittenlocher, Rolf Cc:Xymon at xymon.com Betreff: Re: [Xymon] Monitoring network traffic

Hi Rolf,

Schrittenlocher, Rolf schrieb am Thu, Apr 04, 2024 at 07:45:58AM +0000:

...
Our challenge at moment is how to monitor traffic quantity in/out in order to detect suspicious activities on Solaris 10. Is there are way to do this with xymon?

Definitely. ;-)

For our own use (in a university, too :-) and published via Debian's hobbit-plugins package, I've written a plugin simply called "net" which can check many network interface characteristics including monitoring network traffic (calculating bytes/second average from the rx/tx difference of 10 seconds), but so far it's just for Linux and uses common Linux commandline tools and /proc/ links:

https://salsa.debian.org/debian/hobbit-plugins/-/blob/master/src/usr/lib/xym...

(It also uses the Hobbit.pm Perl module from the same package: https://salsa.debian.org/debian/hobbit-plugins/-/blob/master/src/usr/share/p...)

It though shouldn't be too hard to adapt it to some Solaris commandline tools and their output. I'm just not sure how to convert the /proc/ stuff. Maybe there's a Linux compat mode like in FreeBSD? (Haven't touched any Solaris for like 20 years or so, back when I was a student.)
            Regards, Axel
-- PGP: 2FF9CD59612616B5 /~\ Plain Text Ribbon Campaign, http://arc.pasp.de/ <http://arc.pasp.de/> Mail: abe at deuxchevaux.org \ / Gegen HTML in E-Mails und Usenet Mail+Jabber: abe at noone.org X https://axel.beckert.ch/ / \ I love long mails: https://email.is-not-s.ms/

Xymon mailing list Xymon at xymon.com http://lists.xymon.com/mailman/listinfo/xymon
Xymon mailing list Xymon at xymon.com http://lists.xymon.com/mailman/listinfo/xymon

jeremy＠laidman.org

7 Apr 7 Apr

9:06 a.m.

Check out the DS option in analysis.cfg. This can perform a threshold operation on an RRD file value.

On Fri, 5 Apr 2024, 19:46 Jeremy Ruffer, <jeremy.ruffer at gmail.com> wrote:

...

Hi Rolf,

You could try using rrdfetch to get the data that Trends uses.

HTH

Jeremy

------ Original Message ------ From: "Schrittenlocher, Rolf" <R.Schrittenlocher at ub.uni-frankfurt.de> To: "nor krie" <norkrie at gmail.com>; "Josh Luthman" < josh at imaginenetworksllc.com> Cc: "Xymon at xymon.com" <Xymon at xymon.com> Sent: 05/04/2024 05:32:01 Subject: Re: [Xymon] Monitoring network traffic

Hi,

@Josh : Yes I saw it, I hoped there's an easy way to reuse the data used for the trends presentation

@Norbert : Thanks's a lot that helps a lot. I'll adapt it to our needs

Kind regards

Rolf

Rolf Schrittenlocher

Bibliotheksmanagementsystem IT | IT-Services (ITS)

Universit?tsbibliothek Johann Christian Senckenberg

Goethe-Universit?t Frankfurt | Campus Bockenheim

Zentralbibliothek | Freimannplatz 1

60325 Frankfurt am Main | GERMANY

Telefon Sammelnummer +49 (0)69 798 28830

Telefon pers?nlich +49 (0)69 798 28908

E-Mail: lbs-it at ub.uni-frankfurt.de

E-Mail (pers?nlich) r.schrittenlocher at ub.uni-frankfurt.de

Website: https://www.ub.uni-frankfurt.de

*Von:* nor krie <norkrie at gmail.com> *Gesendet:* Donnerstag, 4. April 2024 23:27 *An:* Josh Luthman *Cc:* Schrittenlocher, Rolf; Xymon at xymon.com *Betreff:* Re: [Xymon] Monitoring network traffic

Hi,

I created a server side script for all the *nix servers where I extract the network info from the clientlog. The script identifies all server with a ssh column (this is clearly a *nix server) and then loops over all these targets to create a "nic" column with interface info. Nothing to configure especially, a new *nic server will be automatically identified and get the column with detailed info and some graphs.

Some snippets to get the idea:

grab all client info

get_all_info(){

$XYMONBIN localhost "clientlog $TARGET"

}

ALLINFO=get_all_info

##################################################

grab the nic details

get_nic_info(){

echo "$ALLINFO" | \
$NAWK '/^\[ifconfig/,/^\[route/' | \

$GREP -v "^\["
}

##################################################

grab the route

get_route_info(){

echo "$ALLINFO" | \
$NAWK '/^\[route/,/^\[netstat/' | \

$GREP -v "^\["
}

##################################################

grab the ports

get_ports_info(){

ALLPORTS=`echo "$ALLINFO" | \
$NAWK '/^\[ports/,/^\[ifstat/' | \

$GREP -v "^\["`
PORTSTATUS=`echo "$ALLPORTS" | \
$NAWK '/^tcp/{print $NF}' | \

$SORT -u`
for stat in $PORTSTATUS

do
NUM=`echo "$ALLPORTS" | \

     $NAWK 'BEGIN{i=0}

            /'$stat'/{i++};BEGIN{i=0}

            END{print i}'`

echo "tcp ports in status $stat: $NUM"
done

}

create the output to send to xymon
echo "&lt;h4>interface info&lt;/h4>"

get_nic_info



echo "&lt;h4>route info&lt;/h4>"

get_route_info



echo "&lt;h4>active tcp connections&lt;/h4>"

get_ports_info


showgraph ifstat_kB
All these data are then send to the xymon server daemon and create a nic column.

A complete run over 500 servers will take approx. 60 secs (but you can run more scripts in parallel if needed).

HTH

Norbert

Am Do., 4. Apr. 2024 um 19:21 Uhr schrieb Josh Luthman < josh at imaginenetworksllc.com>:

...
The clientlog includes [netstat] which has a snapshot of activity in text

The trends puts it in a pretty graph stored in rrd.

On Thu, Apr 4, 2024 at 4:30?AM Schrittenlocher, Rolf < R.Schrittenlocher at ub.uni-frankfurt.de> wrote:

...
Hi,

thanks Axel. I just saw that "trends" shows network traffic. So the data is already collected and available on the server. xymon server is Linux, only the clients are Solaris. So someone can tell me how I can access the data either with a client script or on server side?

kind regards

Rolf

Rolf Schrittenlocher

Bibliotheksmanagementsystem IT | IT-Services (ITS)

Universit?tsbibliothek Johann Christian Senckenberg

Goethe-Universit?t Frankfurt | Campus Bockenheim

Zentralbibliothek | Freimannplatz 1

60325 Frankfurt am Main | GERMANY

Telefon Sammelnummer +49 (0)69 798 28830

Telefon pers?nlich +49 (0)69 798 28908

E-Mail: lbs-it at ub.uni-frankfurt.de

E-Mail (pers?nlich) r.schrittenlocher at ub.uni-frankfurt.de

Website: https://www.ub.uni-frankfurt.de

*Von:* Axel Beckert <abe at deuxchevaux.org> *Gesendet:* Donnerstag, 4. April 2024 10:17 *An:* Schrittenlocher, Rolf *Cc:* Xymon at xymon.com *Betreff:* Re: [Xymon] Monitoring network traffic

Hi Rolf,

Schrittenlocher, Rolf schrieb am Thu, Apr 04, 2024 at 07:45:58AM +0000:

...
Our challenge at moment is how to monitor traffic quantity in/out in order to detect suspicious activities on Solaris 10. Is there are way to do this with xymon?

Definitely. ;-)

For our own use (in a university, too :-) and published via Debian's hobbit-plugins package, I've written a plugin simply called "net" which can check many network interface characteristics including monitoring network traffic (calculating bytes/second average from the rx/tx difference of 10 seconds), but so far it's just for Linux and uses common Linux commandline tools and /proc/ links:

https://salsa.debian.org/debian/hobbit-plugins/-/blob/master/src/usr/lib/xym...

(It also uses the Hobbit.pm Perl module from the same package:

https://salsa.debian.org/debian/hobbit-plugins/-/blob/master/src/usr/share/p... )

It though shouldn't be too hard to adapt it to some Solaris commandline tools and their output. I'm just not sure how to convert the /proc/ stuff. Maybe there's a Linux compat mode like in FreeBSD? (Haven't touched any Solaris for like 20 years or so, back when I was a student.)
            Regards, Axel
-- PGP: 2FF9CD59612616B5 /~\ Plain Text Ribbon Campaign, http://arc.pasp.de/ Mail: abe at deuxchevaux.org \ / Gegen HTML in E-Mails und Usenet Mail+Jabber: abe at noone.org X https://axel.beckert.ch/ / \ I love long mails: https://email.is-not-s.ms/

Xymon mailing list Xymon at xymon.com http://lists.xymon.com/mailman/listinfo/xymon
Xymon mailing list Xymon at xymon.com http://lists.xymon.com/mailman/listinfo/xymon
Xymon mailing list Xymon at xymon.com http://lists.xymon.com/mailman/listinfo/xymon

R.Schrittenlocher＠ub.uni-frankfurt.de

8 Apr 8 Apr

4:58 a.m.

Good morning,

thanks, I found DS on the man page (the explanations in analysis.cfg's comments doent't show it). As far as I understand it is unfortunately not suitable:

"NOTE: This rule uses the raw data value from a client to examine the rules. So this type of test is only really suitable for datasets that are of the "GAUGE" type. It cannot be used meaningfully for datasets that use "COUNTER" or "DERIVE" - e.g. the datasets that are used to capture network packet traffic - because the data stored in the RRD for COUNTER-based datasets undergo a transformation (calculation) when going into the RRD. Xymon does not have direct access to the calculated data."

Bad luck,

cheers

Rolf

Rolf Schrittenlocher

Bibliotheksmanagementsystem IT | IT-Services (ITS)

Universit?tsbibliothek Johann Christian Senckenberg

Goethe-Universit?t Frankfurt | Campus Bockenheim

Zentralbibliothek | Freimannplatz 1

60325 Frankfurt am Main | GERMANY

Telefon Sammelnummer +49 (0)69 798 28830

Telefon pers?nlich +49 (0)69 798 28908

E-Mail: lbs-it at ub.uni-frankfurt.de

E-Mail (pers?nlich) r.schrittenlocher at ub.uni-frankfurt.de

Website: https://www.ub.uni-frankfurt.de

Von: Xymon <xymon-bounces at xymon.com> im Auftrag von Jeremy Laidman <jeremy at laidman.org> Gesendet: Sonntag, 7. April 2024 11:06 An: xymon at xymon.com Betreff: Re: [Xymon] Monitoring network traffic

Check out the DS option in analysis.cfg. This can perform a threshold operation on an RRD file value.

On Fri, 5 Apr 2024, 19:46 Jeremy Ruffer, <jeremy.ruffer at gmail.com<mailto:jeremy.ruffer at gmail.com>> wrote: Hi Rolf,

You could try using rrdfetch to get the data that Trends uses.

HTH

Jeremy

------ Original Message ------ From: "Schrittenlocher, Rolf" <R.Schrittenlocher at ub.uni-frankfurt.de<mailto:R.Schrittenlocher at ub.uni-frankfurt.de>> To: "nor krie" <norkrie at gmail.com<mailto:norkrie at gmail.com>>; "Josh Luthman" <josh at imaginenetworksllc.com<mailto:josh at imaginenetworksllc.com>> Cc: "Xymon at xymon.com<mailto:Xymon at xymon.com>" <Xymon at xymon.com<mailto:Xymon at xymon.com>> Sent: 05/04/2024 05:32:01 Subject: Re: [Xymon] Monitoring network traffic

Hi,

@Josh : Yes I saw it, I hoped there's an easy way to reuse the data used for the trends presentation

@Norbert : Thanks's a lot that helps a lot. I'll adapt it to our needs

Kind regards

Rolf

Rolf Schrittenlocher

Bibliotheksmanagementsystem IT | IT-Services (ITS)

Universit?tsbibliothek Johann Christian Senckenberg

Goethe-Universit?t Frankfurt | Campus Bockenheim

Zentralbibliothek | Freimannplatz 1

60325 Frankfurt am Main | GERMANY

Telefon Sammelnummer +49 (0)69 798 28830

Telefon pers?nlich +49 (0)69 798 28908

E-Mail: lbs-it at ub.uni-frankfurt.de<mailto:lbs-it at ub.uni-frankfurt.de>

E-Mail (pers?nlich) r.schrittenlocher at ub.uni-frankfurt.de<mailto:r.schrittenlocher at ub.uni-frankfurt.de>

Website: https://www.ub.uni-frankfurt.de

Von: nor krie <norkrie at gmail.com<mailto:norkrie at gmail.com>> Gesendet: Donnerstag, 4. April 2024 23:27 An: Josh Luthman Cc: Schrittenlocher, Rolf; Xymon at xymon.com<mailto:Xymon at xymon.com> Betreff: Re: [Xymon] Monitoring network traffic

Hi,

Some snippets to get the idea:

grab all client info

get_all_info(){

$XYMONBIN localhost "clientlog $TARGET"

}

ALLINFO=get_all_info

##################################################

grab the nic details

get_nic_info(){

echo "$ALLINFO" | \

$NAWK '/^\[ifconfig/,/^\[route/' | \

$GREP -v "^\["

}

##################################################

grab the route

get_route_info(){

echo "$ALLINFO" | \

$NAWK '/^\[route/,/^\[netstat/' | \

$GREP -v "^\["

}

##################################################

grab the ports

get_ports_info(){

ALLPORTS=`echo "$ALLINFO" | \

$NAWK '/^\[ports/,/^\[ifstat/' | \

$GREP -v "^\["`

PORTSTATUS=`echo "$ALLPORTS" | \

$NAWK '/^tcp/{print $NF}' | \

$SORT -u`

for stat in $PORTSTATUS

NUM=`echo "$ALLPORTS" | \

     $NAWK 'BEGIN{i=0}

            /'$stat'/{i++};BEGIN{i=0}

            END{print i}'`

echo "tcp ports in status $stat: $NUM"

done

}

create the output to send to xymon

echo "&lt;h4>interface info&lt;/h4>"

get_nic_info



echo "&lt;h4>route info&lt;/h4>"

get_route_info



echo "&lt;h4>active tcp connections&lt;/h4>"

get_ports_info


showgraph ifstat_kB

All these data are then send to the xymon server daemon and create a nic column.

A complete run over 500 servers will take approx. 60 secs (but you can run more scripts in parallel if needed).

HTH

Norbert

The trends puts it in a pretty graph stored in rrd.

On Thu, Apr 4, 2024 at 4:30?AM Schrittenlocher, Rolf <R.Schrittenlocher at ub.uni-frankfurt.de<mailto:R.Schrittenlocher at ub.uni-frankfurt.de>> wrote:

Hi,

kind regards

Rolf

Rolf Schrittenlocher

Bibliotheksmanagementsystem IT | IT-Services (ITS)

Universit?tsbibliothek Johann Christian Senckenberg

Goethe-Universit?t Frankfurt | Campus Bockenheim

Zentralbibliothek | Freimannplatz 1

60325 Frankfurt am Main | GERMANY

Telefon Sammelnummer +49 (0)69 798 28830

Telefon pers?nlich +49 (0)69 798 28908

E-Mail: lbs-it at ub.uni-frankfurt.de<mailto:lbs-it at ub.uni-frankfurt.de>

E-Mail (pers?nlich) r.schrittenlocher at ub.uni-frankfurt.de<mailto:r.schrittenlocher at ub.uni-frankfurt.de>

Website: https://www.ub.uni-frankfurt.de

Hi Rolf,

Schrittenlocher, Rolf schrieb am Thu, Apr 04, 2024 at 07:45:58AM +0000:

...

Our challenge at moment is how to monitor traffic quantity in/out in order to detect suspicious activities on Solaris 10. Is there are way to do this with xymon?

Definitely. ;-)

https://salsa.debian.org/debian/hobbit-plugins/-/blob/master/src/usr/lib/xym...

(It also uses the Hobbit.pm Perl module from the same package: https://salsa.debian.org/debian/hobbit-plugins/-/blob/master/src/usr/share/p...)

            Regards, Axel

Xymon mailing list Xymon at xymon.com<mailto:Xymon at xymon.com> http://lists.xymon.com/mailman/listinfo/xymon

807

Age (days ago)

811

Last active (days ago)

List overview

Download

9 comments

6 participants

participants (6)

abe＠deuxchevaux.org
jeremy.ruffer＠gmail.com
jeremy＠laidman.org
josh＠imaginenetworksllc.com
norkrie＠gmail.com
R.Schrittenlocher＠ub.uni-frankfurt.de

Monitoring network traffic

R.Schrittenlocher＠ub.uni-frankfurt.de

abe＠deuxchevaux.org

R.Schrittenlocher＠ub.uni-frankfurt.de

abe＠deuxchevaux.org

josh＠imaginenetworksllc.com

norkrie＠gmail.com

grab all client info

grab the nic details

grab the route

grab the ports

create the output to send to xymon

R.Schrittenlocher＠ub.uni-frankfurt.de

grab all client info

grab the nic details

grab the route

grab the ports

create the output to send to xymon

jeremy.ruffer＠gmail.com

grab all client info

grab the nic details

grab the route

grab the ports

create the output to send to xymon

jeremy＠laidman.org

grab all client info

grab the nic details

grab the route

grab the ports

create the output to send to xymon

R.Schrittenlocher＠ub.uni-frankfurt.de

grab all client info

grab the nic details

grab the route

grab the ports

create the output to send to xymon

tags

participants (6)