HPE BL460c Gen9 SNMP tests going clear
Hi all
All of our Gen9 servers are flicking between green and clear for Devmon SNMP tests. (Posting here as Devmon seems a dead project.)
The attached screen region grab shows what I'm trying to say.
I've updated the firmware (including iLO) on one server and it made no difference.
Anyone else seen this?
Thanks
CC
On 3/12/19 9:58 PM, Colin Coe wrote:
Hi all
All of our Gen9 servers are flicking between green and clear for Devmon SNMP tests. (Posting here as Devmon seems a dead project.)
The attached screen region grab shows what I'm trying to say.
I've updated the firmware (including iLO) on one server and it made no difference.
Anyone else seen this?
Thanks
CC
Xymon mailing list Xymon at xymon.com http://lists.xymon.com/mailman/listinfo/xymon I see a lot of odd things with devmon. I have seen an issue something like on my Dell R710 with OMSA. It patches itself into snmpd via a loaded library to report on Dell storage and sometimes the response from that get's really slow and blocks response to devmon. An update a while back to OMSA cleared that up.
I also have to run a cron job that does a kill -9 on it every two hours and then restart it... It likes to gobble memory like it's going out of style.
People DO respond on the devmon list, just a little slowly.
I run my devmon in multi node mode with MySQL backing. when I do see gross errors like this I turn on debug and watch the devmon log to see what errors are thrown.
What OS and SNMPD is running on the HP? Especially the SNMPD... It matters. OS X and pFSense don't report some OIDs correctly due to the snmpd they run. It's one of those "snmpd/ya just gotta know about it" things.
Hi Bruce and thanks for the quick response.
These Gen9's are a mix of RHEL6 and Windows 2016, and interestingly, both behave the same. The RHEL boxen are running 'net-snmp' and for Windows 2016 it's the stock WIndows SNMP service. We don't do SNMP traps. These machines also have the HP Service Pack for Proliant installed. The RHEL config can be summarized as:
cat /etc/snmp/snmpd.conf dlmod cmaX /usr/lib64/libcmaX64.so rocommunity secret
Thanks again
On Wed, Mar 13, 2019 at 2:02 PM Bruce Ferrell <bferrell at baywinds.org> wrote:
On 3/12/19 9:58 PM, Colin Coe wrote:
Hi all
All of our Gen9 servers are flicking between green and clear for Devmon SNMP tests. (Posting here as Devmon seems a dead project.)
The attached screen region grab shows what I'm trying to say.
I've updated the firmware (including iLO) on one server and it made no difference.
Anyone else seen this?
Thanks
CC
Xymon mailing list Xymon at xymon.com http://lists.xymon.com/mailman/listinfo/xymon I see a lot of odd things with devmon. I have seen an issue something like on my Dell R710 with OMSA. It patches itself into snmpd via a loaded library to report on Dell storage and sometimes the response from that get's really slow and blocks response to devmon. An update a while back to OMSA cleared that up.
I also have to run a cron job that does a kill -9 on it every two hours and then restart it... It likes to gobble memory like it's going out of style.
People DO respond on the devmon list, just a little slowly.
I run my devmon in multi node mode with MySQL backing. when I do see gross errors like this I turn on debug and watch the devmon log to see what errors are thrown.
What OS and SNMPD is running on the HP? Especially the SNMPD... It matters. OS X and pFSense don't report some OIDs correctly due to the snmpd they run. It's one of those "snmpd/ya just gotta know about it" things.
Colin
Are all your different servers losing numbers all around the same time? The fact that both Windows and Linux systems (with probably different snmpd code) are having the same issues, suggests that it's a devmon problem.
It's been a while since I did any devmon stuff (mine has been running for years but a low node/OID count seems to keep it humming along), but I seem to recall there was some issue with 32-bit vs 64-bit SNMP request/response IDs. What can happen is that a negative 32-bit request ID gets turned into a positive and large 64-bit response ID that is actually the same, but parsed as different by the SNMP library used by devmon. This might be a problem only where there is a 32-bit SNMP library installed on a 64-bit OS, or that might just be a red herring.
If this is a problem, a "tcpdump" with "-v" will show request/response IDs, and you can confirm if you only get responses for positive request IDs.
There's an SNMP_Session API variable you can set in the devmon script to avoid negative request IDs. See https://lists.oetiker.ch/pipermail/mrtg-developers/2002-September/000103.htm... and https://xymon.xymon.narkive.com/6ltXE2nB/devmon-tests-clear-but-snmpwalk-wor... .
Another issue might be the use of SNMPBulkGet, which results in lots of packets in the response, all fragments of a large datagram. One day to disable Bulk requests is to set SNMP version (snmpver) to 1 in your devmon/templates/<devtype>/specs file. Most templates have "snmpver : 2".
The last time I tried to add a bunch of hosts to devmon, it failed, not only for the hosts I added, but also for the hosts that were there before the addition. I ended up reverse-engineering the as-yet-undocumented Xymon SNMP features, and it seems to work well enough. I do sometimes get no response from some hosts from time to time, but it's better than nothing.
In case you wanted to go down this path, it's just a matter of adding entries into /etc/xymon/snmphosts.cfg like this:
[hostname.example.net] version=2 community=secret ip=192.0.2.19 systemmib ifmib=(*) icmpmib hrsystem hrstorage=(*)
The entries after "ip=" line are references to stanzas in the /etc/xymon/snmpmibs.cfg file. There's special black magic in the snmpmibs.cfg file that I haven't worked out, and probably needs code inspection to understand, so I haven't dared to touch this file, with the exception of one apparent typo; diff to fix it is here:
[hrstorage]
storage has data for both memory- and disk-storage
keyidx (HOST-RESOURCES-MIB::hrStorageDescr)keyidx [(HOST-RESOURCES-MIB::hrStorageType]
/rrd:GAUGEkeyidx [HOST-RESOURCES-MIB::hrStorageType] Type = HOST-RESOURCES-MIB::hrStorageType Description = HOST-RESOURCES-MIB::hrStorageDescr Units = HOST-RESOURCES-MIB::hrStorageAllocationUnits
Hmm, perhaps I should try setting "version=1" and see if that helps my occasional missing data...
J
On Wed, 13 Mar 2019 at 17:12, Colin Coe <colin.coe at gmail.com> wrote:
Hi Bruce and thanks for the quick response.
These Gen9's are a mix of RHEL6 and Windows 2016, and interestingly, both behave the same. The RHEL boxen are running 'net-snmp' and for Windows 2016 it's the stock WIndows SNMP service. We don't do SNMP traps. These machines also have the HP Service Pack for Proliant installed. The RHEL config can be summarized as:
cat /etc/snmp/snmpd.conf dlmod cmaX /usr/lib64/libcmaX64.so rocommunity secret
Thanks again
On Wed, Mar 13, 2019 at 2:02 PM Bruce Ferrell <bferrell at baywinds.org> wrote:
On 3/12/19 9:58 PM, Colin Coe wrote:
Hi all
All of our Gen9 servers are flicking between green and clear for Devmon SNMP tests. (Posting here as Devmon seems a dead project.)
The attached screen region grab shows what I'm trying to say.
I've updated the firmware (including iLO) on one server and it made no difference.
Anyone else seen this?
Thanks
CC
Xymon mailing list Xymon at xymon.com http://lists.xymon.com/mailman/listinfo/xymon I see a lot of odd things with devmon. I have seen an issue something like on my Dell R710 with OMSA. It patches itself into snmpd via a loaded library to report on Dell storage and sometimes the response from that get's really slow and blocks response to devmon. An update a while back to OMSA cleared that up.
I also have to run a cron job that does a kill -9 on it every two hours and then restart it... It likes to gobble memory like it's going out of style.
People DO respond on the devmon list, just a little slowly.
I run my devmon in multi node mode with MySQL backing. when I do see gross errors like this I turn on debug and watch the devmon log to see what errors are thrown.
What OS and SNMPD is running on the HP? Especially the SNMPD... It matters. OS X and pFSense don't report some OIDs correctly due to the snmpd they run. It's one of those "snmpd/ya just gotta know about it" things.
Xymon mailing list Xymon at xymon.com http://lists.xymon.com/mailman/listinfo/xymon
Hi Jeremy
Looks like I've had similar issues in the past that I'd completely forgotten about... I've already got "$SNMP_Session::default_avoid_negative_request_ids = 1;" in /usr/share/perl5/SNMP_Session.pm
While multiple servers are affected, they don't always go clear at the same time.
I've changed the snmpver from 2 to 1 but that didn't improve things. having tcpdump running in another ssh session, I can see that requests are being send and responses are being received. I see no negative requests, and ofen the Devmon tests go clear even though SNMP data is received. The devmon.log file reflects devmon's view, not the reality of the SNMP responses.
I'm thinking about using the baked in Xymon SNMP tests but how would I translate the Proliant server tests, for example "compaq-servernohspare/raid"?
Thanks for the pointers
CC
On Wed, Mar 13, 2019 at 6:38 PM Jeremy Laidman <jeremy at laidman.org> wrote:
Colin
Are all your different servers losing numbers all around the same time? The fact that both Windows and Linux systems (with probably different snmpd code) are having the same issues, suggests that it's a devmon problem.
It's been a while since I did any devmon stuff (mine has been running for years but a low node/OID count seems to keep it humming along), but I seem to recall there was some issue with 32-bit vs 64-bit SNMP request/response IDs. What can happen is that a negative 32-bit request ID gets turned into a positive and large 64-bit response ID that is actually the same, but parsed as different by the SNMP library used by devmon. This might be a problem only where there is a 32-bit SNMP library installed on a 64-bit OS, or that might just be a red herring.
If this is a problem, a "tcpdump" with "-v" will show request/response IDs, and you can confirm if you only get responses for positive request IDs.
There's an SNMP_Session API variable you can set in the devmon script to avoid negative request IDs. See https://lists.oetiker.ch/pipermail/mrtg-developers/2002-September/000103.htm... and https://xymon.xymon.narkive.com/6ltXE2nB/devmon-tests-clear-but-snmpwalk-wor... .
Another issue might be the use of SNMPBulkGet, which results in lots of packets in the response, all fragments of a large datagram. One day to disable Bulk requests is to set SNMP version (snmpver) to 1 in your devmon/templates/<devtype>/specs file. Most templates have "snmpver : 2".
The last time I tried to add a bunch of hosts to devmon, it failed, not only for the hosts I added, but also for the hosts that were there before the addition. I ended up reverse-engineering the as-yet-undocumented Xymon SNMP features, and it seems to work well enough. I do sometimes get no response from some hosts from time to time, but it's better than nothing.
In case you wanted to go down this path, it's just a matter of adding entries into /etc/xymon/snmphosts.cfg like this:
[hostname.example.net] version=2 community=secret ip=192.0.2.19 systemmib ifmib=(*) icmpmib hrsystem hrstorage=(*)
The entries after "ip=" line are references to stanzas in the /etc/xymon/snmpmibs.cfg file. There's special black magic in the snmpmibs.cfg file that I haven't worked out, and probably needs code inspection to understand, so I haven't dared to touch this file, with the exception of one apparent typo; diff to fix it is here:
[hrstorage]
storage has data for both memory- and disk-storage
keyidx (HOST-RESOURCES-MIB::hrStorageDescr)
keyidx [(HOST-RESOURCES-MIB::hrStorageType]
keyidx [HOST-RESOURCES-MIB::hrStorageType] Type = HOST-RESOURCES-MIB::hrStorageType Description = HOST-RESOURCES-MIB::hrStorageDescr Units = HOST-RESOURCES-MIB::hrStorageAllocationUnitsHmm, perhaps I should try setting "version=1" and see if that helps my occasional missing data...
J
On Wed, 13 Mar 2019 at 17:12, Colin Coe <colin.coe at gmail.com> wrote:
Hi Bruce and thanks for the quick response.
These Gen9's are a mix of RHEL6 and Windows 2016, and interestingly, both behave the same. The RHEL boxen are running 'net-snmp' and for Windows 2016 it's the stock WIndows SNMP service. We don't do SNMP traps. These machines also have the HP Service Pack for Proliant installed. The RHEL config can be summarized as:
cat /etc/snmp/snmpd.conf dlmod cmaX /usr/lib64/libcmaX64.so rocommunity secret
Thanks again
On Wed, Mar 13, 2019 at 2:02 PM Bruce Ferrell <bferrell at baywinds.org> wrote:
On 3/12/19 9:58 PM, Colin Coe wrote:
Hi all
All of our Gen9 servers are flicking between green and clear for Devmon SNMP tests. (Posting here as Devmon seems a dead project.)
The attached screen region grab shows what I'm trying to say.
I've updated the firmware (including iLO) on one server and it made no difference.
Anyone else seen this?
Thanks
CC
Xymon mailing list Xymon at xymon.com http://lists.xymon.com/mailman/listinfo/xymon I see a lot of odd things with devmon. I have seen an issue something like on my Dell R710 with OMSA. It patches itself into snmpd via a loaded library to report on Dell storage and sometimes the response from that get's really slow and blocks response to devmon. An update a while back to OMSA cleared that up.
I also have to run a cron job that does a kill -9 on it every two hours and then restart it... It likes to gobble memory like it's going out of style.
People DO respond on the devmon list, just a little slowly.
I run my devmon in multi node mode with MySQL backing. when I do see gross errors like this I turn on debug and watch the devmon log to see what errors are thrown.
What OS and SNMPD is running on the HP? Especially the SNMPD... It matters. OS X and pFSense don't report some OIDs correctly due to the snmpd they run. It's one of those "snmpd/ya just gotta know about it" things.
Xymon mailing list Xymon at xymon.com http://lists.xymon.com/mailman/listinfo/xymon
On Thu, 14 Mar 2019 at 14:17, Colin Coe <colin.coe at gmail.com> wrote:
I've changed the snmpver from 2 to 1 but that didn't improve things. having tcpdump running in another ssh session, I can see that requests are being send and responses are being received. I see no negative requests, and ofen the Devmon tests go clear even though SNMP data is received. The devmon.log file reflects devmon's view, not the reality of the SNMP responses.
Hmm, might be a devmon parsing issue.
I'm thinking about using the baked in Xymon SNMP tests but how would I
translate the Proliant server tests, for example "compaq-servernohspare/raid"?
I guess you'd need to configure up a new entry in snmpmibs.conf. I wouldn't know how to do this TBH, on my servers, I let devmon do the RAID stuff and I relieve it from its duty for monitoring interface,ICMP and storage stats.
Sorry, I had meant to add that all our HPE Proliant Gen7s are reporting
fine, its only the Gen9 servers that have this problem.
Curious. I'd be looking at the differences between gen7 and gen9 responses to see if you can detect anything there.
J
On Thu, 14 Mar 2019 at 16:10, Jeremy Laidman <jeremy at laidman.org> wrote:
On Thu, 14 Mar 2019 at 14:17, Colin Coe <colin.coe at gmail.com> wrote:
I'm thinking about using the baked in Xymon SNMP tests but how would I translate the Proliant server tests, for example "compaq-servernohspare/raid"?
I guess you'd need to configure up a new entry in snmpmibs.conf. I wouldn't know how to do this TBH, on my servers, I let devmon do the RAID stuff and I relieve it from its duty for monitoring interface,ICMP and storage stats.
Actually, I cheated with this. I started with the basic stuff in snmpmibs.conf, and had a few things in devmon. For anything critical that wasn't covered, I wrote a shell script that retrieved the client data, parsed it, and created fake devmon status messages.
participants (3)
-
bferrell＠baywinds.org
-
colin.coe＠gmail.com
-
jeremy＠laidman.org