I have a Xymon install that was monitoring (among other things) an HTTPS site. That site moved to a new IP and while I did update the public DNS records it was a few days (or maybe two weeks?) before I remembered to update the internal DNS records. By then, Xymon was reporting red for HTTPS and purple for SSL.
I've corrected the DNS records and the VM running Xymon is resolving it correctly, but now it is still showing the same red and purple alerts. Every now and then, they'll switch to green for a little while and then back to red and purple. I'm not sure why.
Xymon's web interface shows that the certificate is valid for more than 2 months. So that isn't it. I've restarted the Xymon server process and even the whole OS, but this issue remains.
Any ideas on what I could check?
Thanks, Jaime
Hi Jaime
On Thu, 20 Mar 2025 at 02:11, Jaime Kikpole via Xymon <xymon@xymon.com> wrote:
I have a Xymon install that was monitoring (among other things) an HTTPS site. That site moved to a new IP and while I did update the public DNS records it was a few days (or maybe two weeks?) before I remembered to update the internal DNS records. By then, Xymon was reporting red for HTTPS and purple for SSL.
I've corrected the DNS records and the VM running Xymon is resolving it correctly, but now it is still showing the same red and purple alerts. Every now and then, they'll switch to green for a little while and then back to red and purple. I'm not sure why.
Xymon's web interface shows that the certificate is valid for more than 2 months. So that isn't it. I've restarted the Xymon server process and even the whole OS, but this issue remains.
Any ideas on what I could check?
First, review the content of the https status page. There may be non-obvious clues here. There's a difference between "connection refused" and "connection timed out" and this can lead you in different directions to identify the cause. For example, if you have a content check configured ("cont=..." in hosts.cfg), which is failing due to new content on the new webserver, this will be explained on the status page.
Second, I would check that your Xymon server can actually connect to the new IP address on port 443. I usually use "telnet <IP> <port>" and see if it connects, gets a connection refused, or a timeout, but netcat/ncat/nc works too. For an HTTPS website (as distinct from HTTP) I sometimes use an openssl command to see if the TLS/SSL interaction works, which tells me that there's no issue on the webserver itself, something like "openssl s_client -connect <IP>:<port> </dev/null". Even simpler, run curl or wget to fetch a webpage, but if you have a proxy set anywhere, this may give you an invalid result.
Thirdly, I would check the xymonnet.log file for any indications of problems, which might suggest a cause. Having xymonnet in debug mode might help (adding "--debug" into the CMD line of the [xymonnet] block in tasks.cfg). Alternatively, you could run xymonnet directly yourself, and look at the output. Something like this: xymoncmd xymonnet --no-update --debug <hostname>
Good luck with finding the problem.
Cheers Jeremy
Thanks for the ideas. That gave me more to try and I made some progress, but not much.
I discovered that hosts.cfg still had the old IP address. I updated that, but there was no change. I restarted the xymon server process and there was no change. I did a "xymon 127.0.0.1 drop <IP address>" and there was no change. Would you suggest a "xymon 127.0.0.1 drop <FQDN> http" or something like that?
I see what I would expect when using "curl https://<FQDN>" and the logs only show one interesting line within 2025, so I suspect that it all comes down to that one hosts.cfg line with the old IP address.
When I used "grep <server> * | grep 2025 | less", I see the following line and then everything else was from notifications.log about emailing my department's outages notification email address.
history.log:2025-03-18 14:54:12.042214 Will not update /usr/local/www/xymon/data/hist/server,fqdn,redacted.sslcert - color unchanged (purple)
What do you think? Should I just deal with the loss of historical data and drop the host by FQDN?
Jaime Kikpole
Director of Technology Ichabod Crane Central School District (518) 758-7575, x5425
On Wed, Mar 19, 2025 at 7:49 PM Jeremy Laidman <jeremy@laidman.org> wrote:
Hi Jaime
On Thu, 20 Mar 2025 at 02:11, Jaime Kikpole via Xymon <xymon@xymon.com> wrote:
I have a Xymon install that was monitoring (among other things) an HTTPS site. That site moved to a new IP and while I did update the public DNS records it was a few days (or maybe two weeks?) before I remembered to update the internal DNS records. By then, Xymon was reporting red for HTTPS and purple for SSL.
I've corrected the DNS records and the VM running Xymon is resolving it correctly, but now it is still showing the same red and purple alerts. Every now and then, they'll switch to green for a little while and then back to red and purple. I'm not sure why.
Xymon's web interface shows that the certificate is valid for more than 2 months. So that isn't it. I've restarted the Xymon server process and even the whole OS, but this issue remains.
Any ideas on what I could check?
First, review the content of the https status page. There may be non-obvious clues here. There's a difference between "connection refused" and "connection timed out" and this can lead you in different directions to identify the cause. For example, if you have a content check configured ("cont=..." in hosts.cfg), which is failing due to new content on the new webserver, this will be explained on the status page.
Second, I would check that your Xymon server can actually connect to the new IP address on port 443. I usually use "telnet <IP> <port>" and see if it connects, gets a connection refused, or a timeout, but netcat/ncat/nc works too. For an HTTPS website (as distinct from HTTP) I sometimes use an openssl command to see if the TLS/SSL interaction works, which tells me that there's no issue on the webserver itself, something like "openssl s_client -connect <IP>:<port> </dev/null". Even simpler, run curl or wget to fetch a webpage, but if you have a proxy set anywhere, this may give you an invalid result.
Thirdly, I would check the xymonnet.log file for any indications of problems, which might suggest a cause. Having xymonnet in debug mode might help (adding "--debug" into the CMD line of the [xymonnet] block in tasks.cfg). Alternatively, you could run xymonnet directly yourself, and look at the output. Something like this: xymoncmd xymonnet --no-update --debug <hostname>
Good luck with finding the problem.
Cheers Jeremy
The IP in the first column is used in the event the name can't be resolved (or # testip is appended).
The purple test means there is no report for the test (for half an hour I think?) In this case you're testing from the Xymon Server so it is responsible for doing the test.
Are you certain the name of the host matches? Are there any ghosts?
Are you looking at the "sslcert" column? What does your hosts.cfg line look like - does it have an https url?
On Thu, Mar 20, 2025 at 3:38 PM Jaime Kikpole via Xymon <xymon@xymon.com> wrote:
Thanks for the ideas. That gave me more to try and I made some progress, but not much.
I discovered that hosts.cfg still had the old IP address. I updated that, but there was no change. I restarted the xymon server process and there was no change. I did a "xymon 127.0.0.1 drop <IP address>" and there was no change. Would you suggest a "xymon 127.0.0.1 drop <FQDN> http" or something like that?
I see what I would expect when using "curl https://<FQDN>" and the logs only show one interesting line within 2025, so I suspect that it all comes down to that one hosts.cfg line with the old IP address.
When I used "grep <server> * | grep 2025 | less", I see the following line and then everything else was from notifications.log about emailing my department's outages notification email address.
history.log:2025-03-18 14:54:12.042214 Will not update /usr/local/www/xymon/data/hist/server,fqdn,redacted.sslcert - color unchanged (purple)
What do you think? Should I just deal with the loss of historical data and drop the host by FQDN?
Jaime Kikpole
Director of Technology Ichabod Crane Central School District (518) 758-7575, x5425
On Wed, Mar 19, 2025 at 7:49 PM Jeremy Laidman <jeremy@laidman.org> wrote:
Hi Jaime
On Thu, 20 Mar 2025 at 02:11, Jaime Kikpole via Xymon <xymon@xymon.com> wrote:
I have a Xymon install that was monitoring (among other things) an HTTPS site. That site moved to a new IP and while I did update the public DNS records it was a few days (or maybe two weeks?) before I remembered to update the internal DNS records. By then, Xymon was reporting red for HTTPS and purple for SSL.
I've corrected the DNS records and the VM running Xymon is resolving it correctly, but now it is still showing the same red and purple alerts. Every now and then, they'll switch to green for a little while and then back to red and purple. I'm not sure why.
Xymon's web interface shows that the certificate is valid for more than 2 months. So that isn't it. I've restarted the Xymon server process and even the whole OS, but this issue remains.
Any ideas on what I could check?
First, review the content of the https status page. There may be non-obvious clues here. There's a difference between "connection refused" and "connection timed out" and this can lead you in different directions to identify the cause. For example, if you have a content check configured ("cont=..." in hosts.cfg), which is failing due to new content on the new webserver, this will be explained on the status page.
Second, I would check that your Xymon server can actually connect to the new IP address on port 443. I usually use "telnet <IP> <port>" and see if it connects, gets a connection refused, or a timeout, but netcat/ncat/nc works too. For an HTTPS website (as distinct from HTTP) I sometimes use an openssl command to see if the TLS/SSL interaction works, which tells me that there's no issue on the webserver itself, something like "openssl s_client -connect <IP>:<port> </dev/null". Even simpler, run curl or wget to fetch a webpage, but if you have a proxy set anywhere, this may give you an invalid result.
Thirdly, I would check the xymonnet.log file for any indications of problems, which might suggest a cause. Having xymonnet in debug mode might help (adding "--debug" into the CMD line of the [xymonnet] block in tasks.cfg). Alternatively, you could run xymonnet directly yourself, and look at the output. Something like this: xymoncmd xymonnet --no-update --debug <hostname>
Good luck with finding the problem.
Cheers Jeremy
Xymon mailing list -- xymon@xymon.com To unsubscribe send an email to xymon-leave@xymon.com
From hosts.cfg, with redactions between < and >: <IP address> <hostname>.ichabodcrane.org # route:10.5.0.1 https:// <hostname>.ichabodcrane.org noconn
The ghosts report shows no clients.
The hostname definitely resolves (tested on the VM with the xymon server) to the IP address. The purple alert is in a column labeled "sslcert" and the red is in "http". The data in the page with the details about sslcert shows the correct dates and a green alert. The data in the page with the http details shows the message "SSL error" after the https://.... URL. When viewing that URL, it redirects path on the same server. This hasn't given me trouble over the last several years and the only thing that changed on the monitored system is that its IP address changed at the VM hosting service.
Jaime Kikpole
Director of Technology Ichabod Crane Central School District (518) 758-7575, x5425
On Thu, Mar 20, 2025 at 4:33 PM Josh Luthman <josh@imaginenetworksllc.com> wrote:
The IP in the first column is used in the event the name can't be resolved (or # testip is appended).
The purple test means there is no report for the test (for half an hour I think?) In this case you're testing from the Xymon Server so it is responsible for doing the test.
Are you certain the name of the host matches? Are there any ghosts?
Are you looking at the "sslcert" column? What does your hosts.cfg line look like - does it have an https url?
On Thu, Mar 20, 2025 at 3:38 PM Jaime Kikpole via Xymon <xymon@xymon.com> wrote:
Thanks for the ideas. That gave me more to try and I made some progress, but not much.
I discovered that hosts.cfg still had the old IP address. I updated that, but there was no change. I restarted the xymon server process and there was no change. I did a "xymon 127.0.0.1 drop <IP address>" and there was no change. Would you suggest a "xymon 127.0.0.1 drop <FQDN> http" or something like that?
I see what I would expect when using "curl https://<FQDN>" and the logs only show one interesting line within 2025, so I suspect that it all comes down to that one hosts.cfg line with the old IP address.
When I used "grep <server> * | grep 2025 | less", I see the following line and then everything else was from notifications.log about emailing my department's outages notification email address.
history.log:2025-03-18 14:54:12.042214 Will not update /usr/local/www/xymon/data/hist/server,fqdn,redacted.sslcert - color unchanged (purple)
What do you think? Should I just deal with the loss of historical data and drop the host by FQDN?
Jaime Kikpole
Director of Technology Ichabod Crane Central School District (518) 758-7575, x5425
On Wed, Mar 19, 2025 at 7:49 PM Jeremy Laidman <jeremy@laidman.org> wrote:
Hi Jaime
On Thu, 20 Mar 2025 at 02:11, Jaime Kikpole via Xymon <xymon@xymon.com> wrote:
I have a Xymon install that was monitoring (among other things) an HTTPS site. That site moved to a new IP and while I did update the public DNS records it was a few days (or maybe two weeks?) before I remembered to update the internal DNS records. By then, Xymon was reporting red for HTTPS and purple for SSL.
I've corrected the DNS records and the VM running Xymon is resolving it correctly, but now it is still showing the same red and purple alerts. Every now and then, they'll switch to green for a little while and then back to red and purple. I'm not sure why.
Xymon's web interface shows that the certificate is valid for more than 2 months. So that isn't it. I've restarted the Xymon server process and even the whole OS, but this issue remains.
Any ideas on what I could check?
First, review the content of the https status page. There may be non-obvious clues here. There's a difference between "connection refused" and "connection timed out" and this can lead you in different directions to identify the cause. For example, if you have a content check configured ("cont=..." in hosts.cfg), which is failing due to new content on the new webserver, this will be explained on the status page.
Second, I would check that your Xymon server can actually connect to the new IP address on port 443. I usually use "telnet <IP> <port>" and see if it connects, gets a connection refused, or a timeout, but netcat/ncat/nc works too. For an HTTPS website (as distinct from HTTP) I sometimes use an openssl command to see if the TLS/SSL interaction works, which tells me that there's no issue on the webserver itself, something like "openssl s_client -connect <IP>:<port> </dev/null". Even simpler, run curl or wget to fetch a webpage, but if you have a proxy set anywhere, this may give you an invalid result.
Thirdly, I would check the xymonnet.log file for any indications of problems, which might suggest a cause. Having xymonnet in debug mode might help (adding "--debug" into the CMD line of the [xymonnet] block in tasks.cfg). Alternatively, you could run xymonnet directly yourself, and look at the output. Something like this: xymoncmd xymonnet --no-update --debug <hostname>
Good luck with finding the problem.
Cheers Jeremy
Xymon mailing list -- xymon@xymon.com To unsubscribe send an email to xymon-leave@xymon.com
Why is the http test red?
On Thu, Mar 20, 2025 at 4:46 PM Jaime Kikpole <jkikpole@ichabodcrane.org> wrote:
From hosts.cfg, with redactions between < and >: <IP address> <hostname>.ichabodcrane.org # route:10.5.0.1 https:// <hostname>.ichabodcrane.org noconn
The ghosts report shows no clients.
The hostname definitely resolves (tested on the VM with the xymon server) to the IP address. The purple alert is in a column labeled "sslcert" and the red is in "http". The data in the page with the details about sslcert shows the correct dates and a green alert. The data in the page with the http details shows the message "SSL error" after the https://.... URL. When viewing that URL, it redirects path on the same server. This hasn't given me trouble over the last several years and the only thing that changed on the monitored system is that its IP address changed at the VM hosting service.
Jaime Kikpole
Director of Technology Ichabod Crane Central School District (518) 758-7575, x5425
On Thu, Mar 20, 2025 at 4:33 PM Josh Luthman <josh@imaginenetworksllc.com> wrote:
The IP in the first column is used in the event the name can't be resolved (or # testip is appended).
The purple test means there is no report for the test (for half an hour I think?) In this case you're testing from the Xymon Server so it is responsible for doing the test.
Are you certain the name of the host matches? Are there any ghosts?
Are you looking at the "sslcert" column? What does your hosts.cfg line look like - does it have an https url?
On Thu, Mar 20, 2025 at 3:38 PM Jaime Kikpole via Xymon <xymon@xymon.com> wrote:
Thanks for the ideas. That gave me more to try and I made some progress, but not much.
I discovered that hosts.cfg still had the old IP address. I updated that, but there was no change. I restarted the xymon server process and there was no change. I did a "xymon 127.0.0.1 drop <IP address>" and there was no change. Would you suggest a "xymon 127.0.0.1 drop <FQDN> http" or something like that?
I see what I would expect when using "curl https://<FQDN>" and the logs only show one interesting line within 2025, so I suspect that it all comes down to that one hosts.cfg line with the old IP address.
When I used "grep <server> * | grep 2025 | less", I see the following line and then everything else was from notifications.log about emailing my department's outages notification email address.
history.log:2025-03-18 14:54:12.042214 Will not update /usr/local/www/xymon/data/hist/server,fqdn,redacted.sslcert - color unchanged (purple)
What do you think? Should I just deal with the loss of historical data and drop the host by FQDN?
Jaime Kikpole
Director of Technology Ichabod Crane Central School District (518) 758-7575, x5425
On Wed, Mar 19, 2025 at 7:49 PM Jeremy Laidman <jeremy@laidman.org> wrote:
Hi Jaime
On Thu, 20 Mar 2025 at 02:11, Jaime Kikpole via Xymon <xymon@xymon.com> wrote:
I have a Xymon install that was monitoring (among other things) an HTTPS site. That site moved to a new IP and while I did update the public DNS records it was a few days (or maybe two weeks?) before I remembered to update the internal DNS records. By then, Xymon was reporting red for HTTPS and purple for SSL.
I've corrected the DNS records and the VM running Xymon is resolving it correctly, but now it is still showing the same red and purple alerts. Every now and then, they'll switch to green for a little while and then back to red and purple. I'm not sure why.
Xymon's web interface shows that the certificate is valid for more than 2 months. So that isn't it. I've restarted the Xymon server process and even the whole OS, but this issue remains.
Any ideas on what I could check?
First, review the content of the https status page. There may be non-obvious clues here. There's a difference between "connection refused" and "connection timed out" and this can lead you in different directions to identify the cause. For example, if you have a content check configured ("cont=..." in hosts.cfg), which is failing due to new content on the new webserver, this will be explained on the status page.
Second, I would check that your Xymon server can actually connect to the new IP address on port 443. I usually use "telnet <IP> <port>" and see if it connects, gets a connection refused, or a timeout, but netcat/ncat/nc works too. For an HTTPS website (as distinct from HTTP) I sometimes use an openssl command to see if the TLS/SSL interaction works, which tells me that there's no issue on the webserver itself, something like "openssl s_client -connect <IP>:<port> </dev/null". Even simpler, run curl or wget to fetch a webpage, but if you have a proxy set anywhere, this may give you an invalid result.
Thirdly, I would check the xymonnet.log file for any indications of problems, which might suggest a cause. Having xymonnet in debug mode might help (adding "--debug" into the CMD line of the [xymonnet] block in tasks.cfg). Alternatively, you could run xymonnet directly yourself, and look at the output. Something like this: xymoncmd xymonnet --no-update --debug <hostname>
Good luck with finding the problem.
Cheers Jeremy
Xymon mailing list -- xymon@xymon.com To unsubscribe send an email to xymon-leave@xymon.com
Josh's point about how the IP address used for probes comes from DNS rather than hosts.cfg is a good one. It would be worth adding "testip" to the hosts.cfg line, to isolate any issues with DNS. (Most of the servers I monitor are DNS servers, so they all have "testip" because I need to be able to monitor all hosts even if the DNS service is broken.)
Changes to hosts.cfg should be picked up automatically, although not immediately. The hosts.cfg file is checked for updates every 5 minutes by default. You can see when this happens because the xymond.log says "Reloading hostnames". You can trigger an immediate reload of the hosts.cfg file by sending a HUP signal to the xymond process (eg "sudo -u xymon pkill -HUP -x xymond"). All other Xymon processes such as xymonnet (unless configured otherwise) get their list of hosts from interactions with xymond, so as long as xymond has reloaded the contents of hosts.cfg, any updates should be live everywhere.
You can verify that changes to the hosts.cfg file have been picked up by asking the xymond process to show you what it has: xymon 127.1 "hostinfo host=hostname"
The "drop" command lines you provided don't have quotes after 127.0.0.1. The xymon command takes at most two arguments, so you need to wrap the "drop hostname" in quotes so that it's treated as a single argument:
$ xymon 127.0.0.1 "drop hostname sslcert"
Also, the hostname must match the second field in hosts.cfg, so you can't use an IP address.
It should be sufficient to drop the "http" and "sslcert" test status, rather than all statuses.
One last thought is that I had an issue with a stuck status once, which I had to remedy by modifying the xymond checkpoint file. The xymond process periodically dumps its state to this file, so that if it crashes, it can restore the state. The file is called xymond.chk. In my case, I recall there was a state that had a timestamp in the future, or something silly like that. It might be interesting to search for the hostname in that file, and see what the state looks like.
To clear a faulty state, you'd need to stop the xymond process (probably by stopping all of Xymon) and then edit/delete the state file, and then start the xymond process back up again. Editing the host's entry in the file would be less disruptive than deleting the whole file, but the fields aren't well documented so the simplest thing would be to find and delete the line with the troublesome hostname.
J
On Fri, 21 Mar 2025 at 07:56, Josh Luthman <josh@imaginenetworksllc.com> wrote:
Why is the http test red?
On Thu, Mar 20, 2025 at 4:46 PM Jaime Kikpole <jkikpole@ichabodcrane.org> wrote:
From hosts.cfg, with redactions between < and >: <IP address> <hostname>.ichabodcrane.org # route:10.5.0.1 https:// <hostname>.ichabodcrane.org noconn
The ghosts report shows no clients.
The hostname definitely resolves (tested on the VM with the xymon server) to the IP address. The purple alert is in a column labeled "sslcert" and the red is in "http". The data in the page with the details about sslcert shows the correct dates and a green alert. The data in the page with the http details shows the message "SSL error" after the https://.... URL. When viewing that URL, it redirects path on the same server. This hasn't given me trouble over the last several years and the only thing that changed on the monitored system is that its IP address changed at the VM hosting service.
Jaime Kikpole
Director of Technology Ichabod Crane Central School District (518) 758-7575, x5425
On Thu, Mar 20, 2025 at 4:33 PM Josh Luthman <josh@imaginenetworksllc.com> wrote:
The IP in the first column is used in the event the name can't be resolved (or # testip is appended).
The purple test means there is no report for the test (for half an hour I think?) In this case you're testing from the Xymon Server so it is responsible for doing the test.
Are you certain the name of the host matches? Are there any ghosts?
Are you looking at the "sslcert" column? What does your hosts.cfg line look like - does it have an https url?
On Thu, Mar 20, 2025 at 3:38 PM Jaime Kikpole via Xymon <xymon@xymon.com> wrote:
Thanks for the ideas. That gave me more to try and I made some progress, but not much.
I discovered that hosts.cfg still had the old IP address. I updated that, but there was no change. I restarted the xymon server process and there was no change. I did a "xymon 127.0.0.1 drop <IP address>" and there was no change. Would you suggest a "xymon 127.0.0.1 drop <FQDN> http" or something like that?
I see what I would expect when using "curl https://<FQDN>" and the logs only show one interesting line within 2025, so I suspect that it all comes down to that one hosts.cfg line with the old IP address.
When I used "grep <server> * | grep 2025 | less", I see the following line and then everything else was from notifications.log about emailing my department's outages notification email address.
history.log:2025-03-18 14:54:12.042214 Will not update /usr/local/www/xymon/data/hist/server,fqdn,redacted.sslcert - color unchanged (purple)
What do you think? Should I just deal with the loss of historical data and drop the host by FQDN?
Jaime Kikpole
Director of Technology Ichabod Crane Central School District (518) 758-7575, x5425
On Wed, Mar 19, 2025 at 7:49 PM Jeremy Laidman <jeremy@laidman.org> wrote:
Hi Jaime
On Thu, 20 Mar 2025 at 02:11, Jaime Kikpole via Xymon <xymon@xymon.com> wrote:
I have a Xymon install that was monitoring (among other things) an HTTPS site. That site moved to a new IP and while I did update the public DNS records it was a few days (or maybe two weeks?) before I remembered to update the internal DNS records. By then, Xymon was reporting red for HTTPS and purple for SSL.
I've corrected the DNS records and the VM running Xymon is resolving it correctly, but now it is still showing the same red and purple alerts. Every now and then, they'll switch to green for a little while and then back to red and purple. I'm not sure why.
Xymon's web interface shows that the certificate is valid for more than 2 months. So that isn't it. I've restarted the Xymon server process and even the whole OS, but this issue remains.
Any ideas on what I could check?
First, review the content of the https status page. There may be non-obvious clues here. There's a difference between "connection refused" and "connection timed out" and this can lead you in different directions to identify the cause. For example, if you have a content check configured ("cont=..." in hosts.cfg), which is failing due to new content on the new webserver, this will be explained on the status page.
Second, I would check that your Xymon server can actually connect to the new IP address on port 443. I usually use "telnet <IP> <port>" and see if it connects, gets a connection refused, or a timeout, but netcat/ncat/nc works too. For an HTTPS website (as distinct from HTTP) I sometimes use an openssl command to see if the TLS/SSL interaction works, which tells me that there's no issue on the webserver itself, something like "openssl s_client -connect <IP>:<port> </dev/null". Even simpler, run curl or wget to fetch a webpage, but if you have a proxy set anywhere, this may give you an invalid result.
Thirdly, I would check the xymonnet.log file for any indications of problems, which might suggest a cause. Having xymonnet in debug mode might help (adding "--debug" into the CMD line of the [xymonnet] block in tasks.cfg). Alternatively, you could run xymonnet directly yourself, and look at the output. Something like this: xymoncmd xymonnet --no-update --debug <hostname>
Good luck with finding the problem.
Cheers Jeremy
Xymon mailing list -- xymon@xymon.com To unsubscribe send an email to xymon-leave@xymon.com
Hi,
If I remember correctly: If the http test is red because Xymon cannot connect to the server on port 443, then Xymon will never retrieve the SSL certificate (no connection = no SSL data). This will cause the sslcert status to be purple, because it cannot be updated.
Med venlig hilsen / Best regards
Henrik Juul Størner Senior IT Arkitekt - IT-sikkerhed
Fra: Josh Luthman <josh@imaginenetworksllc.com> Sendt: 20. marts 2025 21:56 Til: Jaime Kikpole <jkikpole@ichabodcrane.org> Cc: Xymon mailinglist <xymon@xymon.com>; Jeremy Laidman <jeremy@laidman.org> Emne: [Xymon] Re: SSL stuck on purple
OBS: Denne e-mail er ikke fra en Danske Spil-afsender! Brug kun links eller vedhæftede filer, hvis du mener, de er OK.
Why is the http test red?
On Thu, Mar 20, 2025 at 4:46 PM Jaime Kikpole <jkikpole@ichabodcrane.org<mailto:jkikpole@ichabodcrane.org>> wrote: From hosts.cfg, with redactions between < and >: <IP address> <hostname>.ichabodcrane.org<http://ichabodcrane.org> # route:10.5.0.1 https://<hostname>.ichabodcrane.org<http://ichabodcrane.org> noconn
The ghosts report shows no clients.
The hostname definitely resolves (tested on the VM with the xymon server) to the IP address. The purple alert is in a column labeled "sslcert" and the red is in "http". The data in the page with the details about sslcert shows the correct dates and a green alert. The data in the page with the http details shows the message "SSL error" after the https://.... URL. When viewing that URL, it redirects path on the same server. This hasn't given me trouble over the last several years and the only thing that changed on the monitored system is that its IP address changed at the VM hosting service.
Jaime Kikpole
Director of Technology Ichabod Crane Central School District (518) 758-7575, x5425
On Thu, Mar 20, 2025 at 4:33 PM Josh Luthman <josh@imaginenetworksllc.com<mailto:josh@imaginenetworksllc.com>> wrote: The IP in the first column is used in the event the name can't be resolved (or # testip is appended).
The purple test means there is no report for the test (for half an hour I think?) In this case you're testing from the Xymon Server so it is responsible for doing the test.
Are you certain the name of the host matches? Are there any ghosts?
Are you looking at the "sslcert" column? What does your hosts.cfg line look like - does it have an https url?
On Thu, Mar 20, 2025 at 3:38 PM Jaime Kikpole via Xymon <xymon@xymon.com<mailto:xymon@xymon.com>> wrote: Thanks for the ideas. That gave me more to try and I made some progress, but not much.
I discovered that hosts.cfg still had the old IP address. I updated that, but there was no change. I restarted the xymon server process and there was no change. I did a "xymon 127.0.0.1 drop <IP address>" and there was no change. Would you suggest a "xymon 127.0.0.1 drop <FQDN> http" or something like that?
I see what I would expect when using "curl https://<FQDN>" and the logs only show one interesting line within 2025, so I suspect that it all comes down to that one hosts.cfg line with the old IP address.
When I used "grep <server> * | grep 2025 | less", I see the following line and then everything else was from notifications.log about emailing my department's outages notification email address.
history.log:2025-03-18 14:54:12.042214 Will not update /usr/local/www/xymon/data/hist/server,fqdn,redacted.sslcert - color unchanged (purple)
What do you think? Should I just deal with the loss of historical data and drop the host by FQDN?
Jaime Kikpole
Director of Technology Ichabod Crane Central School District (518) 758-7575, x5425
On Wed, Mar 19, 2025 at 7:49 PM Jeremy Laidman <jeremy@laidman.org<mailto:jeremy@laidman.org>> wrote: Hi Jaime<mailto:Hi%20Jaime%0d%0dOn%20Thu,%20> <mailto:Hi%20Jaime%0d%0dOn%20Thu,%20> On Thu, <mailto:Hi%20Jaime%0d%0dOn%20Thu,%20> 20 Mar 2025 at 02:11, Jaime Kikpole via Xymon <xymon@xymon.com<mailto:xymon@xymon.com>> wrote: I have a Xymon ins<mailto:ave%20a%20Xymon%20ins>tall that was monitoring (among other things) an HTTPS site. That site moved to a new IP and while I did update the public DNS records it was a few days (or maybe two weeks?) before I remembered to update the internal DNS records. By then, Xymon was reporting red for HTTPS and purple for SSL.
I've corrected the DNS records and the VM running Xymon is resolving it correctly, but now it is still showing the same red and purple alerts. Every now and then, they'll switch to green for a little while and then back to red and purple. I'm not sure why.
Xymon's web interface shows that the certificate is valid for more than 2 months. So that isn't it. I've restarted the Xymon server process and even the whole OS, but this issue remains.
Any ideas on what I could check?
First, review the content of the https status page. There may be non-obvious clues here. There's a difference between "connection refused" and "connection timed out" and this can lead you in different directions to identify the cause. For example, if you have a content check configured ("cont=..." in hosts.cfg), which is failing due to new content on the new webserver, this will be explained on the status page.
Second, I would check that your Xymon server can actually connect to the new IP address on port 443. I usually use "telnet <IP> <port>" and see if it connects, gets a connection refused, or a timeout, but netcat/ncat/nc works too. For an HTTPS website (as distinct from HTTP) I sometimes use an openssl command to see if the TLS/SSL interaction works, which tells me that there's no issue on the webserver itself, something like "openssl s_client -connect <IP>:<port> </dev/null". Even simpler, run curl or wget to fetch a webpage, but if you have a proxy set anywhere, this may give you an invalid result.
Thirdly, I would check the xymonnet.log file for any indications of problems, which might suggest a cause. Having xymonnet in debug mode might help (adding "--debug" into the CMD line of the [xymonnet] block in tasks.cfg). Alternatively, you could run xymonnet directly yourself, and look at the output. Something like this: xymoncmd xymonnet --no-update --debug <hostname>
Good luck with finding the problem.
Cheers Jeremy
Xymon mailing list -- xymon@xymon.com<mailto:xymon@xymon.com> To unsubscribe send an ema<mailto:ibe%20send%20an%20ema>il to xymon-leave@xymon.com<mailto:xymon-leave@xymon.com>
The sslcert detail oage is green because it was last time it was retrieved. There should be a timestamp for the last update
Nicola
Il giorno 21 mar 2025, 10:01, alle ore 10:01, "Henrik Juul Størner via Xymon" <xymon@xymon.com> ha scritto:
Hi,
If I remember correctly: If the http test is red because Xymon cannot connect to the server on port 443, then Xymon will never retrieve the SSL certificate (no connection = no SSL data). This will cause the sslcert status to be purple, because it cannot be updated.
Med venlig hilsen / Best regards
Henrik Juul Størner Senior IT Arkitekt - IT-sikkerhed
Fra: Josh Luthman <josh@imaginenetworksllc.com> Sendt: 20. marts 2025 21:56 Til: Jaime Kikpole <jkikpole@ichabodcrane.org> Cc: Xymon mailinglist <xymon@xymon.com>; Jeremy Laidman <jeremy@laidman.org> Emne: [Xymon] Re: SSL stuck on purple
OBS: Denne e-mail er ikke fra en Danske Spil-afsender! Brug kun links eller vedhæftede filer, hvis du mener, de er OK.
Why is the http test red?
On Thu, Mar 20, 2025 at 4:46 PM Jaime Kikpole <jkikpole@ichabodcrane.org<mailto:jkikpole@ichabodcrane.org>> wrote: From hosts.cfg, with redactions between < and >: <IP address> <hostname>.ichabodcrane.org<http://ichabodcrane.org> # route:10.5.0.1 https://<hostname>.ichabodcrane.org<http://ichabodcrane.org> noconn
The ghosts report shows no clients.
The hostname definitely resolves (tested on the VM with the xymon server) to the IP address. The purple alert is in a column labeled "sslcert" and the red is in "http". The data in the page with the details about sslcert shows the correct dates and a green alert. The data in the page with the http details shows the message "SSL error" after the https://.... URL. When viewing that URL, it redirects path on the same server. This hasn't given me trouble over the last several years and the only thing that changed on the monitored system is that its IP address changed at the VM hosting service.
Jaime Kikpole
Director of Technology Ichabod Crane Central School District (518) 758-7575, x5425
On Thu, Mar 20, 2025 at 4:33 PM Josh Luthman <josh@imaginenetworksllc.com<mailto:josh@imaginenetworksllc.com>> wrote: The IP in the first column is used in the event the name can't be resolved (or # testip is appended).
The purple test means there is no report for the test (for half an hour I think?) In this case you're testing from the Xymon Server so it is responsible for doing the test.
Are you certain the name of the host matches? Are there any ghosts?
Are you looking at the "sslcert" column? What does your hosts.cfg line look like - does it have an https url?
On Thu, Mar 20, 2025 at 3:38 PM Jaime Kikpole via Xymon <xymon@xymon.com<mailto:xymon@xymon.com>> wrote: Thanks for the ideas. That gave me more to try and I made some progress, but not much.
I discovered that hosts.cfg still had the old IP address. I updated that, but there was no change. I restarted the xymon server process and there was no change. I did a "xymon 127.0.0.1 drop <IP address>" and there was no change. Would you suggest a "xymon 127.0.0.1 drop <FQDN> http" or something like that?
I see what I would expect when using "curl https://<FQDN>" and the logs only show one interesting line within 2025, so I suspect that it all comes down to that one hosts.cfg line with the old IP address.
When I used "grep <server> * | grep 2025 | less", I see the following line and then everything else was from notifications.log about emailing my department's outages notification email address.
history.log:2025-03-18 14:54:12.042214 Will not update /usr/local/www/xymon/data/hist/server,fqdn,redacted.sslcert - color unchanged (purple)
What do you think? Should I just deal with the loss of historical data and drop the host by FQDN?
Jaime Kikpole
Director of Technology Ichabod Crane Central School District (518) 758-7575, x5425
On Wed, Mar 19, 2025 at 7:49 PM Jeremy Laidman <jeremy@laidman.org<mailto:jeremy@laidman.org>> wrote: Hi Jaime<mailto:Hi%20Jaime%0d%0dOn%20Thu,%20> <mailto:Hi%20Jaime%0d%0dOn%20Thu,%20> On Thu, <mailto:Hi%20Jaime%0d%0dOn%20Thu,%20> 20 Mar 2025 at 02:11, Jaime Kikpole via Xymon <xymon@xymon.com<mailto:xymon@xymon.com>> wrote: I have a Xymon ins<mailto:ave%20a%20Xymon%20ins>tall that was monitoring (among other things) an HTTPS site. That site moved to a new IP and while I did update the public DNS records it was a few days (or maybe two weeks?) before I remembered to update the internal DNS records. By then, Xymon was reporting red for HTTPS and purple for SSL.
I've corrected the DNS records and the VM running Xymon is resolving it correctly, but now it is still showing the same red and purple alerts. Every now and then, they'll switch to green for a little while and then back to red and purple. I'm not sure why.
Xymon's web interface shows that the certificate is valid for more than 2 months. So that isn't it. I've restarted the Xymon server process and even the whole OS, but this issue remains.
Any ideas on what I could check?
First, review the content of the https status page. There may be non-obvious clues here. There's a difference between "connection refused" and "connection timed out" and this can lead you in different directions to identify the cause. For example, if you have a content check configured ("cont=..." in hosts.cfg), which is failing due to new content on the new webserver, this will be explained on the status page.
Second, I would check that your Xymon server can actually connect to the new IP address on port 443. I usually use "telnet <IP> <port>" and see if it connects, gets a connection refused, or a timeout, but netcat/ncat/nc works too. For an HTTPS website (as distinct from HTTP) I sometimes use an openssl command to see if the TLS/SSL interaction works, which tells me that there's no issue on the webserver itself, something like "openssl s_client -connect <IP>:<port> </dev/null". Even simpler, run curl or wget to fetch a webpage, but if you have a proxy set anywhere, this may give you an invalid result.
Thirdly, I would check the xymonnet.log file for any indications of problems, which might suggest a cause. Having xymonnet in debug mode might help (adding "--debug" into the CMD line of the [xymonnet] block in tasks.cfg). Alternatively, you could run xymonnet directly yourself, and look at the output. Something like this: xymoncmd xymonnet --no-update --debug <hostname>
Good luck with finding the problem.
Cheers Jeremy
Xymon mailing list -- xymon@xymon.com<mailto:xymon@xymon.com> To unsubscribe send an ema<mailto:ibe%20send%20an%20ema>il to xymon-leave@xymon.com<mailto:xymon-leave@xymon.com>
Xymon mailing list -- xymon@xymon.com To unsubscribe send an email to xymon-leave@xymon.com
The SSL and http tests are not directly connected. The http retrieve a page, which will return red if not found, and the SSL retrieve the SSL certificate. So the SSL can be green and the http red and vis versa.
On Fri, 21 Mar 2025, 10:18 Nicola, <canne74@gmail.com> wrote:
The sslcert detail oage is green because it was last time it was retrieved. There should be a timestamp for the last update
Nicola Il giorno 21 mar 2025, alle ore 10:01, "Henrik Juul Størner via Xymon" < xymon@xymon.com> ha scritto:
Hi,
If I remember correctly: If the http test is red because Xymon cannot connect to the server on port 443, then Xymon will never retrieve the SSL certificate (no connection = no SSL data). This will cause the sslcert status to be purple, because it cannot be updated.
Med venlig hilsen / Best regards
*Henrik Juul Størner* Senior IT Arkitekt - IT-sikkerhed
*Fra:* Josh Luthman <josh@imaginenetworksllc.com> *Sendt:* 20. marts 2025 21:56 *Til:* Jaime Kikpole <jkikpole@ichabodcrane.org> *Cc:* Xymon mailinglist <xymon@xymon.com>; Jeremy Laidman < jeremy@laidman.org> *Emne:* [Xymon] Re: SSL stuck on purple
OBS: Denne e-mail er *ikke* fra en Danske Spil-afsender! Brug kun links eller vedhæftede filer, hvis du mener, de er OK.
Why is the http test red?
On Thu, Mar 20, 2025 at 4:46 PM Jaime Kikpole <jkikpole@ichabodcrane.org> wrote:
From hosts.cfg, with redactions between < and >:
<IP address> <hostname>.ichabodcrane.org # route:10.5.0.1 https:// <hostname>.ichabodcrane.org noconn
The ghosts report shows no clients.
The hostname definitely resolves (tested on the VM with the xymon server) to the IP address. The purple alert is in a column labeled "sslcert" and the red is in "http". The data in the page with the details about sslcert shows the correct dates and a green alert. The data in the page with the http details shows the message "SSL error" after the https://.... URL. When viewing that URL, it redirects path on the same server. This hasn't given me trouble over the last several years and the only thing that changed on the monitored system is that its IP address changed at the VM hosting service.
*Jaime Kikpole*
*Director of Technology* *Ichabod Crane Central School District* (518) 758-7575, x5425
On Thu, Mar 20, 2025 at 4:33 PM Josh Luthman <josh@imaginenetworksllc.com> wrote:
The IP in the first column is used in the event the name can't be resolved (or # testip is appended).
The purple test means there is no report for the test (for half an hour I think?) In this case you're testing from the Xymon Server so it is responsible for doing the test.
Are you certain the name of the host matches? Are there any ghosts?
Are you looking at the "sslcert" column? What does your hosts.cfg line look like - does it have an https url?
On Thu, Mar 20, 2025 at 3:38 PM Jaime Kikpole via Xymon <xymon@xymon.com> wrote:
Thanks for the ideas. That gave me more to try and I made some progress, but not much.
I discovered that hosts.cfg still had the old IP address. I updated that, but there was no change. I restarted the xymon server process and there was no change. I did a "xymon 127.0.0.1 drop <IP address>" and there was no change. Would you suggest a "xymon 127.0.0.1 drop <FQDN> http" or something like that?
I see what I would expect when using "curl https://<FQDN>" and the logs only show one interesting line within 2025, so I suspect that it all comes down to that one hosts.cfg line with the old IP address.
When I used "grep <server> * | grep 2025 | less", I see the following line and then everything else was from notifications.log about emailing my department's outages notification email address.
history.log:2025-03-18 14:54:12.042214 Will not update /usr/local/www/xymon/data/hist/server,fqdn,redacted.sslcert - color unchanged (purple)
What do you think? Should I just deal with the loss of historical data and drop the host by FQDN?
*Jaime Kikpole*
*Director of Technology* *Ichabod Crane Central School District* (518) 758-7575, x5425
On Wed, Mar 19, 2025 at 7:49 PM Jeremy Laidman <jeremy@laidman.org> wrote:
Hi Jaime <Hi%20Jaime%0d%0dOn%20Thu,%20>
<Hi%20Jaime%0d%0dOn%20Thu,%20>
<Hi%20Jaime%0d%0dOn%20Thu,%20>
<Hi%20Jaime%0d%0dOn%20Thu,%20>
On Thu, <Hi%20Jaime%0d%0dOn%20Thu,%20>20 Mar 2025 at 02:11, Jaime Kikpole via Xymon <xymon@xymon.com> wrote:
I have a Xymon install that was monitoring (among other things) an HTTPS site. That site moved to a new IP and while I did update the public DNS records it was a few days (or maybe two weeks?) before I remembered to update the internal DNS records. By then, Xymon was reporting red for HTTPS and purple for SSL.
I've corrected the DNS records and the VM running Xymon is resolving it correctly, but now it is still showing the same red and purple alerts. Every now and then, they'll switch to green for a little while and then back to red and purple. I'm not sure why.
Xymon's web interface shows that the certificate is valid for more than 2 months. So that isn't it. I've restarted the Xymon server process and even the whole OS, but this issue remains.
Any ideas on what I could check?
First, review the content of the https status page. There may be non-obvious clues here. There's a difference between "connection refused" and "connection timed out" and this can lead you in different directions to identify the cause. For example, if you have a content check configured ("cont=..." in hosts.cfg), which is failing due to new content on the new webserver, this will be explained on the status page.
Second, I would check that your Xymon server can actually connect to the new IP address on port 443. I usually use "telnet <IP> <port>" and see if it connects, gets a connection refused, or a timeout, but netcat/ncat/nc works too. For an HTTPS website (as distinct from HTTP) I sometimes use an openssl command to see if the TLS/SSL interaction works, which tells me that there's no issue on the webserver itself, something like "openssl s_client -connect <IP>:<port> </dev/null". Even simpler, run curl or wget to fetch a webpage, but if you have a proxy set anywhere, this may give you an invalid result.
Thirdly, I would check the xymonnet.log file for any indications of problems, which might suggest a cause. Having xymonnet in debug mode might help (adding "--debug" into the CMD line of the [xymonnet] block in tasks.cfg). Alternatively, you could run xymonnet directly yourself, and look at the output. Something like this: xymoncmd xymonnet --no-update --debug <hostname>
Good luck with finding the problem.
Cheers
Jeremy
Xymon mailing list -- xymon@xymon.com To unsubscribe send an email to xymon-leave@xymon.com
Xymon mailing list -- xymon@xymon.com To unsubscribe send an email to xymon-leave@xymon.com
Xymon mailing list -- xymon@xymon.com To unsubscribe send an email to xymon-leave@xymon.com
Thanks for the suggestions and info so far. If I understand, my real issue is that the https test is failing and the sslcert test just doesn't have data to evaluate. Is that correct?
I think I'm going to have to mention the FQDN in order to nail this down, even though I was trying to avoid it. Maybe the issue will be obvious to one of you. I'm trying to test https://server.fw.ichabdocrane.org. (Another valid test would be https://server.fw.ichabodcrane.org/filewave.)
I noticed it was giving a 302 redirect, so I tried monitoring https://server.fw.ichabodcrane.org/filewave. Still had the same results. The web page for the http test now says:
https://server.fw.ichabodcrane.org/filewave - SSL error Seconds: 0.045584000
I ran a test using https://www.sslshopper.com/ssl-checker.html#hostname=https://server.fw.ichab... and it said that the certificate was good and that the IP address resolved to the IP that I expected it to point to. Note that this is the new IP. This issue started when the IP address of the hosted service was changed by the vendor a few weeks ago.
I tried to check it using curl from the system running Xymon.
curl --cert-status -I https://server.fw.ichabodcrane.org/filewave|more
% Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 3548 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 HTTP/1.1 200 OK Date: Mon, 24 Mar 2025 16:03:18 GMT Server: Apache/2.4.62 (Unix) OpenSSL/3.3.3 mod_wsgi/5.0.0 Python/3.12 Strict-Transport-Security: max-age=31536000; includeSubDomains X-Frame-Options: SAMEORIGIN Last-Modified: Wed, 12 Feb 2025 20:37:26 GMT Accept-Ranges: bytes Content-Length: 3548 X-Content-Type-Options: nosniff Pragma: no-cache Expires: Thu, 1 Jan 1970 00:00:00 GMT Cache-Control: max-age=0, no-store, no-cache, must-revalidate Content-Type: text/html
I also tried to check that the VM with Xymon installer on it was resolving things correctly.
host server.fw.ichabodcrane.org
server.fw.ichabodcrane.org has address 34.236.200.223
I also checked /etc/resolv.conf for the DNS resolvers and checked the FQDN resolving against both of those resolvers. The answer was the same as above.
So as far as I can tell, the targeted URL is accessible from the CLI on that host, but Xymon thinks it isn't. Any ideas?
Jaime Kikpole
Director of Technology Ichabod Crane Central School District (518) 758-7575, x5425
On Fri, Mar 21, 2025 at 11:29 AM Ron Cohen <rcohen55@gmail.com> wrote:
The SSL and http tests are not directly connected. The http retrieve a page, which will return red if not found, and the SSL retrieve the SSL certificate. So the SSL can be green and the http red and vis versa.
On Fri, 21 Mar 2025, 10:18 Nicola, <canne74@gmail.com> wrote:
The sslcert detail oage is green because it was last time it was retrieved. There should be a timestamp for the last update
Nicola Il giorno 21 mar 2025, alle ore 10:01, "Henrik Juul Størner via Xymon" < xymon@xymon.com> ha scritto:
Hi,
If I remember correctly: If the http test is red because Xymon cannot connect to the server on port 443, then Xymon will never retrieve the SSL certificate (no connection = no SSL data). This will cause the sslcert status to be purple, because it cannot be updated.
Med venlig hilsen / Best regards
*Henrik Juul Størner* Senior IT Arkitekt - IT-sikkerhed
*Fra:* Josh Luthman <josh@imaginenetworksllc.com> *Sendt:* 20. marts 2025 21:56 *Til:* Jaime Kikpole <jkikpole@ichabodcrane.org> *Cc:* Xymon mailinglist <xymon@xymon.com>; Jeremy Laidman < jeremy@laidman.org> *Emne:* [Xymon] Re: SSL stuck on purple
OBS: Denne e-mail er *ikke* fra en Danske Spil-afsender! Brug kun links eller vedhæftede filer, hvis du mener, de er OK.
Why is the http test red?
On Thu, Mar 20, 2025 at 4:46 PM Jaime Kikpole <jkikpole@ichabodcrane.org> wrote:
From hosts.cfg, with redactions between < and >:
<IP address> <hostname>.ichabodcrane.org # route:10.5.0.1 https:// <hostname>.ichabodcrane.org noconn
The ghosts report shows no clients.
The hostname definitely resolves (tested on the VM with the xymon server) to the IP address. The purple alert is in a column labeled "sslcert" and the red is in "http". The data in the page with the details about sslcert shows the correct dates and a green alert. The data in the page with the http details shows the message "SSL error" after the https://.... URL. When viewing that URL, it redirects path on the same server. This hasn't given me trouble over the last several years and the only thing that changed on the monitored system is that its IP address changed at the VM hosting service.
*Jaime Kikpole*
*Director of Technology* *Ichabod Crane Central School District* (518) 758-7575, x5425
On Thu, Mar 20, 2025 at 4:33 PM Josh Luthman < josh@imaginenetworksllc.com> wrote:
The IP in the first column is used in the event the name can't be resolved (or # testip is appended).
The purple test means there is no report for the test (for half an hour I think?) In this case you're testing from the Xymon Server so it is responsible for doing the test.
Are you certain the name of the host matches? Are there any ghosts?
Are you looking at the "sslcert" column? What does your hosts.cfg line look like - does it have an https url?
On Thu, Mar 20, 2025 at 3:38 PM Jaime Kikpole via Xymon <xymon@xymon.com> wrote:
Thanks for the ideas. That gave me more to try and I made some progress, but not much.
I discovered that hosts.cfg still had the old IP address. I updated that, but there was no change. I restarted the xymon server process and there was no change. I did a "xymon 127.0.0.1 drop <IP address>" and there was no change. Would you suggest a "xymon 127.0.0.1 drop <FQDN> http" or something like that?
I see what I would expect when using "curl https://<FQDN>" and the logs only show one interesting line within 2025, so I suspect that it all comes down to that one hosts.cfg line with the old IP address.
When I used "grep <server> * | grep 2025 | less", I see the following line and then everything else was from notifications.log about emailing my department's outages notification email address.
history.log:2025-03-18 14:54:12.042214 Will not update /usr/local/www/xymon/data/hist/server,fqdn,redacted.sslcert - color unchanged (purple)
What do you think? Should I just deal with the loss of historical data and drop the host by FQDN?
*Jaime Kikpole*
*Director of Technology* *Ichabod Crane Central School District* (518) 758-7575, x5425
On Wed, Mar 19, 2025 at 7:49 PM Jeremy Laidman <jeremy@laidman.org> wrote:
Hi Jaime <Hi%20Jaime%0d%0dOn%20Thu,%20>
<Hi%20Jaime%0d%0dOn%20Thu,%20>
<Hi%20Jaime%0d%0dOn%20Thu,%20>
<Hi%20Jaime%0d%0dOn%20Thu,%20>
On Thu, <Hi%20Jaime%0d%0dOn%20Thu,%20>20 Mar 2025 at 02:11, Jaime Kikpole via Xymon <xymon@xymon.com> wrote:
I have a Xymon install that was monitoring (among other things) an HTTPS site. That site moved to a new IP and while I did update the public DNS records it was a few days (or maybe two weeks?) before I remembered to update the internal DNS records. By then, Xymon was reporting red for HTTPS and purple for SSL.
I've corrected the DNS records and the VM running Xymon is resolving it correctly, but now it is still showing the same red and purple alerts. Every now and then, they'll switch to green for a little while and then back to red and purple. I'm not sure why.
Xymon's web interface shows that the certificate is valid for more than 2 months. So that isn't it. I've restarted the Xymon server process and even the whole OS, but this issue remains.
Any ideas on what I could check?
First, review the content of the https status page. There may be non-obvious clues here. There's a difference between "connection refused" and "connection timed out" and this can lead you in different directions to identify the cause. For example, if you have a content check configured ("cont=..." in hosts.cfg), which is failing due to new content on the new webserver, this will be explained on the status page.
Second, I would check that your Xymon server can actually connect to the new IP address on port 443. I usually use "telnet <IP> <port>" and see if it connects, gets a connection refused, or a timeout, but netcat/ncat/nc works too. For an HTTPS website (as distinct from HTTP) I sometimes use an openssl command to see if the TLS/SSL interaction works, which tells me that there's no issue on the webserver itself, something like "openssl s_client -connect <IP>:<port> </dev/null". Even simpler, run curl or wget to fetch a webpage, but if you have a proxy set anywhere, this may give you an invalid result.
Thirdly, I would check the xymonnet.log file for any indications of problems, which might suggest a cause. Having xymonnet in debug mode might help (adding "--debug" into the CMD line of the [xymonnet] block in tasks.cfg). Alternatively, you could run xymonnet directly yourself, and look at the output. Something like this: xymoncmd xymonnet --no-update --debug <hostname>
Good luck with finding the problem.
Cheers
Jeremy
Xymon mailing list -- xymon@xymon.com To unsubscribe send an email to xymon-leave@xymon.com
Xymon mailing list -- xymon@xymon.com To unsubscribe send an email to xymon-leave@xymon.com
Xymon mailing list -- xymon@xymon.com To unsubscribe send an email to xymon-leave@xymon.com
Hello,
You might need to try one of the other https methods listed for hosts.cfg? https://xymon.sourceforge.io/xymon/help/manpages/man5/hosts.cfg.5.html
- "2", e.g. https2://www.sample.com/ : use only SSLv2
- "3", e.g. https3://www.sample.com/ : use only SSLv3
- "t", e.g. httpst://www.sample.com/ : use only TLSv1.0
- "a", e.g. httpsa://www.sample.com/ : use only TLSv1.0
- "b", e.g. httpsb://www.sample.com/ : use only TLSv1.1
- "c", e.g. httpsc://www.sample.com/ : use only TLSv1.2
- "m", e.g. httpsm://www.sample.com/ : use only 128-bit ciphers
- "h", e.g. httpsh://www.sample.com/ : use only >128-bit ciphers
- "10", e.g. http10://www.sample.com/ : use HTTP 1.0
- "11", e.g. http11://www.sample.com/ : use HTTP 1.1
ie... httpsc://server.fw.ichabodcrane.org/filewave <https://server.fw.ichabodcrane.org/filewave>
Thanks, Matt
On Mon, Mar 24, 2025 at 1:22 PM Jaime Kikpole via Xymon <xymon@xymon.com> wrote:
Thanks for the suggestions and info so far. If I understand, my real issue is that the https test is failing and the sslcert test just doesn't have data to evaluate. Is that correct?
I think I'm going to have to mention the FQDN in order to nail this down, even though I was trying to avoid it. Maybe the issue will be obvious to one of you. I'm trying to test https://server.fw.ichabdocrane.org. (Another valid test would be https://server.fw.ichabodcrane.org/filewave.)
I noticed it was giving a 302 redirect, so I tried monitoring https://server.fw.ichabodcrane.org/filewave. Still had the same results. The web page for the http test now says:
https://server.fw.ichabodcrane.org/filewave - SSL error Seconds: 0.045584000
I ran a test using https://www.sslshopper.com/ssl-checker.html#hostname=https://server.fw.ichab... and it said that the certificate was good and that the IP address resolved to the IP that I expected it to point to. Note that this is the new IP. This issue started when the IP address of the hosted service was changed by the vendor a few weeks ago.
I tried to check it using curl from the system running Xymon.
curl --cert-status -I https://server.fw.ichabodcrane.org/filewave|more
% Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 3548 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 HTTP/1.1 200 OK Date: Mon, 24 Mar 2025 16:03:18 GMT Server: Apache/2.4.62 (Unix) OpenSSL/3.3.3 mod_wsgi/5.0.0 Python/3.12 Strict-Transport-Security: max-age=31536000; includeSubDomains X-Frame-Options: SAMEORIGIN Last-Modified: Wed, 12 Feb 2025 20:37:26 GMT Accept-Ranges: bytes Content-Length: 3548 X-Content-Type-Options: nosniff Pragma: no-cache Expires: Thu, 1 Jan 1970 00:00:00 GMT Cache-Control: max-age=0, no-store, no-cache, must-revalidate Content-Type: text/html
I also tried to check that the VM with Xymon installer on it was resolving things correctly.
host server.fw.ichabodcrane.org
server.fw.ichabodcrane.org has address 34.236.200.223
I also checked /etc/resolv.conf for the DNS resolvers and checked the FQDN resolving against both of those resolvers. The answer was the same as above.
So as far as I can tell, the targeted URL is accessible from the CLI on that host, but Xymon thinks it isn't. Any ideas?
Jaime Kikpole
Director of Technology Ichabod Crane Central School District (518) 758-7575, x5425
On Fri, Mar 21, 2025 at 11:29 AM Ron Cohen <rcohen55@gmail.com> wrote:
The SSL and http tests are not directly connected. The http retrieve a page, which will return red if not found, and the SSL retrieve the SSL certificate. So the SSL can be green and the http red and vis versa.
On Fri, 21 Mar 2025, 10:18 Nicola, <canne74@gmail.com> wrote:
The sslcert detail oage is green because it was last time it was retrieved. There should be a timestamp for the last update
Nicola Il giorno 21 mar 2025, alle ore 10:01, "Henrik Juul Størner via Xymon" < xymon@xymon.com> ha scritto:
Hi,
If I remember correctly: If the http test is red because Xymon cannot connect to the server on port 443, then Xymon will never retrieve the SSL certificate (no connection = no SSL data). This will cause the sslcert status to be purple, because it cannot be updated.
Med venlig hilsen / Best regards
*Henrik Juul Størner* Senior IT Arkitekt - IT-sikkerhed
*Fra:* Josh Luthman <josh@imaginenetworksllc.com> *Sendt:* 20. marts 2025 21:56 *Til:* Jaime Kikpole <jkikpole@ichabodcrane.org> *Cc:* Xymon mailinglist <xymon@xymon.com>; Jeremy Laidman < jeremy@laidman.org> *Emne:* [Xymon] Re: SSL stuck on purple
OBS: Denne e-mail er *ikke* fra en Danske Spil-afsender! Brug kun links eller vedhæftede filer, hvis du mener, de er OK.
Why is the http test red?
On Thu, Mar 20, 2025 at 4:46 PM Jaime Kikpole < jkikpole@ichabodcrane.org> wrote:
From hosts.cfg, with redactions between < and >:
<IP address> <hostname>.ichabodcrane.org # route:10.5.0.1 https:// <hostname>.ichabodcrane.org noconn
The ghosts report shows no clients.
The hostname definitely resolves (tested on the VM with the xymon server) to the IP address. The purple alert is in a column labeled "sslcert" and the red is in "http". The data in the page with the details about sslcert shows the correct dates and a green alert. The data in the page with the http details shows the message "SSL error" after the https://.... URL. When viewing that URL, it redirects path on the same server. This hasn't given me trouble over the last several years and the only thing that changed on the monitored system is that its IP address changed at the VM hosting service.
*Jaime Kikpole*
*Director of Technology* *Ichabod Crane Central School District* (518) 758-7575, x5425
On Thu, Mar 20, 2025 at 4:33 PM Josh Luthman < josh@imaginenetworksllc.com> wrote:
The IP in the first column is used in the event the name can't be resolved (or # testip is appended).
The purple test means there is no report for the test (for half an hour I think?) In this case you're testing from the Xymon Server so it is responsible for doing the test.
Are you certain the name of the host matches? Are there any ghosts?
Are you looking at the "sslcert" column? What does your hosts.cfg line look like - does it have an https url?
On Thu, Mar 20, 2025 at 3:38 PM Jaime Kikpole via Xymon < xymon@xymon.com> wrote:
Thanks for the ideas. That gave me more to try and I made some progress, but not much.
I discovered that hosts.cfg still had the old IP address. I updated that, but there was no change. I restarted the xymon server process and there was no change. I did a "xymon 127.0.0.1 drop <IP address>" and there was no change. Would you suggest a "xymon 127.0.0.1 drop <FQDN> http" or something like that?
I see what I would expect when using "curl https://<FQDN>" and the logs only show one interesting line within 2025, so I suspect that it all comes down to that one hosts.cfg line with the old IP address.
When I used "grep <server> * | grep 2025 | less", I see the following line and then everything else was from notifications.log about emailing my department's outages notification email address.
history.log:2025-03-18 14:54:12.042214 Will not update /usr/local/www/xymon/data/hist/server,fqdn,redacted.sslcert - color unchanged (purple)
What do you think? Should I just deal with the loss of historical data and drop the host by FQDN?
*Jaime Kikpole*
*Director of Technology* *Ichabod Crane Central School District* (518) 758-7575, x5425
On Wed, Mar 19, 2025 at 7:49 PM Jeremy Laidman <jeremy@laidman.org> wrote:
Hi Jaime <Hi%20Jaime%0d%0dOn%20Thu,%20>
<Hi%20Jaime%0d%0dOn%20Thu,%20>
<Hi%20Jaime%0d%0dOn%20Thu,%20>
<Hi%20Jaime%0d%0dOn%20Thu,%20>
On Thu, <Hi%20Jaime%0d%0dOn%20Thu,%20>20 Mar 2025 at 02:11, Jaime Kikpole via Xymon <xymon@xymon.com> wrote:
I have a Xymon install that was monitoring (among other things) an HTTPS site. That site moved to a new IP and while I did update the public DNS records it was a few days (or maybe two weeks?) before I remembered to update the internal DNS records. By then, Xymon was reporting red for HTTPS and purple for SSL.
I've corrected the DNS records and the VM running Xymon is resolving it correctly, but now it is still showing the same red and purple alerts. Every now and then, they'll switch to green for a little while and then back to red and purple. I'm not sure why.
Xymon's web interface shows that the certificate is valid for more than 2 months. So that isn't it. I've restarted the Xymon server process and even the whole OS, but this issue remains.
Any ideas on what I could check?
First, review the content of the https status page. There may be non-obvious clues here. There's a difference between "connection refused" and "connection timed out" and this can lead you in different directions to identify the cause. For example, if you have a content check configured ("cont=..." in hosts.cfg), which is failing due to new content on the new webserver, this will be explained on the status page.
Second, I would check that your Xymon server can actually connect to the new IP address on port 443. I usually use "telnet <IP> <port>" and see if it connects, gets a connection refused, or a timeout, but netcat/ncat/nc works too. For an HTTPS website (as distinct from HTTP) I sometimes use an openssl command to see if the TLS/SSL interaction works, which tells me that there's no issue on the webserver itself, something like "openssl s_client -connect <IP>:<port> </dev/null". Even simpler, run curl or wget to fetch a webpage, but if you have a proxy set anywhere, this may give you an invalid result.
Thirdly, I would check the xymonnet.log file for any indications of problems, which might suggest a cause. Having xymonnet in debug mode might help (adding "--debug" into the CMD line of the [xymonnet] block in tasks.cfg). Alternatively, you could run xymonnet directly yourself, and look at the output. Something like this: xymoncmd xymonnet --no-update --debug <hostname>
Good luck with finding the problem.
Cheers
Jeremy
Xymon mailing list -- xymon@xymon.com To unsubscribe send an email to xymon-leave@xymon.com
Xymon mailing list -- xymon@xymon.com To unsubscribe send an email to xymon-leave@xymon.com
Xymon mailing list -- xymon@xymon.com To unsubscribe send an email to xymon-leave@xymon.com
Xymon mailing list -- xymon@xymon.com To unsubscribe send an email to xymon-leave@xymon.com
-- Matthew Goebel : m <goebel@emunix.emich.edu>goebel@emich.edu : Unix Jockey @ EMU : Hail Eris Neo-Student, Net Lurker, Donut consumer, and procrastinating medher... "Always with the negative waves, Moriarty" - Oddball "Comfort the troubled, and trouble the comfortable." - Dietrich Bonhoeffer
Thanks. How can you tell which version of TLS it is?
Jaime Kikpole
Director of Technology Ichabod Crane Central School District (518) 758-7575, x5425
On Mon, Mar 24, 2025 at 2:48 PM Matthew Goebel <mgoebel@emich.edu> wrote:
Hello,
You might need to try one of the other https methods listed for hosts.cfg? https://xymon.sourceforge.io/xymon/help/manpages/man5/hosts.cfg.5.html
- "2", e.g. https2://www.sample.com/ : use only SSLv2
- "3", e.g. https3://www.sample.com/ : use only SSLv3
- "t", e.g. httpst://www.sample.com/ : use only TLSv1.0
- "a", e.g. httpsa://www.sample.com/ : use only TLSv1.0
- "b", e.g. httpsb://www.sample.com/ : use only TLSv1.1
- "c", e.g. httpsc://www.sample.com/ : use only TLSv1.2
- "m", e.g. httpsm://www.sample.com/ : use only 128-bit ciphers
- "h", e.g. httpsh://www.sample.com/ : use only >128-bit ciphers
- "10", e.g. http10://www.sample.com/ : use HTTP 1.0
- "11", e.g. http11://www.sample.com/ : use HTTP 1.1
ie... httpsc://server.fw.ichabodcrane.org/filewave <https://server.fw.ichabodcrane.org/filewave>
Thanks, Matt
On Mon, Mar 24, 2025 at 1:22 PM Jaime Kikpole via Xymon <xymon@xymon.com> wrote:
Thanks for the suggestions and info so far. If I understand, my real issue is that the https test is failing and the sslcert test just doesn't have data to evaluate. Is that correct?
I think I'm going to have to mention the FQDN in order to nail this down, even though I was trying to avoid it. Maybe the issue will be obvious to one of you. I'm trying to test https://server.fw.ichabdocrane.org. (Another valid test would be https://server.fw.ichabodcrane.org/filewave .)
I noticed it was giving a 302 redirect, so I tried monitoring https://server.fw.ichabodcrane.org/filewave. Still had the same results. The web page for the http test now says:
https://server.fw.ichabodcrane.org/filewave - SSL error Seconds: 0.045584000
I ran a test using https://www.sslshopper.com/ssl-checker.html#hostname=https://server.fw.ichab... and it said that the certificate was good and that the IP address resolved to the IP that I expected it to point to. Note that this is the new IP. This issue started when the IP address of the hosted service was changed by the vendor a few weeks ago.
I tried to check it using curl from the system running Xymon.
curl --cert-status -I https://server.fw.ichabodcrane.org/filewave|more
% Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 3548 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 HTTP/1.1 200 OK Date: Mon, 24 Mar 2025 16:03:18 GMT Server: Apache/2.4.62 (Unix) OpenSSL/3.3.3 mod_wsgi/5.0.0 Python/3.12 Strict-Transport-Security: max-age=31536000; includeSubDomains X-Frame-Options: SAMEORIGIN Last-Modified: Wed, 12 Feb 2025 20:37:26 GMT Accept-Ranges: bytes Content-Length: 3548 X-Content-Type-Options: nosniff Pragma: no-cache Expires: Thu, 1 Jan 1970 00:00:00 GMT Cache-Control: max-age=0, no-store, no-cache, must-revalidate Content-Type: text/html
I also tried to check that the VM with Xymon installer on it was resolving things correctly.
host server.fw.ichabodcrane.org
server.fw.ichabodcrane.org has address 34.236.200.223
I also checked /etc/resolv.conf for the DNS resolvers and checked the FQDN resolving against both of those resolvers. The answer was the same as above.
So as far as I can tell, the targeted URL is accessible from the CLI on that host, but Xymon thinks it isn't. Any ideas?
Jaime Kikpole
Director of Technology Ichabod Crane Central School District (518) 758-7575, x5425
On Fri, Mar 21, 2025 at 11:29 AM Ron Cohen <rcohen55@gmail.com> wrote:
The SSL and http tests are not directly connected. The http retrieve a page, which will return red if not found, and the SSL retrieve the SSL certificate. So the SSL can be green and the http red and vis versa.
On Fri, 21 Mar 2025, 10:18 Nicola, <canne74@gmail.com> wrote:
The sslcert detail oage is green because it was last time it was retrieved. There should be a timestamp for the last update
Nicola Il giorno 21 mar 2025, alle ore 10:01, "Henrik Juul Størner via Xymon" < xymon@xymon.com> ha scritto:
Hi,
If I remember correctly: If the http test is red because Xymon cannot connect to the server on port 443, then Xymon will never retrieve the SSL certificate (no connection = no SSL data). This will cause the sslcert status to be purple, because it cannot be updated.
Med venlig hilsen / Best regards
*Henrik Juul Størner* Senior IT Arkitekt - IT-sikkerhed
*Fra:* Josh Luthman <josh@imaginenetworksllc.com> *Sendt:* 20. marts 2025 21:56 *Til:* Jaime Kikpole <jkikpole@ichabodcrane.org> *Cc:* Xymon mailinglist <xymon@xymon.com>; Jeremy Laidman < jeremy@laidman.org> *Emne:* [Xymon] Re: SSL stuck on purple
OBS: Denne e-mail er *ikke* fra en Danske Spil-afsender! Brug kun links eller vedhæftede filer, hvis du mener, de er OK.
Why is the http test red?
On Thu, Mar 20, 2025 at 4:46 PM Jaime Kikpole < jkikpole@ichabodcrane.org> wrote:
From hosts.cfg, with redactions between < and >:
<IP address> <hostname>.ichabodcrane.org # route:10.5.0.1 https:// <hostname>.ichabodcrane.org noconn
The ghosts report shows no clients.
The hostname definitely resolves (tested on the VM with the xymon server) to the IP address. The purple alert is in a column labeled "sslcert" and the red is in "http". The data in the page with the details about sslcert shows the correct dates and a green alert. The data in the page with the http details shows the message "SSL error" after the https://.... URL. When viewing that URL, it redirects path on the same server. This hasn't given me trouble over the last several years and the only thing that changed on the monitored system is that its IP address changed at the VM hosting service.
*Jaime Kikpole*
*Director of Technology* *Ichabod Crane Central School District* (518) 758-7575, x5425
On Thu, Mar 20, 2025 at 4:33 PM Josh Luthman < josh@imaginenetworksllc.com> wrote:
The IP in the first column is used in the event the name can't be resolved (or # testip is appended).
The purple test means there is no report for the test (for half an hour I think?) In this case you're testing from the Xymon Server so it is responsible for doing the test.
Are you certain the name of the host matches? Are there any ghosts?
Are you looking at the "sslcert" column? What does your hosts.cfg line look like - does it have an https url?
On Thu, Mar 20, 2025 at 3:38 PM Jaime Kikpole via Xymon < xymon@xymon.com> wrote:
Thanks for the ideas. That gave me more to try and I made some progress, but not much.
I discovered that hosts.cfg still had the old IP address. I updated that, but there was no change. I restarted the xymon server process and there was no change. I did a "xymon 127.0.0.1 drop <IP address>" and there was no change. Would you suggest a "xymon 127.0.0.1 drop <FQDN> http" or something like that?
I see what I would expect when using "curl https://<FQDN>" and the logs only show one interesting line within 2025, so I suspect that it all comes down to that one hosts.cfg line with the old IP address.
When I used "grep <server> * | grep 2025 | less", I see the following line and then everything else was from notifications.log about emailing my department's outages notification email address.
history.log:2025-03-18 14:54:12.042214 Will not update /usr/local/www/xymon/data/hist/server,fqdn,redacted.sslcert - color unchanged (purple)
What do you think? Should I just deal with the loss of historical data and drop the host by FQDN?
*Jaime Kikpole*
*Director of Technology* *Ichabod Crane Central School District* (518) 758-7575, x5425
On Wed, Mar 19, 2025 at 7:49 PM Jeremy Laidman <jeremy@laidman.org> wrote:
Hi Jaime <Hi%20Jaime%0d%0dOn%20Thu,%20>
<Hi%20Jaime%0d%0dOn%20Thu,%20>
<Hi%20Jaime%0d%0dOn%20Thu,%20>
<Hi%20Jaime%0d%0dOn%20Thu,%20>
On Thu, <Hi%20Jaime%0d%0dOn%20Thu,%20>20 Mar 2025 at 02:11, Jaime Kikpole via Xymon <xymon@xymon.com> wrote:
I have a Xymon install that was monitoring (among other things) an HTTPS site. That site moved to a new IP and while I did update the public DNS records it was a few days (or maybe two weeks?) before I remembered to update the internal DNS records. By then, Xymon was reporting red for HTTPS and purple for SSL.
I've corrected the DNS records and the VM running Xymon is resolving it correctly, but now it is still showing the same red and purple alerts. Every now and then, they'll switch to green for a little while and then back to red and purple. I'm not sure why.
Xymon's web interface shows that the certificate is valid for more than 2 months. So that isn't it. I've restarted the Xymon server process and even the whole OS, but this issue remains.
Any ideas on what I could check?
First, review the content of the https status page. There may be non-obvious clues here. There's a difference between "connection refused" and "connection timed out" and this can lead you in different directions to identify the cause. For example, if you have a content check configured ("cont=..." in hosts.cfg), which is failing due to new content on the new webserver, this will be explained on the status page.
Second, I would check that your Xymon server can actually connect to the new IP address on port 443. I usually use "telnet <IP> <port>" and see if it connects, gets a connection refused, or a timeout, but netcat/ncat/nc works too. For an HTTPS website (as distinct from HTTP) I sometimes use an openssl command to see if the TLS/SSL interaction works, which tells me that there's no issue on the webserver itself, something like "openssl s_client -connect <IP>:<port> </dev/null". Even simpler, run curl or wget to fetch a webpage, but if you have a proxy set anywhere, this may give you an invalid result.
Thirdly, I would check the xymonnet.log file for any indications of problems, which might suggest a cause. Having xymonnet in debug mode might help (adding "--debug" into the CMD line of the [xymonnet] block in tasks.cfg). Alternatively, you could run xymonnet directly yourself, and look at the output. Something like this: xymoncmd xymonnet --no-update --debug <hostname>
Good luck with finding the problem.
Cheers
Jeremy
Xymon mailing list -- xymon@xymon.com To unsubscribe send an email to xymon-leave@xymon.com
Xymon mailing list -- xymon@xymon.com To unsubscribe send an email to xymon-leave@xymon.com
Xymon mailing list -- xymon@xymon.com To unsubscribe send an email to xymon-leave@xymon.com
Xymon mailing list -- xymon@xymon.com To unsubscribe send an email to xymon-leave@xymon.com
-- Matthew Goebel : m <goebel@emunix.emich.edu>goebel@emich.edu : Unix Jockey @ EMU : Hail Eris Neo-Student, Net Lurker, Donut consumer, and procrastinating medher... "Always with the negative waves, Moriarty" - Oddball "Comfort the troubled, and trouble the comfortable." - Dietrich Bonhoeffer
You have TLS 1.2
From your xymon server, what does running this show:
openssl s_client -connect server.fw.ichabodcrane.org:443
On Mon, 24 Mar 2025, Jaime Kikpole via Xymon wrote:
Date: Mon, 24 Mar 2025 14:55:02 -0400 From: Jaime Kikpole via Xymon <xymon@xymon.com> To: Matthew Goebel <mgoebel@emich.edu> Cc: Xymon mailinglist <xymon@xymon.com>, Jaime Kikpole <jkikpole@ichabodcrane.org> Subject: [Xymon] Re: SSL stuck on purple
Thanks. How can you tell which version of TLS it is?
Jaime Kikpole
Director of Technology Ichabod Crane Central School District (518) 758-7575, x5425
On Mon, Mar 24, 2025 at 2:48 PM Matthew Goebel <mgoebel@emich.edu> wrote:
Hello,
You might need to try one of the other https methods listed for hosts.cfg? https://xymon.sourceforge.io/xymon/help/manpages/man5/hosts.cfg.5.html
- "2", e.g. https2://www.sample.com/ : use only SSLv2
- "3", e.g. https3://www.sample.com/ : use only SSLv3
- "t", e.g. httpst://www.sample.com/ : use only TLSv1.0
- "a", e.g. httpsa://www.sample.com/ : use only TLSv1.0
- "b", e.g. httpsb://www.sample.com/ : use only TLSv1.1
- "c", e.g. httpsc://www.sample.com/ : use only TLSv1.2
- "m", e.g. httpsm://www.sample.com/ : use only 128-bit ciphers
- "h", e.g. httpsh://www.sample.com/ : use only >128-bit ciphers
- "10", e.g. http10://www.sample.com/ : use HTTP 1.0
- "11", e.g. http11://www.sample.com/ : use HTTP 1.1
ie... httpsc://server.fw.ichabodcrane.org/filewave <https://server.fw.ichabodcrane.org/filewave>
Thanks, Matt
On Mon, Mar 24, 2025 at 1:22 PM Jaime Kikpole via Xymon <xymon@xymon.com> wrote:
Thanks for the suggestions and info so far. If I understand, my real issue is that the https test is failing and the sslcert test just doesn't have data to evaluate. Is that correct?
I think I'm going to have to mention the FQDN in order to nail this down, even though I was trying to avoid it. Maybe the issue will be obvious to one of you. I'm trying to test https://server.fw.ichabdocrane.org. (Another valid test would be https://server.fw.ichabodcrane.org/filewave .)
I noticed it was giving a 302 redirect, so I tried monitoring https://server.fw.ichabodcrane.org/filewave. Still had the same results. The web page for the http test now says:
https://server.fw.ichabodcrane.org/filewave - SSL error Seconds: 0.045584000
I ran a test using https://www.sslshopper.com/ssl-checker.html#hostname=https://server.fw.ichab... and it said that the certificate was good and that the IP address resolved to the IP that I expected it to point to. Note that this is the new IP. This issue started when the IP address of the hosted service was changed by the vendor a few weeks ago.
I tried to check it using curl from the system running Xymon.
curl --cert-status -I https://server.fw.ichabodcrane.org/filewave|more
% Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 3548 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 HTTP/1.1 200 OK Date: Mon, 24 Mar 2025 16:03:18 GMT Server: Apache/2.4.62 (Unix) OpenSSL/3.3.3 mod_wsgi/5.0.0 Python/3.12 Strict-Transport-Security: max-age=31536000; includeSubDomains X-Frame-Options: SAMEORIGIN Last-Modified: Wed, 12 Feb 2025 20:37:26 GMT Accept-Ranges: bytes Content-Length: 3548 X-Content-Type-Options: nosniff Pragma: no-cache Expires: Thu, 1 Jan 1970 00:00:00 GMT Cache-Control: max-age=0, no-store, no-cache, must-revalidate Content-Type: text/html
I also tried to check that the VM with Xymon installer on it was resolving things correctly.
host server.fw.ichabodcrane.org
server.fw.ichabodcrane.org has address 34.236.200.223
I also checked /etc/resolv.conf for the DNS resolvers and checked the FQDN resolving against both of those resolvers. The answer was the same as above.
So as far as I can tell, the targeted URL is accessible from the CLI on that host, but Xymon thinks it isn't. Any ideas?
Jaime Kikpole
Director of Technology Ichabod Crane Central School District (518) 758-7575, x5425
On Fri, Mar 21, 2025 at 11:29 AM Ron Cohen <rcohen55@gmail.com> wrote:
The SSL and http tests are not directly connected. The http retrieve a page, which will return red if not found, and the SSL retrieve the SSL certificate. So the SSL can be green and the http red and vis versa.
On Fri, 21 Mar 2025, 10:18 Nicola, <canne74@gmail.com> wrote:
The sslcert detail oage is green because it was last time it was retrieved. There should be a timestamp for the last update
Nicola Il giorno 21 mar 2025, alle ore 10:01, "Henrik Juul Størner via Xymon" < xymon@xymon.com> ha scritto:
Hi,
If I remember correctly: If the http test is red because Xymon cannot connect to the server on port 443, then Xymon will never retrieve the SSL certificate (no connection = no SSL data). This will cause the sslcert status to be purple, because it cannot be updated.
Med venlig hilsen / Best regards
*Henrik Juul Størner* Senior IT Arkitekt - IT-sikkerhed
*Fra:* Josh Luthman <josh@imaginenetworksllc.com> *Sendt:* 20. marts 2025 21:56 *Til:* Jaime Kikpole <jkikpole@ichabodcrane.org> *Cc:* Xymon mailinglist <xymon@xymon.com>; Jeremy Laidman < jeremy@laidman.org> *Emne:* [Xymon] Re: SSL stuck on purple
OBS: Denne e-mail er *ikke* fra en Danske Spil-afsender! Brug kun links eller vedhæftede filer, hvis du mener, de er OK.
Why is the http test red?
On Thu, Mar 20, 2025 at 4:46 PM Jaime Kikpole < jkikpole@ichabodcrane.org> wrote:
From hosts.cfg, with redactions between < and >:
<IP address> <hostname>.ichabodcrane.org # route:10.5.0.1 https:// <hostname>.ichabodcrane.org noconn
The ghosts report shows no clients.
The hostname definitely resolves (tested on the VM with the xymon server) to the IP address. The purple alert is in a column labeled "sslcert" and the red is in "http". The data in the page with the details about sslcert shows the correct dates and a green alert. The data in the page with the http details shows the message "SSL error" after the https://.... URL. When viewing that URL, it redirects path on the same server. This hasn't given me trouble over the last several years and the only thing that changed on the monitored system is that its IP address changed at the VM hosting service.
*Jaime Kikpole*
*Director of Technology* *Ichabod Crane Central School District* (518) 758-7575, x5425
On Thu, Mar 20, 2025 at 4:33 PM Josh Luthman < josh@imaginenetworksllc.com> wrote:
The IP in the first column is used in the event the name can't be resolved (or # testip is appended).
The purple test means there is no report for the test (for half an hour I think?) In this case you're testing from the Xymon Server so it is responsible for doing the test.
Are you certain the name of the host matches? Are there any ghosts?
Are you looking at the "sslcert" column? What does your hosts.cfg line look like - does it have an https url?
On Thu, Mar 20, 2025 at 3:38 PM Jaime Kikpole via Xymon < xymon@xymon.com> wrote:
Thanks for the ideas. That gave me more to try and I made some progress, but not much.
I discovered that hosts.cfg still had the old IP address. I updated that, but there was no change. I restarted the xymon server process and there was no change. I did a "xymon 127.0.0.1 drop <IP address>" and there was no change. Would you suggest a "xymon 127.0.0.1 drop <FQDN> http" or something like that?
I see what I would expect when using "curl https://<FQDN>" and the logs only show one interesting line within 2025, so I suspect that it all comes down to that one hosts.cfg line with the old IP address.
When I used "grep <server> * | grep 2025 | less", I see the following line and then everything else was from notifications.log about emailing my department's outages notification email address.
history.log:2025-03-18 14:54:12.042214 Will not update /usr/local/www/xymon/data/hist/server,fqdn,redacted.sslcert - color unchanged (purple)
What do you think? Should I just deal with the loss of historical data and drop the host by FQDN?
*Jaime Kikpole*
*Director of Technology* *Ichabod Crane Central School District* (518) 758-7575, x5425
On Wed, Mar 19, 2025 at 7:49 PM Jeremy Laidman <jeremy@laidman.org> wrote:
Hi Jaime <Hi%20Jaime%0d%0dOn%20Thu,%20>
<Hi%20Jaime%0d%0dOn%20Thu,%20>
<Hi%20Jaime%0d%0dOn%20Thu,%20>
<Hi%20Jaime%0d%0dOn%20Thu,%20>
On Thu, <Hi%20Jaime%0d%0dOn%20Thu,%20>20 Mar 2025 at 02:11, Jaime Kikpole via Xymon <xymon@xymon.com> wrote:
I have a Xymon install that was monitoring (among other things) an HTTPS site. That site moved to a new IP and while I did update the public DNS records it was a few days (or maybe two weeks?) before I remembered to update the internal DNS records. By then, Xymon was reporting red for HTTPS and purple for SSL.
I've corrected the DNS records and the VM running Xymon is resolving it correctly, but now it is still showing the same red and purple alerts. Every now and then, they'll switch to green for a little while and then back to red and purple. I'm not sure why.
Xymon's web interface shows that the certificate is valid for more than 2 months. So that isn't it. I've restarted the Xymon server process and even the whole OS, but this issue remains.
Any ideas on what I could check?
First, review the content of the https status page. There may be non-obvious clues here. There's a difference between "connection refused" and "connection timed out" and this can lead you in different directions to identify the cause. For example, if you have a content check configured ("cont=..." in hosts.cfg), which is failing due to new content on the new webserver, this will be explained on the status page.
Second, I would check that your Xymon server can actually connect to the new IP address on port 443. I usually use "telnet <IP> <port>" and see if it connects, gets a connection refused, or a timeout, but netcat/ncat/nc works too. For an HTTPS website (as distinct from HTTP) I sometimes use an openssl command to see if the TLS/SSL interaction works, which tells me that there's no issue on the webserver itself, something like "openssl s_client -connect <IP>:<port> </dev/null". Even simpler, run curl or wget to fetch a webpage, but if you have a proxy set anywhere, this may give you an invalid result.
Thirdly, I would check the xymonnet.log file for any indications of problems, which might suggest a cause. Having xymonnet in debug mode might help (adding "--debug" into the CMD line of the [xymonnet] block in tasks.cfg). Alternatively, you could run xymonnet directly yourself, and look at the output. Something like this: xymoncmd xymonnet --no-update --debug <hostname>
Good luck with finding the problem.
Cheers
Jeremy
Xymon mailing list -- xymon@xymon.com To unsubscribe send an email to xymon-leave@xymon.com
Xymon mailing list -- xymon@xymon.com To unsubscribe send an email to xymon-leave@xymon.com
Xymon mailing list -- xymon@xymon.com To unsubscribe send an email to xymon-leave@xymon.com
Xymon mailing list -- xymon@xymon.com To unsubscribe send an email to xymon-leave@xymon.com
-- Matthew Goebel : m <goebel@emunix.emich.edu>goebel@emich.edu : Unix Jockey @ EMU : Hail Eris Neo-Student, Net Lurker, Donut consumer, and procrastinating medher... "Always with the negative waves, Moriarty" - Oddball "Comfort the troubled, and trouble the comfortable." - Dietrich Bonhoeffer
Thanks. I see the protocol entry now that I look through the output of that command. I'll make that edit when I get to a better endpoint than my phone.
Jaime
On Mon, Mar 24, 2025, 4:07 PM Stephane Bakhos <nuitari@nuitari.net> wrote:
You have TLS 1.2
From your xymon server, what does running this show:
openssl s_client -connect server.fw.ichabodcrane.org:443
On Mon, 24 Mar 2025, Jaime Kikpole via Xymon wrote:
Date: Mon, 24 Mar 2025 14:55:02 -0400 From: Jaime Kikpole via Xymon <xymon@xymon.com> To: Matthew Goebel <mgoebel@emich.edu> Cc: Xymon mailinglist <xymon@xymon.com>, Jaime Kikpole <jkikpole@ichabodcrane.org> Subject: [Xymon] Re: SSL stuck on purple
Thanks. How can you tell which version of TLS it is?
Jaime Kikpole
Director of Technology Ichabod Crane Central School District (518) 758-7575, x5425
On Mon, Mar 24, 2025 at 2:48 PM Matthew Goebel <mgoebel@emich.edu> wrote:
Hello,
You might need to try one of the other https methods listed for hosts.cfg? https://xymon.sourceforge.io/xymon/help/manpages/man5/hosts.cfg.5.html
- "2", e.g. https2://www.sample.com/ : use only SSLv2
- "3", e.g. https3://www.sample.com/ : use only SSLv3
- "t", e.g. httpst://www.sample.com/ : use only TLSv1.0
- "a", e.g. httpsa://www.sample.com/ : use only TLSv1.0
- "b", e.g. httpsb://www.sample.com/ : use only TLSv1.1
- "c", e.g. httpsc://www.sample.com/ : use only TLSv1.2
- "m", e.g. httpsm://www.sample.com/ : use only 128-bit ciphers
- "h", e.g. httpsh://www.sample.com/ : use only >128-bit ciphers
- "10", e.g. http10://www.sample.com/ : use HTTP 1.0
- "11", e.g. http11://www.sample.com/ : use HTTP 1.1
ie... httpsc://server.fw.ichabodcrane.org/filewave <https://server.fw.ichabodcrane.org/filewave>
Thanks, Matt
On Mon, Mar 24, 2025 at 1:22 PM Jaime Kikpole via Xymon < xymon@xymon.com> wrote:
Thanks for the suggestions and info so far. If I understand, my real issue is that the https test is failing and the sslcert test just doesn't have data to evaluate. Is that correct?
I think I'm going to have to mention the FQDN in order to nail this down, even though I was trying to avoid it. Maybe the issue will be obvious to one of you. I'm trying to test https://server.fw.ichabdocrane.org. (Another valid test would be https://server.fw.ichabodcrane.org/filewave .)
I noticed it was giving a 302 redirect, so I tried monitoring https://server.fw.ichabodcrane.org/filewave. Still had the same results. The web page for the http test now says:
https://server.fw.ichabodcrane.org/filewave - SSL error Seconds: 0.045584000
I ran a test using
https://www.sslshopper.com/ssl-checker.html#hostname=https://server.fw.ichab...
and it said that the certificate was good and that the IP address resolved to the IP that I expected it to point to. Note that this is the new IP. This issue started when the IP address of the hosted service was changed by the vendor a few weeks ago.
I tried to check it using curl from the system running Xymon.
curl --cert-status -I
https://server.fw.ichabodcrane.org/filewave|more % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 3548 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 HTTP/1.1 200 OK Date: Mon, 24 Mar 2025 16:03:18 GMT Server: Apache/2.4.62 (Unix) OpenSSL/3.3.3 mod_wsgi/5.0.0 Python/3.12 Strict-Transport-Security: max-age=31536000; includeSubDomains X-Frame-Options: SAMEORIGIN Last-Modified: Wed, 12 Feb 2025 20:37:26 GMT Accept-Ranges: bytes Content-Length: 3548 X-Content-Type-Options: nosniff Pragma: no-cache Expires: Thu, 1 Jan 1970 00:00:00 GMT Cache-Control: max-age=0, no-store, no-cache, must-revalidate Content-Type: text/html
I also tried to check that the VM with Xymon installer on it was resolving things correctly.
host server.fw.ichabodcrane.org
server.fw.ichabodcrane.org has address 34.236.200.223
I also checked /etc/resolv.conf for the DNS resolvers and checked the FQDN resolving against both of those resolvers. The answer was the same as above.
So as far as I can tell, the targeted URL is accessible from the CLI on that host, but Xymon thinks it isn't. Any ideas?
Jaime Kikpole
Director of Technology Ichabod Crane Central School District (518) 758-7575, x5425
On Fri, Mar 21, 2025 at 11:29 AM Ron Cohen <rcohen55@gmail.com> wrote:
The SSL and http tests are not directly connected. The http retrieve a page, which will return red if not found, and the SSL retrieve the SSL certificate. So the SSL can be green and the http red and vis versa.
On Fri, 21 Mar 2025, 10:18 Nicola, <canne74@gmail.com> wrote:
The sslcert detail oage is green because it was last time it was retrieved. There should be a timestamp for the last update
Nicola Il giorno 21 mar 2025, alle ore 10:01, "Henrik Juul Størner via Xymon" < xymon@xymon.com> ha scritto: > > Hi, > > > > If I remember correctly: If the http test is red because Xymon cannot > connect to the server on port 443, then Xymon will never retrieve the SSL > certificate (no connection = no SSL data). This will cause the sslcert > status to be purple, because it cannot be updated. > > > > Med venlig hilsen / Best regards > > > > *Henrik Juul Størner* > Senior IT Arkitekt - IT-sikkerhed > > > > *Fra:* Josh Luthman <josh@imaginenetworksllc.com> > *Sendt:* 20. marts 2025 21:56 > *Til:* Jaime Kikpole <jkikpole@ichabodcrane.org> > *Cc:* Xymon mailinglist <xymon@xymon.com>; Jeremy Laidman < > jeremy@laidman.org> > *Emne:* [Xymon] Re: SSL stuck on purple > > > > OBS: Denne e-mail er *ikke* fra en Danske Spil-afsender! Brug kun > links eller vedhæftede filer, hvis du mener, de er OK. > > > > Why is the http test red? > > > > On Thu, Mar 20, 2025 at 4:46 PM Jaime Kikpole < > jkikpole@ichabodcrane.org> wrote: > > From hosts.cfg, with redactions between < and >: > > <IP address> <hostname>.ichabodcrane.org # route:10.5.0.1 https:// > <hostname>.ichabodcrane.org noconn > > > > The ghosts report shows no clients. > > > > The hostname definitely resolves (tested on the VM with the xymon > server) to the IP address. The purple alert is in a column labeled > "sslcert" and the red is in "http". The data in the page with the details > about sslcert shows the correct dates and a green alert. The data in the > page with the http details shows the message "SSL error" after the > https://.... URL. When viewing that URL, it redirects path on the > same server. This hasn't given me trouble over the last several years and > the only thing that changed on the monitored system is that its IP address > changed at the VM hosting service. > > > > > > > > *Jaime Kikpole* > > *Director of Technology* > *Ichabod Crane Central School District* > (518) 758-7575, x5425 > > > > > > On Thu, Mar 20, 2025 at 4:33 PM Josh Luthman < > josh@imaginenetworksllc.com> wrote: > > The IP in the first column is used in the event the name can't be > resolved (or # testip is appended). > > > > The purple test means there is no report for the test (for half an > hour I think?) In this case you're testing from the Xymon Server so it is > responsible for doing the test. > > > > Are you certain the name of the host matches? Are there any ghosts? > > > > Are you looking at the "sslcert" column? What does your hosts.cfg > line look like - does it have an https url? > > > > On Thu, Mar 20, 2025 at 3:38 PM Jaime Kikpole via Xymon < > xymon@xymon.com> wrote: > > Thanks for the ideas. That gave me more to try and I made some > progress, but not much. > > > > I discovered that hosts.cfg still had the old IP address. I updated > that, but there was no change. I restarted the xymon server process and > there was no change. I did a "xymon 127.0.0.1 drop <IP address>" and there > was no change. Would you suggest a "xymon 127.0.0.1 drop <FQDN> http" or > something like that? > > > > I see what I would expect when using "curl https://<FQDN>" and the > logs only show one interesting line within 2025, so I suspect that it > all comes down to that one hosts.cfg line with the old IP address. > > > > When I used "grep <server> * | grep 2025 | less", I see the following > line and then everything else was from notifications.log about emailing my > department's outages notification email address. > > > > history.log:2025-03-18 14:54:12.042214 Will not update > /usr/local/www/xymon/data/hist/server,fqdn,redacted.sslcert - color > unchanged (purple) > > > > What do you think? Should I just deal with the loss of historical > data and drop the host by FQDN? > > > > > > *Jaime Kikpole* > > *Director of Technology* > *Ichabod Crane Central School District* > (518) 758-7575, x5425 > > > > > > On Wed, Mar 19, 2025 at 7:49 PM Jeremy Laidman <jeremy@laidman.org> > wrote: > > Hi Jaime <Hi%20Jaime%0d%0dOn%20Thu,%20> > > <Hi%20Jaime%0d%0dOn%20Thu,%20> > > <Hi%20Jaime%0d%0dOn%20Thu,%20> > > <Hi%20Jaime%0d%0dOn%20Thu,%20> > > On Thu, <Hi%20Jaime%0d%0dOn%20Thu,%20>20 Mar 2025 at 02:11, Jaime > Kikpole via Xymon <xymon@xymon.com> wrote: > > I have a Xymon install that was monitoring (among other things) an > HTTPS site. That site moved to a new IP and while I did update the public > DNS records it was a few days (or maybe two weeks?) before I remembered to > update the internal DNS records. By then, Xymon was reporting red for > HTTPS and purple for SSL. > > > > I've corrected the DNS records and the VM running Xymon is resolving > it correctly, but now it is still showing the same red and purple alerts. > Every now and then, they'll switch to green for a little while and then > back to red and purple. I'm not sure why. > > > > Xymon's web interface shows that the certificate is valid for more > than 2 months. So that isn't it. I've restarted the Xymon server process > and even the whole OS, but this issue remains. > > > > Any ideas on what I could check? > > > > First, review the content of the https status page. There may be > non-obvious clues here. There's a difference between "connection refused" > and "connection timed out" and this can lead you in different directions to > identify the cause. For example, if you have a content check configured > ("cont=..." in hosts.cfg), which is failing due to new content on the new > webserver, this will be explained on the status page. > > > > Second, I would check that your Xymon server can actually connect to > the new IP address on port 443. I usually use "telnet <IP> <port>" and see > if it connects, gets a connection refused, or a timeout, but netcat/ncat/nc > works too. For an HTTPS website (as distinct from HTTP) I sometimes use an > openssl command to see if the TLS/SSL interaction works, which tells me > that there's no issue on the webserver itself, something like "openssl > s_client -connect <IP>:<port> </dev/null". Even simpler, run curl or wget > to fetch a webpage, but if you have a proxy set anywhere, this may give you > an invalid result. > > > > Thirdly, I would check the xymonnet.log file for any indications of > problems, which might suggest a cause. Having xymonnet in debug mode might > help (adding "--debug" into the CMD line of the [xymonnet] block in > tasks.cfg). Alternatively, you could run xymonnet directly yourself, and > look at the output. Something like this: xymoncmd xymonnet --no-update > --debug <hostname> > > > > Good luck with finding the problem. > > > > Cheers > > Jeremy > > _______________________________________________ > Xymon mailing list -- xymon@xymon.com > To unsubscribe send an email to xymon-leave@xymon.com > > ------------------------------ > > Xymon mailing list -- xymon@xymon.com > To unsubscribe send an email to xymon-leave@xymon.com > > _______________________________________________ Xymon mailing list -- xymon@xymon.com To unsubscribe send an email to xymon-leave@xymon.com
Xymon mailing list -- xymon@xymon.com To unsubscribe send an email to xymon-leave@xymon.com
-- Matthew Goebel : m <goebel@emunix.emich.edu>goebel@emich.edu : Unix Jockey @ EMU : Hail Eris Neo-Student, Net Lurker, Donut consumer, and procrastinating medher... "Always with the negative waves, Moriarty" - Oddball "Comfort the troubled, and trouble the comfortable." - Dietrich Bonhoeffer
I just tried it and gave it 15+ minutes to change from red or at least give another status message. It didn't change the alert color or the text on the alert's page.
Any other suggestions?
Jaime Kikpole
Director of Technology Ichabod Crane Central School District (518) 758-7575, x5425
On Mon, Mar 24, 2025 at 5:24 PM Jaime Kikpole <jkikpole@ichabodcrane.org> wrote:
Thanks. I see the protocol entry now that I look through the output of that command. I'll make that edit when I get to a better endpoint than my phone.
Jaime
On Mon, Mar 24, 2025, 4:07 PM Stephane Bakhos <nuitari@nuitari.net> wrote:
You have TLS 1.2
From your xymon server, what does running this show:
openssl s_client -connect server.fw.ichabodcrane.org:443
On Mon, 24 Mar 2025, Jaime Kikpole via Xymon wrote:
Date: Mon, 24 Mar 2025 14:55:02 -0400 From: Jaime Kikpole via Xymon <xymon@xymon.com> To: Matthew Goebel <mgoebel@emich.edu> Cc: Xymon mailinglist <xymon@xymon.com>, Jaime Kikpole <jkikpole@ichabodcrane.org> Subject: [Xymon] Re: SSL stuck on purple
Thanks. How can you tell which version of TLS it is?
Jaime Kikpole
Director of Technology Ichabod Crane Central School District (518) 758-7575, x5425
On Mon, Mar 24, 2025 at 2:48 PM Matthew Goebel <mgoebel@emich.edu> wrote:
Hello,
You might need to try one of the other https methods listed for hosts.cfg? https://xymon.sourceforge.io/xymon/help/manpages/man5/hosts.cfg.5.html
- "2", e.g. https2://www.sample.com/ : use only SSLv2
- "3", e.g. https3://www.sample.com/ : use only SSLv3
- "t", e.g. httpst://www.sample.com/ : use only TLSv1.0
- "a", e.g. httpsa://www.sample.com/ : use only TLSv1.0
- "b", e.g. httpsb://www.sample.com/ : use only TLSv1.1
- "c", e.g. httpsc://www.sample.com/ : use only TLSv1.2
- "m", e.g. httpsm://www.sample.com/ : use only 128-bit ciphers
- "h", e.g. httpsh://www.sample.com/ : use only >128-bit ciphers
- "10", e.g. http10://www.sample.com/ : use HTTP 1.0
- "11", e.g. http11://www.sample.com/ : use HTTP 1.1
ie... httpsc://server.fw.ichabodcrane.org/filewave <https://server.fw.ichabodcrane.org/filewave>
Thanks, Matt
On Mon, Mar 24, 2025 at 1:22 PM Jaime Kikpole via Xymon < xymon@xymon.com> wrote:
Thanks for the suggestions and info so far. If I understand, my real issue is that the https test is failing and the sslcert test just doesn't have data to evaluate. Is that correct?
I think I'm going to have to mention the FQDN in order to nail this down, even though I was trying to avoid it. Maybe the issue will be obvious to one of you. I'm trying to test https://server.fw.ichabdocrane.org. (Another valid test would be https://server.fw.ichabodcrane.org/filewave .)
I noticed it was giving a 302 redirect, so I tried monitoring https://server.fw.ichabodcrane.org/filewave. Still had the same results. The web page for the http test now says:
https://server.fw.ichabodcrane.org/filewave - SSL error Seconds: 0.045584000
I ran a test using
https://www.sslshopper.com/ssl-checker.html#hostname=https://server.fw.ichab...
and it said that the certificate was good and that the IP address resolved to the IP that I expected it to point to. Note that this is the new IP. This issue started when the IP address of the hosted service was changed by the vendor a few weeks ago.
I tried to check it using curl from the system running Xymon.
curl --cert-status -I
https://server.fw.ichabodcrane.org/filewave|more % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 3548 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 HTTP/1.1 200 OK Date: Mon, 24 Mar 2025 16:03:18 GMT Server: Apache/2.4.62 (Unix) OpenSSL/3.3.3 mod_wsgi/5.0.0 Python/3.12 Strict-Transport-Security: max-age=31536000; includeSubDomains X-Frame-Options: SAMEORIGIN Last-Modified: Wed, 12 Feb 2025 20:37:26 GMT Accept-Ranges: bytes Content-Length: 3548 X-Content-Type-Options: nosniff Pragma: no-cache Expires: Thu, 1 Jan 1970 00:00:00 GMT Cache-Control: max-age=0, no-store, no-cache, must-revalidate Content-Type: text/html
I also tried to check that the VM with Xymon installer on it was resolving things correctly.
host server.fw.ichabodcrane.org
server.fw.ichabodcrane.org has address 34.236.200.223
I also checked /etc/resolv.conf for the DNS resolvers and checked the FQDN resolving against both of those resolvers. The answer was the same as above.
So as far as I can tell, the targeted URL is accessible from the CLI on that host, but Xymon thinks it isn't. Any ideas?
Jaime Kikpole
Director of Technology Ichabod Crane Central School District (518) 758-7575, x5425
On Fri, Mar 21, 2025 at 11:29 AM Ron Cohen <rcohen55@gmail.com> wrote:
The SSL and http tests are not directly connected. The http retrieve a page, which will return red if not found, and the SSL retrieve the SSL certificate. So the SSL can be green and the http red and vis versa.
On Fri, 21 Mar 2025, 10:18 Nicola, <canne74@gmail.com> wrote:
> The sslcert detail oage is green because it was last time it was > retrieved. > There should be a timestamp for the last update > > Nicola > Il giorno 21 mar 2025, alle ore 10:01, "Henrik Juul Størner via Xymon" < > xymon@xymon.com> ha scritto: >> >> Hi, >> >> >> >> If I remember correctly: If the http test is red because Xymon cannot >> connect to the server on port 443, then Xymon will never retrieve the SSL >> certificate (no connection = no SSL data). This will cause the sslcert >> status to be purple, because it cannot be updated. >> >> >> >> Med venlig hilsen / Best regards >> >> >> >> *Henrik Juul Størner* >> Senior IT Arkitekt - IT-sikkerhed >> >> >> >> *Fra:* Josh Luthman <josh@imaginenetworksllc.com> >> *Sendt:* 20. marts 2025 21:56 >> *Til:* Jaime Kikpole <jkikpole@ichabodcrane.org> >> *Cc:* Xymon mailinglist <xymon@xymon.com>; Jeremy Laidman < >> jeremy@laidman.org> >> *Emne:* [Xymon] Re: SSL stuck on purple >> >> >> >> OBS: Denne e-mail er *ikke* fra en Danske Spil-afsender! Brug kun >> links eller vedhæftede filer, hvis du mener, de er OK. >> >> >> >> Why is the http test red? >> >> >> >> On Thu, Mar 20, 2025 at 4:46 PM Jaime Kikpole < >> jkikpole@ichabodcrane.org> wrote: >> >> From hosts.cfg, with redactions between < and >: >> >> <IP address> <hostname>.ichabodcrane.org # route:10.5.0.1 https:// >> <hostname>.ichabodcrane.org noconn >> >> >> >> The ghosts report shows no clients. >> >> >> >> The hostname definitely resolves (tested on the VM with the xymon >> server) to the IP address. The purple alert is in a column labeled >> "sslcert" and the red is in "http". The data in the page with the details >> about sslcert shows the correct dates and a green alert. The data in the >> page with the http details shows the message "SSL error" after the >> https://.... URL. When viewing that URL, it redirects path on the >> same server. This hasn't given me trouble over the last several years and >> the only thing that changed on the monitored system is that its IP address >> changed at the VM hosting service. >> >> >> >> >> >> >> >> *Jaime Kikpole* >> >> *Director of Technology* >> *Ichabod Crane Central School District* >> (518) 758-7575, x5425 >> >> >> >> >> >> On Thu, Mar 20, 2025 at 4:33 PM Josh Luthman < >> josh@imaginenetworksllc.com> wrote: >> >> The IP in the first column is used in the event the name can't be >> resolved (or # testip is appended). >> >> >> >> The purple test means there is no report for the test (for half an >> hour I think?) In this case you're testing from the Xymon Server so it is >> responsible for doing the test. >> >> >> >> Are you certain the name of the host matches? Are there any ghosts? >> >> >> >> Are you looking at the "sslcert" column? What does your hosts.cfg >> line look like - does it have an https url? >> >> >> >> On Thu, Mar 20, 2025 at 3:38 PM Jaime Kikpole via Xymon < >> xymon@xymon.com> wrote: >> >> Thanks for the ideas. That gave me more to try and I made some >> progress, but not much. >> >> >> >> I discovered that hosts.cfg still had the old IP address. I updated >> that, but there was no change. I restarted the xymon server process and >> there was no change. I did a "xymon 127.0.0.1 drop <IP address>" and there >> was no change. Would you suggest a "xymon 127.0.0.1 drop <FQDN> http" or >> something like that? >> >> >> >> I see what I would expect when using "curl https://<FQDN>" and the >> logs only show one interesting line within 2025, so I suspect that it >> all comes down to that one hosts.cfg line with the old IP address. >> >> >> >> When I used "grep <server> * | grep 2025 | less", I see the following >> line and then everything else was from notifications.log about emailing my >> department's outages notification email address. >> >> >> >> history.log:2025-03-18 14:54:12.042214 Will not update >> /usr/local/www/xymon/data/hist/server,fqdn,redacted.sslcert - color >> unchanged (purple) >> >> >> >> What do you think? Should I just deal with the loss of historical >> data and drop the host by FQDN? >> >> >> >> >> >> *Jaime Kikpole* >> >> *Director of Technology* >> *Ichabod Crane Central School District* >> (518) 758-7575, x5425 >> >> >> >> >> >> On Wed, Mar 19, 2025 at 7:49 PM Jeremy Laidman <jeremy@laidman.org
>> wrote: >> >> Hi Jaime <Hi%20Jaime%0d%0dOn%20Thu,%20> >> >> <Hi%20Jaime%0d%0dOn%20Thu,%20> >> >> <Hi%20Jaime%0d%0dOn%20Thu,%20> >> >> <Hi%20Jaime%0d%0dOn%20Thu,%20> >> >> On Thu, <Hi%20Jaime%0d%0dOn%20Thu,%20>20 Mar 2025 at 02:11, Jaime >> Kikpole via Xymon <xymon@xymon.com> wrote: >> >> I have a Xymon install that was monitoring (among other things) an >> HTTPS site. That site moved to a new IP and while I did update the public >> DNS records it was a few days (or maybe two weeks?) before I remembered to >> update the internal DNS records. By then, Xymon was reporting red for >> HTTPS and purple for SSL. >> >> >> >> I've corrected the DNS records and the VM running Xymon is resolving >> it correctly, but now it is still showing the same red and purple alerts. >> Every now and then, they'll switch to green for a little while and then >> back to red and purple. I'm not sure why. >> >> >> >> Xymon's web interface shows that the certificate is valid for more >> than 2 months. So that isn't it. I've restarted the Xymon server process >> and even the whole OS, but this issue remains. >> >> >> >> Any ideas on what I could check? >> >> >> >> First, review the content of the https status page. There may be >> non-obvious clues here. There's a difference between "connection refused" >> and "connection timed out" and this can lead you in different directions to >> identify the cause. For example, if you have a content check configured >> ("cont=..." in hosts.cfg), which is failing due to new content on the new >> webserver, this will be explained on the status page. >> >> >> >> Second, I would check that your Xymon server can actually connect to >> the new IP address on port 443. I usually use "telnet <IP> <port>" and see >> if it connects, gets a connection refused, or a timeout, but netcat/ncat/nc >> works too. For an HTTPS website (as distinct from HTTP) I sometimes use an >> openssl command to see if the TLS/SSL interaction works, which tells me >> that there's no issue on the webserver itself, something like "openssl >> s_client -connect <IP>:<port> </dev/null". Even simpler, run curl or wget >> to fetch a webpage, but if you have a proxy set anywhere, this may give you >> an invalid result. >> >> >> >> Thirdly, I would check the xymonnet.log file for any indications of >> problems, which might suggest a cause. Having xymonnet in debug mode might >> help (adding "--debug" into the CMD line of the [xymonnet] block in >> tasks.cfg). Alternatively, you could run xymonnet directly yourself, and >> look at the output. Something like this: xymoncmd xymonnet --no-update >> --debug <hostname> >> >> >> >> Good luck with finding the problem. >> >> >> >> Cheers >> >> Jeremy >> >> _______________________________________________ >> Xymon mailing list -- xymon@xymon.com >> To unsubscribe send an email to xymon-leave@xymon.com >> >> ------------------------------ >> >> Xymon mailing list -- xymon@xymon.com >> To unsubscribe send an email to xymon-leave@xymon.com >> >> _______________________________________________ > Xymon mailing list -- xymon@xymon.com > To unsubscribe send an email to xymon-leave@xymon.com >
Xymon mailing list -- xymon@xymon.com To unsubscribe send an email to xymon-leave@xymon.com
-- Matthew Goebel : m <goebel@emunix.emich.edu>goebel@emich.edu : Unix Jockey @ EMU : Hail Eris Neo-Student, Net Lurker, Donut consumer, and procrastinating medher... "Always with the negative waves, Moriarty" - Oddball "Comfort the troubled, and trouble the comfortable." - Dietrich Bonhoeffer
Just jumping into the middle of this thread, but have you tried adding sni at the end of your https test in your hosts file? *https://server.fw.ichabodcrane.org sni*
I've had issues with some ssl tests not working on some specific servers, and this sni tag solved my issues.
Search for 'sni' to read about it in the man page. https://xymon.sourceforge.io/xymon/help/manpages/man5/hosts.cfg.5.html
Thank You, Kris Springer Systems Admin I/O Network Administration
On 3/26/25 11:24, Jaime Kikpole via Xymon wrote:
I just tried it and gave it 15+ minutes to change from red or at least give another status message. It didn't change the alert color or the text on the alert's page.
Any other suggestions?
Jaime Kikpole
Director of Technology Ichabod Crane Central School District (518) 758-7575, x5425
On Mon, Mar 24, 2025 at 5:24 PM Jaime Kikpole <jkikpole@ichabodcrane.org> wrote:
Thanks. I see the protocol entry now that I look through the output of that command. I'll make that edit when I get to a better endpoint than my phone. Jaime On Mon, Mar 24, 2025, 4:07 PM Stephane Bakhos <nuitari@nuitari.net> wrote: You have TLS 1.2 >From your xymon server, what does running this show: openssl s_client -connect server.fw.ichabodcrane.org:443 <http://server.fw.ichabodcrane.org:443> On Mon, 24 Mar 2025, Jaime Kikpole via Xymon wrote: > Date: Mon, 24 Mar 2025 14:55:02 -0400 > From: Jaime Kikpole via Xymon <xymon@xymon.com> > To: Matthew Goebel <mgoebel@emich.edu> > Cc: Xymon mailinglist <xymon@xymon.com>, > Jaime Kikpole <jkikpole@ichabodcrane.org> > Subject: [Xymon] Re: SSL stuck on purple > > Thanks. How can you tell which version of TLS it is? > > > > Jaime Kikpole > > Director of Technology > Ichabod Crane Central School District > (518) 758-7575, x5425 > > > On Mon, Mar 24, 2025 at 2:48 PM Matthew Goebel <mgoebel@emich.edu> wrote: > >> Hello, >> >> You might need to try one of the other https methods listed for hosts.cfg? >> https://xymon.sourceforge.io/xymon/help/manpages/man5/hosts.cfg.5.html >> >> * "2", e.g. https2://www.sample.com/ <http://www.sample.com/> : use only SSLv2 >> * "3", e.g. https3://www.sample.com/ <http://www.sample.com/> : use only SSLv3 >> * "t", e.g. httpst://www.sample.com/ <http://www.sample.com/> : use only TLSv1.0 >> * "a", e.g. httpsa://www.sample.com/ <http://www.sample.com/> : use only TLSv1.0 >> * "b", e.g. httpsb://www.sample.com/ <http://www.sample.com/> : use only TLSv1.1 >> * "c", e.g. httpsc://www.sample.com/ <http://www.sample.com/> : use only TLSv1.2 >> * "m", e.g. httpsm://www.sample.com/ <http://www.sample.com/> : use only 128-bit ciphers >> * "h", e.g. httpsh://www.sample.com/ <http://www.sample.com/> : use only >128-bit ciphers >> * "10", e.g. http10://www.sample.com/ <http://www.sample.com/> : use HTTP 1.0 >> * "11", e.g. http11://www.sample.com/ <http://www.sample.com/> : use HTTP 1.1 >> >> ie... >> httpsc://server.fw.ichabodcrane.org/filewave <http://server.fw.ichabodcrane.org/filewave> >> <https://server.fw.ichabodcrane.org/filewave> >> >> Thanks, >> Matt >> >> >> On Mon, Mar 24, 2025 at 1:22 PM Jaime Kikpole via Xymon <xymon@xymon.com> >> wrote: >> >>> Thanks for the suggestions and info so far. If I understand, my real >>> issue is that the https test is failing and the sslcert test just doesn't >>> have data to evaluate. Is that correct? >>> >>> I think I'm going to have to mention the FQDN in order to nail this down, >>> even though I was trying to avoid it. Maybe the issue will be obvious to >>> one of you. I'm trying to test https://server.fw.ichabdocrane.org. >>> (Another valid test would be https://server.fw.ichabodcrane.org/filewave >>> .) >>> >>> I noticed it was giving a 302 redirect, so I tried monitoring >>> https://server.fw.ichabodcrane.org/filewave. Still had the same >>> results. The web page for the http test now says: >>> >>> https://server.fw.ichabodcrane.org/filewave - SSL error >>> Seconds: 0.045584000 >>> >>> I ran a test using >>> https://www.sslshopper.com/ssl-checker.html#hostname=https://server.fw.ichabodcrane.org >>> and it said that the certificate was good and that the IP address resolved >>> to the IP that I expected it to point to. Note that this is the new IP. >>> This issue started when the IP address of the hosted service was changed by >>> the vendor a few weeks ago. >>> >>> I tried to check it using curl from the system running Xymon. >>> >>> # curl --cert-status -I https://server.fw.ichabodcrane.org/filewave|more <https://server.fw.ichabodcrane.org/filewave%7Cmore> >>> % Total % Received % Xferd Average Speed Time Time Time >>> Current >>> Dload Upload Total Spent Left >>> Speed >>> 0 3548 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- >>> 0 >>> HTTP/1.1 200 OK >>> Date: Mon, 24 Mar 2025 16:03:18 GMT >>> Server: Apache/2.4.62 (Unix) OpenSSL/3.3.3 mod_wsgi/5.0.0 Python/3.12 >>> Strict-Transport-Security: max-age=31536000; includeSubDomains >>> X-Frame-Options: SAMEORIGIN >>> Last-Modified: Wed, 12 Feb 2025 20:37:26 GMT >>> Accept-Ranges: bytes >>> Content-Length: 3548 >>> X-Content-Type-Options: nosniff >>> Pragma: no-cache >>> Expires: Thu, 1 Jan 1970 00:00:00 GMT >>> Cache-Control: max-age=0, no-store, no-cache, must-revalidate >>> Content-Type: text/html >>> >>> >>> I also tried to check that the VM with Xymon installer on it was >>> resolving things correctly. >>> >>> # host server.fw.ichabodcrane.org <http://server.fw.ichabodcrane.org> >>> server.fw.ichabodcrane.org <http://server.fw.ichabodcrane.org> has address 34.236.200.223 >>> >>> >>> I also checked /etc/resolv.conf for the DNS resolvers and checked the >>> FQDN resolving against both of those resolvers. The answer was the same as >>> above. >>> >>> So as far as I can tell, the targeted URL is accessible from the CLI on >>> that host, but Xymon thinks it isn't. Any ideas? >>> >>> >>> >>> Jaime Kikpole >>> >>> Director of Technology >>> Ichabod Crane Central School District >>> (518) 758-7575, x5425 >>> >>> >>> On Fri, Mar 21, 2025 at 11:29 AM Ron Cohen <rcohen55@gmail.com> wrote: >>> >>>> The SSL and http tests are not directly connected. The http retrieve a >>>> page, which will return red if not found, and the SSL retrieve the SSL >>>> certificate. So the SSL can be green and the http red and vis versa. >>>> >>>> On Fri, 21 Mar 2025, 10:18 Nicola, <canne74@gmail.com> wrote: >>>> >>>>> The sslcert detail oage is green because it was last time it was >>>>> retrieved. >>>>> There should be a timestamp for the last update >>>>> >>>>> Nicola >>>>> Il giorno 21 mar 2025, alle ore 10:01, "Henrik Juul Størner via Xymon" < >>>>> xymon@xymon.com> ha scritto: >>>>>> >>>>>> Hi, >>>>>> >>>>>> >>>>>> >>>>>> If I remember correctly: If the http test is red because Xymon cannot >>>>>> connect to the server on port 443, then Xymon will never retrieve the SSL >>>>>> certificate (no connection = no SSL data). This will cause the sslcert >>>>>> status to be purple, because it cannot be updated. >>>>>> >>>>>> >>>>>> >>>>>> Med venlig hilsen / Best regards >>>>>> >>>>>> >>>>>> >>>>>> *Henrik Juul Størner* >>>>>> Senior IT Arkitekt - IT-sikkerhed >>>>>> >>>>>> >>>>>> >>>>>> *Fra:* Josh Luthman <josh@imaginenetworksllc.com> >>>>>> *Sendt:* 20. marts 2025 21:56 >>>>>> *Til:* Jaime Kikpole <jkikpole@ichabodcrane.org> >>>>>> *Cc:* Xymon mailinglist <xymon@xymon.com>; Jeremy Laidman < >>>>>> jeremy@laidman.org> >>>>>> *Emne:* [Xymon] Re: SSL stuck on purple >>>>>> >>>>>> >>>>>> >>>>>> OBS: Denne e-mail er *ikke* fra en Danske Spil-afsender! Brug kun >>>>>> links eller vedhæftede filer, hvis du mener, de er OK. >>>>>> >>>>>> >>>>>> >>>>>> Why is the http test red? >>>>>> >>>>>> >>>>>> >>>>>> On Thu, Mar 20, 2025 at 4:46 PM Jaime Kikpole < >>>>>> jkikpole@ichabodcrane.org> wrote: >>>>>> >>>>>> From hosts.cfg, with redactions between < and >: >>>>>> >>>>>> <IP address> <hostname>.ichabodcrane.org <http://ichabodcrane.org> # route:10.5.0.1 https:// >>>>>> <hostname>.ichabodcrane.org <http://ichabodcrane.org> noconn >>>>>> >>>>>> >>>>>> >>>>>> The ghosts report shows no clients. >>>>>> >>>>>> >>>>>> >>>>>> The hostname definitely resolves (tested on the VM with the xymon >>>>>> server) to the IP address. The purple alert is in a column labeled >>>>>> "sslcert" and the red is in "http". The data in the page with the details >>>>>> about sslcert shows the correct dates and a green alert. The data in the >>>>>> page with the http details shows the message "SSL error" after the >>>>>> https://.... URL. When viewing that URL, it redirects path on the >>>>>> same server. This hasn't given me trouble over the last several years and >>>>>> the only thing that changed on the monitored system is that its IP address >>>>>> changed at the VM hosting service. >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> *Jaime Kikpole* >>>>>> >>>>>> *Director of Technology* >>>>>> *Ichabod Crane Central School District* >>>>>> (518) 758-7575, x5425 >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> On Thu, Mar 20, 2025 at 4:33 PM Josh Luthman < >>>>>> josh@imaginenetworksllc.com> wrote: >>>>>> >>>>>> The IP in the first column is used in the event the name can't be >>>>>> resolved (or # testip is appended). >>>>>> >>>>>> >>>>>> >>>>>> The purple test means there is no report for the test (for half an >>>>>> hour I think?) In this case you're testing from the Xymon Server so it is >>>>>> responsible for doing the test. >>>>>> >>>>>> >>>>>> >>>>>> Are you certain the name of the host matches? Are there any ghosts? >>>>>> >>>>>> >>>>>> >>>>>> Are you looking at the "sslcert" column? What does your hosts.cfg >>>>>> line look like - does it have an https url? >>>>>> >>>>>> >>>>>> >>>>>> On Thu, Mar 20, 2025 at 3:38 PM Jaime Kikpole via Xymon < >>>>>> xymon@xymon.com> wrote: >>>>>> >>>>>> Thanks for the ideas. That gave me more to try and I made some >>>>>> progress, but not much. >>>>>> >>>>>> >>>>>> >>>>>> I discovered that hosts.cfg still had the old IP address. I updated >>>>>> that, but there was no change. I restarted the xymon server process and >>>>>> there was no change. I did a "xymon 127.0.0.1 drop <IP address>" and there >>>>>> was no change. Would you suggest a "xymon 127.0.0.1 drop <FQDN> http" or >>>>>> something like that? >>>>>> >>>>>> >>>>>> >>>>>> I see what I would expect when using "curl https://<FQDN>" and the >>>>>> logs only show one interesting line within 2025, so I suspect that it >>>>>> all comes down to that one hosts.cfg line with the old IP address. >>>>>> >>>>>> >>>>>> >>>>>> When I used "grep <server> * | grep 2025 | less", I see the following >>>>>> line and then everything else was from notifications.log about emailing my >>>>>> department's outages notification email address. >>>>>> >>>>>> >>>>>> >>>>>> history.log:2025-03-18 14:54:12.042214 Will not update >>>>>> /usr/local/www/xymon/data/hist/server,fqdn,redacted.sslcert - color >>>>>> unchanged (purple) >>>>>> >>>>>> >>>>>> >>>>>> What do you think? Should I just deal with the loss of historical >>>>>> data and drop the host by FQDN? >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> *Jaime Kikpole* >>>>>> >>>>>> *Director of Technology* >>>>>> *Ichabod Crane Central School District* >>>>>> (518) 758-7575, x5425 >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> On Wed, Mar 19, 2025 at 7:49 PM Jeremy Laidman <jeremy@laidman.org> >>>>>> wrote: >>>>>> >>>>>> Hi Jaime <Hi%20Jaime%0d%0dOn%20Thu,%20> >>>>>> >>>>>> <Hi%20Jaime%0d%0dOn%20Thu,%20> >>>>>> >>>>>> <Hi%20Jaime%0d%0dOn%20Thu,%20> >>>>>> >>>>>> <Hi%20Jaime%0d%0dOn%20Thu,%20> >>>>>> >>>>>> On Thu, <Hi%20Jaime%0d%0dOn%20Thu,%20>20 Mar 2025 at 02:11, Jaime >>>>>> Kikpole via Xymon <xymon@xymon.com> wrote: >>>>>> >>>>>> I have a Xymon install that was monitoring (among other things) an >>>>>> HTTPS site. That site moved to a new IP and while I did update the public >>>>>> DNS records it was a few days (or maybe two weeks?) before I remembered to >>>>>> update the internal DNS records. By then, Xymon was reporting red for >>>>>> HTTPS and purple for SSL. >>>>>> >>>>>> >>>>>> >>>>>> I've corrected the DNS records and the VM running Xymon is resolving >>>>>> it correctly, but now it is still showing the same red and purple alerts. >>>>>> Every now and then, they'll switch to green for a little while and then >>>>>> back to red and purple. I'm not sure why. >>>>>> >>>>>> >>>>>> >>>>>> Xymon's web interface shows that the certificate is valid for more >>>>>> than 2 months. So that isn't it. I've restarted the Xymon server process >>>>>> and even the whole OS, but this issue remains. >>>>>> >>>>>> >>>>>> >>>>>> Any ideas on what I could check? >>>>>> >>>>>> >>>>>> >>>>>> First, review the content of the https status page. There may be >>>>>> non-obvious clues here. There's a difference between "connection refused" >>>>>> and "connection timed out" and this can lead you in different directions to >>>>>> identify the cause. For example, if you have a content check configured >>>>>> ("cont=..." in hosts.cfg), which is failing due to new content on the new >>>>>> webserver, this will be explained on the status page. >>>>>> >>>>>> >>>>>> >>>>>> Second, I would check that your Xymon server can actually connect to >>>>>> the new IP address on port 443. I usually use "telnet <IP> <port>" and see >>>>>> if it connects, gets a connection refused, or a timeout, but netcat/ncat/nc >>>>>> works too. For an HTTPS website (as distinct from HTTP) I sometimes use an >>>>>> openssl command to see if the TLS/SSL interaction works, which tells me >>>>>> that there's no issue on the webserver itself, something like "openssl >>>>>> s_client -connect <IP>:<port> </dev/null". Even simpler, run curl or wget >>>>>> to fetch a webpage, but if you have a proxy set anywhere, this may give you >>>>>> an invalid result. >>>>>> >>>>>> >>>>>> >>>>>> Thirdly, I would check the xymonnet.log file for any indications of >>>>>> problems, which might suggest a cause. Having xymonnet in debug mode might >>>>>> help (adding "--debug" into the CMD line of the [xymonnet] block in >>>>>> tasks.cfg). Alternatively, you could run xymonnet directly yourself, and >>>>>> look at the output. Something like this: xymoncmd xymonnet --no-update >>>>>> --debug <hostname> >>>>>> >>>>>> >>>>>> >>>>>> Good luck with finding the problem. >>>>>> >>>>>> >>>>>> >>>>>> Cheers >>>>>> >>>>>> Jeremy >>>>>> >>>>>> _______________________________________________ >>>>>> Xymon mailing list -- xymon@xymon.com >>>>>> To unsubscribe send an email to xymon-leave@xymon.com >>>>>> >>>>>> ------------------------------ >>>>>> >>>>>> Xymon mailing list -- xymon@xymon.com >>>>>> To unsubscribe send an email to xymon-leave@xymon.com >>>>>> >>>>>> _______________________________________________ >>>>> Xymon mailing list -- xymon@xymon.com >>>>> To unsubscribe send an email to xymon-leave@xymon.com >>>>> >>>> _______________________________________________ >>> Xymon mailing list -- xymon@xymon.com >>> To unsubscribe send an email to xymon-leave@xymon.com >>> >> >> >> -- >> Matthew Goebel : m <goebel@emunix.emich.edu>goebel@emich.edu : Unix >> Jockey @ EMU : Hail Eris >> Neo-Student, Net Lurker, Donut consumer, and procrastinating medher... >> "Always with the negative waves, Moriarty" - Oddball >> "Comfort the troubled, and trouble the comfortable." - Dietrich Bonhoeffer >> >> >> >
Xymon mailing list --xymon@xymon.com To unsubscribe send an email toxymon-leave@xymon.com
That was it! Thank you for pointing out SNI.
Jaime Kikpole
Director of Technology Ichabod Crane Central School District (518) 758-7575, x5425
On Wed, Mar 26, 2025 at 1:48 PM IO Support <support@ionetworkadmin.com> wrote:
Just jumping into the middle of this thread, but have you tried adding sni at the end of your https test in your hosts file? *https://server.fw.ichabodcrane.org <https://server.fw.ichabodcrane.org> sni*
I've had issues with some ssl tests not working on some specific servers, and this sni tag solved my issues.
Search for 'sni' to read about it in the man page. https://xymon.sourceforge.io/xymon/help/manpages/man5/hosts.cfg.5.html
Thank You, Kris Springer Systems Admin I/O Network Administration
On 3/26/25 11:24, Jaime Kikpole via Xymon wrote:
I just tried it and gave it 15+ minutes to change from red or at least give another status message. It didn't change the alert color or the text on the alert's page.
Any other suggestions?
Jaime Kikpole
Director of Technology Ichabod Crane Central School District (518) 758-7575, x5425
On Mon, Mar 24, 2025 at 5:24 PM Jaime Kikpole <jkikpole@ichabodcrane.org> wrote:
Thanks. I see the protocol entry now that I look through the output of that command. I'll make that edit when I get to a better endpoint than my phone.
Jaime
On Mon, Mar 24, 2025, 4:07 PM Stephane Bakhos <nuitari@nuitari.net> wrote:
You have TLS 1.2
From your xymon server, what does running this show:
openssl s_client -connect server.fw.ichabodcrane.org:443
On Mon, 24 Mar 2025, Jaime Kikpole via Xymon wrote:
Date: Mon, 24 Mar 2025 14:55:02 -0400 From: Jaime Kikpole via Xymon <xymon@xymon.com> To: Matthew Goebel <mgoebel@emich.edu> Cc: Xymon mailinglist <xymon@xymon.com>, Jaime Kikpole <jkikpole@ichabodcrane.org> Subject: [Xymon] Re: SSL stuck on purple
Thanks. How can you tell which version of TLS it is?
Jaime Kikpole
Director of Technology Ichabod Crane Central School District (518) 758-7575, x5425
On Mon, Mar 24, 2025 at 2:48 PM Matthew Goebel <mgoebel@emich.edu> wrote:
Hello,
You might need to try one of the other https methods listed for hosts.cfg?
https://xymon.sourceforge.io/xymon/help/manpages/man5/hosts.cfg.5.html
- "2", e.g. https2://www.sample.com/ : use only SSLv2
- "3", e.g. https3://www.sample.com/ : use only SSLv3
- "t", e.g. httpst://www.sample.com/ : use only TLSv1.0
- "a", e.g. httpsa://www.sample.com/ : use only TLSv1.0
- "b", e.g. httpsb://www.sample.com/ : use only TLSv1.1
- "c", e.g. httpsc://www.sample.com/ : use only TLSv1.2
- "m", e.g. httpsm://www.sample.com/ : use only 128-bit ciphers
- "h", e.g. httpsh://www.sample.com/ : use only >128-bit ciphers
- "10", e.g. http10://www.sample.com/ : use HTTP 1.0
- "11", e.g. http11://www.sample.com/ : use HTTP 1.1
ie... httpsc://server.fw.ichabodcrane.org/filewave <https://server.fw.ichabodcrane.org/filewave>
Thanks, Matt
On Mon, Mar 24, 2025 at 1:22 PM Jaime Kikpole via Xymon <
xymon@xymon.com>
wrote:
Thanks for the suggestions and info so far. If I understand, my real issue is that the https test is failing and the sslcert test just doesn't have data to evaluate. Is that correct?
I think I'm going to have to mention the FQDN in order to nail this down, even though I was trying to avoid it. Maybe the issue will be obvious to one of you. I'm trying to test https://server.fw.ichabdocrane.org. (Another valid test would be https://server.fw.ichabodcrane.org/filewave .)
I noticed it was giving a 302 redirect, so I tried monitoring https://server.fw.ichabodcrane.org/filewave. Still had the same results. The web page for the http test now says:
https://server.fw.ichabodcrane.org/filewave - SSL error Seconds: 0.045584000
I ran a test using
https://www.sslshopper.com/ssl-checker.html#hostname=https://server.fw.ichab...
and it said that the certificate was good and that the IP address resolved to the IP that I expected it to point to. Note that this is the new IP. This issue started when the IP address of the hosted service was changed by the vendor a few weeks ago.
I tried to check it using curl from the system running Xymon.
curl --cert-status -I
https://server.fw.ichabodcrane.org/filewave|more % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 3548 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 HTTP/1.1 200 OK Date: Mon, 24 Mar 2025 16:03:18 GMT Server: Apache/2.4.62 (Unix) OpenSSL/3.3.3 mod_wsgi/5.0.0 Python/3.12 Strict-Transport-Security: max-age=31536000; includeSubDomains X-Frame-Options: SAMEORIGIN Last-Modified: Wed, 12 Feb 2025 20:37:26 GMT Accept-Ranges: bytes Content-Length: 3548 X-Content-Type-Options: nosniff Pragma: no-cache Expires: Thu, 1 Jan 1970 00:00:00 GMT Cache-Control: max-age=0, no-store, no-cache, must-revalidate Content-Type: text/html
I also tried to check that the VM with Xymon installer on it was resolving things correctly.
host server.fw.ichabodcrane.org
server.fw.ichabodcrane.org has address 34.236.200.223
I also checked /etc/resolv.conf for the DNS resolvers and checked the FQDN resolving against both of those resolvers. The answer was the same as above.
So as far as I can tell, the targeted URL is accessible from the CLI on that host, but Xymon thinks it isn't. Any ideas?
Jaime Kikpole
Director of Technology Ichabod Crane Central School District (518) 758-7575, x5425
On Fri, Mar 21, 2025 at 11:29 AM Ron Cohen <rcohen55@gmail.com> wrote:
> The SSL and http tests are not directly connected. The http retrieve a > page, which will return red if not found, and the SSL retrieve the SSL > certificate. So the SSL can be green and the http red and vis versa. > > On Fri, 21 Mar 2025, 10:18 Nicola, <canne74@gmail.com> wrote: > >> The sslcert detail oage is green because it was last time it was >> retrieved. >> There should be a timestamp for the last update >> >> Nicola >> Il giorno 21 mar 2025, alle ore 10:01, "Henrik Juul Størner via Xymon" < >> xymon@xymon.com> ha scritto: >>> >>> Hi, >>> >>> >>> >>> If I remember correctly: If the http test is red because Xymon cannot >>> connect to the server on port 443, then Xymon will never retrieve the SSL >>> certificate (no connection = no SSL data). This will cause the sslcert >>> status to be purple, because it cannot be updated. >>> >>> >>> >>> Med venlig hilsen / Best regards >>> >>> >>> >>> *Henrik Juul Størner* >>> Senior IT Arkitekt - IT-sikkerhed >>> >>> >>> >>> *Fra:* Josh Luthman <josh@imaginenetworksllc.com> >>> *Sendt:* 20. marts 2025 21:56 >>> *Til:* Jaime Kikpole <jkikpole@ichabodcrane.org> >>> *Cc:* Xymon mailinglist <xymon@xymon.com>; Jeremy Laidman < >>> jeremy@laidman.org> >>> *Emne:* [Xymon] Re: SSL stuck on purple >>> >>> >>> >>> OBS: Denne e-mail er *ikke* fra en Danske Spil-afsender! Brug kun >>> links eller vedhæftede filer, hvis du mener, de er OK. >>> >>> >>> >>> Why is the http test red? >>> >>> >>> >>> On Thu, Mar 20, 2025 at 4:46 PM Jaime Kikpole < >>> jkikpole@ichabodcrane.org> wrote: >>> >>> From hosts.cfg, with redactions between < and >: >>> >>> <IP address> <hostname>.ichabodcrane.org # route:10.5.0.1 https:// >>> <hostname>.ichabodcrane.org noconn >>> >>> >>> >>> The ghosts report shows no clients. >>> >>> >>> >>> The hostname definitely resolves (tested on the VM with the xymon >>> server) to the IP address. The purple alert is in a column labeled >>> "sslcert" and the red is in "http". The data in the page with the details >>> about sslcert shows the correct dates and a green alert. The data in the >>> page with the http details shows the message "SSL error" after the >>> https://.... URL. When viewing that URL, it redirects path on the >>> same server. This hasn't given me trouble over the last several years and >>> the only thing that changed on the monitored system is that its IP address >>> changed at the VM hosting service. >>> >>> >>> >>> >>> >>> >>> >>> *Jaime Kikpole* >>> >>> *Director of Technology* >>> *Ichabod Crane Central School District* >>> (518) 758-7575, x5425 >>> >>> >>> >>> >>> >>> On Thu, Mar 20, 2025 at 4:33 PM Josh Luthman < >>> josh@imaginenetworksllc.com> wrote: >>> >>> The IP in the first column is used in the event the name can't be >>> resolved (or # testip is appended). >>> >>> >>> >>> The purple test means there is no report for the test (for half an >>> hour I think?) In this case you're testing from the Xymon Server so it is >>> responsible for doing the test. >>> >>> >>> >>> Are you certain the name of the host matches? Are there any ghosts? >>> >>> >>> >>> Are you looking at the "sslcert" column? What does your hosts.cfg >>> line look like - does it have an https url? >>> >>> >>> >>> On Thu, Mar 20, 2025 at 3:38 PM Jaime Kikpole via Xymon < >>> xymon@xymon.com> wrote: >>> >>> Thanks for the ideas. That gave me more to try and I made some >>> progress, but not much. >>> >>> >>> >>> I discovered that hosts.cfg still had the old IP address. I updated >>> that, but there was no change. I restarted the xymon server process and >>> there was no change. I did a "xymon 127.0.0.1 drop <IP address>" and there >>> was no change. Would you suggest a "xymon 127.0.0.1 drop <FQDN> http" or >>> something like that? >>> >>> >>> >>> I see what I would expect when using "curl https://<FQDN>" and the >>> logs only show one interesting line within 2025, so I suspect that it >>> all comes down to that one hosts.cfg line with the old IP address. >>> >>> >>> >>> When I used "grep <server> * | grep 2025 | less", I see the following >>> line and then everything else was from notifications.log about emailing my >>> department's outages notification email address. >>> >>> >>> >>> history.log:2025-03-18 14:54:12.042214 Will not update >>> /usr/local/www/xymon/data/hist/server,fqdn,redacted.sslcert - color >>> unchanged (purple) >>> >>> >>> >>> What do you think? Should I just deal with the loss of historical >>> data and drop the host by FQDN? >>> >>> >>> >>> >>> >>> *Jaime Kikpole* >>> >>> *Director of Technology* >>> *Ichabod Crane Central School District* >>> (518) 758-7575, x5425 >>> >>> >>> >>> >>> >>> On Wed, Mar 19, 2025 at 7:49 PM Jeremy Laidman < jeremy@laidman.org> >>> wrote: >>> >>> Hi Jaime <Hi%20Jaime%0d%0dOn%20Thu,%20> >>> >>> <Hi%20Jaime%0d%0dOn%20Thu,%20> >>> >>> <Hi%20Jaime%0d%0dOn%20Thu,%20> >>> >>> <Hi%20Jaime%0d%0dOn%20Thu,%20> >>> >>> On Thu, <Hi%20Jaime%0d%0dOn%20Thu,%20>20 Mar 2025 at 02:11, Jaime >>> Kikpole via Xymon <xymon@xymon.com> wrote: >>> >>> I have a Xymon install that was monitoring (among other things) an >>> HTTPS site. That site moved to a new IP and while I did update the public >>> DNS records it was a few days (or maybe two weeks?) before I remembered to >>> update the internal DNS records. By then, Xymon was reporting red for >>> HTTPS and purple for SSL. >>> >>> >>> >>> I've corrected the DNS records and the VM running Xymon is resolving >>> it correctly, but now it is still showing the same red and purple alerts. >>> Every now and then, they'll switch to green for a little while and then >>> back to red and purple. I'm not sure why. >>> >>> >>> >>> Xymon's web interface shows that the certificate is valid for more >>> than 2 months. So that isn't it. I've restarted the Xymon server process >>> and even the whole OS, but this issue remains. >>> >>> >>> >>> Any ideas on what I could check? >>> >>> >>> >>> First, review the content of the https status page. There may be >>> non-obvious clues here. There's a difference between "connection refused" >>> and "connection timed out" and this can lead you in different directions to >>> identify the cause. For example, if you have a content check configured >>> ("cont=..." in hosts.cfg), which is failing due to new content on the new >>> webserver, this will be explained on the status page. >>> >>> >>> >>> Second, I would check that your Xymon server can actually connect to >>> the new IP address on port 443. I usually use "telnet <IP> <port>" and see >>> if it connects, gets a connection refused, or a timeout, but netcat/ncat/nc >>> works too. For an HTTPS website (as distinct from HTTP) I sometimes use an >>> openssl command to see if the TLS/SSL interaction works, which tells me >>> that there's no issue on the webserver itself, something like "openssl >>> s_client -connect <IP>:<port> </dev/null". Even simpler, run curl or wget >>> to fetch a webpage, but if you have a proxy set anywhere, this may give you >>> an invalid result. >>> >>> >>> >>> Thirdly, I would check the xymonnet.log file for any indications of >>> problems, which might suggest a cause. Having xymonnet in debug mode might >>> help (adding "--debug" into the CMD line of the [xymonnet] block in >>> tasks.cfg). Alternatively, you could run xymonnet directly yourself, and >>> look at the output. Something like this: xymoncmd xymonnet --no-update >>> --debug <hostname> >>> >>> >>> >>> Good luck with finding the problem. >>> >>> >>> >>> Cheers >>> >>> Jeremy >>> >>> _______________________________________________ >>> Xymon mailing list -- xymon@xymon.com >>> To unsubscribe send an email to xymon-leave@xymon.com >>> >>> ------------------------------ >>> >>> Xymon mailing list -- xymon@xymon.com >>> To unsubscribe send an email to xymon-leave@xymon.com >>> >>> _______________________________________________ >> Xymon mailing list -- xymon@xymon.com >> To unsubscribe send an email to xymon-leave@xymon.com >> > _______________________________________________ Xymon mailing list -- xymon@xymon.com To unsubscribe send an email to xymon-leave@xymon.com
-- Matthew Goebel : m <goebel@emunix.emich.edu>goebel@emich.edu : Unix Jockey @ EMU : Hail Eris Neo-Student, Net Lurker, Donut consumer, and procrastinating medher... "Always with the negative waves, Moriarty" - Oddball "Comfort the troubled, and trouble the comfortable." - Dietrich Bonhoeffer
Xymon mailing list -- xymon@xymon.com To unsubscribe send an email to xymon-leave@xymon.com
participants (9)
-
Henrik Juul Størner
-
IO Support
-
Jaime Kikpole
-
Jeremy Laidman
-
Josh Luthman
-
Matthew Goebel
-
Nicola
-
Ron Cohen
-
Stephane Bakhos