Upgrade to 4.3.21, https tests seem to no longer work.
I upgraded to 4.3.21 today, from 4.3.14. It was built using this command:
build/makerpm.sh 4.3.21
After working my way through recovering my configs that were pushed out of the way as .rpmsave files, I started it. Immediately I got http alarms failing.
The most interesting one was this. I have xymon monitoring its own URL, which is secured with SSL:
red Tue Aug 25 11:38:40 2015: <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
&red https://xymon.REDACTED.com/xymon/ - <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"> <html><head> <title>400 Bad Request</title> </head><body> <h1>Bad Request</h1> <p>Your browser sent a request that this server could not understand.<br /> Reason: You're speaking plain HTTP to an SSL-enabled server port.<br /> Instead use the HTTPS scheme to access this URL, please.<br /> <blockquote>Hint: <a href="https://xymon.REDACTED.com/"><b>https://xymon.REDACTED.com/</b></a></blockquote></p> <hr> <address>Apache/2.2.15 (CentOS) Server at xymon.REDACTED.com Port 443</address> </body></html>
It looks like what's happening is that xymon's http test has forgotten how to speak HTTPS. It actually looks like it has forgotten how to speal SSL entirely -- imaps and pop3s are showing yellow status with "Unexpected service response" messages.
I am now also seeing purple alarms from sslcert, because none of the SSL stuff is working right.
How can I fix this?
Thanks, Shawn
On 8/25/2015 11:53 AM, Shawn Heisey wrote:
I upgraded to 4.3.21 today, from 4.3.14. It was built using this command:
build/makerpm.sh 4.3.21
<snip>
It looks like what's happening is that xymon's http test has forgotten how to speak HTTPS. It actually looks like it has forgotten how to speal SSL entirely -- imaps and pop3s are showing yellow status with "Unexpected service response" messages.
I am now also seeing purple alarms from sslcert, because none of the SSL stuff is working right.
How can I fix this?
Followup: Downgrading to 4.3.14 again (and once again fighting with RPM's insistence that my configs need to be replaced) has restored everything.
Side note: 4.3.21 didn't fix my ongoing problems with ATT0001.bin attachments on msgs alarms from BBWin clients. I *have* implemented the known fix using the "tr" command to eliminate carriage returns, but apparently that isn't working.
Thanks, Shawn
On Tue, August 25, 2015 10:53 am, Shawn Heisey wrote:
I upgraded to 4.3.21 today, from 4.3.14. It was built using this command:
build/makerpm.sh 4.3.21
After working my way through recovering my configs that were pushed out of the way as .rpmsave files, I started it. Immediately I got http alarms failing.
The most interesting one was this. I have xymon monitoring its own URL, which is secured with SSL:
red Tue Aug 25 11:38:40 2015: <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
&red https://xymon.REDACTED.com/xymon/ - <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"> <html><head> <title>400 Bad Request</title> </head><body> <h1>Bad Request</h1> <p>Your browser sent a request that this server could not understand.<br /> Reason: You're speaking plain HTTP to an SSL-enabled server port.<br /> Instead use the HTTPS scheme to access this URL, please.<br /> <blockquote>Hint: <a href="https://xymon.REDACTED.com/"><b>https://xymon.REDACTED.com/</b></a></blockquote></p> <hr> <address>Apache/2.2.15 (CentOS) Server at xymon.REDACTED.com Port 443</address> </body></html>
It looks like what's happening is that xymon's http test has forgotten how to speak HTTPS. It actually looks like it has forgotten how to speal SSL entirely -- imaps and pop3s are showing yellow status with "Unexpected service response" messages.
I am now also seeing purple alarms from sslcert, because none of the SSL stuff is working right.
How can I fix this?
Shawn,
Hmm. Most likely, xymon wasn't built with OpenSSL included. The quickest way to tell is to look at your "xymonnet" test (a snapshot from when 4.3.21 was running). If the OpenSSL version isn't indicated in the first few lines, that's definitely the problem.
Do you have build logs from when this was originally run? And can you indicate your distro and openssl-devel version?
Regards,
-jc
On 8/25/2015 1:13 PM, J.C. Cleaver wrote:
Hmm. Most likely, xymon wasn't built with OpenSSL included. The quickest way to tell is to look at your "xymonnet" test (a snapshot from when 4.3.21 was running). If the OpenSSL version isn't indicated in the first few lines, that's definitely the problem.
Do you have build logs from when this was originally run? And can you indicate your distro and openssl-devel version?
I completely deleted the extracted source archive, re-extracted it, and then did these commands:
cd xymon-4.3.21 build/makerpm.sh 4.3.21 > build.out.txt 2>&1
In the build.out.txt file, I see these lines:
make: *** [test-link] Error 1 Warning: Cannot link with SSL library OpenSSL include- or library-files not found.
If you want the full gory details, the logfile can be found here:
https://www.dropbox.com/s/g6f4d2scf8h3f7q/build.out.txt?dl=0
OpenSSL *is* installed, this is CentOS 6.7:
[root at mcp xymon-4.3.21]# rpm -qa | grep openssl openssl-devel-1.0.1e-42.el6.x86_64 openssl-1.0.1e-42.el6.x86_64
Also, I built 4.3.14 on this same system and it did work. This was back in January of 2014, a few days after I did the initial OS install on this server.
Running "yum distro-sync" did not install any relevant package updates, but I would imagine that OpenSSL has been upgraded at least once in the year and a half since 4.3.14 was built.
Resolving Dependencies --> Running transaction check ---> Package httpd.x86_64 0:2.2.15-45.el6.centos will be updated ---> Package httpd.x86_64 0:2.2.15-47.el6.centos will be an update ---> Package httpd-tools.x86_64 0:2.2.15-45.el6.centos will be updated ---> Package httpd-tools.x86_64 0:2.2.15-47.el6.centos will be an update ---> Package mod_ssl.x86_64 1:2.2.15-45.el6.centos will be updated ---> Package mod_ssl.x86_64 1:2.2.15-47.el6.centos will be an update ---> Package pam.x86_64 0:1.1.1-20.el6 will be updated ---> Package pam.x86_64 0:1.1.1-20.el6_7.1 will be an update ---> Package sqlite.x86_64 0:3.6.20-1.el6 will be updated ---> Package sqlite.x86_64 0:3.6.20-1.el6_7.2 will be an update ---> Package subversion.x86_64 0:1.6.11-14.el6 will be updated ---> Package subversion.x86_64 0:1.6.11-15.el6_7 will be an update ---> Package tzdata.noarch 0:2015e-1.el6 will be updated ---> Package tzdata.noarch 0:2015f-1.el6 will be an update ---> Package tzdata-java.noarch 0:2015e-1.el6 will be updated ---> Package tzdata-java.noarch 0:2015f-1.el6 will be an update --> Finished Dependency Resolution
Suggestion: As soon as the openssl error was encountered, the entire build should have been aborted. There should not have been an RPM for me to install. I did not think to check the log for errors ... I figured that since the RPM was created, everything was fine.
Thanks, Shawn
Do you have the openssl-devel package installed? You need it.
-----Original Message----- From: Xymon [mailto:xymon-bounces at xymon.com] On Behalf Of Shawn Heisey Sent: Tuesday, August 25, 2015 3:30 PM To: xymon at xymon.com Subject: Re: [Xymon] Upgrade to 4.3.21, https tests seem to no longer work.
On 8/25/2015 1:13 PM, J.C. Cleaver wrote:
Hmm. Most likely, xymon wasn't built with OpenSSL included. The quickest way to tell is to look at your "xymonnet" test (a snapshot from when 4.3.21 was running). If the OpenSSL version isn't indicated in the first few lines, that's definitely the problem.
Do you have build logs from when this was originally run? And can you indicate your distro and openssl-devel version?
I completely deleted the extracted source archive, re-extracted it, and then did these commands:
cd xymon-4.3.21 build/makerpm.sh 4.3.21 > build.out.txt 2>&1
In the build.out.txt file, I see these lines:
make: *** [test-link] Error 1 Warning: Cannot link with SSL library OpenSSL include- or library-files not found.
If you want the full gory details, the logfile can be found here:
https://www.dropbox.com/s/g6f4d2scf8h3f7q/build.out.txt?dl=0
OpenSSL *is* installed, this is CentOS 6.7:
[root at mcp xymon-4.3.21]# rpm -qa | grep openssl openssl-devel-1.0.1e-42.el6.x86_64 openssl-1.0.1e-42.el6.x86_64
Also, I built 4.3.14 on this same system and it did work. This was back in January of 2014, a few days after I did the initial OS install on this server.
Running "yum distro-sync" did not install any relevant package updates, but I would imagine that OpenSSL has been upgraded at least once in the year and a half since 4.3.14 was built.
Resolving Dependencies --> Running transaction check ---> Package httpd.x86_64 0:2.2.15-45.el6.centos will be updated ---> Package httpd.x86_64 0:2.2.15-47.el6.centos will be an update ---> Package httpd-tools.x86_64 0:2.2.15-45.el6.centos will be updated ---> Package httpd-tools.x86_64 0:2.2.15-47.el6.centos will be an update ---> Package mod_ssl.x86_64 1:2.2.15-45.el6.centos will be updated ---> Package mod_ssl.x86_64 1:2.2.15-47.el6.centos will be an update ---> Package pam.x86_64 0:1.1.1-20.el6 will be updated ---> Package pam.x86_64 0:1.1.1-20.el6_7.1 will be an update ---> Package sqlite.x86_64 0:3.6.20-1.el6 will be updated ---> Package sqlite.x86_64 0:3.6.20-1.el6_7.2 will be an update ---> Package subversion.x86_64 0:1.6.11-14.el6 will be updated ---> Package subversion.x86_64 0:1.6.11-15.el6_7 will be an update ---> Package tzdata.noarch 0:2015e-1.el6 will be updated ---> Package tzdata.noarch 0:2015f-1.el6 will be an update ---> Package tzdata-java.noarch 0:2015e-1.el6 will be updated ---> Package tzdata-java.noarch 0:2015f-1.el6 will be an update --> Finished Dependency Resolution
Suggestion: As soon as the openssl error was encountered, the entire build should have been aborted. There should not have been an RPM for me to install. I did not think to check the log for errors ... I figured that since the RPM was created, everything was fine.
Thanks, Shawn
Xymon mailing list Xymon at xymon.com http://lists.xymon.com/mailman/listinfo/xymon This communication is the property of CenturyLink and may contain confidential or privileged information. Unauthorized use of this communication is strictly prohibited and may be unlawful. If you have received this communication in error, please immediately notify the sender by reply e-mail and destroy all copies of the communication and any attachments.
Do you need to install openssl-devel?
g
-----Original Message----- From: Xymon [mailto:xymon-bounces at xymon.com] On Behalf Of Shawn Heisey Sent: Tuesday, August 25, 2015 15:30 To: xymon at xymon.com Subject: Re: [Xymon] Upgrade to 4.3.21, https tests seem to no longer work.
On 8/25/2015 1:13 PM, J.C. Cleaver wrote:
Hmm. Most likely, xymon wasn't built with OpenSSL included. The quickest way to tell is to look at your "xymonnet" test (a snapshot from when 4.3.21 was running). If the OpenSSL version isn't indicated in the first few lines, that's definitely the problem.
Do you have build logs from when this was originally run? And can you indicate your distro and openssl-devel version?
I completely deleted the extracted source archive, re-extracted it, and then did these commands:
cd xymon-4.3.21 build/makerpm.sh 4.3.21 > build.out.txt 2>&1
In the build.out.txt file, I see these lines:
make: *** [test-link] Error 1 Warning: Cannot link with SSL library OpenSSL include- or library-files not found.
If you want the full gory details, the logfile can be found here:
https://www.dropbox.com/s/g6f4d2scf8h3f7q/build.out.txt?dl=0
OpenSSL *is* installed, this is CentOS 6.7:
[root at mcp xymon-4.3.21]# rpm -qa | grep openssl openssl-devel-1.0.1e-42.el6.x86_64 openssl-1.0.1e-42.el6.x86_64
Also, I built 4.3.14 on this same system and it did work. This was back in January of 2014, a few days after I did the initial OS install on this server.
Running "yum distro-sync" did not install any relevant package updates, but I would imagine that OpenSSL has been upgraded at least once in the year and a half since 4.3.14 was built.
Resolving Dependencies --> Running transaction check ---> Package httpd.x86_64 0:2.2.15-45.el6.centos will be updated ---> Package httpd.x86_64 0:2.2.15-47.el6.centos will be an update ---> Package httpd-tools.x86_64 0:2.2.15-45.el6.centos will be updated ---> Package httpd-tools.x86_64 0:2.2.15-47.el6.centos will be an update ---> Package mod_ssl.x86_64 1:2.2.15-45.el6.centos will be updated ---> Package mod_ssl.x86_64 1:2.2.15-47.el6.centos will be an update ---> Package pam.x86_64 0:1.1.1-20.el6 will be updated ---> Package pam.x86_64 0:1.1.1-20.el6_7.1 will be an update ---> Package sqlite.x86_64 0:3.6.20-1.el6 will be updated ---> Package sqlite.x86_64 0:3.6.20-1.el6_7.2 will be an update ---> Package subversion.x86_64 0:1.6.11-14.el6 will be updated ---> Package subversion.x86_64 0:1.6.11-15.el6_7 will be an update ---> Package tzdata.noarch 0:2015e-1.el6 will be updated ---> Package tzdata.noarch 0:2015f-1.el6 will be an update ---> Package tzdata-java.noarch 0:2015e-1.el6 will be updated ---> Package tzdata-java.noarch 0:2015f-1.el6 will be an update --> Finished Dependency Resolution
Suggestion: As soon as the openssl error was encountered, the entire build should have been aborted. There should not have been an RPM for me to install. I did not think to check the log for errors ... I figured that since the RPM was created, everything was fine.
Thanks, Shawn
Xymon mailing list Xymon at xymon.com http://lists.xymon.com/mailman/listinfo/xymon
On 8/25/2015 2:30 PM, Shawn Heisey wrote:
I completely deleted the extracted source archive, re-extracted it, and then did these commands:
cd xymon-4.3.21 build/makerpm.sh 4.3.21 > build.out.txt 2>&1
In the build.out.txt file, I see these lines:
make: *** [test-link] Error 1 Warning: Cannot link with SSL library OpenSSL include- or library-files not found.
If you want the full gory details, the logfile can be found here:
https://www.dropbox.com/s/g6f4d2scf8h3f7q/build.out.txt?dl=0
OpenSSL *is* installed, this is CentOS 6.7:
[root at mcp xymon-4.3.21]# rpm -qa | grep openssl openssl-devel-1.0.1e-42.el6.x86_64 openssl-1.0.1e-42.el6.x86_64
Just to be complete, I repeated these action with the 4.3.14 source archive. The same link error happens with the older source, so it's definitely a problem with the RH/CentOS openssl-devel package.
It's a good thing I still had the RPM from when I initially built 4.3.14 a year and a half ago!
To those who replied asking if I installed openssl-devel ... you didn't read everything I wrote. It's included above, but just in case that's not obvious, here it is again:
OpenSSL *is* installed, this is CentOS 6.7:
[root at mcp xymon-4.3.21]# rpm -qa | grep openssl openssl-devel-1.0.1e-42.el6.x86_64 openssl-1.0.1e-42.el6.x86_64
Thanks, Shawn
You might want to try
rpm--verify openssl-develin case something happened that made the files go away, or become inaccessible.
Ralph Mitchell
On Tue, Aug 25, 2015 at 4:46 PM, Shawn Heisey <hobbit at elyograg.org> wrote:
On 8/25/2015 2:30 PM, Shawn Heisey wrote:
I completely deleted the extracted source archive, re-extracted it, and then did these commands:
cd xymon-4.3.21 build/makerpm.sh 4.3.21 > build.out.txt 2>&1
In the build.out.txt file, I see these lines:
make: *** [test-link] Error 1 Warning: Cannot link with SSL library OpenSSL include- or library-files not found.
If you want the full gory details, the logfile can be found here:
https://www.dropbox.com/s/g6f4d2scf8h3f7q/build.out.txt?dl=0
OpenSSL *is* installed, this is CentOS 6.7:
[root at mcp xymon-4.3.21]# rpm -qa | grep openssl openssl-devel-1.0.1e-42.el6.x86_64 openssl-1.0.1e-42.el6.x86_64
Just to be complete, I repeated these action with the 4.3.14 source archive. The same link error happens with the older source, so it's definitely a problem with the RH/CentOS openssl-devel package.
It's a good thing I still had the RPM from when I initially built 4.3.14 a year and a half ago!
To those who replied asking if I installed openssl-devel ... you didn't read everything I wrote. It's included above, but just in case that's not obvious, here it is again:
OpenSSL *is* installed, this is CentOS 6.7:
[root at mcp xymon-4.3.21]# rpm -qa | grep openssl openssl-devel-1.0.1e-42.el6.x86_64 openssl-1.0.1e-42.el6.x86_64
Thanks, Shawn
Xymon mailing list Xymon at xymon.com http://lists.xymon.com/mailman/listinfo/xymon
On 8/25/2015 3:37 PM, Ralph Mitchell wrote:
You might want to try
rpm--verify openssl-develin case something happened that made the files go away, or become inaccessible.
That command returned no output. On a Linux machine, that usually means everything's good.
I also did "yum reinstall openssl openssl-devel" which downloaded the packages and reinstalled them. Xymon still wouldn't link right.
I do have OpenSSL 1.0.2a installed in /usr/local/ssl. There is a symlink at /usr/local/bin/ossl pointing to /usr/local/ssl/bin/openssl. I would not expect this to cause any problems, but do you think it might be causing a problem? I only installed this so I could run a newer version when I create certificate signing requests. Nothing on this server is linked against that OpenSSL installation.
On the off chance that the additional openssl install was causing problems, I did these commands:
cd /usr/local tar zcf ssl.tar.gz ssl rm -rf ssl
When I tried the build again, suddenly it all worked. Looking at the build log a second time, now I notice that it is finding libraries in /usr/local/ssl, which I didn't notice the first time.
Is there any way to make this work right? If it detected the install in /usr/local/ssl, shouldn't it have used it? Is Xymon not compatible with the latest OpenSSL versions?
I probably compiled the additional OpenSSL so it is statically linked. I lifted the commandline to compile OpenSSL from my haproxy servers. I needed the latest openssl for haproxy, and haproxy will not work right if you build it with a locally compiled openssl but also have the openssl package (NOT the dev package!) from the distro installed.
Thanks, Shawn
On Tue, August 25, 2015 3:12 pm, Shawn Heisey wrote:
On 8/25/2015 3:37 PM, Ralph Mitchell wrote:
You might want to try
rpm--verify openssl-develin case something happened that made the files go away, or become inaccessible.
That command returned no output. On a Linux machine, that usually means everything's good.
I also did "yum reinstall openssl openssl-devel" which downloaded the packages and reinstalled them. Xymon still wouldn't link right.
I do have OpenSSL 1.0.2a installed in /usr/local/ssl. There is a symlink at /usr/local/bin/ossl pointing to /usr/local/ssl/bin/openssl. I would not expect this to cause any problems, but do you think it might be causing a problem? I only installed this so I could run a newer version when I create certificate signing requests. Nothing on this server is linked against that OpenSSL installation.
On the off chance that the additional openssl install was causing problems, I did these commands:
cd /usr/local tar zcf ssl.tar.gz ssl rm -rf ssl
When I tried the build again, suddenly it all worked. Looking at the build log a second time, now I notice that it is finding libraries in /usr/local/ssl, which I didn't notice the first time.
Is there any way to make this work right? If it detected the install in /usr/local/ssl, shouldn't it have used it? Is Xymon not compatible with the latest OpenSSL versions?
I probably compiled the additional OpenSSL so it is statically linked. I lifted the commandline to compile OpenSSL from my haproxy servers. I needed the latest openssl for haproxy, and haproxy will not work right if you build it with a locally compiled openssl but also have the openssl package (NOT the dev package!) from the distro installed.
Thanks, Shawn
Generally speaking, having multiple copies of the library -- some static, some dynamic -- is rather likely to cause runtime problems.
Instead of makerpm.sh, can you run the ./configure --server script manually (followed by make, etc)? That should be able to tell us more precisely what's happening.
The specific path order that's cycled through for locating the libraries will be in build/ssl.sh in the tarball distribution.
For the record, there are also pre-built RPMs for RHEL/Fedora at http://terabithia.org/rpms/xymon/, although there are deltas between that and the originating tarball. I'm curious if either a) those binary RPMs work for you, or b) the SRPM can rebuilt cleanly.
Regards,
-jc
participants (5)
-
cleaver＠terabithia.org
-
glauber.ribeiro＠experian.com
-
hobbit＠elyograg.org
-
Paul.Root＠CenturyLink.com
-
ralphmitchell＠gmail.com