fedora upgrade from 13 to 14 crashed hobbitd_rrd
- Program crashed
Fatal signal caught!
Anyone know how I can resolve this issue? The tail end of rrd-status.log looks like this:
2010-11-11 14:43:34 RRD error updating /home/hobbit/data/rrd/xxx.yyy.com/disk,H.rrd from 192.168.0.0: This RRD was created on another architecture
The funny thing is, the architecture never changed. I just upgraded Fedora Core from version 13 to 14.
Jason Chambers IT Help Desk Associate
GEOSOFT INC. freedom to explore T +1 416.369.0111 #344 F +1 416.369.9599
Visit our site at www.geosoft.com
Just by curiosity, I tried creating a new rrd directory and things didn't improve at all:
rrd_dontneed: Argument 'rrd_file' is NULL. 2010-11-11 12:40:44 Peer at 0.0.0.0:0 failed: Broken pipe 2010-11-11 12:40:45 Peer not up, flushing message queue 2010-11-11 12:40:46 Peer not up, flushing message queue 2010-11-11 12:40:47 Peer not up, flushing message queue 2010-11-11 12:40:47 Peer not up, flushing message queue 2010-11-11 12:40:49 Peer not up, flushing message queue
Jason Chambers IT Help Desk Associate
GEOSOFT INC. freedom to explore T +1 416.369.0111 #344 F +1 416.369.9599
Visit our site at www.geosoft.com
From: Jason Chambers [mailto:Jason.Chambers at geosoft.com] Sent: November-11-10 2:45 PM To: 'xymon at xymon.com' Subject: [xymon] fedora upgrade from 13 to 14 crashed hobbitd_rrd
- Program crashed
Fatal signal caught!
Anyone know how I can resolve this issue? The tail end of rrd-status.log looks like this:
2010-11-11 14:43:34 RRD error updating /home/hobbit/data/rrd/xxx.yyy.com/disk,H.rrd from 192.168.0.0: This RRD was created on another architecture
The funny thing is, the architecture never changed. I just upgraded Fedora Core from version 13 to 14.
Jason Chambers IT Help Desk Associate
GEOSOFT INC. freedom to explore T +1 416.369.0111 #344 F +1 416.369.9599
Visit our site at www.geosoft.com<http://www.geosoft.com>
I get the same issue on RedHat AS 5.2 (running 4.3.0-0.beta2) once in a while. It happened two days ago. I find, if I don't restart all of the Xymon processes, it happens more frequently and some of my graphs get very screwed up.
......Bruce
Bruce White Senior Enterprise Systems Engineer | Phone: 630-671-5169 | Fax: 630-893-1648 | bewhite at fellowes.com | http://www.fellowes.com/ Disclaimer: The information contained in this message may be privileged and confidential and protected from disclosure. If the reader of this message is not the intended recipient or an employee or agent responsible for delivering this message to the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you have received this communication in error, please notify us immediately by replying to the message and deleting it from your computer. Thank you. Fellowes, Inc.
From: Jason Chambers [mailto:Jason.Chambers at geosoft.com] Sent: Thursday, November 11, 2010 1:45 PM To: 'xymon at xymon.com' Subject: [xymon] fedora upgrade from 13 to 14 crashed hobbitd_rrd
- Program crashed
Fatal signal caught!
Anyone know how I can resolve this issue? The tail end of rrd-status.log looks like this:
2010-11-11 14:43:34 RRD error updating /home/hobbit/data/rrd/xxx.yyy.com/disk,H.rrd from 192.168.0.0: This RRD was created on another architecture
The funny thing is, the architecture never changed. I just upgraded Fedora Core from version 13 to 14.
Jason Chambers IT Help Desk Associate
GEOSOFT INC. freedom to explore T +1 416.369.0111 #344 F +1 416.369.9599
Visit our site at www.geosoft.com
Hi Bruce,
I have actually tried restarting the service all the time. The RRD daemon just crashes immediately. Quite confusing.
Jason Chambers IT Help Desk Associate
GEOSOFT INC. freedom to explore T +1 416.369.0111 #344 F +1 416.369.9599
Visit our site at www.geosoft.com
From: White, Bruce [mailto:bewhite at fellowes.com] Sent: November-11-10 3:39 PM To: xymon at xymon.com Subject: RE: [xymon] fedora upgrade from 13 to 14 crashed hobbitd_rrd
I get the same issue on RedHat AS 5.2 (running 4.3.0-0.beta2) once in a while. It happened two days ago. I find, if I don’t restart all of the Xymon processes, it happens more frequently and some of my graphs get very screwed up.
……BruceBruce White
Senior Enterprise Systems Engineer | Phone: 630-671-5169 | Fax: 630-893-1648 | bewhite at fellowes.com<mailto:bewhite at fellowes.com> | www.fellowes.com<http://www.fellowes.com/>
[cid:image001.jpg at 01CB81BC.20EC1A50]
Disclaimer: The information contained in this message may be privileged and confidential and protected from disclosure. If the reader of this message is not the intended recipient or an employee or agent responsible for delivering this message to the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you have received this communication in error, please notify us immediately by replying to the message and deleting it from your computer. Thank you. Fellowes, Inc.
From: Jason Chambers [mailto:Jason.Chambers at geosoft.com] Sent: Thursday, November 11, 2010 1:45 PM To: 'xymon at xymon.com' Subject: [xymon] fedora upgrade from 13 to 14 crashed hobbitd_rrd
- Program crashed
Fatal signal caught!
Anyone know how I can resolve this issue? The tail end of rrd-status.log looks like this:
2010-11-11 14:43:34 RRD error updating /home/hobbit/data/rrd/xxx.yyy.com/disk,H.rrd from 192.168.0.0: This RRD was created on another architecture
The funny thing is, the architecture never changed. I just upgraded Fedora Core from version 13 to 14.
Jason Chambers IT Help Desk Associate
GEOSOFT INC. freedom to explore T +1 416.369.0111 #344 F +1 416.369.9599
Visit our site at www.geosoft.com<http://www.geosoft.com>
In <018528ADB17A124D84DD0D081A4C82D22B5C61FB at exchange.geosoft.com> Jason Chambers <Jason.Chambers at geosoft.com> writes:
- Program crashed
Fatal signal caught!
Anyone know how I can resolve this issue? The tail end of rrd-status.log lo= oks like this:
2010-11-11 14:43:34 RRD error updating /home/hobbit/data/rrd/xxx.yyy.com/di= sk,H.rrd from 192.168.0.0: This RRD was created on another architecture
The funny thing is, the architecture never changed. I just upgraded Fedora = Core from version 13 to 14.
Could it be an upgrade from a 32-bit to 64-bit version of the RRD library?
To begin with, I would definitely re-compile Xymon with the versions of librrd that are now in place on your new system. Just to rule out any problems with the runtime library having changed from the compile- time setup.
Regards, Henrik
Hi Henrik,
This is actually an in place upgrade using their PreUpgrade tool described here: http://fedoraproject.org/wiki/PreUpgrade . The bit version did not change. I also used the GUI interface (of yum I would assume) to uninstall and reinstall the RRD tool without any luck.
I have also done a make clean and started a new configuration of the xymon compilation without any luck. This is using the 4.3 beta2 code.
Thankfully I am getting alerts and the xymon tool itself is working. Just no RRD right now.
Jason Chambers IT Help Desk Associate
GEOSOFT INC. freedom to explore T +1 416.369.0111 #344 F +1 416.369.9599
Visit our site at www.geosoft.com
-----Original Message----- From: Henrik Størner [mailto:henrik at hswn.dk] Sent: November-11-10 3:45 PM To: xymon at xymon.com Subject: Re: [xymon] fedora upgrade from 13 to 14 crashed hobbitd_rrd
In <018528ADB17A124D84DD0D081A4C82D22B5C61FB at exchange.geosoft.com> Jason Chambers <Jason.Chambers at geosoft.com> writes:
- Program crashed
Fatal signal caught!
Anyone know how I can resolve this issue? The tail end of rrd-status.log lo= oks like this:
2010-11-11 14:43:34 RRD error updating /home/hobbit/data/rrd/xxx.yyy.com/di= sk,H.rrd from 192.168.0.0: This RRD was created on another architecture
The funny thing is, the architecture never changed. I just upgraded Fedora = Core from version 13 to 14.
Could it be an upgrade from a 32-bit to 64-bit version of the RRD library?
To begin with, I would definitely re-compile Xymon with the versions of librrd that are now in place on your new system. Just to rule out any problems with the runtime library having changed from the compile- time setup.
Regards, Henrik
To unsubscribe from the xymon list, send an e-mail to xymon-unsubscribe at xymon.com
On Thu, 11 Nov 2010 20:45:24 +0000, Henrik "Størner" wrote:
In <018528ADB17A124D84DD0D081A4C82D22B5C61FB at exchange.geosoft.com> Jason Chambers <Jason.Chambers at geosoft.com> writes:
- Program crashed
Fatal signal caught!
Anyone know how I can resolve this issue? The tail end of rrd-status.log lo= oks like this:
2010-11-11 14:43:34 RRD error updating /home/hobbit/data/rrd/xxx.yyy.com/di= sk,H.rrd from 192.168.0.0: This RRD was created on another architecture
The funny thing is, the architecture never changed. I just upgraded Fedora = Core from version 13 to 14.
Could it be an upgrade from a 32-bit to 64-bit version of the RRD library?
I think the problem has been found, but it might be difficult to fix. Look at the Bugzilla discussion in the Fedora mailing list: https://bugzilla.redhat.com/show_bug.cgi?id=638477#c29
Apparently the glibc folks implemented a new version of the glibc routine that does memory-copies fin Fedora 14. And unlike the old version, this one corrupts data when copying between memory areas that overlap.
Stricly speaking, doing that is not valid. But it is apparently quite common - the Bugzilla entry lists quite a few apps that break because of this change - and it has worked until the change in glibc. So the glibc maintainers is taking the "not a bug - won't fix" stance.
In which case the problem has to be fixed in all of the applications and libraries that do the invalid memcpy, i.e. the rrdtool library in this case.
Regards, Henrik
On Thu, November 11, 2010 17:06, Henrik Størner wrote:
On Thu, 11 Nov 2010 20:45:24 +0000, Henrik "Størner" wrote:
In <018528ADB17A124D84DD0D081A4C82D22B5C61FB at exchange.geosoft.com> Jason Chambers <Jason.Chambers at geosoft.com> writes:
- Program crashed
Fatal signal caught!
Anyone know how I can resolve this issue? The tail end of rrd-status.log lo= oks like this:
2010-11-11 14:43:34 RRD error updating /home/hobbit/data/rrd/xxx.yyy.com/di= sk,H.rrd from 192.168.0.0: This RRD was created on another architecture
The funny thing is, the architecture never changed. I just upgraded Fedora = Core from version 13 to 14.
Could it be an upgrade from a 32-bit to 64-bit version of the RRD library?
I think the problem has been found, but it might be difficult to fix. Look at the Bugzilla discussion in the Fedora mailing list: https://bugzilla.redhat.com/show_bug.cgi?id=638477#c29
Apparently the glibc folks implemented a new version of the glibc routine that does memory-copies fin Fedora 14. And unlike the old version, this one corrupts data when copying between memory areas that overlap.
Interesting bug thread. So the new glibc broke Flash (and rrd), and who knows what else. And even Linus Torvalds thinks it's wrong and unnecessary. And the glibc team seem to be adamant about not changing it.
BSD, anyone?
*face palm*
OK, Let's see if I can figure out how to downgrade glibc.
Note: not a linux user :)
Thanks for this information guys.
Jason Chambers IT Help Desk Associate
GEOSOFT INC. freedom to explore T +1 416.369.0111 #344 F +1 416.369.9599
Visit our site at www.geosoft.com
-----Original Message----- From: Xymon User in Richmond [mailto:hobbit at epperson.homelinux.net] Sent: November-11-10 6:48 PM To: xymon at xymon.com Subject: Re: [xymon] fedora upgrade from 13 to 14 crashed hobbitd_rrd
On Thu, November 11, 2010 17:06, Henrik Størner wrote:
On Thu, 11 Nov 2010 20:45:24 +0000, Henrik "Størner" wrote:
In <018528ADB17A124D84DD0D081A4C82D22B5C61FB at exchange.geosoft.com> Jason Chambers <Jason.Chambers at geosoft.com> writes:
- Program crashed
Fatal signal caught!
Anyone know how I can resolve this issue? The tail end of rrd-status.log lo= oks like this:
2010-11-11 14:43:34 RRD error updating /home/hobbit/data/rrd/xxx.yyy.com/di= sk,H.rrd from 192.168.0.0: This RRD was created on another architecture
The funny thing is, the architecture never changed. I just upgraded Fedora = Core from version 13 to 14.
Could it be an upgrade from a 32-bit to 64-bit version of the RRD library?
I think the problem has been found, but it might be difficult to fix. Look at the Bugzilla discussion in the Fedora mailing list: https://bugzilla.redhat.com/show_bug.cgi?id=638477#c29
Apparently the glibc folks implemented a new version of the glibc routine that does memory-copies fin Fedora 14. And unlike the old version, this one corrupts data when copying between memory areas that overlap.
Interesting bug thread. So the new glibc broke Flash (and rrd), and who knows what else. And even Linus Torvalds thinks it's wrong and unnecessary. And the glibc team seem to be adamant about not changing it.
BSD, anyone?
To unsubscribe from the xymon list, send an e-mail to xymon-unsubscribe at xymon.com
Hi Jason,
I'm certainly no expert on this, but in the Bugzilla discussion in the Fedora mailing list: https://bugzilla.redhat.com/show_bug.cgi?id=638477#c29 That Henrik quoted below, there is a work-around posted originally by Linus Torvalds (and reposted a bit more friendly here): https://bugzilla.redhat.com/show_bug.cgi?id=638477#c55 This may also work for rrd. Try replacing /usr/bin/firefox in that post with the command to execute hobbitd_rrd, or perhaps more precisely, try prepending: LD_PRELOAD=$HOME/Downloads/linusmemcpy.so to that command. Off the top of my head, I'm not sure where you change that command though in the hobbit config or code. I presume that it can be done though?
Kind regards,
SebA
-----Original Message----- From: Jason Chambers [mailto:Jason.Chambers at geosoft.com] Sent: 12 November 2010 13:53 To: 'xymon at xymon.com' Subject: RE: [xymon] fedora upgrade from 13 to 14 crashed hobbitd_rrd
*face palm*
OK, Let's see if I can figure out how to downgrade glibc.
Note: not a linux user :)
Thanks for this information guys.
Jason Chambers IT Help Desk Associate
GEOSOFT INC. freedom to explore T +1 416.369.0111 #344 F +1 416.369.9599
Visit our site at www.geosoft.com
-----Original Message----- From: Xymon User in Richmond [mailto:hobbit at epperson.homelinux.net] Sent: November-11-10 6:48 PM To: xymon at xymon.com Subject: Re: [xymon] fedora upgrade from 13 to 14 crashed hobbitd_rrd
On Thu, November 11, 2010 17:06, Henrik Størner wrote:
On Thu, 11 Nov 2010 20:45:24 +0000, Henrik "Størner" wrote:
In <018528ADB17A124D84DD0D081A4C82D22B5C61FB at exchange.geosoft.com> Jason Chambers <Jason.Chambers at geosoft.com> writes:
- Program crashed
Fatal signal caught!
Anyone know how I can resolve this issue? The tail end of rrd-status.log lo= oks like this:
2010-11-11 14:43:34 RRD error updating /home/hobbit/data/rrd/xxx.yyy.com/di= sk,H.rrd from 192.168.0.0: This RRD was created on another architecture
The funny thing is, the architecture never changed. I just upgraded Fedora = Core from version 13 to 14.
Could it be an upgrade from a 32-bit to 64-bit version of the RRD library?
I think the problem has been found, but it might be difficult to fix. Look at the Bugzilla discussion in the Fedora mailing list: https://bugzilla.redhat.com/show_bug.cgi?id=638477#c29
Apparently the glibc folks implemented a new version of the glibc routine that does memory-copies fin Fedora 14. And unlike the old version, this one corrupts data when copying between memory areas that overlap.
Interesting bug thread. So the new glibc broke Flash (and rrd), and who knows what else. And even Linus Torvalds thinks it's wrong and unnecessary. And the glibc team seem to be adamant about not changing it.
BSD, anyone?
To unsubscribe from the xymon list, send an e-mail to xymon-unsubscribe at xymon.com
To unsubscribe from the xymon list, send an e-mail to xymon-unsubscribe at xymon.com
No virus found in this incoming message. Checked by AVG - www.avg.com Version: 8.5.449 / Virus Database: 271.1.1/3238 - Release Date: 11/07/10 19:34:00
No virus found in this outgoing message. Checked by AVG - www.avg.com Version: 8.5.449 / Virus Database: 271.1.1/3238 - Release Date: 11/07/10 19:34:00
On Fri, November 12, 2010 08:53, Jason Chambers wrote:
*face palm*
OK, Let's see if I can figure out how to downgrade glibc.
Note: not a linux user :)
Off the top of my head:
Copy /etc/yum.repos.d/fedora-updates.repo to /etc/yum.repos.d/fedora-updates12.repo
Edit the new file, replacing all "$releasever" with "13" and set "enabled=1" to "enabled=0". Change every "updates" inside square brackets to "updates12". Then use yum --enablerepo=updates12 downgrade install glibc-2.12.1-4 glibc-common-2.12.1-4 glibc-devel-2.12.1-4 glibc-headers-2.12.1-4 to do the actual downgrade.
I don't have the means to test this for you right now, so ymmv.
Next time: clone your Xymon server on a PC with similar processor architecture and bring it up on a different IP. Point a couple of clients at it (they can feed multiple Xymon servers each). Upgrade it and test things out. And don't use Fedora on production machines unless you're well prepared to troubleshoot things that are broken by cutting-edge stuff. I use Fedora at home, but Centos/RHEL/OEL at work.
Good news.
RRD Group released a fix for this problem. I'm able to see graphs again, but obviously lost a few weeks of data there when it didn't work. No major problem for me here. Just thought I send an update just in case someone else is using Fedora.
Jason Chambers IT Help Desk Associate
GEOSOFT INC. freedom to explore T +1 416.369.0111 #344 F +1 416.369.9599
Visit our site at www.geosoft.com
From: Jason Chambers [mailto:Jason.Chambers at geosoft.com] Sent: November-11-10 2:45 PM To: 'xymon at xymon.com' Subject: [xymon] fedora upgrade from 13 to 14 crashed hobbitd_rrd
- Program crashed
Fatal signal caught!
Anyone know how I can resolve this issue? The tail end of rrd-status.log looks like this:
2010-11-11 14:43:34 RRD error updating /home/hobbit/data/rrd/xxx.yyy.com/disk,H.rrd from 192.168.0.0: This RRD was created on another architecture
The funny thing is, the architecture never changed. I just upgraded Fedora Core from version 13 to 14.
Jason Chambers IT Help Desk Associate
GEOSOFT INC. freedom to explore T +1 416.369.0111 #344 F +1 416.369.9599
Visit our site at www.geosoft.com<http://www.geosoft.com>
participants (5)
-
bewhite＠fellowes.com
-
henrik＠hswn.dk
-
hobbit＠epperson.homelinux.net
-
Jason.Chambers＠geosoft.com
-
spa＠syntec.co.uk