[hobbit] hobbit.sh not stopping server correctly
Hi Buchan,
Apologies, I'll explain it in more detail. We are running on Solaris 10 and trying to configure Xymon as an SMF service (which essentially uses stop/start scripts). The stop script is basically:
kill -TERM cat /export/home/hobbit/logs/hobbitlaunch.pid
Where hobbitlaunch.pid is the PID of the hobbitlaunch process. When running this command we are left with many stray processes, not just a vmstat. For example:
/usr/local/bin/ssh -l root 10.6.6.6 sysstat -m -c1 290 sh -c (/usr/local/bin/ssh -l root 10.6.6.5 sysstat -x -c1 290 >/export/home/hob
Doing a bit more digging it seems these are related to an ext/netapp monitor perl script. Taking this out leaves less processes (although I don't want to leave it out forever so I'll have to do a bit of digging around to find out how to kill these gracefully) however because the iostats do not finish the SMF reports a failure rather than a disable when I try to stop the monitor. This means the SMF can't restart the bbserver if it ever crashes.
----- Original Message ----- From: "Buchan Milne" <bgmilne at staff.telkomsa.net> To: <hobbit at hswn.dk> Cc: <j.sansford at ntlworld.com> Sent: Monday, August 17, 2009 3:52 PM Subject: Re: [hobbit] hobbit.sh not stopping server correctly
On Monday, 17 August 2009 11:04:36 j.sansford at ntlworld.com wrote:
Hi,
I think I have an issue with 4.2.3 where the hobbit.sh is not stopping the server correctly. Running "./hobbit.sh stop" doesn't stop all the hobbit processes
Which processes remain? A vmstat running as the hobbit / xymon user?
and so I can't install this as a service currently. Could anyone help me to debug why this is happening? I'm not really sure where to go from here.
It's a bit difficult with no information. A stray vmstat process after stopping xymon is expected, and using the example init scripts provided in the debian or rpm directories works on many distros ...
Regards, Buchan
To unsubscribe from the hobbit list, send an e-mail to hobbit-unsubscribe at hswn.dk
Hi Buchan,
Just a stuppid question. But have you stopped and the hobbit server and the hobbit client on that machine or only the server?
Regards, Jef Jagers Systems Engineer Thomson CompuMark
Thomson Reuters
T +32 3 220 76 02
-----Original Message----- From: j.sansford at ntlworld.com [mailto:j.sansford at ntlworld.com] Sent: dinsdag, augustus 18, 2009 13:25 To: hobbit at hswn.dk Subject: Re: [hobbit] hobbit.sh not stopping server correctly
Hi Buchan,
Apologies, I'll explain it in more detail. We are running on Solaris 10 and trying to configure Xymon as an SMF service (which essentially uses stop/start scripts). The stop script is basically:
kill -TERM cat /export/home/hobbit/logs/hobbitlaunch.pid
Where hobbitlaunch.pid is the PID of the hobbitlaunch process. When running this command we are left with many stray processes, not just a vmstat. For example:
/usr/local/bin/ssh -l root 10.6.6.6 sysstat -m -c1 290 sh -c (/usr/local/bin/ssh -l root 10.6.6.5 sysstat -x -c1 290 >/export/home/hob
Doing a bit more digging it seems these are related to an ext/netapp monitor perl script. Taking this out leaves less processes (although I don't want to leave it out forever so I'll have to do a bit of digging around to find out how to kill these gracefully) however because the iostats do not finish the SMF reports a failure rather than a disable when I try to stop the monitor. This means the SMF can't restart the bbserver if it ever crashes.
----- Original Message ----- From: "Buchan Milne" <bgmilne at staff.telkomsa.net> To: <hobbit at hswn.dk> Cc: <j.sansford at ntlworld.com> Sent: Monday, August 17, 2009 3:52 PM Subject: Re: [hobbit] hobbit.sh not stopping server correctly
On Monday, 17 August 2009 11:04:36 j.sansford at ntlworld.com wrote:
Hi,
I think I have an issue with 4.2.3 where the hobbit.sh is not stopping the server correctly. Running "./hobbit.sh stop" doesn't stop all the hobbit processes
Which processes remain? A vmstat running as the hobbit / xymon user?
and so I can't install this as a service currently. Could anyone help me to debug why this is happening? I'm not really sure where to go from here.
It's a bit difficult with no information. A stray vmstat process after stopping xymon is expected, and using the example init scripts provided in the debian or rpm directories works on many distros ...
Regards, Buchan
To unsubscribe from the hobbit list, send an e-mail to hobbit-unsubscribe at hswn.dk
To unsubscribe from the hobbit list, send an e-mail to hobbit-unsubscribe at hswn.dk
Hi there Jef,
I assume that question was aimed at me? The stop script stops "hobbitlaunch" and not a client - however the client is owned by hobbitlaunch as it is configured within hobbitlaunch.cfg - we don't run a separately owned client on the system.
---- jef.jagers at thomsonreuters.com wrote:
Hi Buchan,
Just a stuppid question. But have you stopped and the hobbit server and the hobbit client on that machine or only the server?
Regards, Jef Jagers Systems Engineer Thomson CompuMark
Thomson Reuters
T +32 3 220 76 02
-----Original Message----- From: j.sansford at ntlworld.com [mailto:j.sansford at ntlworld.com] Sent: dinsdag, augustus 18, 2009 13:25 To: hobbit at hswn.dk Subject: Re: [hobbit] hobbit.sh not stopping server correctly
Hi Buchan,
Apologies, I'll explain it in more detail. We are running on Solaris 10 and trying to configure Xymon as an SMF service (which essentially uses stop/start scripts). The stop script is basically:
kill -TERM
cat /export/home/hobbit/logs/hobbitlaunch.pidWhere hobbitlaunch.pid is the PID of the hobbitlaunch process. When running this command we are left with many stray processes, not just a vmstat. For example:
/usr/local/bin/ssh -l root 10.6.6.6 sysstat -m -c1 290 sh -c (/usr/local/bin/ssh -l root 10.6.6.5 sysstat -x -c1 290 >/export/home/hob
Doing a bit more digging it seems these are related to an ext/netapp monitor perl script. Taking this out leaves less processes (although I don't want to leave it out forever so I'll have to do a bit of digging around to find out how to kill these gracefully) however because the iostats do not finish the SMF reports a failure rather than a disable when I try to stop the monitor. This means the SMF can't restart the bbserver if it ever crashes.
----- Original Message ----- From: "Buchan Milne" <bgmilne at staff.telkomsa.net> To: <hobbit at hswn.dk> Cc: <j.sansford at ntlworld.com> Sent: Monday, August 17, 2009 3:52 PM Subject: Re: [hobbit] hobbit.sh not stopping server correctly
On Monday, 17 August 2009 11:04:36 j.sansford at ntlworld.com wrote:
Hi,
I think I have an issue with 4.2.3 where the hobbit.sh is not stopping the server correctly. Running "./hobbit.sh stop" doesn't stop all the hobbit processes
Which processes remain? A vmstat running as the hobbit / xymon user?
and so I can't install this as a service currently. Could anyone help me to debug why this is happening? I'm not really sure where to go from here.
It's a bit difficult with no information. A stray vmstat process after stopping xymon is expected, and using the example init scripts provided in the debian or rpm directories works on many distros ...
Regards, Buchan
To unsubscribe from the hobbit list, send an e-mail to hobbit-unsubscribe at hswn.dk
To unsubscribe from the hobbit list, send an e-mail to hobbit-unsubscribe at hswn.dk
How long are you waiting after you kill the main Hobbit process for all the other processes to die off? Sometimes it can take several minutes for all the iostats and vmstats and such to complete -- they must be independent of their parent. Add-ons, especially from deadcat (written for BB, not Hobbit/Xymon) must be considered separately...
GLH
On 8/18/09, j.sansford at ntlworld.com <j.sansford at ntlworld.com> wrote:
Hi there Jef,
I assume that question was aimed at me? The stop script stops "hobbitlaunch" and not a client - however the client is owned by hobbitlaunch as it is configured within hobbitlaunch.cfg - we don't run a separately owned client on the system.
---- jef.jagers at thomsonreuters.com wrote:
Hi Buchan,
Just a stuppid question. But have you stopped and the hobbit server and the hobbit client on that machine or only the server?
Regards, Jef Jagers Systems Engineer Thomson CompuMark
Thomson Reuters
T +32 3 220 76 02
-----Original Message----- From: j.sansford at ntlworld.com [mailto:j.sansford at ntlworld.com] Sent: dinsdag, augustus 18, 2009 13:25 To: hobbit at hswn.dk Subject: Re: [hobbit] hobbit.sh not stopping server correctly
Hi Buchan,
Apologies, I'll explain it in more detail. We are running on Solaris 10 and trying to configure Xymon as an SMF service (which essentially uses stop/start scripts). The stop script is basically:
kill -TERM
cat /export/home/hobbit/logs/hobbitlaunch.pidWhere hobbitlaunch.pid is the PID of the hobbitlaunch process. When running this command we are left with many stray processes, not just a vmstat. For example:
/usr/local/bin/ssh -l root 10.6.6.6 sysstat -m -c1 290 sh -c (/usr/local/bin/ssh -l root 10.6.6.5 sysstat -x -c1 290 /export/home/hob
Doing a bit more digging it seems these are related to an ext/netapp monitor perl script. Taking this out leaves less processes (although I don't want to leave it out forever so I'll have to do a bit of digging around to find out how to kill these gracefully) however because the iostats do not finish the SMF reports a failure rather than a disable when I try to stop the monitor. This means the SMF can't restart the bbserver if it ever crashes.
----- Original Message ----- From: "Buchan Milne" <bgmilne at staff.telkomsa.net> To: <hobbit at hswn.dk> Cc: <j.sansford at ntlworld.com> Sent: Monday, August 17, 2009 3:52 PM Subject: Re: [hobbit] hobbit.sh not stopping server correctly
On Monday, 17 August 2009 11:04:36 j.sansford at ntlworld.com wrote:
Hi,
I think I have an issue with 4.2.3 where the hobbit.sh is not stopping the server correctly. Running "./hobbit.sh stop" doesn't stop all the hobbit processes
Which processes remain? A vmstat running as the hobbit / xymon user?
and so I can't install this as a service currently. Could anyone help me to debug why this is happening? I'm not really sure where to go from here.
It's a bit difficult with no information. A stray vmstat process after stopping xymon is expected, and using the example init scripts provided in the debian or rpm directories works on many distros ...
Regards, Buchan
To unsubscribe from the hobbit list, send an e-mail to hobbit-unsubscribe at hswn.dk
To unsubscribe from the hobbit list, send an e-mail to hobbit-unsubscribe at hswn.dk
To unsubscribe from the hobbit list, send an e-mail to hobbit-unsubscribe at hswn.dk
Hi Greg,
I've tried waiting over 5 minutes. I'll try waiting a little longer...:)
---- Greg Hubbard <glh.forums at gmail.com> wrote:
How long are you waiting after you kill the main Hobbit process for all the other processes to die off? Sometimes it can take several minutes for all the iostats and vmstats and such to complete -- they must be independent of their parent. Add-ons, especially from deadcat (written for BB, not Hobbit/Xymon) must be considered separately...
GLH
On 8/18/09, j.sansford at ntlworld.com <j.sansford at ntlworld.com> wrote:
Hi there Jef,
I assume that question was aimed at me? The stop script stops "hobbitlaunch" and not a client - however the client is owned by hobbitlaunch as it is configured within hobbitlaunch.cfg - we don't run a separately owned client on the system.
---- jef.jagers at thomsonreuters.com wrote:
Hi Buchan,
Just a stuppid question. But have you stopped and the hobbit server and the hobbit client on that machine or only the server?
Regards, Jef Jagers Systems Engineer Thomson CompuMark
Thomson Reuters
T +32 3 220 76 02
-----Original Message----- From: j.sansford at ntlworld.com [mailto:j.sansford at ntlworld.com] Sent: dinsdag, augustus 18, 2009 13:25 To: hobbit at hswn.dk Subject: Re: [hobbit] hobbit.sh not stopping server correctly
Hi Buchan,
Apologies, I'll explain it in more detail. We are running on Solaris 10 and trying to configure Xymon as an SMF service (which essentially uses stop/start scripts). The stop script is basically:
kill -TERM
cat /export/home/hobbit/logs/hobbitlaunch.pidWhere hobbitlaunch.pid is the PID of the hobbitlaunch process. When running this command we are left with many stray processes, not just a vmstat. For example:
/usr/local/bin/ssh -l root 10.6.6.6 sysstat -m -c1 290 sh -c (/usr/local/bin/ssh -l root 10.6.6.5 sysstat -x -c1 290 /export/home/hob
Doing a bit more digging it seems these are related to an ext/netapp monitor perl script. Taking this out leaves less processes (although I don't want to leave it out forever so I'll have to do a bit of digging around to find out how to kill these gracefully) however because the iostats do not finish the SMF reports a failure rather than a disable when I try to stop the monitor. This means the SMF can't restart the bbserver if it ever crashes.
----- Original Message ----- From: "Buchan Milne" <bgmilne at staff.telkomsa.net> To: <hobbit at hswn.dk> Cc: <j.sansford at ntlworld.com> Sent: Monday, August 17, 2009 3:52 PM Subject: Re: [hobbit] hobbit.sh not stopping server correctly
On Monday, 17 August 2009 11:04:36 j.sansford at ntlworld.com wrote:
Hi,
I think I have an issue with 4.2.3 where the hobbit.sh is not stopping the server correctly. Running "./hobbit.sh stop" doesn't stop all the hobbit processes
Which processes remain? A vmstat running as the hobbit / xymon user?
and so I can't install this as a service currently. Could anyone help me to debug why this is happening? I'm not really sure where to go from here.
It's a bit difficult with no information. A stray vmstat process after stopping xymon is expected, and using the example init scripts provided in the debian or rpm directories works on many distros ...
Regards, Buchan
To unsubscribe from the hobbit list, send an e-mail to hobbit-unsubscribe at hswn.dk
To unsubscribe from the hobbit list, send an e-mail to hobbit-unsubscribe at hswn.dk
To unsubscribe from the hobbit list, send an e-mail to hobbit-unsubscribe at hswn.dk
Well whadaya know...set the service timeout to around 7-8 minutes and it works. It may be slow...but at least it now has automatic resiliance.
Many thanks all.
James.
---- j.sansford at ntlworld.com wrote:
Hi Greg,
I've tried waiting over 5 minutes. I'll try waiting a little longer...:)
---- Greg Hubbard <glh.forums at gmail.com> wrote:
How long are you waiting after you kill the main Hobbit process for all the other processes to die off? Sometimes it can take several minutes for all the iostats and vmstats and such to complete -- they must be independent of their parent. Add-ons, especially from deadcat (written for BB, not Hobbit/Xymon) must be considered separately...
GLH
On 8/18/09, j.sansford at ntlworld.com <j.sansford at ntlworld.com> wrote:
Hi there Jef,
I assume that question was aimed at me? The stop script stops "hobbitlaunch" and not a client - however the client is owned by hobbitlaunch as it is configured within hobbitlaunch.cfg - we don't run a separately owned client on the system.
---- jef.jagers at thomsonreuters.com wrote:
Hi Buchan,
Just a stuppid question. But have you stopped and the hobbit server and the hobbit client on that machine or only the server?
Regards, Jef Jagers Systems Engineer Thomson CompuMark
Thomson Reuters
T +32 3 220 76 02
-----Original Message----- From: j.sansford at ntlworld.com [mailto:j.sansford at ntlworld.com] Sent: dinsdag, augustus 18, 2009 13:25 To: hobbit at hswn.dk Subject: Re: [hobbit] hobbit.sh not stopping server correctly
Hi Buchan,
Apologies, I'll explain it in more detail. We are running on Solaris 10 and trying to configure Xymon as an SMF service (which essentially uses stop/start scripts). The stop script is basically:
kill -TERM
cat /export/home/hobbit/logs/hobbitlaunch.pidWhere hobbitlaunch.pid is the PID of the hobbitlaunch process. When running this command we are left with many stray processes, not just a vmstat. For example:
/usr/local/bin/ssh -l root 10.6.6.6 sysstat -m -c1 290 sh -c (/usr/local/bin/ssh -l root 10.6.6.5 sysstat -x -c1 290 /export/home/hob
Doing a bit more digging it seems these are related to an ext/netapp monitor perl script. Taking this out leaves less processes (although I don't want to leave it out forever so I'll have to do a bit of digging around to find out how to kill these gracefully) however because the iostats do not finish the SMF reports a failure rather than a disable when I try to stop the monitor. This means the SMF can't restart the bbserver if it ever crashes.
----- Original Message ----- From: "Buchan Milne" <bgmilne at staff.telkomsa.net> To: <hobbit at hswn.dk> Cc: <j.sansford at ntlworld.com> Sent: Monday, August 17, 2009 3:52 PM Subject: Re: [hobbit] hobbit.sh not stopping server correctly
On Monday, 17 August 2009 11:04:36 j.sansford at ntlworld.com wrote:
Hi,
I think I have an issue with 4.2.3 where the hobbit.sh is not stopping the server correctly. Running "./hobbit.sh stop" doesn't stop all the hobbit processes
Which processes remain? A vmstat running as the hobbit / xymon user?
and so I can't install this as a service currently. Could anyone help me to debug why this is happening? I'm not really sure where to go from here.
It's a bit difficult with no information. A stray vmstat process after stopping xymon is expected, and using the example init scripts provided in the debian or rpm directories works on many distros ...
Regards, Buchan
To unsubscribe from the hobbit list, send an e-mail to hobbit-unsubscribe at hswn.dk
To unsubscribe from the hobbit list, send an e-mail to hobbit-unsubscribe at hswn.dk
To unsubscribe from the hobbit list, send an e-mail to hobbit-unsubscribe at hswn.dk
To unsubscribe from the hobbit list, send an e-mail to hobbit-unsubscribe at hswn.dk
On Tuesday, 18 August 2009 12:25:26 j.sansford at ntlworld.com wrote:
Hi Buchan,
Apologies, I'll explain it in more detail. We are running on Solaris 10 and trying to configure Xymon as an SMF service (which essentially uses stop/start scripts). The stop script is basically:
kill -TERM
cat /export/home/hobbit/logs/hobbitlaunch.pidWhere hobbitlaunch.pid is the PID of the hobbitlaunch process. When running this command we are left with many stray processes, not just a vmstat. For example:
/usr/local/bin/ssh -l root 10.6.6.6 sysstat -m -c1 290 sh -c (/usr/local/bin/ssh -l root 10.6.6.5 sysstat -x -c1 290
/export/home/hob
Doing a bit more digging it seems these are related to an ext/netapp monitor perl script.
I haven't (yet) monitored any netapp devices with devmon (most likely I will soon) ... but there is a template, which has tests named 'cluster', 'if_load','netapptemp','quota','status' and 'volume'. You may want to consider trying it instead of this script. If the teplate is missing something which is available by SNMP, we can add support for it.
BTW, are you running this out of cron, or from hobbitlaunch?
Taking this out leaves less processes (although I don't want to leave it out forever so I'll have to do a bit of digging around to find out how to kill these gracefully) however because the iostats do not finish the SMF reports a failure rather than a disable when I try to stop the monitor.
Well, then it seems SMF is a bit brain-dead ... however, you could run something equivalent to: pkill -u xymon vmstat
as well, which on my test box results in no processes running as the xymon user (after stopping the service via the init script on a Linux box).
This means the SMF can't restart the bbserver if it ever crashes.
I don't think I've ever seen hobbitlaunch crash ...
Regards, Buchan
participants (4)
-
bgmilne＠staff.telkomsa.net
-
glh.forums＠gmail.com
-
j.sansford＠ntlworld.com
-
jef.jagers＠thomsonreuters.com