no vmstat data for 8 hours on weekdays?
Hello, all.
I've got something odd going on...I have one AIX 5.3 system for which the CPU Utilization (vmstat) graph is blank from 9AM to 5PM, only on weekdays. None of my other AIX systems (5.3 or 6.1) are displaying the same behavior.
I've installed the same Hobbit client on all of my AIX instances. The vmstat processes are running as they should, and all of the other graphs in the trends area get data all day long.
I've already looked and I have no "TIME" options set for this specific for this server in either the localclient.cfg file on the server nor the analysis.cfg on the Xymon server...in fact, other than this system's entry in the hosts.cfg, there is no mention of this system in any config file EXCEPT hosts.cfg.
Any thoughts would be appreciated.
--
Mike Burger
AIX Administrator
<http://freedomhome.fhmc.local/intranet/main.jsp>
Phone (317) 537-3680, Fax (317) 537-4680, Cell (317) 797-2040
E-mail: Mike.Burger at FreedomMortgage.com <mailto:Anthony.House at freedommortgage.com>
"Once word leaks out that a pirate's gone soft, people begin to disobey you and then it's nothing but work, work, work, all the time." --Westley/The Dread Pirate Roberts
CONFIDENTIALITY NOTICE: This e-mail message, including all attachments, is for the sole use of the intended recipient(s) and may contain confidential information. Any unauthorized review, use, disclosure, alteration or distribution is strictly prohibited and may violate state or federal law. If you are not the intended recipient, please contact the sender by reply e-mail, delete this email and destroy all copies of the message.
On 13 November 2012 08:33, Mike Burger <Mike.Burger at freedommortgage.com>wrote:
I’ve got something odd going on…I have one AIX 5.3 system for which the CPU Utilization (vmstat) graph is blank from 9AM to 5PM, only on weekdays. None of my other AIX systems (5.3 or 6.1) are displaying the same behavior.
Odd. Is the blank section EXACTLY (within 5 minutes) between 9am and 5pm, or just approximate?
Are you getting updates for your client data? Check out the [vmstat] section both inside and outside of the 9-5 range, and see if they look different.
Perhaps during business hours, there is a process that runs, or a general increase in load, that increases the numbers in the vmstat output, such that the columns are no longer aligned the same. I know that there is at least one Xymon client data parser that fail if the columns aren't aligned as expected (procs, perhaps).
J
On Nov 12, 2012, at 6:52 PM, "Jeremy Laidman" <jlaidman at rebel-it.com.au> wrote:
On 13 November 2012 08:33, Mike Burger <Mike.Burger at freedommortgage.com> wrote: I’ve got something odd going on…I have one AIX 5.3 system for which the CPU Utilization (vmstat) graph is blank from 9AM to 5PM, only on weekdays. None of my other AIX systems (5.3 or 6.1) are displaying the same behavior.
Odd. Is the blank section EXACTLY (within 5 minutes) between 9am and 5pm, or just approximate?
Are you getting updates for your client data? Check out the [vmstat] section both inside and outside of the 9-5 range, and see if they look different.
Perhaps during business hours, there is a process that runs, or a general increase in load, that increases the numbers in the vmstat output, such that the columns are no longer aligned the same. I know that there is at least one Xymon client data parser that fail if the columns aren't aligned as expected (procs, perhaps). Friday of last week, on the 9AM mark. This morning 9:15AM and hadn't resumed graphing vmstat data as of 5:07 when I left the office (will need to check when I get hone tonight or in the morning at the office, but I'm expecting the data to have started around 5:15PM).
Over the weekend, there were no breaks in data graphing.
I'll have to also look at a continuous vmstat output, just in case.
Sent from my iPhone...please excuse any typos or short answers.
It's always suicide-mission this save-the-planet that. No one ever stops by just to say hi anymore. -Col. Jack O'Neill
CONFIDENTIALITY NOTICE: This e-mail message, including all attachments, is for the sole use of the intended recipient(s) and may contain confidential information. Any unauthorized review, use, disclosure, alteration or distribution is strictly prohibited and may violate state or federal law. If you are not the intended recipient, please contact the sender by reply e-mail, delete this email and destroy all copies of the message.
Ok…I can confirm that the graph started again at the 5:15PM mark, yesterday.
Does the parser look for specific column placement? If it does, shouldn’t it look for white/blank space(s) as data column delimiters?
In the meantime, I’m capturing a vmstat run, now, and will capture another if/when the graphing ceases again…I’m curious, though…the vmstat command line appears to be “vmstat 300 2”…that’s 300 captures in 2 second intervals (10 minutes worth of captures). As there are two vmstat processes running at any time, with a secondary command of “mv /usr/local/hobbit/tmp/hobbit_vmstat.PID /usr/local/hobbit/tmp/hobbit_vmstat.HOSTNAME”, is there a chance that the files are clobbering each other when they’re moved? Granted, each process have start times at 5 minute intervals, but I’m wondering.
From: Mike Burger Sent: Monday, November 12, 2012 8:42 PM To: Jeremy Laidman Cc: Mike Burger; xymon at xymon.com Subject: Re: [Xymon] no vmstat data for 8 hours on weekdays?
On Nov 12, 2012, at 6:52 PM, "Jeremy Laidman" <jlaidman at rebel-it.com.au> wrote:
On 13 November 2012 08:33, Mike Burger <Mike.Burger at freedommortgage.com> wrote:
I’ve got something odd going on…I have one AIX 5.3 system for which the CPU Utilization (vmstat) graph is blank from 9AM to 5PM, only on weekdays. None of my other AIX systems (5.3 or 6.1) are displaying the same behavior.
Odd. Is the blank section EXACTLY (within 5 minutes) between 9am and 5pm, or just approximate?
Are you getting updates for your client data? Check out the [vmstat] section both inside and outside of the 9-5 range, and see if they look different.
Perhaps during business hours, there is a process that runs, or a general increase in load, that increases the numbers in the vmstat output, such that the columns are no longer aligned the same. I know that there is at least one Xymon client data parser that fail if the columns aren't aligned as expected (procs, perhaps).Friday of last week, on the 9AM mark. This morning 9:15AM and hadn't resumed graphing vmstat data as of 5:07 when I left the office (will need to check when I get hone tonight or in the morning at the office, but I'm expecting the data to have started around 5:15PM).
Over the weekend, there were no breaks in data graphing.
I'll have to also look at a continuous vmstat output, just in case.
Sent from my iPhone...please excuse any typos or short answers.
It's always suicide-mission this save-the-planet that. No one ever stops by just to say hi anymore. -Col. Jack O'Neill
CONFIDENTIALITY NOTICE: This e-mail message, including all attachments, is for the sole use of the intended recipient(s) and may contain confidential information. Any unauthorized review, use, disclosure, alteration or distribution is strictly prohibited and may violate state or federal law. If you are not the intended recipient, please contact the sender by reply e-mail, delete this email and destroy all copies of the message.
I’m also noticing that even though I’m getting data in my graphs, right now, I’m not seeing much of anything in the actual vmstat output files in /usr/local/hobbit/tmp.
From: Mike Burger Sent: Tuesday, November 13, 2012 8:23 AM To: Mike Burger; Jeremy Laidman Cc: xymon at xymon.com Subject: RE: [Xymon] no vmstat data for 8 hours on weekdays?
Ok…I can confirm that the graph started again at the 5:15PM mark, yesterday.
Does the parser look for specific column placement? If it does, shouldn’t it look for white/blank space(s) as data column delimiters?
In the meantime, I’m capturing a vmstat run, now, and will capture another if/when the graphing ceases again…I’m curious, though…the vmstat command line appears to be “vmstat 300 2”…that’s 300 captures in 2 second intervals (10 minutes worth of captures). As there are two vmstat processes running at any time, with a secondary command of “mv /usr/local/hobbit/tmp/hobbit_vmstat.PID /usr/local/hobbit/tmp/hobbit_vmstat.HOSTNAME”, is there a chance that the files are clobbering each other when they’re moved? Granted, each process have start times at 5 minute intervals, but I’m wondering.
From: Mike Burger Sent: Monday, November 12, 2012 8:42 PM To: Jeremy Laidman Cc: Mike Burger; xymon at xymon.com Subject: Re: [Xymon] no vmstat data for 8 hours on weekdays?
On Nov 12, 2012, at 6:52 PM, "Jeremy Laidman" <jlaidman at rebel-it.com.au> wrote:
On 13 November 2012 08:33, Mike Burger <Mike.Burger at freedommortgage.com> wrote:
I’ve got something odd going on…I have one AIX 5.3 system for which the CPU Utilization (vmstat) graph is blank from 9AM to 5PM, only on weekdays. None of my other AIX systems (5.3 or 6.1) are displaying the same behavior.
Odd. Is the blank section EXACTLY (within 5 minutes) between 9am and 5pm, or just approximate?
Are you getting updates for your client data? Check out the [vmstat] section both inside and outside of the 9-5 range, and see if they look different.
Perhaps during business hours, there is a process that runs, or a general increase in load, that increases the numbers in the vmstat output, such that the columns are no longer aligned the same. I know that there is at least one Xymon client data parser that fail if the columns aren't aligned as expected (procs, perhaps).Friday of last week, on the 9AM mark. This morning 9:15AM and hadn't resumed graphing vmstat data as of 5:07 when I left the office (will need to check when I get hone tonight or in the morning at the office, but I'm expecting the data to have started around 5:15PM).
Over the weekend, there were no breaks in data graphing.
I'll have to also look at a continuous vmstat output, just in case.
Sent from my iPhone...please excuse any typos or short answers.
It's always suicide-mission this save-the-planet that. No one ever stops by just to say hi anymore. -Col. Jack O'Neill
CONFIDENTIALITY NOTICE: This e-mail message, including all attachments, is for the sole use of the intended recipient(s) and may contain confidential information. Any unauthorized review, use, disclosure, alteration or distribution is strictly prohibited and may violate state or federal law. If you are not the intended recipient, please contact the sender by reply e-mail, delete this email and destroy all copies of the message.
And, again, after the 9AM graph update, nothing going into the graphs. The output columns don’t seem to be drifting any more than they were before the 9AM time frame, as I’m watching the vmstat output on my screen.
From: Mike Burger Sent: Tuesday, November 13, 2012 8:24 AM To: Mike Burger; 'Jeremy Laidman' Cc: 'xymon at xymon.com' Subject: RE: [Xymon] no vmstat data for 8 hours on weekdays?
I’m also noticing that even though I’m getting data in my graphs, right now, I’m not seeing much of anything in the actual vmstat output files in /usr/local/hobbit/tmp.
From: Mike Burger Sent: Tuesday, November 13, 2012 8:23 AM To: Mike Burger; Jeremy Laidman Cc: xymon at xymon.com Subject: RE: [Xymon] no vmstat data for 8 hours on weekdays?
Ok…I can confirm that the graph started again at the 5:15PM mark, yesterday.
Does the parser look for specific column placement? If it does, shouldn’t it look for white/blank space(s) as data column delimiters?
In the meantime, I’m capturing a vmstat run, now, and will capture another if/when the graphing ceases again…I’m curious, though…the vmstat command line appears to be “vmstat 300 2”…that’s 300 captures in 2 second intervals (10 minutes worth of captures). As there are two vmstat processes running at any time, with a secondary command of “mv /usr/local/hobbit/tmp/hobbit_vmstat.PID /usr/local/hobbit/tmp/hobbit_vmstat.HOSTNAME”, is there a chance that the files are clobbering each other when they’re moved? Granted, each process have start times at 5 minute intervals, but I’m wondering.
From: Mike Burger Sent: Monday, November 12, 2012 8:42 PM To: Jeremy Laidman Cc: Mike Burger; xymon at xymon.com Subject: Re: [Xymon] no vmstat data for 8 hours on weekdays?
On Nov 12, 2012, at 6:52 PM, "Jeremy Laidman" <jlaidman at rebel-it.com.au> wrote:
On 13 November 2012 08:33, Mike Burger <Mike.Burger at freedommortgage.com> wrote:
I’ve got something odd going on…I have one AIX 5.3 system for which the CPU Utilization (vmstat) graph is blank from 9AM to 5PM, only on weekdays. None of my other AIX systems (5.3 or 6.1) are displaying the same behavior.
Odd. Is the blank section EXACTLY (within 5 minutes) between 9am and 5pm, or just approximate?
Are you getting updates for your client data? Check out the [vmstat] section both inside and outside of the 9-5 range, and see if they look different.
Perhaps during business hours, there is a process that runs, or a general increase in load, that increases the numbers in the vmstat output, such that the columns are no longer aligned the same. I know that there is at least one Xymon client data parser that fail if the columns aren't aligned as expected (procs, perhaps).Friday of last week, on the 9AM mark. This morning 9:15AM and hadn't resumed graphing vmstat data as of 5:07 when I left the office (will need to check when I get hone tonight or in the morning at the office, but I'm expecting the data to have started around 5:15PM).
Over the weekend, there were no breaks in data graphing.
I'll have to also look at a continuous vmstat output, just in case.
Sent from my iPhone...please excuse any typos or short answers.
It's always suicide-mission this save-the-planet that. No one ever stops by just to say hi anymore. -Col. Jack O'Neill
CONFIDENTIALITY NOTICE: This e-mail message, including all attachments, is for the sole use of the intended recipient(s) and may contain confidential information. Any unauthorized review, use, disclosure, alteration or distribution is strictly prohibited and may violate state or federal law. If you are not the intended recipient, please contact the sender by reply e-mail, delete this email and destroy all copies of the message.
Ok…I may have spoken too soon…9:05 data…waiting for 9:10 data. (hangs head)
From: Mike Burger Sent: Tuesday, November 13, 2012 9:09 AM To: Mike Burger; 'Jeremy Laidman' Cc: 'xymon at xymon.com' Subject: RE: [Xymon] no vmstat data for 8 hours on weekdays?
And, again, after the 9AM graph update, nothing going into the graphs. The output columns don’t seem to be drifting any more than they were before the 9AM time frame, as I’m watching the vmstat output on my screen.
From: Mike Burger Sent: Tuesday, November 13, 2012 8:24 AM To: Mike Burger; 'Jeremy Laidman' Cc: 'xymon at xymon.com' Subject: RE: [Xymon] no vmstat data for 8 hours on weekdays?
I’m also noticing that even though I’m getting data in my graphs, right now, I’m not seeing much of anything in the actual vmstat output files in /usr/local/hobbit/tmp.
From: Mike Burger Sent: Tuesday, November 13, 2012 8:23 AM To: Mike Burger; Jeremy Laidman Cc: xymon at xymon.com Subject: RE: [Xymon] no vmstat data for 8 hours on weekdays?
Ok…I can confirm that the graph started again at the 5:15PM mark, yesterday.
Does the parser look for specific column placement? If it does, shouldn’t it look for white/blank space(s) as data column delimiters?
In the meantime, I’m capturing a vmstat run, now, and will capture another if/when the graphing ceases again…I’m curious, though…the vmstat command line appears to be “vmstat 300 2”…that’s 300 captures in 2 second intervals (10 minutes worth of captures). As there are two vmstat processes running at any time, with a secondary command of “mv /usr/local/hobbit/tmp/hobbit_vmstat.PID /usr/local/hobbit/tmp/hobbit_vmstat.HOSTNAME”, is there a chance that the files are clobbering each other when they’re moved? Granted, each process have start times at 5 minute intervals, but I’m wondering.
From: Mike Burger Sent: Monday, November 12, 2012 8:42 PM To: Jeremy Laidman Cc: Mike Burger; xymon at xymon.com Subject: Re: [Xymon] no vmstat data for 8 hours on weekdays?
On Nov 12, 2012, at 6:52 PM, "Jeremy Laidman" <jlaidman at rebel-it.com.au> wrote:
On 13 November 2012 08:33, Mike Burger <Mike.Burger at freedommortgage.com> wrote:
I’ve got something odd going on…I have one AIX 5.3 system for which the CPU Utilization (vmstat) graph is blank from 9AM to 5PM, only on weekdays. None of my other AIX systems (5.3 or 6.1) are displaying the same behavior.
Odd. Is the blank section EXACTLY (within 5 minutes) between 9am and 5pm, or just approximate?
Are you getting updates for your client data? Check out the [vmstat] section both inside and outside of the 9-5 range, and see if they look different.
Perhaps during business hours, there is a process that runs, or a general increase in load, that increases the numbers in the vmstat output, such that the columns are no longer aligned the same. I know that there is at least one Xymon client data parser that fail if the columns aren't aligned as expected (procs, perhaps).Friday of last week, on the 9AM mark. This morning 9:15AM and hadn't resumed graphing vmstat data as of 5:07 when I left the office (will need to check when I get hone tonight or in the morning at the office, but I'm expecting the data to have started around 5:15PM).
Over the weekend, there were no breaks in data graphing.
I'll have to also look at a continuous vmstat output, just in case.
Sent from my iPhone...please excuse any typos or short answers.
It's always suicide-mission this save-the-planet that. No one ever stops by just to say hi anymore. -Col. Jack O'Neill
CONFIDENTIALITY NOTICE: This e-mail message, including all attachments, is for the sole use of the intended recipient(s) and may contain confidential information. Any unauthorized review, use, disclosure, alteration or distribution is strictly prohibited and may violate state or federal law. If you are not the intended recipient, please contact the sender by reply e-mail, delete this email and destroy all copies of the message.
Last data recorded, now, is from 9:10.
I’ve got vmstat output that I’d be happy to share if it helps anyone understand why this is happening.
From: Mike Burger Sent: Tuesday, November 13, 2012 9:11 AM To: Mike Burger; 'Jeremy Laidman' Cc: 'xymon at xymon.com' Subject: RE: [Xymon] no vmstat data for 8 hours on weekdays?
Ok…I may have spoken too soon…9:05 data…waiting for 9:10 data. (hangs head)
From: Mike Burger Sent: Tuesday, November 13, 2012 9:09 AM To: Mike Burger; 'Jeremy Laidman' Cc: 'xymon at xymon.com' Subject: RE: [Xymon] no vmstat data for 8 hours on weekdays?
And, again, after the 9AM graph update, nothing going into the graphs. The output columns don’t seem to be drifting any more than they were before the 9AM time frame, as I’m watching the vmstat output on my screen.
From: Mike Burger Sent: Tuesday, November 13, 2012 8:24 AM To: Mike Burger; 'Jeremy Laidman' Cc: 'xymon at xymon.com' Subject: RE: [Xymon] no vmstat data for 8 hours on weekdays?
I’m also noticing that even though I’m getting data in my graphs, right now, I’m not seeing much of anything in the actual vmstat output files in /usr/local/hobbit/tmp.
From: Mike Burger Sent: Tuesday, November 13, 2012 8:23 AM To: Mike Burger; Jeremy Laidman Cc: xymon at xymon.com Subject: RE: [Xymon] no vmstat data for 8 hours on weekdays?
Ok…I can confirm that the graph started again at the 5:15PM mark, yesterday.
Does the parser look for specific column placement? If it does, shouldn’t it look for white/blank space(s) as data column delimiters?
In the meantime, I’m capturing a vmstat run, now, and will capture another if/when the graphing ceases again…I’m curious, though…the vmstat command line appears to be “vmstat 300 2”…that’s 300 captures in 2 second intervals (10 minutes worth of captures). As there are two vmstat processes running at any time, with a secondary command of “mv /usr/local/hobbit/tmp/hobbit_vmstat.PID /usr/local/hobbit/tmp/hobbit_vmstat.HOSTNAME”, is there a chance that the files are clobbering each other when they’re moved? Granted, each process have start times at 5 minute intervals, but I’m wondering.
From: Mike Burger Sent: Monday, November 12, 2012 8:42 PM To: Jeremy Laidman Cc: Mike Burger; xymon at xymon.com Subject: Re: [Xymon] no vmstat data for 8 hours on weekdays?
On Nov 12, 2012, at 6:52 PM, "Jeremy Laidman" <jlaidman at rebel-it.com.au> wrote:
On 13 November 2012 08:33, Mike Burger <Mike.Burger at freedommortgage.com> wrote:
I’ve got something odd going on…I have one AIX 5.3 system for which the CPU Utilization (vmstat) graph is blank from 9AM to 5PM, only on weekdays. None of my other AIX systems (5.3 or 6.1) are displaying the same behavior.
Odd. Is the blank section EXACTLY (within 5 minutes) between 9am and 5pm, or just approximate?
Are you getting updates for your client data? Check out the [vmstat] section both inside and outside of the 9-5 range, and see if they look different.
Perhaps during business hours, there is a process that runs, or a general increase in load, that increases the numbers in the vmstat output, such that the columns are no longer aligned the same. I know that there is at least one Xymon client data parser that fail if the columns aren't aligned as expected (procs, perhaps).Friday of last week, on the 9AM mark. This morning 9:15AM and hadn't resumed graphing vmstat data as of 5:07 when I left the office (will need to check when I get hone tonight or in the morning at the office, but I'm expecting the data to have started around 5:15PM).
Over the weekend, there were no breaks in data graphing.
I'll have to also look at a continuous vmstat output, just in case.
Sent from my iPhone...please excuse any typos or short answers.
It's always suicide-mission this save-the-planet that. No one ever stops by just to say hi anymore. -Col. Jack O'Neill
CONFIDENTIALITY NOTICE: This e-mail message, including all attachments, is for the sole use of the intended recipient(s) and may contain confidential information. Any unauthorized review, use, disclosure, alteration or distribution is strictly prohibited and may violate state or federal law. If you are not the intended recipient, please contact the sender by reply e-mail, delete this email and destroy all copies of the message.
FWIW, I’m attaching vmstat output that ends pre-dropoff and another that starts pre-dropoff. With any luck, something will stand out.
Xymon server is 4.3.10, Hobbit client on AIX is 4.2.0, if that makes any difference.
From: Mike Burger Sent: Tuesday, November 13, 2012 9:40 AM To: Mike Burger; 'Jeremy Laidman' Cc: 'xymon at xymon.com' Subject: RE: [Xymon] no vmstat data for 8 hours on weekdays?
Last data recorded, now, is from 9:10.
I’ve got vmstat output that I’d be happy to share if it helps anyone understand why this is happening.
From: Mike Burger Sent: Tuesday, November 13, 2012 9:11 AM To: Mike Burger; 'Jeremy Laidman' Cc: 'xymon at xymon.com' Subject: RE: [Xymon] no vmstat data for 8 hours on weekdays?
Ok…I may have spoken too soon…9:05 data…waiting for 9:10 data. (hangs head)
From: Mike Burger Sent: Tuesday, November 13, 2012 9:09 AM To: Mike Burger; 'Jeremy Laidman' Cc: 'xymon at xymon.com' Subject: RE: [Xymon] no vmstat data for 8 hours on weekdays?
And, again, after the 9AM graph update, nothing going into the graphs. The output columns don’t seem to be drifting any more than they were before the 9AM time frame, as I’m watching the vmstat output on my screen.
From: Mike Burger Sent: Tuesday, November 13, 2012 8:24 AM To: Mike Burger; 'Jeremy Laidman' Cc: 'xymon at xymon.com' Subject: RE: [Xymon] no vmstat data for 8 hours on weekdays?
I’m also noticing that even though I’m getting data in my graphs, right now, I’m not seeing much of anything in the actual vmstat output files in /usr/local/hobbit/tmp.
From: Mike Burger Sent: Tuesday, November 13, 2012 8:23 AM To: Mike Burger; Jeremy Laidman Cc: xymon at xymon.com Subject: RE: [Xymon] no vmstat data for 8 hours on weekdays?
Ok…I can confirm that the graph started again at the 5:15PM mark, yesterday.
Does the parser look for specific column placement? If it does, shouldn’t it look for white/blank space(s) as data column delimiters?
In the meantime, I’m capturing a vmstat run, now, and will capture another if/when the graphing ceases again…I’m curious, though…the vmstat command line appears to be “vmstat 300 2”…that’s 300 captures in 2 second intervals (10 minutes worth of captures). As there are two vmstat processes running at any time, with a secondary command of “mv /usr/local/hobbit/tmp/hobbit_vmstat.PID /usr/local/hobbit/tmp/hobbit_vmstat.HOSTNAME”, is there a chance that the files are clobbering each other when they’re moved? Granted, each process have start times at 5 minute intervals, but I’m wondering.
From: Mike Burger Sent: Monday, November 12, 2012 8:42 PM To: Jeremy Laidman Cc: Mike Burger; xymon at xymon.com Subject: Re: [Xymon] no vmstat data for 8 hours on weekdays?
On Nov 12, 2012, at 6:52 PM, "Jeremy Laidman" <jlaidman at rebel-it.com.au> wrote:
On 13 November 2012 08:33, Mike Burger <Mike.Burger at freedommortgage.com> wrote:
I’ve got something odd going on…I have one AIX 5.3 system for which the CPU Utilization (vmstat) graph is blank from 9AM to 5PM, only on weekdays. None of my other AIX systems (5.3 or 6.1) are displaying the same behavior.
Odd. Is the blank section EXACTLY (within 5 minutes) between 9am and 5pm, or just approximate?
Are you getting updates for your client data? Check out the [vmstat] section both inside and outside of the 9-5 range, and see if they look different.
Perhaps during business hours, there is a process that runs, or a general increase in load, that increases the numbers in the vmstat output, such that the columns are no longer aligned the same. I know that there is at least one Xymon client data parser that fail if the columns aren't aligned as expected (procs, perhaps).Friday of last week, on the 9AM mark. This morning 9:15AM and hadn't resumed graphing vmstat data as of 5:07 when I left the office (will need to check when I get hone tonight or in the morning at the office, but I'm expecting the data to have started around 5:15PM).
Over the weekend, there were no breaks in data graphing.
I'll have to also look at a continuous vmstat output, just in case.
Sent from my iPhone...please excuse any typos or short answers.
It's always suicide-mission this save-the-planet that. No one ever stops by just to say hi anymore. -Col. Jack O'Neill
CONFIDENTIALITY NOTICE: This e-mail message, including all attachments, is for the sole use of the intended recipient(s) and may contain confidential information. Any unauthorized review, use, disclosure, alteration or distribution is strictly prohibited and may violate state or federal law. If you are not the intended recipient, please contact the sender by reply e-mail, delete this email and destroy all copies of the message.
It would likely help if I would have attached the file. *facepalm*
From: Mike Burger Sent: Tuesday, November 13, 2012 11:13 AM To: Mike Burger; 'Jeremy Laidman' Cc: 'xymon at xymon.com' Subject: RE: [Xymon] no vmstat data for 8 hours on weekdays?
FWIW, I’m attaching vmstat output that ends pre-dropoff and another that starts pre-dropoff. With any luck, something will stand out.
Xymon server is 4.3.10, Hobbit client on AIX is 4.2.0, if that makes any difference.
From: Mike Burger Sent: Tuesday, November 13, 2012 9:40 AM To: Mike Burger; 'Jeremy Laidman' Cc: 'xymon at xymon.com' Subject: RE: [Xymon] no vmstat data for 8 hours on weekdays?
Last data recorded, now, is from 9:10.
I’ve got vmstat output that I’d be happy to share if it helps anyone understand why this is happening.
From: Mike Burger Sent: Tuesday, November 13, 2012 9:11 AM To: Mike Burger; 'Jeremy Laidman' Cc: 'xymon at xymon.com' Subject: RE: [Xymon] no vmstat data for 8 hours on weekdays?
Ok…I may have spoken too soon…9:05 data…waiting for 9:10 data. (hangs head)
From: Mike Burger Sent: Tuesday, November 13, 2012 9:09 AM To: Mike Burger; 'Jeremy Laidman' Cc: 'xymon at xymon.com' Subject: RE: [Xymon] no vmstat data for 8 hours on weekdays?
And, again, after the 9AM graph update, nothing going into the graphs. The output columns don’t seem to be drifting any more than they were before the 9AM time frame, as I’m watching the vmstat output on my screen.
From: Mike Burger Sent: Tuesday, November 13, 2012 8:24 AM To: Mike Burger; 'Jeremy Laidman' Cc: 'xymon at xymon.com' Subject: RE: [Xymon] no vmstat data for 8 hours on weekdays?
I’m also noticing that even though I’m getting data in my graphs, right now, I’m not seeing much of anything in the actual vmstat output files in /usr/local/hobbit/tmp.
From: Mike Burger Sent: Tuesday, November 13, 2012 8:23 AM To: Mike Burger; Jeremy Laidman Cc: xymon at xymon.com Subject: RE: [Xymon] no vmstat data for 8 hours on weekdays?
Ok…I can confirm that the graph started again at the 5:15PM mark, yesterday.
Does the parser look for specific column placement? If it does, shouldn’t it look for white/blank space(s) as data column delimiters?
In the meantime, I’m capturing a vmstat run, now, and will capture another if/when the graphing ceases again…I’m curious, though…the vmstat command line appears to be “vmstat 300 2”…that’s 300 captures in 2 second intervals (10 minutes worth of captures). As there are two vmstat processes running at any time, with a secondary command of “mv /usr/local/hobbit/tmp/hobbit_vmstat.PID /usr/local/hobbit/tmp/hobbit_vmstat.HOSTNAME”, is there a chance that the files are clobbering each other when they’re moved? Granted, each process have start times at 5 minute intervals, but I’m wondering.
From: Mike Burger Sent: Monday, November 12, 2012 8:42 PM To: Jeremy Laidman Cc: Mike Burger; xymon at xymon.com Subject: Re: [Xymon] no vmstat data for 8 hours on weekdays?
On Nov 12, 2012, at 6:52 PM, "Jeremy Laidman" <jlaidman at rebel-it.com.au> wrote:
On 13 November 2012 08:33, Mike Burger <Mike.Burger at freedommortgage.com> wrote:
I’ve got something odd going on…I have one AIX 5.3 system for which the CPU Utilization (vmstat) graph is blank from 9AM to 5PM, only on weekdays. None of my other AIX systems (5.3 or 6.1) are displaying the same behavior.
Odd. Is the blank section EXACTLY (within 5 minutes) between 9am and 5pm, or just approximate?
Are you getting updates for your client data? Check out the [vmstat] section both inside and outside of the 9-5 range, and see if they look different.
Perhaps during business hours, there is a process that runs, or a general increase in load, that increases the numbers in the vmstat output, such that the columns are no longer aligned the same. I know that there is at least one Xymon client data parser that fail if the columns aren't aligned as expected (procs, perhaps).Friday of last week, on the 9AM mark. This morning 9:15AM and hadn't resumed graphing vmstat data as of 5:07 when I left the office (will need to check when I get hone tonight or in the morning at the office, but I'm expecting the data to have started around 5:15PM).
Over the weekend, there were no breaks in data graphing.
I'll have to also look at a continuous vmstat output, just in case.
Sent from my iPhone...please excuse any typos or short answers.
It's always suicide-mission this save-the-planet that. No one ever stops by just to say hi anymore. -Col. Jack O'Neill
CONFIDENTIALITY NOTICE: This e-mail message, including all attachments, is for the sole use of the intended recipient(s) and may contain confidential information. Any unauthorized review, use, disclosure, alteration or distribution is strictly prohibited and may violate state or federal law. If you are not the intended recipient, please contact the sender by reply e-mail, delete this email and destroy all copies of the message.
On 14 November 2012 03:14, Mike Burger <Mike.Burger at freedommortgage.com>wrote:
It would likely help if I would have attached the file. **facepalm**
I'm not entirely sure what I'm looking at. I expected only two lines of output. Nevertheless, the columns do seem to be a little irregular.
Can you attach the [vmstat] section from the client data for the host?
If you have xymond running with "--store-clientlogs" then you can find historical data snapshots in $XYMONVAR/hostdata/name-of-host/.
If not, you can get the [vmstat] section of the client data, at the appropriate times, with this command run on the Xymon server:
xymoncmd xymon localhost 'clientlog name-of-host section=vmstat'
J
Sorry...that was the output from my own vmstat collections, rather than the file(s) being collected by the Hobbit Client.
From: Jeremy Laidman [mailto:jlaidman at rebel-it.com.au] Sent: Tuesday, November 13, 2012 6:29 PM To: Mike Burger Cc: xymon at xymon.com Subject: Re: [Xymon] no vmstat data for 8 hours on weekdays?
On 14 November 2012 03:14, Mike Burger <Mike.Burger at freedommortgage.com> wrote:
It would likely help if I would have attached the file.*facepalm*
I'm not entirely sure what I'm looking at. I expected only two lines of output. Nevertheless, the columns do seem to be a little irregular.
Can you attach the [vmstat] section from the client data for the host?
If you have xymond running with "--store-clientlogs" then you can find historical data snapshots in $XYMONVAR/hostdata/name-of-host/.
If not, you can get the [vmstat] section of the client data, at the appropriate times, with this command run on the Xymon server:
xymoncmd xymon localhost 'clientlog name-of-host section=vmstat'
J
CONFIDENTIALITY NOTICE: This e-mail message, including all attachments, is for the sole use of the intended recipient(s) and may contain confidential information. Any unauthorized review, use, disclosure, alteration or distribution is strictly prohibited and may violate state or federal law. If you are not the intended recipient, please contact the sender by reply e-mail, delete this email and destroy all copies of the message.
On 14 November 2012 00:22, Mike Burger <Mike.Burger at freedommortgage.com>wrote:
Does the parser look for specific column placement? If it does, shouldn’t it look for white/blank space(s) as data column delimiters?
Some parsers don't use whitespace delimeters, but instead use column position. But I don't know if the vmstat parser does the same.
vmstat command line appears to be “vmstat 300 2”…that’s 300 captures in 2
second intervals (10 minutes worth of captures).
No, I think you'll find that it's two captures in 300 second intervals. The first sample is the total/average since last boot, and the second sample is the total/average since the first one. The first sample is discarded.
As there are two vmstat processes running at any time, with a secondary command of “mv /usr/local/hobbit/tmp/hobbit_vmstat.PID /usr/local/hobbit/tmp/hobbit_vmstat.HOSTNAME”, is there a chance that the files are clobbering each other when they’re moved?
Yes, perhaps. I only have one vmstat command running at any one time. You might try killing both off and see what happens.
J
On 14 November 2012 10:09, Jeremy Laidman <jlaidman at rebel-it.com.au> wrote:
Some parsers don't use whitespace delimeters, but instead use column position. But I don't know if the vmstat parser does the same.
I just checked the code, and I think the vmstat parser (in do_vmstat.c function do_vmstat_rrd()) splits on whitespace, thus uses column number rather than column position. So this shouldn't be a problem for you.
Ok...that's good news on one front...I'd love to know what the heck is going on then.
I'm going to stop the hobbit client, kill any lingering vmstat processes and then restart the hobbit client and see what happens.
From: Jeremy Laidman [mailto:jlaidman at rebel-it.com.au] Sent: Tuesday, November 13, 2012 6:38 PM To: Mike Burger Cc: xymon at xymon.com Subject: Re: [Xymon] no vmstat data for 8 hours on weekdays?
On 14 November 2012 10:09, Jeremy Laidman <jlaidman at rebel-it.com.au> wrote:
Some parsers don't use whitespace delimeters, but instead usecolumn position. But I don't know if the vmstat parser does the same.
I just checked the code, and I think the vmstat parser (in do_vmstat.c function do_vmstat_rrd()) splits on whitespace, thus uses column number rather than column position. So this shouldn't be a problem for you.
CONFIDENTIALITY NOTICE: This e-mail message, including all attachments, is for the sole use of the intended recipient(s) and may contain confidential information. Any unauthorized review, use, disclosure, alteration or distribution is strictly prohibited and may violate state or federal law. If you are not the intended recipient, please contact the sender by reply e-mail, delete this email and destroy all copies of the message.
Silly question time...is that code in the server or client?
From: Jeremy Laidman [mailto:jlaidman at rebel-it.com.au] Sent: Tuesday, November 13, 2012 6:38 PM To: Mike Burger Cc: xymon at xymon.com Subject: Re: [Xymon] no vmstat data for 8 hours on weekdays?
On 14 November 2012 10:09, Jeremy Laidman <jlaidman at rebel-it.com.au> wrote:
Some parsers don't use whitespace delimeters, but instead usecolumn position. But I don't know if the vmstat parser does the same.
I just checked the code, and I think the vmstat parser (in do_vmstat.c function do_vmstat_rrd()) splits on whitespace, thus uses column number rather than column position. So this shouldn't be a problem for you.
CONFIDENTIALITY NOTICE: This e-mail message, including all attachments, is for the sole use of the intended recipient(s) and may contain confidential information. Any unauthorized review, use, disclosure, alteration or distribution is strictly prohibited and may violate state or federal law. If you are not the intended recipient, please contact the sender by reply e-mail, delete this email and destroy all copies of the message.
Good to know...I was trying (unsuccessfully) to compile the 4.3.10 client for AIX, in case it was the client side.
On the other side...I did stop the client, killed any remaining vmstat processes and started the client back up. Initially, only one vmstat process was running, and then eventually there were two at any given time, again. They kick off every 5 minutes, outputting to a vmstat file tied to that process' PID (hobbit_vmstat.<hostname>.PID), and at the end of the run, the PID specific vmstat file is supposed to be renamed to hobbit_vmstat.<hostname>. The one thing I notice is that I never see a file hobbit_vmstat.<hostname>...even now, while there is still data actually being reported in the graph (it's only 7:44AM at present), there is no vmstat file without the PID as part of the file name present. Not that I think that is the problem.
Is there anything I can do to increase verbosity in the client and/or server logging to determine if there's some sort of communication issue (or lack thereof)?
From: Jeremy Laidman [mailto:jlaidman at rebel-it.com.au] Sent: Wednesday, November 14, 2012 6:20 PM To: Mike Burger Cc: xymon at xymon.com Subject: Re: [Xymon] no vmstat data for 8 hours on weekdays?
On 15 November 2012 02:04, Mike Burger <Mike.Burger at freedommortgage.com> wrote:
Silly question time...is that code in the server or client?There are no silly questions: all questions lead to enlightenment!
The parser runs on the server.
J
CONFIDENTIALITY NOTICE: This e-mail message, including all attachments, is for the sole use of the intended recipient(s) and may contain confidential information. Any unauthorized review, use, disclosure, alteration or distribution is strictly prohibited and may violate state or federal law. If you are not the intended recipient, please contact the sender by reply e-mail, delete this email and destroy all copies of the message.
I think I've found it in the server's xymond.log:
Oversize status msg from <client IP> for <hostname>:procs truncated (n=<some #>, limit=262144)
Oversize data/client msg from <client IP <truncated (n=<some #>, limit=524288).
Now, I need to figure out how to change those limits.
From: xymon-bounces at xymon.com [mailto:xymon-bounces at xymon.com] On Behalf Of Mike Burger Sent: Thursday, November 15, 2012 7:46 AM To: Jeremy Laidman Cc: xymon at xymon.com Subject: Re: [Xymon] no vmstat data for 8 hours on weekdays?
Good to know...I was trying (unsuccessfully) to compile the 4.3.10 client for AIX, in case it was the client side.
On the other side...I did stop the client, killed any remaining vmstat processes and started the client back up. Initially, only one vmstat process was running, and then eventually there were two at any given time, again. They kick off every 5 minutes, outputting to a vmstat file tied to that process' PID (hobbit_vmstat.<hostname>.PID), and at the end of the run, the PID specific vmstat file is supposed to be renamed to hobbit_vmstat.<hostname>. The one thing I notice is that I never see a file hobbit_vmstat.<hostname>...even now, while there is still data actually being reported in the graph (it's only 7:44AM at present), there is no vmstat file without the PID as part of the file name present. Not that I think that is the problem.
Is there anything I can do to increase verbosity in the client and/or server logging to determine if there's some sort of communication issue (or lack thereof)?
From: Jeremy Laidman [mailto:jlaidman at rebel-it.com.au] Sent: Wednesday, November 14, 2012 6:20 PM To: Mike Burger Cc: xymon at xymon.com Subject: Re: [Xymon] no vmstat data for 8 hours on weekdays?
On 15 November 2012 02:04, Mike Burger <Mike.Burger at freedommortgage.com> wrote:
Silly question time...is that code in the server or client?There are no silly questions: all questions lead to enlightenment!
The parser runs on the server.
J
CONFIDENTIALITY NOTICE: This e-mail message, including all attachments, is for the sole use of the intended recipient(s) and may contain confidential information. Any unauthorized review, use, disclosure, alteration or distribution is strictly prohibited and may violate state or federal law. If you are not the intended recipient, please contact the sender by reply e-mail, delete this email and destroy all copies of the message.
CONFIDENTIALITY NOTICE: This e-mail message, including all attachments, is for the sole use of the intended recipient(s) and may contain confidential information. Any unauthorized review, use, disclosure, alteration or distribution is strictly prohibited and may violate state or federal law. If you are not the intended recipient, please contact the sender by reply e-mail, delete this email and destroy all copies of the message.
Ok...I've modified the following options in the xymonserver.cfg file:
MAXMSG_CLIENT changed from 512 to 1024
MAXMSG_STATUS changed from 256 to 512
MAXMSG_DATA changed from 256 to 512
I'll be watching to see how things look as we get to/past the 9AM hour.
From: Mike Burger Sent: Thursday, November 15, 2012 8:03 AM To: Mike Burger; Jeremy Laidman Cc: xymon at xymon.com Subject: RE: [Xymon] no vmstat data for 8 hours on weekdays?
I think I've found it in the server's xymond.log:
Oversize status msg from <client IP> for <hostname>:procs truncated (n=<some #>, limit=262144)
Oversize data/client msg from <client IP <truncated (n=<some #>, limit=524288).
Now, I need to figure out how to change those limits.
From: xymon-bounces at xymon.com [mailto:xymon-bounces at xymon.com] On Behalf Of Mike Burger Sent: Thursday, November 15, 2012 7:46 AM To: Jeremy Laidman Cc: xymon at xymon.com Subject: Re: [Xymon] no vmstat data for 8 hours on weekdays?
Good to know...I was trying (unsuccessfully) to compile the 4.3.10 client for AIX, in case it was the client side.
On the other side...I did stop the client, killed any remaining vmstat processes and started the client back up. Initially, only one vmstat process was running, and then eventually there were two at any given time, again. They kick off every 5 minutes, outputting to a vmstat file tied to that process' PID (hobbit_vmstat.<hostname>.PID), and at the end of the run, the PID specific vmstat file is supposed to be renamed to hobbit_vmstat.<hostname>. The one thing I notice is that I never see a file hobbit_vmstat.<hostname>...even now, while there is still data actually being reported in the graph (it's only 7:44AM at present), there is no vmstat file without the PID as part of the file name present. Not that I think that is the problem.
Is there anything I can do to increase verbosity in the client and/or server logging to determine if there's some sort of communication issue (or lack thereof)?
From: Jeremy Laidman [mailto:jlaidman at rebel-it.com.au] Sent: Wednesday, November 14, 2012 6:20 PM To: Mike Burger Cc: xymon at xymon.com Subject: Re: [Xymon] no vmstat data for 8 hours on weekdays?
On 15 November 2012 02:04, Mike Burger <Mike.Burger at freedommortgage.com> wrote:
Silly question time...is that code in the server or client?There are no silly questions: all questions lead to enlightenment!
The parser runs on the server.
J
CONFIDENTIALITY NOTICE: This e-mail message, including all attachments, is for the sole use of the intended recipient(s) and may contain confidential information. Any unauthorized review, use, disclosure, alteration or distribution is strictly prohibited and may violate state or federal law. If you are not the intended recipient, please contact the sender by reply e-mail, delete this email and destroy all copies of the message.
CONFIDENTIALITY NOTICE: This e-mail message, including all attachments, is for the sole use of the intended recipient(s) and may contain confidential information. Any unauthorized review, use, disclosure, alteration or distribution is strictly prohibited and may violate state or federal law. If you are not the intended recipient, please contact the sender by reply e-mail, delete this email and destroy all copies of the message.
That did it. I'm not sure if MAXMSG_STATUS or MAXMSG_DATA is the culprit, though I suspect it's _DATA.
From: Mike Burger Sent: Thursday, November 15, 2012 8:08 AM To: Mike Burger; 'Jeremy Laidman' Cc: 'xymon at xymon.com' Subject: RE: [Xymon] no vmstat data for 8 hours on weekdays?
Ok...I've modified the following options in the xymonserver.cfg file:
MAXMSG_CLIENT changed from 512 to 1024
MAXMSG_STATUS changed from 256 to 512
MAXMSG_DATA changed from 256 to 512
I'll be watching to see how things look as we get to/past the 9AM hour.
From: Mike Burger Sent: Thursday, November 15, 2012 8:03 AM To: Mike Burger; Jeremy Laidman Cc: xymon at xymon.com Subject: RE: [Xymon] no vmstat data for 8 hours on weekdays?
I think I've found it in the server's xymond.log:
Oversize status msg from <client IP> for <hostname>:procs truncated (n=<some #>, limit=262144)
Oversize data/client msg from <client IP <truncated (n=<some #>, limit=524288).
Now, I need to figure out how to change those limits.
From: xymon-bounces at xymon.com [mailto:xymon-bounces at xymon.com] On Behalf Of Mike Burger Sent: Thursday, November 15, 2012 7:46 AM To: Jeremy Laidman Cc: xymon at xymon.com Subject: Re: [Xymon] no vmstat data for 8 hours on weekdays?
Good to know...I was trying (unsuccessfully) to compile the 4.3.10 client for AIX, in case it was the client side.
On the other side...I did stop the client, killed any remaining vmstat processes and started the client back up. Initially, only one vmstat process was running, and then eventually there were two at any given time, again. They kick off every 5 minutes, outputting to a vmstat file tied to that process' PID (hobbit_vmstat.<hostname>.PID), and at the end of the run, the PID specific vmstat file is supposed to be renamed to hobbit_vmstat.<hostname>. The one thing I notice is that I never see a file hobbit_vmstat.<hostname>...even now, while there is still data actually being reported in the graph (it's only 7:44AM at present), there is no vmstat file without the PID as part of the file name present. Not that I think that is the problem.
Is there anything I can do to increase verbosity in the client and/or server logging to determine if there's some sort of communication issue (or lack thereof)?
From: Jeremy Laidman [mailto:jlaidman at rebel-it.com.au] Sent: Wednesday, November 14, 2012 6:20 PM To: Mike Burger Cc: xymon at xymon.com Subject: Re: [Xymon] no vmstat data for 8 hours on weekdays?
On 15 November 2012 02:04, Mike Burger <Mike.Burger at freedommortgage.com> wrote:
Silly question time...is that code in the server or client?There are no silly questions: all questions lead to enlightenment!
The parser runs on the server.
J
CONFIDENTIALITY NOTICE: This e-mail message, including all attachments, is for the sole use of the intended recipient(s) and may contain confidential information. Any unauthorized review, use, disclosure, alteration or distribution is strictly prohibited and may violate state or federal law. If you are not the intended recipient, please contact the sender by reply e-mail, delete this email and destroy all copies of the message.
CONFIDENTIALITY NOTICE: This e-mail message, including all attachments, is for the sole use of the intended recipient(s) and may contain confidential information. Any unauthorized review, use, disclosure, alteration or distribution is strictly prohibited and may violate state or federal law. If you are not the intended recipient, please contact the sender by reply e-mail, delete this email and destroy all copies of the message.
On 16 November 2012 02:48, Mike Burger <Mike.Burger at freedommortgage.com>wrote:
That did it. I’m not sure if MAXMSG_STATUS or MAXMSG_DATA is the culprit, though I suspect it’s _DATA.
Nice one. Pretty sure it would be MAXMSG_CLIENT, which relates to the client messages. I have this set to 2048 on my server because I have some particularly fast-moving log files I want to monitor on a couple of my servers.
Rather than increasing the limit, you might want to find out why the client message is so large and try to reduce the size if you can. Perhaps the server is logging heeeeaps of messages into a logfile that's being monitored. Have a browse through the client data and see if there's one section that's really big and can probably be adjusted or eliminated.
$ xymoncmd xymon localhost 'clientlog name-of-client' | less
If it's log data, and you don't actually report on log messages, you might consider removing the "log:" entry from client-local.cfg on the Xymon server - either from the [aix] section, or (if it exist) from the [name-of-host] section.
Or you can have the client put a cap on the size of log data being sent back in its client message by adjusting the "log:" entry in the file client-local.cfg on the server. This limit is fetched by the client from the server after it reports its client data. So an adjustment on the server can take up to 10 minutes to be honoured by the client. The default for AIX syslog.log is 10k, so it's unlikely that this alone could push your client messages beyond 512k.
J
I think Mike said yesterday that is was his "procs" section that was causing the client to generate oversized client message. This test precedes the vmstat test in xymonclient-aix.sh
From: xymon-bounces at xymon.com [mailto:xymon-bounces at xymon.com]On Behalf Of Jeremy Laidman Sent: 15 November 2012 23:17 To: Mike Burger Cc: xymon at xymon.com Subject: Re: [Xymon] no vmstat data for 8 hours on weekdays?
On 16 November 2012 02:48, Mike Burger<Mike.Burger at freedommortgage.com> wrote:
That did it. I'm not sure if MAXMSG_STATUS orMAXMSG_DATA is the culprit, though I suspect it's _DATA.
Nice one. Pretty sure it would be MAXMSG_CLIENT, which relatesto the client messages. I have this set to 2048 on my server because I have some particularly fast-moving log files I want to monitor on a couple of my servers.
Rather than increasing the limit, you might want to find out whythe client message is so large and try to reduce the size if you can. Perhaps the server is logging heeeeaps of messages into a logfile that's being monitored. Have a browse through the client data and see if there's one section that's really big and can probably be adjusted or eliminated.
$ xymoncmd xymon localhost 'clientlog name-of-client' | less
If it's log data, and you don't actually report on log messages,you might consider removing the "log:" entry from client-local.cfg on the Xymon server - either from the [aix] section, or (if it exist) from the [name-of-host] section.
Or you can have the client put a cap on the size of log databeing sent back in its client message by adjusting the "log:" entry in the file client-local.cfg on the server. This limit is fetched by the client from the server after it reports its client data. So an adjustment on the server can take up to 10 minutes to be honoured by the client. The default for AIX syslog.log is 10k, so it's unlikely that this alone could push your client messages beyond 512k.
JThe information contained in this email is intended only for the use of the intended recipient at the email address to which it has been addressed. If the reader of this message is not an intended recipient, you are hereby notified that you have received this document in error and that any review, dissemination or copying of the message or associated attachments is strictly prohibited. If you have received this email in error, please contact the sender by return email or call 01793 877777 and ask for the sender and then delete it immediately from your system.Please note that neither the RWE Group of Companies nor the sender accepts any responsibility for viruses and it is your responsibility to scan attachments (if any).
Right...it wasn't logging, but CPU utilization data that was missing in the graphs. Logging was fine, and all other graphs were fine...just the CPU utilization graph was having issues. J
From: Chris.Morris at rwe.com [mailto:Chris.Morris at rwe.com] Sent: Friday, November 16, 2012 4:02 AM To: jlaidman at rebel-it.com.au; Mike Burger Cc: xymon at xymon.com Subject: RE: [Xymon] no vmstat data for 8 hours on weekdays?
I think Mike said yesterday that is was his "procs" section that was causing the client to generate oversized client message. This test precedes the vmstat test in xymonclient-aix.sh
From: xymon-bounces at xymon.com [mailto:xymon-bounces at xymon.com]On Behalf Of Jeremy Laidman Sent: 15 November 2012 23:17 To: Mike Burger Cc: xymon at xymon.com Subject: Re: [Xymon] no vmstat data for 8 hours on weekdays?
On 16 November 2012 02:48, Mike Burger<Mike.Burger at freedommortgage.com> wrote:
That did it. I'm not sure if MAXMSG_STATUS orMAXMSG_DATA is the culprit, though I suspect it's _DATA.
Nice one. Pretty sure it would be MAXMSG_CLIENT, which relatesto the client messages. I have this set to 2048 on my server because I have some particularly fast-moving log files I want to monitor on a couple of my servers.
Rather than increasing the limit, you might want to find out whythe client message is so large and try to reduce the size if you can. Perhaps the server is logging heeeeaps of messages into a logfile that's being monitored. Have a browse through the client data and see if there's one section that's really big and can probably be adjusted or eliminated.
$ xymoncmd xymon localhost 'clientlog name-of-client' | less
If it's log data, and you don't actually report on log messages,you might consider removing the "log:" entry from client-local.cfg on the Xymon server - either from the [aix] section, or (if it exist) from the [name-of-host] section.
Or you can have the client put a cap on the size of log databeing sent back in its client message by adjusting the "log:" entry in the file client-local.cfg on the server. This limit is fetched by the client from the server after it reports its client data. So an adjustment on the server can take up to 10 minutes to be honoured by the client. The default for AIX syslog.log is 10k, so it's unlikely that this alone could push your client messages beyond 512k.
J
The information contained in this email is intended only for the use of the intended recipient at the email address to which it has been addressed. If the reader of this message is not an intended recipient, you are hereby notified that you have received this document in error and that any review, dissemination or copying of the message or associated attachments is strictly prohibited. If you have received this email in error, please contact the sender by return email or call 01793 877777 and ask for the sender and then delete it immediately from your system.Please note that neither the RWE Group of Companies nor the sender accepts any responsibility for viruses and it is your responsibility to scan attachments (if any).
CONFIDENTIALITY NOTICE: This e-mail message, including all attachments, is for the sole use of the intended recipient(s) and may contain confidential information. Any unauthorized review, use, disclosure, alteration or distribution is strictly prohibited and may violate state or federal law. If you are not the intended recipient, please contact the sender by reply e-mail, delete this email and destroy all copies of the message.
On 16 November 2012 20:01, <Chris.Morris at rwe.com> wrote:
** I think Mike said yesterday that is was his "procs" section that was causing the client to generate oversized client message. This test precedes the vmstat test in xymonclient-aix.sh
Yes you are correct. I was confusing "client messages" with "client data".
J
participants (3)
-
Chris.Morris＠rwe.com
-
jlaidman＠rebel-it.com.au
-
Mike.Burger＠FreedomMortgage.com