In message <139E0D6D-28B0-4576-A033-3525AD2970CA at PacketPushers.com>, Scott Walters writes: } }> On Mon, Dec 12, 2005 at 01:12:20PM -0600, Jeff Newman wrote: }>> }>> I wanted to move from a 5 minute interval on all my clients to a 1
}>> minute }>> interval. }>> } }In all my years of Systems Administration, things that run every
}minute all the time usually end up being a "Bad Idea". } }How will a smaller sampling period improve the service you provide?

We run pretty much all of our big brother tests every minute. On our new hobbit servers, we're running them at the default intervals.

BB shows us that our primary name server is going out for less than a minute, about every 62 minutes. Hobbit is missing most of those outages, although the longer "xxxx events received in the last xxx minutes" is what helped us spot the problem, as a whole bunch of machines' services don't respond well when our primary name server is out, and having a mass of servers go yellow then green, in unison, is sort of eye catching.

Tracy J. Di Marco White Information Technology Services Iowa State University

In message <F098822C-19A2-42CC-B6BA-2AB4E71D18BA at PacketPushers.com>, Scott Walters writes: }> }> We run pretty much all of our big brother tests every minute. On }> our new hobbit servers, we're running them at the default intervals. }> }> BB shows us that our primary name server is going out for less than }> a minute, about every 62 minutes. }> Hobbit is missing most of those }> outages, although the longer "xxxx events received in the last xxx }> minutes" is what helped us spot the problem, as a whole bunch of }> machines' services don't respond well when our primary name server }> is out, and having a mass of servers go yellow then green, in }> unison, is sort of eye catching. } }So hobbit with the xxx events (running every 5m) did provide enough
}information to indicate an intermittent problem with DNS?

Hobbit's non-green page, with last xxx events, gave us a large enough view that we could see all the machine services going yellow at the same time dns went red. We're monitoring a bit over 260 machines with a whole lot of difference services, so there's often something going red or yellow. With BB's older default of the last 25 events, there wasn't ever that much on screen to notice a group of swings to yellow, then back to green.

}Things running every 5m will collide with a problem that happens for
}a minute frequently enough to 'show up on the radar'

Sure, but we'd see up to 13 hours between dns 'red', when BB would get several in that period.

I haven't changed hobbit yet to 1 minute checks. I've even made an explicit explanation that I wasn't planning to shorten it to 1 minute checks when we officially switched over, and that was agreed to. However, with the fact that the 1 minute checks did actually make a difference in tracking down and solving the problem with DNS, I may yet have to work on that change. We'll see what kind of feedback I get after today. Even then, the only thing I'd really be willing to shorten to that frequency of checks are the remote checks, over the network.

}But every site has different requirements. It's just been my
}experience that sampling more frequently than 5m hits the knee-bend
}of diminishing returns. It also increases the potential for state
}changes, which chews up the filesystem with the history info.

I thought it was unnecessary when I originally brought BB into production years ago, but it was one of the requirements I ended up with to sell switching to BB. Some things can't be checked every minute, I have raid checks that can take more than a minute to run.

Tracy J. Di Marco White Information Technology Services Iowa State University

In message <7921F8E0-E87F-4A95-895C-A38ABB1D7831 at PacketPushers.com>, Scott Walters writes: }> }> Sure, but we'd see up to 13 hours between dns 'red', when BB would }> get several in that period. }> }> I haven't changed hobbit yet to 1 minute checks. I've even made }> an explicit explanation that I wasn't planning to shorten it to }> 1 minute checks when we officially switched over, and that was }> agreed to. However, with the fact that the 1 minute checks did }> actually make a difference in tracking down and solving the problem }> with DNS, I may yet have to work on that change. } }It sounds like your shop is so tidy that you would have found it and
}fixed it anyway. It was just a little brighter with the shorter
}interval.

I agree we would have found it. I'm amused at the thought of our shop being tidy, but thanks.

}> We'll see what }> kind of feedback I get after today. Even then, the only thing I'd }> really be willing to shorten to that frequency of checks are the }> remote checks, over the network. } }I believe hobbit has great 're-test' logic. So if it is down, it
}will test more frequently . . . .

And that's how I sold the 5 minute testing interval. And how I think I'll not have to shorten the interval for hobbit now.

}> I thought it was unnecessary when I originally brought BB into }> production years ago, but it was one of the requirements I ended }> up with to sell switching to BB. Some things can't be checked }> every minute, I have raid checks that can take more than a }> minute to run. } }On the client, 1m samples are opening Pandora's Box . . . .

Everyone really needs to consider what all the effects are of the frequency of the monitoring.

Tracy J. Di Marco White Information Technology Services Iowa State University

Scott,

I wanted to respond to you regarding technical reasons on a decreased interval.

I agree that in most cases where people would want an increase in frequency it would be for real-time performance analysis, whereas hobbit/bb are more for capacity planning/trending.

In my business, we deal with recieving all financial data and pushing that data around servers. a graph would have little data until the stock market opens, then the floodgates open :-) The graph then fluctuates with another surge at market close.

The interval being at 1 minute for specifically CPU and network is important to us for capacity planning purposes because during, say, market open, there are huge peaks that a 5m interval doesn't catch. We need to plan capacity based around those spikes, as those are indicative of future market trends in stock volume. It's not that the 5m interval does nothing, indeed it is helpful, but from a business perspective, a 1m interval allows us to plan capacity because it helps us catch the spikes that we want to see.

So something like a low-interval cpu/network column would be beneficial. Those tests could use seperate rrd files etc...

I recently integrated mrtg into hobbit. I assume that the 5m interval "issue" (not really an issue I know) exists with it as well since it utilizes the same rrd structure? Or can I set the interval of mrtg to be 1 minute? That would solve my networking interval problem.

Anyway, I hope I have explained the business reason well enough, feel free to ask any questions. I feel that while not all circumstances are ideal for a 1m polling sample, there are some situations where this is ideal.

-Jeff

On 12/15/05, Scott Walters <scott at packetpushers.com> wrote:

On Thu, 15 Dec 2005, Tracy J. Di Marco White wrote:

...

Everyone really needs to consider what all the effects are of the frequency of the monitoring.

I understand a more frequent sampling period is an easy sell, but I don't think it is a valid one when the rubber meets the road.

Plus, I try and make sure all technical decisions have a business reason.

I dislike technology and its advocates that try and drive the business. I guess I've gotten old and 'kewl' is no longer good enough ;)

Businessmen don't think in terms of technology. It's our job as professionals to make technology help the business. If we cannot clearly articulate how technology (or architecture changes) can help the business, it probably won't.

Unfortunately, mailing lists are not the best forum for these discussions.

-- Scott Walters -PacketPusher

To unsubscribe from the hobbit list, send an e-mail to hobbit-unsubscribe at hswn.dk

This helps *immensely*. Now we'll be able to justify shiny new gear
to management to reliably provide an IT infrastructure capable of
meeting the long term growth of trade volumes. On Dec 23, 2005, at 10:19 AM, Jeff Newman wrote:

servers. a graph would have little data until the stock market
opens, then the floodgates open :-) The graph then fluctuates with another surge at market close.

gotcha

The interval being at 1 minute for specifically CPU and network is
important to us for capacity planning purposes because during, say, market open,
there are huge peaks that a 5m interval doesn't catch. We need to plan capacity based
around those spikes, as those are indicative of future market
trends in stock volume. It's not that the 5m interval does nothing,
indeed it is helpful, but from a business perspective, a 1m
interval allows us to plan capacity because it helps us catch the
spikes that we want to see.

Absolutely. I am glad to hear you aware you must plan for the peaks.

Busy doesn't mean slow.

The server stats are generally only 1/2 the equation. They are the
impact on the machine. Ideally, for these types of situations, you
are also able to measure the load E.G. trade volumes and their
average execution times.

Knowing the RPMs of your motor doesn't tell you you MPH. If you
could see/prove that when CPU is 100% execution times can grow
outside of SLAs, its easier to convince management you need a bigger/ better environment and/or testing/QA/integration.

I hear there's a few nickels on Wall Street ;)

So something like a low-interval cpu/network column would be
beneficial. Those tests could use seperate rrd files etc...

I am still going to argue this isn't the right way to measure the
data in your environment to provide the information you are looking for.

1. RRD makes the presumption the older data gets, the less important
   it is. In your case that is *not true*. Each 'peak' is a set of
   data where the granularity needs to be preserved. So even if the RRA
   gets configured to keep 1m samples, which might help 'see' the last
   2 days or so of peaks, it won't help when you want to review the
   data set of Black Monday. Those peaks will have been averaged down.

2. One cannot assume causation of UNIX statistics and performance in
   the business environment. If you need to know your servers will
   handle 5 million trades in 5 minutes, you need throw 5 million trades
   at the boxes and see what happens. "If it ain't tested, it doesn't
   work."

3. When environments reach bottlenecks, it's impossible to say what
   the real peak is. If your CPU is at 100%, one cannot know (without
   testing) what the real demand for CPU is . . . .

4. It's the always the code/SQL/CICS anyway ;)

I recently integrated mrtg into hobbit. I assume that the 5m
interval "issue" (not really an issue I know) exists with it as
well since it utilizes the same rrd structure? Or can I set the
interval of mrtg to be 1 minute? That would solve my networking
interval problem.

But that is only one sample per minute. For your application, you
need something *much* more granular.

Anyway, I hope I have explained the business reason well enough,
feel free to ask any questions. I feel that while not all
circumstances are ideal for a 1m polling sample, there are some situations where this is ideal.

You have legitimate business needs for sure, and an idea for a
feature which would be very *useful*.

A high interval/sampling for 'stress testing' impact with the data
being preserved.

That would be a great addition hobbit. I am not sure if RRD is the
right backend, but it might work if the solution is clever.

I'll let it rattle around . . . .

scott

On Wed, 2005-12-14 at 17:31 -0600, Jeff Newman wrote:

On 12/13/05, Scott Walters <scott at packetpushers.com> wrote:

    >> The graph DB's that vmstat feeds data into (the RRD files)
    are
    >> constructed in such a way that a 5-minute interval is what
    makes 
    >> sense. So running them with anything else really just a
    waste of
    >> ressources.
    
    > With the stock larrd/hobbit RRD definitions you are
    correct.  He'll
    > only use one of the five, and whine about the timestamp of
    the other 
    > four.

Firstly, can you explain your comment in more detail? Secondly, im confused as to why you would state that I would "whine" about anything when you have no basis for a conclusion to that effect. It seems to be a rather pointed comment in a discussion that hasn't involved the use of language that would dictate a response like that.

I think he was referring to the error message that would be generated because of the extra data compared to the interval configured in the rrd files.

    > To become flexible enough
    > to handle different sampling rates, the server would need to
    know the 
    > frequency of the tests.  And then changing the RRD in the
    future is
    > 'almost' impossible (very difficult at the least).  And I've
    never
    > seen what happens to 1.5 years of data when you start
    messing with 
    > the RRD.
    
    > In the end, I think you'd get the worst of both worlds.

Well, we could take this the other way, and say that we only wanted to run our tests once every 10 minutes, because it was causing too much overhead to run the tests every 5 minutes. How would we deal with that?

I think there are benefits, on the client side, the client should pass the frequency which it is calling the tests at, to them, so that for example, the vmstat test can adjust how long it will run for to either 600 seconds, or 60 seconds, or whatever else is needed.

Further to that, there is some additional work (which I feel is the real place that all the work is involved, the client side stuff would be quite simple, or so it sounds). On the server, we either need to re-create/adjust the rrd file so that we can insert data more frequently, or else we need to somehow summarise the data before insertion (which means there was no benefit in collecting the data more frequently in the first place). So, the question becomes, how difficult is it to convert an rrd file, which was initially created to store data-points every 300 seconds, such that we can now store data-points every X seconds? The second part to this question, is how does hobbit know how frequently you want to send your reports? ie, it can't be based on 'however often they are received', because that value would change very frequantly, ie, the reports are done every 300 seconds, but by the time the report is submitted/processed by hobbit, it might be 1 or 2 seconds late/early compared to last time..... Could hobbit server 'learn' the frequency from the client (which is where this is configured anyway), because the client would report that value to the server as a part of the vmstat output?

Of course, even once both of those questions are satisfactorily answered, you (yes, you) need to convince somebody to take the effort to actually do it. Simply seeing the methods, and the interest some people have taken in the possibility might be enough for someone with the coding skills to do it, or, sometimes you might need to provide other incentives (even paid). Let me state clearly right now before I go on, no, don't pay me, I don't have that level of coding skills in C to do it for you, nor the knowledge of RRD. No, I don't know who you might pay, or anything else, I have no interest in any of it, I'm just stating a simple fact of life (you want something done, you can't do it, so find someone who can do it, and motivate them to the point where they will do it whether they want to or not)...

    > I disagree.  If real-time performance analysis is needed, I
    would
    > pick other tools --  "vmstat 5"  works for me;)  Or
    construct/fork 
    > the client agent specifically designed for such a task, and
    run it on
    > an as-needed basis.

There are other tools yes. I am trying to leverage hobbit. If it's not possible and nobody wants to do it, then yes, ill look into other tools. On the same token I don't want to kill my performance by running lots of different monitoring on the server. Hobbit is extrodinarily lightweight on the client (as opposed to other solutions out there) so I think something like this is possible without overloading a client.

Agreed, it would be nice to not have to run hobbit plus something else when they are both collecting the same data (just a different frequency).

Just my two cents.

Here's a couple of mine also :)

Regards, Adam

First off, I know I can come off terse in e-mail, but they are not
personal attacks.

It can be a bad idea sometimes, others not (for example, the reply
from the person catching intermittant problems with BB running every
minute)

Who ended up stating the anomaly *was* detected in 5m intervals, but
only once every 13h instead of every hour. But I still don't
understand how it will help *you*.

A smaller sampling period can show things in a more granular
aspect. For example, a process kicks off and 5 minutes later you
see 100 errors (im keeping things generic for illustrative
purposes) Were those 100 errors in the first minute? the last?
constantly throughout the 5 minutes?

The 5m averages over a week would be quite low compared so a single
5m plot. From that, one could extrapolate in the last 5m things have
not been 'normal'.

Im not saying your wrong, simply pointing out that it's not as
black and white as your making it.

And I am disagreeing with you ;) I've been watching the data in
these graphs for many many years now, and I have yet to come across a
situation where having a 1m sampling/graphing period would have
helped me fix/improve something . . .

It's like a story problem with too much information, it makes coming
up with the real answer harder in the end. Most people don't have
time/enegry/brains to be able to sift all the data correctly. If if
they do, the 5m samples are good enough.

Most people (including really smart people that are forgetful) can't
deal with an auto-scaling y-axis.

Something being just interesting initially can sometimes uncover
problems that you didn't see before.

Like I said, if you have job were interesting is worthwhile,
wonderful. In my experience, most folks that are running the BB/ hobbit tools are involved in the operational aspects of
infrastructure, not R&D.

...

With the stock larrd/hobbit RRD definitions you are correct. He'll only use one of the five, and whine about the timestamp of the other four.

Firstly, can you explain your comment in more detail?

RRD interpolates Time Series Data to put a value at a fixed
interval. That is why you hardly ever see integers in the data. If
you sample comes in at 299s, RRD interpolates what that value to what
would have been at 300s. How this is done can be tuned. The default
settings with the RRAs expect data to happen every 300s. RRD will
only insert data one time within that interval.

Secondly, im confused as to why you would state that I would "whine" about
anything when you have no basis for a conclusion to that effect. It seems to
be a rather pointed comment in a discussion that hasn't involved the use of
language that would dictate a response like that.

"He'll whine" meant rrdtool, not you:

ERROR: illegal attempt to update using time 1042731000 when last
update time

is 1043099100 (minimum one second step)

That's whining in my book. Sorry you thought I was speaking about you.

That is a very good point you make. There is a difference between real-time analysis and capacity planning/trending. I don't however
think that it is that far outside of hobbit's scope to try and leverage
it for a more pointed analysis.

From a software development standpoint there is a lot to be said
for: "Do one thing and do it well". If architecting the RRD
framework for RTA breaks trending, bad idea.

My goal isn't to take every machine in my environment and make them into 1 minute sampling period machines. To have the
ability to do so on a machine-by-machine basis could be useful

Which is why I proposed another client collector for this activity.

...

That's my design you inherited and because of the complexity of the parts, I think it is a very solid design.

I don't think anyone is really questioning that.

You are questioning that. And that is fine. I don't take it
personally you think there may be a better way. I know my way may
not be the best, but I sure know exactly *why* I chose it.

Honestly, I don't claim to know anything about the way larrd and
hobbit are coded in the slightest. There are difficulties to be sure, but
part of having a community such as this is to foster ideas and innovation. Just
because you don't think it's useful or that it's hard doesn't mean the same is
true for everyone out there.

Ahhhhh, to the heart of the matter. Don't suggest ideas in a public
forum if you are not prepared to defend them. Fostering ideas comes
from intelligent discussions. I merely wanted to understand why you
felt you needed a higher sampling rate from a business perspective.

scott