TYPO there, 47M not 47G. The files are about 2G/day
On Tue, Nov 22, 2011 at 6:06 PM, Elizabeth Schwartz <betsy.schwartz at gmail.com> wrote:
I've got to monitor some very large log files. They're up to a couple gigs a day and individual lines can be 30800 characters or more , including HTML. (changing the log file format is a project for another day) So my last half hour of one of these files chosen at random is 21,000 lines, 47G.
I want to look at all the lines that start with
2011-11-22 4:15:31 ERROR servicename LotsOfText
I want to ignore lines that start 2011-11-22 17:13:39 LOG NNNNN servicename LotsOfHTML
Ignoring all of those lines would bring it to a manageable size (this particular file is 41 lines, 23k data)
I've been playing around with rules in client-local.cfg like: [mmw2.example.com] log:/var/log/mmb1/MMRequest.log:10240 trigger ERROR ignore LOG
but I'm just not getting the ERROR lines in the log. Is this file just too large and too full of HTML to parse? Any suggestions?
(we can write a custom script, of course, and I'm thinking of bringing in SEC. But it sure would be handy to be able to do this with out of the box xymon)