Help with very large log file - not getting the right lines
I've got to monitor some very large log files. They're up to a couple gigs a day and individual lines can be 30800 characters or more , including HTML. (changing the log file format is a project for another day) So my last half hour of one of these files chosen at random is 21,000 lines, 47G.
I want to look at all the lines that start with
2011-11-22 4:15:31 ERROR servicename LotsOfText
I want to ignore lines that start 2011-11-22 17:13:39 LOG NNNNN servicename LotsOfHTML
Ignoring all of those lines would bring it to a manageable size (this particular file is 41 lines, 23k data)
I've been playing around with rules in client-local.cfg like: [mmw2.example.com] log:/var/log/mmb1/MMRequest.log:10240 trigger ERROR ignore LOG
but I'm just not getting the ERROR lines in the log. Is this file just too large and too full of HTML to parse? Any suggestions?
(we can write a custom script, of course, and I'm thinking of bringing in SEC. But it sure would be handy to be able to do this with out of the box xymon)
TYPO there, 47M not 47G. The files are about 2G/day
On Tue, Nov 22, 2011 at 6:06 PM, Elizabeth Schwartz <betsy.schwartz at gmail.com> wrote:
I've got to monitor some very large log files. They're up to a couple gigs a day and individual lines can be 30800 characters or more , including HTML. (changing the log file format is a project for another day) So my last half hour of one of these files chosen at random is 21,000 lines, 47G.
I want to look at all the lines that start with
2011-11-22 4:15:31 ERROR servicename LotsOfText
I want to ignore lines that start 2011-11-22 17:13:39 LOG NNNNN servicename LotsOfHTML
Ignoring all of those lines would bring it to a manageable size (this particular file is 41 lines, 23k data)
I've been playing around with rules in client-local.cfg like: [mmw2.example.com] log:/var/log/mmb1/MMRequest.log:10240 trigger ERROR ignore LOG
but I'm just not getting the ERROR lines in the log. Is this file just too large and too full of HTML to parse? Any suggestions?
(we can write a custom script, of course, and I'm thinking of bringing in SEC. But it sure would be handy to be able to do this with out of the box xymon)
First of many "quick fixes": could you set up an auto-restarting script to do "tail -f logfile | grep ERROR > errorlog"?? Then watch the aeroflot file.
Ralph Mitchell On Nov 22, 2011 6:07 PM, "Elizabeth Schwartz" <betsy.schwartz at gmail.com> wrote:
I've got to monitor some very large log files. They're up to a couple gigs a day and individual lines can be 30800 characters or more , including HTML. (changing the log file format is a project for another day) So my last half hour of one of these files chosen at random is 21,000 lines, 47G.
I want to look at all the lines that start with
2011-11-22 4:15:31 ERROR servicename LotsOfText
I want to ignore lines that start 2011-11-22 17:13:39 LOG NNNNN servicename LotsOfHTML
Ignoring all of those lines would bring it to a manageable size (this particular file is 41 lines, 23k data)
I've been playing around with rules in client-local.cfg like: [mmw2.example.com] log:/var/log/mmb1/MMRequest.log:10240 trigger ERROR ignore LOG
but I'm just not getting the ERROR lines in the log. Is this file just too large and too full of HTML to parse? Any suggestions?
(we can write a custom script, of course, and I'm thinking of bringing in SEC. But it sure would be handy to be able to do this with out of the box xymon)
Xymon mailing list Xymon at xymon.com http://lists.xymon.com/mailman/listinfo/xymon
My first thought, also. I think you can also train syslog-ng to do that, if you're using syslog-ng.
On Tue, November 22, 2011 18:34, Ralph Mitchell wrote:
First of many "quick fixes": could you set up an auto-restarting script to do "tail -f logfile | grep ERROR > errorlog"?? Then watch the aeroflot file.
Ralph Mitchell On Nov 22, 2011 6:07 PM, "Elizabeth Schwartz" <betsy.schwartz at gmail.com> wrote:
I've got to monitor some very large log files. They're up to a couple gigs a day and individual lines can be 30800 characters or more , including HTML. (changing the log file format is a project for another day) So my last half hour of one of these files chosen at random is 21,000 lines, 47G.
I want to look at all the lines that start with
2011-11-22 4:15:31 ERROR servicename LotsOfText
I want to ignore lines that start 2011-11-22 17:13:39 LOG NNNNN servicename LotsOfHTML
Ignoring all of those lines would bring it to a manageable size (this particular file is 41 lines, 23k data)
I've been playing around with rules in client-local.cfg like: [mmw2.example.com] log:/var/log/mmb1/MMRequest.log:10240 trigger ERROR ignore LOG
but I'm just not getting the ERROR lines in the log. Is this file just too large and too full of HTML to parse? Any suggestions?
(we can write a custom script, of course, and I'm thinking of bringing in SEC. But it sure would be handy to be able to do this with out of the box xymon) _______________________________________________ Xymon mailing list Xymon at xymon.com http://lists.xymon.com/mailman/listinfo/xymon
_______________________________________________ Xymon mailing list Xymon at xymon.com http://lists.xymon.com/mailman/listinfo/xymon
Actually, you can do the "tail" with the default rsyslogd imfile module. http://www.rsyslog.com/doc/imfile.html http://www.rsyslog.com/doc/multi_ruleset.html
This was pointed out to me a while back by Irithori on linuxforums.org, and I had forgotten it.
On Tue, November 22, 2011 20:20, Xymon User in Richmond wrote:
My first thought, also. I think you can also train syslog-ng to do that, if you're using syslog-ng.
On Tue, November 22, 2011 18:34, Ralph Mitchell wrote:
First of many "quick fixes": could you set up an auto-restarting script to do "tail -f logfile | grep ERROR > errorlog"?? Then watch the aeroflot file.
Ralph Mitchell On Nov 22, 2011 6:07 PM, "Elizabeth Schwartz" <betsy.schwartz at gmail.com> wrote:
I've got to monitor some very large log files. They're up to a couple gigs a day and individual lines can be 30800 characters or more , including HTML. (changing the log file format is a project for another day) So my last half hour of one of these files chosen at random is 21,000 lines, 47G.
I want to look at all the lines that start with
2011-11-22 4:15:31 ERROR servicename LotsOfText
I want to ignore lines that start 2011-11-22 17:13:39 LOG NNNNN servicename LotsOfHTML
Ignoring all of those lines would bring it to a manageable size (this particular file is 41 lines, 23k data)
I've been playing around with rules in client-local.cfg like: [mmw2.example.com] log:/var/log/mmb1/MMRequest.log:10240 trigger ERROR ignore LOG
but I'm just not getting the ERROR lines in the log. Is this file just too large and too full of HTML to parse? Any suggestions?
(we can write a custom script, of course, and I'm thinking of bringing in SEC. But it sure would be handy to be able to do this with out of the box xymon) _______________________________________________ Xymon mailing list Xymon at xymon.com http://lists.xymon.com/mailman/listinfo/xymon
_______________________________________________ Xymon mailing list Xymon at xymon.com http://lists.xymon.com/mailman/listinfo/xymon
Xymon mailing list Xymon at xymon.com http://lists.xymon.com/mailman/listinfo/xymon
On Wed, Nov 23, 2011 at 10:06 AM, Elizabeth Schwartz <betsy.schwartz at gmail.com> wrote:
I've got to monitor some very large log files. They're up to a couple
You can use the GNU tail "--retry" and "--follow=name" options to tail a file even through log rotation. Then pipe it into grep (set bufferring to per-line if more than one grep is used) and send the output via "logger" to syslog. If you simply redirect to another file rather than logger/syslog, you have problems when you want to zero/rotate the output file, so using logger/syslog is easier that way.
For example, add this into /etc/inittab:
tail --follow=MMRequest.log | grep "ERROR servicename LotsOfText" | grep --line-buffered -v "LOG NNNNN servicename LotsOfHTML" | logger -p user2.info -t logwatch
Note that some versions of init can't handle inittab lines that are too long, so you'd put it into a script.
but I'm just not getting the ERROR lines in the log. Is this file just too large and too full of HTML to parse? Any suggestions?
Yes, could be. I've had problems using the "log:" stuff with fast-moving logfiles.
Also, I've found bugs in the "logfetch" program in some cases of "trigger" and "ignore".
Cheers Jeremy
participants (4)
-
betsy.schwartz＠gmail.com
-
hobbit＠epperson.homelinux.net
-
jlaidman＠rebel-it.com.au
-
ralphmitchell＠gmail.com