On Wed, Nov 23, 2011 at 10:06 AM, Elizabeth Schwartz <betsy.schwartz at gmail.com> wrote:
I've got to monitor some very large log files. They're up to a couple
You can use the GNU tail "--retry" and "--follow=name" options to tail a file even through log rotation. Then pipe it into grep (set bufferring to per-line if more than one grep is used) and send the output via "logger" to syslog. If you simply redirect to another file rather than logger/syslog, you have problems when you want to zero/rotate the output file, so using logger/syslog is easier that way.
For example, add this into /etc/inittab:
tail --follow=MMRequest.log | grep "ERROR servicename LotsOfText" | grep --line-buffered -v "LOG NNNNN servicename LotsOfHTML" | logger -p user2.info -t logwatch
Note that some versions of init can't handle inittab lines that are too long, so you'd put it into a script.
but I'm just not getting the ERROR lines in the log. Is this file just too large and too full of HTML to parse? Any suggestions?
Yes, could be. I've had problems using the "log:" stuff with fast-moving logfiles.
Also, I've found bugs in the "logfetch" program in some cases of "trigger" and "ignore".
Cheers Jeremy