Logfile monitoring - I'd like some comments
A few days ago, I mentioned that I would like to do logfile monitoring for the next Hobbit release.
I've worked a bit on this and have a prototype solution for it, which you can test with the current snapshots. I'd like some comments on how it works to make sure I haven't overlooked something before committing myself.
There are several objectives:
- As far as is possible, logfile monitoring must be configured centrally, on the Hobbit server. Having to go to each server to (re)configure what logfiles to check and what to look for simply doesn't work.
- The amount of data sent from each client to Hobbit should be small, but it must catch the "important" stuff.
- You rarely know in advance what will be in the logs when you need them the most. So the monitor should give you as much of the log entries as possible, not just those lines that match some pre-defined strings or regex'es.
- Some systems log messages on multiple lines. The system must be able to show all parts of a log entry.
- Logfile entries must appear on the monitor for some time after they show up in the logs, but should also disappear after a while.
In other words: The ideal solution would let you have the entire logfile available on the Hobbit server - but that obviously won't work. So the client should - after weeding out the really irrelevant stuff - send us as much of each logfile as possible.
My proposed solution is this:
- On the Hobbit server, there's a log-monitoring configuration file for the Hobbit clients. This defines which logfiles are monitored for a single client installation, or you can define it for a group of clients. (The idea is to define at least one group for each operating system, since the standard system logs are OS dependant). This configuration lists the log filename, the maximum amount of data to send from this logfile, a regex "noise" filter (i.e. lines that are stripped from the logfile), and *optionally* a regex identifying really interesting stuff in the logfile that should always be reported.
- When a client connects to the Hobbit server and sends the normal client message, the Hobbit server will respond with the logfile configuration for this client. So the client has a copy of the central configuration file, but only the part that it needs for itself. The reason for sending this as a response to the client message is to avoid an extra round-trip from client to server; piggy-backing the config push on the client message means that it is almost without any performance cost on the server side.
- When the client runs, it uses the local copy of the configuration file to determine what logs to look at. For each logfile, it maintains a "where-was-I-the-last-time" status, so it only looks at the entries made to the logfile during the past 30 minutes. First, the client strips off any "noise" messages. Then, if all of the entries fit into the maximum size that can be reported, it sends all of the log to the Hobbit server. If there is more than will fit, it first checks to see of the regex defining the really interesting stuff is present in the log - if it is, then it drops anything before the interesting text. If there is still more than will fit, it keeps the interesting text + a few lines after that (to allow for multi-line log-entries which some OS'es have), and then sends that together with as much of last part the log as will fit inside the max. message size.
This part has been implemented in the Hobbit daemon (hobbitd), and in the clients via a new "logfetch" utility. This utility uses standard regular expressions - not the Perl-compatible ones, because that would require you to install the PCRE library on all of your clients. The standard regex routines are included in all (I think) system libraries used today.
The last part is what happens when the log data arrives on the Hobbit server. Currently, there's a simple processing of this data to just dump it into an always-green "msgs" column. What should happen once I get it coded is:
- Data from each logfile is matched against a set of strings (regex'es) defined in the hobbit-clients.cfg file. Each string determines the color (red, yellow, green) and sets the color of the msgs column accordingly.
When the color has been decided, all of the normal alerting happens automatically. I do plan on making a more fine-grained alert mechanism (for the msgs, procs and disk statuses) so you can direct alerts to different groups depending on exactly which log-message triggered the alert, but that will not be part of this release.
So - how does that sound ? Anything I've missed ?
Regards, Henrik
The feature of the client retrieving its configuration from the server might cause some problems with firewalls, and security types might not be willing to open firewalls for such infrequent messages. I like the idea of having centralized configuration, but it would be nice if it could be implemented without upsetting the firewall admins. It also eliminates the pure "push" of the BigBrother client, which is also something security types like. It might be a good idea to have an option for local client configuration.
Thanks, Larry Barber
On 2/14/06, Henrik Stoerner <henrik at hswn.dk> wrote:
A few days ago, I mentioned that I would like to do logfile monitoring for the next Hobbit release.
I've worked a bit on this and have a prototype solution for it, which you can test with the current snapshots. I'd like some comments on how it works to make sure I haven't overlooked something before committing myself.
There are several objectives:
- As far as is possible, logfile monitoring must be configured centrally, on the Hobbit server. Having to go to each server to (re)configure what logfiles to check and what to look for simply doesn't work.
- The amount of data sent from each client to Hobbit should be small, but it must catch the "important" stuff.
- You rarely know in advance what will be in the logs when you need them the most. So the monitor should give you as much of the log entries as possible, not just those lines that match some pre-defined strings or regex'es.
- Some systems log messages on multiple lines. The system must be able to show all parts of a log entry.
- Logfile entries must appear on the monitor for some time after they show up in the logs, but should also disappear after a while.
In other words: The ideal solution would let you have the entire logfile available on the Hobbit server - but that obviously won't work. So the client should - after weeding out the really irrelevant stuff - send us as much of each logfile as possible.
My proposed solution is this:
- On the Hobbit server, there's a log-monitoring configuration file for the Hobbit clients. This defines which logfiles are monitored for a single client installation, or you can define it for a group of clients. (The idea is to define at least one group for each operating system, since the standard system logs are OS dependant). This configuration lists the log filename, the maximum amount of data to send from this logfile, a regex "noise" filter (i.e. lines that are stripped from the logfile), and *optionally* a regex identifying really interesting stuff in the logfile that should always be reported.
- When a client connects to the Hobbit server and sends the normal client message, the Hobbit server will respond with the logfile configuration for this client. So the client has a copy of the central configuration file, but only the part that it needs for itself. The reason for sending this as a response to the client message is to avoid an extra round-trip from client to server; piggy-backing the config push on the client message means that it is almost without any performance cost on the server side.
- When the client runs, it uses the local copy of the configuration file to determine what logs to look at. For each logfile, it maintains a "where-was-I-the-last-time" status, so it only looks at the entries made to the logfile during the past 30 minutes. First, the client strips off any "noise" messages. Then, if all of the entries fit into the maximum size that can be reported, it sends all of the log to the Hobbit server. If there is more than will fit, it first checks to see of the regex defining the really interesting stuff is present in the log - if it is, then it drops anything before the interesting text. If there is still more than will fit, it keeps the interesting text + a few lines after that (to allow for multi-line log-entries which some OS'es have), and then sends that together with as much of last part the log as will fit inside the max. message size.
This part has been implemented in the Hobbit daemon (hobbitd), and in the clients via a new "logfetch" utility. This utility uses standard regular expressions - not the Perl-compatible ones, because that would require you to install the PCRE library on all of your clients. The standard regex routines are included in all (I think) system libraries used today.
The last part is what happens when the log data arrives on the Hobbit server. Currently, there's a simple processing of this data to just dump it into an always-green "msgs" column. What should happen once I get it coded is:
- Data from each logfile is matched against a set of strings (regex'es) defined in the hobbit-clients.cfg file. Each string determines the color (red, yellow, green) and sets the color of the msgs column accordingly.
When the color has been decided, all of the normal alerting happens automatically. I do plan on making a more fine-grained alert mechanism (for the msgs, procs and disk statuses) so you can direct alerts to different groups depending on exactly which log-message triggered the alert, but that will not be part of this release.
So - how does that sound ? Anything I've missed ?
Regards, Henrik
To unsubscribe from the hobbit list, send an e-mail to hobbit-unsubscribe at hswn.dk
On Tue, Feb 14, 2006 at 04:56:17PM, Larry Barber wrote:
The feature of the client retrieving its configuration from the server might cause some problems with firewalls, and security types might not be willing
BB Client currently uses the config feature to pull info from hobbit server. It does not require an inbound connection rule for the client to pull data from hobbit server. So if you currently use BB client to push the client info to hobbit server on default destination port 1984 then you should be good I think
to open firewalls for such infrequent messages. I like the idea of having centralized configuration, but it would be nice if it could be implemented without upsetting the firewall admins. It also eliminates the pure "push" of the BigBrother client, which is also something security types like. It might be a good idea to have an option for local client configuration.
Thanks, Larry Barber
On 2/14/06, Henrik Stoerner <henrik at hswn.dk> wrote:
A few days ago, I mentioned that I would like to do logfile monitoring for the next Hobbit release.
I've worked a bit on this and have a prototype solution for it, which you can test with the current snapshots. I'd like some comments on how it works to make sure I haven't overlooked something before committing myself.
There are several objectives:
- As far as is possible, logfile monitoring must be configured centrally, on the Hobbit server. Having to go to each server to (re)configure what logfiles to check and what to look for simply doesn't work.
- The amount of data sent from each client to Hobbit should be small, but it must catch the "important" stuff.
- You rarely know in advance what will be in the logs when you need them the most. So the monitor should give you as much of the log entries as possible, not just those lines that match some pre-defined strings or regex'es.
- Some systems log messages on multiple lines. The system must be able to show all parts of a log entry.
- Logfile entries must appear on the monitor for some time after they show up in the logs, but should also disappear after a while.
In other words: The ideal solution would let you have the entire logfile available on the Hobbit server - but that obviously won't work. So the client should - after weeding out the really irrelevant stuff - send us as much of each logfile as possible.
My proposed solution is this:
- On the Hobbit server, there's a log-monitoring configuration file for the Hobbit clients. This defines which logfiles are monitored for a single client installation, or you can define it for a group of clients. (The idea is to define at least one group for each operating system, since the standard system logs are OS dependant). This configuration lists the log filename, the maximum amount of data to send from this logfile, a regex "noise" filter (i.e. lines that are stripped from the logfile), and *optionally* a regex identifying really interesting stuff in the logfile that should always be reported.
- When a client connects to the Hobbit server and sends the normal client message, the Hobbit server will respond with the logfile configuration for this client. So the client has a copy of the central configuration file, but only the part that it needs for itself. The reason for sending this as a response to the client message is to avoid an extra round-trip from client to server; piggy-backing the config push on the client message means that it is almost without any performance cost on the server side.
- When the client runs, it uses the local copy of the configuration file to determine what logs to look at. For each logfile, it maintains a "where-was-I-the-last-time" status, so it only looks at the entries made to the logfile during the past 30 minutes. First, the client strips off any "noise" messages. Then, if all of the entries fit into the maximum size that can be reported, it sends all of the log to the Hobbit server. If there is more than will fit, it first checks to see of the regex defining the really interesting stuff is present in the log - if it is, then it drops anything before the interesting text. If there is still more than will fit, it keeps the interesting text + a few lines after that (to allow for multi-line log-entries which some OS'es have), and then sends that together with as much of last part the log as will fit inside the max. message size.
This part has been implemented in the Hobbit daemon (hobbitd), and in the clients via a new "logfetch" utility. This utility uses standard regular expressions - not the Perl-compatible ones, because that would require you to install the PCRE library on all of your clients. The standard regex routines are included in all (I think) system libraries used today.
The last part is what happens when the log data arrives on the Hobbit server. Currently, there's a simple processing of this data to just dump it into an always-green "msgs" column. What should happen once I get it coded is:
- Data from each logfile is matched against a set of strings (regex'es) defined in the hobbit-clients.cfg file. Each string determines the color (red, yellow, green) and sets the color of the msgs column accordingly.
When the color has been decided, all of the normal alerting happens automatically. I do plan on making a more fine-grained alert mechanism (for the msgs, procs and disk statuses) so you can direct alerts to different groups depending on exactly which log-message triggered the alert, but that will not be part of this release.
So - how does that sound ? Anything I've missed ?
Regards, Henrik
To unsubscribe from the hobbit list, send an e-mail to hobbit-unsubscribe at hswn.dk
-- Asif Iqbal PGP Key: 0xE62693C5 KeyServer: pgp.mit.edu "..there are two kinds of people: those who work and those who take the credit...try to be in the first group;...less competition there." - Indira Gandhi
On Tue, Feb 14, 2006 at 04:56:17PM -0600, Larry Barber wrote:
The feature of the client retrieving its configuration from the server might cause some problems with firewalls, and security types might not be willing to open firewalls for such infrequent messages.
Infrequent? It's part of the client sending its status update, so it happens every 5 minutes.
Since this is just an extension of the protocol that is already being used for sending statuses to the Hobbit server, you won't need any additional firewall openings.
And it's not like you can use it for any kind of file transfer. You'll have to get the data into the Hobbit server first, so security on that server is obviously important (but if you're care about security, you should really care about the security of your monitoring server in the first place). What's sent to the client can only be a part of the hobbit servers' log-configuration.
I guess you shouldn't tell your firewall admin's about the "config" request you can send through the "bb" utility ....
I like the idea of having centralized configuration, but it would be nice if it could be implemented without upsetting the firewall admins. It also eliminates the pure "push" of the BigBrother client, which is also something security types like. It might be a good idea to have an option for local client configuration.
It is possible to run the client using a local configuration. You'll need to install the PCRE libraries on your clients for that, but it can be done.
Regards, Henrik
Hi Henrik,
that sounds wonderful, your ideas are pretty good. I have one suggestion and one point to think about.
For monitoring application-logs the config file soon would become very complex and likely unreadable. There it would be nice if there is a possibilty for include-files. So general definitions -e.g. for OS-related logs - would be made in the general config file, all the individual stuff in <host>.config files.
Second, there are logs which change their names once the server or the application stops. So a "message.log" might become "message<date>.log" while there is a new message.log now. Still the last informations in "message<date>.log" are relevant, especially if the reason for the new log was a crash of the application. I don't know how to deal with this situation as there are multiple ways how logs might change their names but perhaps others have an idea of how to do that.
kind regards Rolf
A few days ago, I mentioned that I would like to do logfile monitoring for the next Hobbit release.
I've worked a bit on this and have a prototype solution for it, which you can test with the current snapshots. I'd like some comments on how it works to make sure I haven't overlooked something before committing myself.
There are several objectives:
- As far as is possible, logfile monitoring must be configured centrally, on the Hobbit server. Having to go to each server to (re)configure what logfiles to check and what to look for simply doesn't work.
- The amount of data sent from each client to Hobbit should be small, but it must catch the "important" stuff.
- You rarely know in advance what will be in the logs when you need them the most. So the monitor should give you as much of the log entries as possible, not just those lines that match some pre-defined strings or regex'es.
- Some systems log messages on multiple lines. The system must be able to show all parts of a log entry.
- Logfile entries must appear on the monitor for some time after they show up in the logs, but should also disappear after a while.
In other words: The ideal solution would let you have the entire logfile available on the Hobbit server - but that obviously won't work. So the client should - after weeding out the really irrelevant stuff - send us as much of each logfile as possible.
My proposed solution is this:
- On the Hobbit server, there's a log-monitoring configuration file for the Hobbit clients. This defines which logfiles are monitored for a single client installation, or you can define it for a group of clients. (The idea is to define at least one group for each operating system, since the standard system logs are OS dependant). This configuration lists the log filename, the maximum amount of data to send from this logfile, a regex "noise" filter (i.e. lines that are stripped from the logfile), and *optionally* a regex identifying really interesting stuff in the logfile that should always be reported.
- When a client connects to the Hobbit server and sends the normal client message, the Hobbit server will respond with the logfile configuration for this client. So the client has a copy of the central configuration file, but only the part that it needs for itself. The reason for sending this as a response to the client message is to avoid an extra round-trip from client to server; piggy-backing the config push on the client message means that it is almost without any performance cost on the server side.
- When the client runs, it uses the local copy of the configuration file to determine what logs to look at. For each logfile, it maintains a "where-was-I-the-last-time" status, so it only looks at the entries made to the logfile during the past 30 minutes. First, the client strips off any "noise" messages. Then, if all of the entries fit into the maximum size that can be reported, it sends all of the log to the Hobbit server. If there is more than will fit, it first checks to see of the regex defining the really interesting stuff is present in the log - if it is, then it drops anything before the interesting text. If there is still more than will fit, it keeps the interesting text + a few lines after that (to allow for multi-line log-entries which some OS'es have), and then sends that together with as much of last part the log as will fit inside the max. message size.
This part has been implemented in the Hobbit daemon (hobbitd), and in the clients via a new "logfetch" utility. This utility uses standard regular expressions - not the Perl-compatible ones, because that would require you to install the PCRE library on all of your clients. The standard regex routines are included in all (I think) system libraries used today.
The last part is what happens when the log data arrives on the Hobbit server. Currently, there's a simple processing of this data to just dump it into an always-green "msgs" column. What should happen once I get it coded is:
- Data from each logfile is matched against a set of strings (regex'es) defined in the hobbit-clients.cfg file. Each string determines the color (red, yellow, green) and sets the color of the msgs column accordingly.
When the color has been decided, all of the normal alerting happens automatically. I do plan on making a more fine-grained alert mechanism (for the msgs, procs and disk statuses) so you can direct alerts to different groups depending on exactly which log-message triggered the alert, but that will not be part of this release.
So - how does that sound ? Anything I've missed ?
Regards, Henrik
To unsubscribe from the hobbit list, send an e-mail to hobbit-unsubscribe at hswn.dk
-- Mit freundlichen Gruessen Rolf Schrittenlocher
HRZ/BDV, Senckenberganlage 31, 60054 Frankfurt Tel: (49) 69 - 798 28908 Fax: (49) 69 798 2881 LBS: lbs-f at mlist.uni-frankfurt.de Persoenlich: schritte at rz.uni-frankfurt.de
Hi Rolf,
On Wed, Feb 15, 2006 at 07:30:11AM +0100, Rolf Schrittenlocher wrote:
Hi Henrik,
that sounds wonderful, your ideas are pretty good.
Thanks.
I have one suggestion and one point to think about.
For monitoring application-logs the config file soon would become very complex and likely unreadable. There it would be nice if there is a possibilty for include-files. So general definitions -e.g. for OS-related logs - would be made in the general config file, all the individual stuff in <host>.config files.
That would make sense, I agree.
Second, there are logs which change their names once the server or the application stops. So a "message.log" might become "message<date>.log" while there is a new message.log now. Still the last informations in "message<date>.log" are relevant, especially if the reason for the new log was a crash of the application. I don't know how to deal with this situation as there are multiple ways how logs might change their names but perhaps others have an idea of how to do that.
It's something I thought about too, although my concerns were with logfiles that get rotated at a specific time of day, not those that change when an application restarts. But like yourself I couldn't find a really good solution.
The file being renamed is not the only problem - it may get compressed, moved to a different directory, ... lots of ways for it to get lost.
So - if anyone has a bright idea, do speak up.
Regards, Henrik
On Wed, Feb 15, 2006 at 07:50:15AM, Henrik Storner wrote:
Hi Rolf,
On Wed, Feb 15, 2006 at 07:30:11AM +0100, Rolf Schrittenlocher wrote:
Hi Henrik,
that sounds wonderful, your ideas are pretty good.
Thanks.
I have one suggestion and one point to think about.
For monitoring application-logs the config file soon would become very complex and likely unreadable. There it would be nice if there is a possibilty for include-files. So general definitions -e.g. for OS-related logs - would be made in the general config file, all the individual stuff in <host>.config files.
That would make sense, I agree.
Second, there are logs which change their names once the server or the application stops. So a "message.log" might become "message<date>.log" while there is a new message.log now. Still the last informations in "message<date>.log" are relevant, especially if the reason for the new log was a crash of the application. I don't know how to deal with this situation as there are multiple ways how logs might change their names but perhaps others have an idea of how to do that.
It's something I thought about too, although my concerns were with logfiles that get rotated at a specific time of day, not those that change when an application restarts. But like yourself I couldn't find a really good solution.
We have similar problem where we actually need to monitor
messsges.date file. With bb-msgtab you cannot do that. I wonder if
hobbit could be able monitor a file with dynamic extension like that.
The file being renamed is not the only problem - it may get compressed, moved to a different directory, ... lots of ways for it to get lost.
So - if anyone has a bright idea, do speak up.
Currently we have created symbolic files like these
messages.020106 --> messages messages.020206 --> messages messages.022806 --> messages
So when the application tries to put files on the specific date file it does not matter since bb-msgtab always reading the `messages' file which never changed.
Thanks
Regards, Henrik
To unsubscribe from the hobbit list, send an e-mail to hobbit-unsubscribe at hswn.dk
-- Asif Iqbal PGP Key: 0xE62693C5 KeyServer: pgp.mit.edu "..there are two kinds of people: those who work and those who take the credit...try to be in the first group;...less competition there." - Indira Gandhi
Hi Henrik,
So on the central server there will be 2 configuration files. One for the log retrival defining interesting items ( I guess this is what today is yellow and red strings) and then a hobbit-client configuration file where you define the stings again ? I am not clear on why you would want to seperate files with some of the same information in. I get the idea of a log retrival configuration file with the log file names (both OS dependant and local files) but why not have it all in 1 file. Then you could have a LOGFILE-CLASS definition to put on a host group or single host in the hobbit-clients file.
Just a thought.
Will this new logfile retrival also be able to look for logfiles with variable file names, ie. logfile.txt-20060215 for today and then a new filename logfile.txt-20060216 tomorrow ? I know its stupid but that's how the vendor creates it.
Best regards, Thomas
Henrik Stoerner wrote:
A few days ago, I mentioned that I would like to do logfile monitoring for the next Hobbit release.
I've worked a bit on this and have a prototype solution for it, which you can test with the current snapshots. I'd like some comments on how it works to make sure I haven't overlooked something before committing myself.
There are several objectives:
- As far as is possible, logfile monitoring must be configured centrally, on the Hobbit server. Having to go to each server to (re)configure what logfiles to check and what to look for simply doesn't work.
- The amount of data sent from each client to Hobbit should be small, but it must catch the "important" stuff.
- You rarely know in advance what will be in the logs when you need them the most. So the monitor should give you as much of the log entries as possible, not just those lines that match some pre-defined strings or regex'es.
- Some systems log messages on multiple lines. The system must be able to show all parts of a log entry.
- Logfile entries must appear on the monitor for some time after they show up in the logs, but should also disappear after a while.
In other words: The ideal solution would let you have the entire logfile available on the Hobbit server - but that obviously won't work. So the client should - after weeding out the really irrelevant stuff - send us as much of each logfile as possible.
My proposed solution is this:
- On the Hobbit server, there's a log-monitoring configuration file for the Hobbit clients. This defines which logfiles are monitored for a single client installation, or you can define it for a group of clients. (The idea is to define at least one group for each operating system, since the standard system logs are OS dependant). This configuration lists the log filename, the maximum amount of data to send from this logfile, a regex "noise" filter (i.e. lines that are stripped from the logfile), and *optionally* a regex identifying really interesting stuff in the logfile that should always be reported.
- When a client connects to the Hobbit server and sends the normal client message, the Hobbit server will respond with the logfile configuration for this client. So the client has a copy of the central configuration file, but only the part that it needs for itself. The reason for sending this as a response to the client message is to avoid an extra round-trip from client to server; piggy-backing the config push on the client message means that it is almost without any performance cost on the server side.
- When the client runs, it uses the local copy of the configuration file to determine what logs to look at. For each logfile, it maintains a "where-was-I-the-last-time" status, so it only looks at the entries made to the logfile during the past 30 minutes. First, the client strips off any "noise" messages. Then, if all of the entries fit into the maximum size that can be reported, it sends all of the log to the Hobbit server. If there is more than will fit, it first checks to see of the regex defining the really interesting stuff is present in the log - if it is, then it drops anything before the interesting text. If there is still more than will fit, it keeps the interesting text + a few lines after that (to allow for multi-line log-entries which some OS'es have), and then sends that together with as much of last part the log as will fit inside the max. message size.
This part has been implemented in the Hobbit daemon (hobbitd), and in the clients via a new "logfetch" utility. This utility uses standard regular expressions - not the Perl-compatible ones, because that would require you to install the PCRE library on all of your clients. The standard regex routines are included in all (I think) system libraries used today.
The last part is what happens when the log data arrives on the Hobbit server. Currently, there's a simple processing of this data to just dump it into an always-green "msgs" column. What should happen once I get it coded is:
- Data from each logfile is matched against a set of strings (regex'es) defined in the hobbit-clients.cfg file. Each string determines the color (red, yellow, green) and sets the color of the msgs column accordingly.
When the color has been decided, all of the normal alerting happens automatically. I do plan on making a more fine-grained alert mechanism (for the msgs, procs and disk statuses) so you can direct alerts to different groups depending on exactly which log-message triggered the alert, but that will not be part of this release.
So - how does that sound ? Anything I've missed ?
Regards, Henrik
To unsubscribe from the hobbit list, send an e-mail to hobbit-unsubscribe at hswn.dk
On Wed, Feb 15, 2006 at 09:23:22AM +0100, Thomas wrote:
Hi Henrik,
So on the central server there will be 2 configuration files. One for the log retrival defining interesting items ( I guess this is what today is yellow and red strings) and then a hobbit-client configuration file where you define the stings again ? I am not clear on why you would want to seperate files with some of the same information in.
No, you wouldn't define the same strings in both files - that would be silly. You define the strings that can trigger a red or yellow status in the hobbit-clients config - that's all.
What you *can* put into the other config are some hints about how to minimize the amount of log data that Hobbit needs to process. So you can setup a regexp of stuff in the logfile that you *never* want to see, and a regexp of stuff that you *always* want to report - regardless of how much the log grows. The last one may be identical to some of what you have in the hobbit-clients config, but it could be different - or you could go without any definition in the second file at all.
An example: you're monitoring an application that logs some data to a logfile, and that you've set a limit on the amount of data you want of 200 bytes (that is probably too small for anything, but just for this example). You know that the application crashes occasionally, but it usually recovers automatically - so you've just configured the hobbit-clients.cfg file to send a warning for the application "Startup complete" message, and an alert for "Startup failed" or "Error".
The log now looks like this:
10:41:03 myapp: Startup complete 10:41:03 myapp: -- MARK -- 10:44:03 myapp: -- MARK -- 10:47:03 myapp: -- MARK -- 10:48:32 myapp: Error reading data, retrying 10:49:19 myapp: Error reading data, retrying 10:49:20 myapp: Error reading data, retrying 10:49:21 myapp: Error reading data, retrying 10:49:22 myapp: Error reading data, retrying 10:49:23 myapp: Error reading data, retrying 10:49:24 myapp: Error reading data, retrying 10:49:37 myapp: Unhandled exception at myapp_service.c:312: I/O error 10:49:37 myapp: Instruction dump follows: 0000000 030460 027060 070155 005147 060504 064556 060543 042012 0000020 071545 072153 070157 042012 060551 071147 066541 027061 0000040 064544 005141 067504 072543 062555 072156 005163 041105 0000060 045512 027123 061145 065552 074545 072163 071157 005145 0000100 052110 046115 052137 050157 043104 031455 031456 072056 0000120 071141 063456 005172 060515 066151 046412 064541 062154 0000140 071151 046412 075157 071141 057564 074563 063155 032137 0000160 057460 064550 064147 066456 031560 046412 071565 065551 0000200 047012 073545 005163 064520 071543 005062 051522 067171 0000220 041143 061541 072553 005160 051523 005114 042530 064160 0000240 066545 060412 061141 060412 071544 005154 063141 163154 0000260 027163 074164 005164 067141 064564 064566 072562 005163 0000300 071141 064143 073151 005145 072541 067564 060563 062566 0000320 060412 064170 066557 005145 075141 071165 072545 005163 0000340 033142 005064 033142 027064 005143 060542 065543 070165 0000360 061012 071541 061551 057563 071550 067167 066056 064544 0000400 005146 060542 064563 071543 071537 061165 062556 027164 0000420 062154 063151 061012 026542 067550 072163 005163 061142 10:49:37 myapp: Initiating recovery restart procedure 10:49:38 myapp: Startup complete
The "-- MARK --" lines are just noise - they just tell os the application is running. So you put those into the "ignore" regexp that is pushed to the client, and the client will filter out those lines before reporting data to Hobbit.
Hobbit would normally report the last 200 bytes of the logfile. But in this case, that would only include the dump data and the "Startup complete" - so you would miss both the fact that the dump was due to an unhandled exception, and the fact that it may have been triggered by a disk error which causes the application to retry I/O operations several times. And the "msgs" status would be yellow.
To catch that, you can tell the Hobbit client to always include certain log entries in the data it sends - e.g. here you could configure it to always include lines containing the word "Error".
Will this new logfile retrival also be able to look for logfiles with variable file names, ie. logfile.txt-20060215 for today and then a new filename logfile.txt-20060216 tomorrow ? I know its stupid but that's how the vendor creates it.
That's one variant I haven't seen yet. It would be tricky to implement; couldn't you just run something like this on the client daily via cron:
cd /var/log/myapp
CURRENTLOG=ls -t logfile.txt-* | head -1
ln -s $CURRENTLOG logfile.txt
and then Hobbit can look at logfile.txt ?
Regards, Henrik
Henrik Stoerner wrote:
the amount of log data that Hobbit needs to process. So you can setup a regexp of stuff in the logfile that you *never* want to see, and a regexp of stuff that you *always* want to report - regardless of how much the log grows.
Well, that covers my comment. I'd much rather give a list of "this stuff is always Good" than try to cover every instance of Bad, so that's awesome.
It would be ideal if the central config was somewhat bb-hosts-ish, and could accept (in addition to aforementioned includes) host-specific directives for what to log. I am assuming that how to react will already be host-specific like every other test, yes?
As far as the logs rotating out... couldn't hobbit look for an environment variable for the format of the rotated logs...? The filenames vary host to host, but on a given host, a quick look at /var/log tells you what to expect, right?
Lastly - being someone who couldn't program his way out of a paper stack, i will now cheekily suggest that on install, hobbit could look at /var/log and guesstimate the format, and ask for human confirmation (as it already does for the hobbit user and homedir).
Even if this automated | dream doesn't happen, could it still be set manually or via config?
-- Rob Munsch Solutions For Progress IT
On Wed, 15 Feb 2006 13:08:04 +0100 henrik at hswn.dk (Henrik Stoerner) wrote:
Will this new logfile retrival also be able to look for logfiles with variable file names, ie. logfile.txt-20060215 for today and then a new filename logfile.txt-20060216 tomorrow ? I know its stupid but that's how the vendor creates it.
That's one variant I haven't seen yet. It would be tricky to implement; couldn't you just run something like this on the client daily via cron:
cd /var/log/myapp CURRENTLOG=
ls -t logfile.txt-* | head -1ln -s $CURRENTLOG logfile.txtand then Hobbit can look at logfile.txt ?
Hi Henrik,
At first, a very big HURRRRRAYY to Henrik and its quick growing up**hobbit**. And thanks for your time and dedication.
A suggestion for this tricky situation (and maybe it could helpin two or tree other situation) would be support something like *date expansion* on the logfile variable. This could be done in two posible ways.
1) logfile = "/var/log/some_log.txt-%Y%m%d"
2) logfile - "/var/log/some_log.txt-$(+%Y%m%d --date '1 day')"
The first version could be build in, and use the standart time,strftime ... functions.
The seccond could be used by these people who have gnu date onits client systems, and hobbit client could expand the pattern using the date utility. I know, spawn/run external programs is not always so easy/secure, but may be an option.
A variation of the seccond option (much less secure, and muchmore risk) could be support some kind of shell expansion, and sou accept one line shell scripts to get the correct name of the logfile. For example in the above case.
logfile = "/var/log/$SHELL(ls -t /var/log/app/my_app_log.txt.* |head -1)"
Anyway, these are only some thinkings... may be they can behelpfull for something in the future.
Regards
Werner
participants (7)
-
henrik＠hswn.dk
-
iqbala-hobbit＠qwestip.net
-
lebarber＠gmail.com
-
rmunsch＠solutionsforprogress.com
-
Schrittenlocher＠rz.uni-frankfurt.de
-
tlp-hobbit＠holme-pedersen.dk
-
wmlist_ext＠terra.com.br