My hobbit server (Fedora FC4) has 1.25 gig of memory installed. When the server is backed, up using Retrospect client, REAL memory usage spikes from 34% to 97% and stays at that level until a reboot. When I check the system performance, using the built in system monitor, user memory is at 18.9%. Dell Open Manage is using the most memory at 3% with a few additional processes between 1% and 2%. Everything else is well under 1%. What exactly is hobbit reporting on when it says that Physical/Real memory is at 97%, Actual memory is at 17%, and Swap is at 0%?
Thanks,
David Gilmore Consultant Stenhouse Consulting, LLC. 4 Traverse St Providence, RI 02906 401.453.6900 401.454.7581 (fax)
I'm not sure on the details, but I believe the Linux kernel loads up a large percentage of available memory with disk buffers as files are read. Those buffers don't really count against any process and will be evicted as soon as an actual process asks for real memory. The idea is that it speeds up disk access for those files that are in memory, and doesn't cost much (in machine time) to keep them around until the space is needed for something else.
Memory usage spikes during a backup because it's reading *all* the files on disk...
Ralph Mitchell
On 2/7/06, David Gilmore <david at stenhouseconsulting.com> wrote:
My hobbit server (Fedora FC4) has 1.25 gig of memory installed. When the server is backed, up using Retrospect client, REAL memory usage spikes from 34% to 97% and stays at that level until a reboot. When I check the system performance, using the built in system monitor, user memory is at 18.9%. Dell Open Manage is using the most memory at 3% with a few additional processes between 1% and 2%. Everything else is well under 1%. What exactly is hobbit reporting on when it says that Physical/Real memory is at 97%, Actual memory is at 17%, and Swap is at 0%?
Thanks,
David Gilmore Consultant Stenhouse Consulting, LLC. 4 Traverse St Providence, RI 02906 401.453.6900 401.454.7581 (fax)
On Tue, Feb 07, 2006 at 07:41:23PM -0500, David Gilmore wrote:
My hobbit server (Fedora FC4) has 1.25 gig of memory installed. When the server is backed, up using Retrospect client, REAL memory usage spikes from 34% to 97% and stays at that level until a reboot. When I check the system performance, using the built in system monitor, user memory is at 18.9%. Dell Open Manage is using the most memory at 3% with a few additional processes between 1% and 2%. Everything else is well under 1%. What exactly is hobbit reporting on when it says that Physical/Real memory is at 97%, Actual memory is at 17%, and Swap is at 0%?
Hobbit reports the output from the "free" command. It probably looks somewhat like this after you've run a backup:
total used free shared buffers cachedMem: 646432 642172 4260 0 167676 136068 -/+ buffers/cache: 338428 308004 Swap: 511992 4 511988
The "Mem" line here tells you that there is 640 MB RAM installed, and all except 4 MB is being "used". However, a lot of that is used for "buffers" and "cache", which is the Linux kernel's dynamically resized disk cache; if an application needs more RAM that is "free", the disk cache/buffers are discarded and the memory made available to the application.
So that's why the "-/+ buffers/cache" line is interesting: This shows the used/free memory count if the buffers/cached is counted as "free" memory. Hobbit report this as the "actual" memory count.
So a Linux system will practically always have a REAL memory usage close to 100% (Linus Torvalds once said that "unused RAM is *wasted* RAM, and there's no reason to spend lots of money on something that isn't used" - quoting from memory). The ACTUAL memory usage (should) be a lot less, and is what you'll want to keep an eye on.
Regards, Henrik
Hi,
On Tue, Feb 07, 2006 at 07:41:23PM -0500, David Gilmore wrote:
My hobbit server (Fedora FC4) has 1.25 gig of memory installed. When the server is backed, up using Retrospect client, REAL memory usage spikes from 34% to 97% and stays at that level until a reboot. When I check the system performance, using the built in system monitor, user memory is at 18.9%. Dell Open Manage is using the most memory at 3% with a few additional processes between 1% and 2%. Everything else is well under 1%. What exactly is hobbit reporting on when it says that Physical/Real memory is at 97%, Actual memory is at 17%, and Swap is at 0%?
Hobbit reports the output from the "free" command. It probably looks somewhat like this after you've run a backup:
total used free shared buffers cachedMem: 646432 642172 4260 0 167676 136068 -/+ buffers/cache: 338428 308004 Swap: 511992 4 511988
The "Mem" line here tells you that there is 640 MB RAM installed, and all except 4 MB is being "used". However, a lot of that is used for "buffers" and "cache", which is the Linux kernel's dynamically resized disk cache; if an application needs more RAM that is "free", the disk cache/buffers are discarded and the memory made available to the application.
So that's why the "-/+ buffers/cache" line is interesting: This shows the used/free memory count if the buffers/cached is counted as "free" memory. Hobbit report this as the "actual" memory count.
So a Linux system will practically always have a REAL memory usage close to 100% (Linus Torvalds once said that "unused RAM is *wasted* RAM, and there's no reason to spend lots of money on something that isn't used" - quoting from memory). The ACTUAL memory usage (should) be a lot less, and is what you'll want to keep an eye on.
We are working with Solaris 9 and for them unfortunately there is no ACTUAL MEMory presented. But PHYSical MEMory is always high on some machines runing a database in warm standby for the same reasons as Henrik explained. In other words: The way it is this test is useless for us. I'd prefer other columns of vmstat checked as io, etc. Did anyone work in this direction or where have I to dig in trying these changes?
regards Rolf
-- Mit freundlichen Gruessen Rolf Schrittenlocher
HRZ/BDV, Senckenberganlage 31, 60054 Frankfurt Tel: (49) 69 - 798 28908 Fax: (49) 69 798 2881 LBS: lbs-f at mlist.uni-frankfurt.de Persoenlich: schritte at rz.uni-frankfurt.de
On Wednesday 08 February 2006 08:34, Rolf Schrittenlocher wrote:
We are working with Solaris 9 and for them unfortunately there is no ACTUAL MEMory presented. But PHYSical MEMory is always high on some machines runing a database in warm standby for the same reasons as Henrik explained. In other words: The way it is this test is useless for us. I'd prefer other columns of vmstat checked as io, etc. Did anyone work in this direction or where have I to dig in trying these changes? Swap is usage is _not_ important. Most systems are a) using all available memory for disk cache, b) swapping not recently used programs to swap in favour for disk cache and c) doing prea-llocation of swap space (reserving disk cache even when it's not yet needed). So, memory usage is 100% and swap usage can be high even if you have enough memory and not using any swap.
What you have to monitor is a) swap-in / swap-out and b) free list (I know this exist for AIX, but I don't know for other os's). -> The swap-in / swap-out statistics determine how much of swap is activly used by the system as memory (in best scenario this is most of the time 0 or at least sometimes 0). -> The free list determines how much memory is free and can be used for programs who really need it very fast (in best scenario this never turns 0).
There are other numbers that are important like page scans and stuff, but page-in and page-out are the most important.
That's why I mailed some days ago to this list to add extra graphs to the memory overview page. The answer was that this will be not so easy to implement.
Stef
On Wed, Feb 08, 2006 at 08:34:21AM +0100, Rolf Schrittenlocher wrote:
We are working with Solaris 9 and for them unfortunately there is no ACTUAL MEMory presented. But PHYSical MEMory is always high on some machines runing a database in warm standby for the same reasons as Henrik explained. In other words: The way it is this test is useless for us. I'd prefer other columns of vmstat checked as io, etc. Did anyone work in this direction or where have I to dig in trying these changes?
Most of the data you want to look at is available in the vmstat data which Hobbit also collects. And it is highly OS dependant - there is no "one-size-fits-all" solution to memory monitoring or memory management.
My plan is to make it possible for any test (cpu, memory ...) to have its status modified by some of the data collected by vmstat (currently, the vmstat data is only used to feed the graph data, not to change a color or trigger alerts). But that will require some serious re-arranging of how a status is handled internally by Hobbit (it needs a feed-back mechanism from the hobbitd modules back into hobbitd, and a mechanism for modifying the color of an existing status without causing huge amounts of history updates due to the color changes. Plus some way of configuring the whole thing. But then it would also let you do neat stuff like using the standard Hobbit configuration tools to configure alerts based on custom-built tests and things like that).
Henrik
Ok I understand the concept. However, I don't want to continue to receive Alerts because Linux is doing exactly what it is designed to do. Does anyone have a script that can clear the buffers and stop hobbit from paging me? Can I modify the script to only alert when REAL memory is at 100% or higher? Or do I have to reboot my server ever morning to resolve this alert? I currently have the alerts disabled, but I am concerned that I could miss a critical error
Dave
-----Original Message----- From: hobbit-return-5584-david=stenhouseconsulting.com at hswn.dk [mailto:hobbit-return-5584-david=stenhouseconsulting.com at hswn. dk] On Behalf Of Henrik Stoerner Sent: Wednesday, February 08, 2006 1:45 AM To: hobbit at hswn.dk Subject: Re: [hobbit] Memory check
On Tue, Feb 07, 2006 at 07:41:23PM -0500, David Gilmore wrote:
My hobbit server (Fedora FC4) has 1.25 gig of memory installed. When the server is backed, up using Retrospect client, REAL memory usage spikes from 34% to 97% and stays at that level until a reboot. When I check the system performance, using the built in system monitor, user memory is at 18.9%. Dell Open Manage is using the most memory at 3% with a few additional processes between 1% and 2%. Everything else is well under 1%. What exactly is hobbit reporting on when it says that Physical/Real memory is at 97%, Actual memory is at 17%, and Swap is at 0%?
Hobbit reports the output from the "free" command. It probably looks somewhat like this after you've run a backup:
total used free shared buffers cachedMem: 646432 642172 4260 0 167676 136068 -/+ buffers/cache: 338428 308004 Swap: 511992 4 511988
The "Mem" line here tells you that there is 640 MB RAM installed, and all except 4 MB is being "used". However, a lot of that is used for "buffers" and "cache", which is the Linux kernel's dynamically resized disk cache; if an application needs more RAM that is "free", the disk cache/buffers are discarded and the memory made available to the application.
So that's why the "-/+ buffers/cache" line is interesting: This shows the used/free memory count if the buffers/cached is counted as "free" memory. Hobbit report this as the "actual" memory count.
So a Linux system will practically always have a REAL memory usage close to 100% (Linus Torvalds once said that "unused RAM is *wasted* RAM, and there's no reason to spend lots of money on something that isn't used" - quoting from memory). The ACTUAL memory usage (should) be a lot less, and is what you'll want to keep an eye on.
Regards, Henrik
To unsubscribe from the hobbit list, send an e-mail to hobbit-unsubscribe at hswn.dk
On Wednesday 08 February 2006 15:26, David Gilmore wrote:
Ok I understand the concept. However, I don't want to continue to receive Alerts because Linux is doing exactly what it is designed to do. Does anyone have a script that can clear the buffers and stop hobbit from paging me? Can I modify the script to only alert when REAL memory is at 100% or higher? Or do I have to reboot my server ever morning to resolve this alert? I currently have the alerts disabled, but I am concerned that I could miss a critical error I'm not sure if it can, but can't you alert on the swap usage?
Stef
On Wed, Feb 08, 2006 at 09:26:31AM -0500, David Gilmore wrote:
Ok I understand the concept. However, I don't want to continue to receive Alerts because Linux is doing exactly what it is designed to do. Does anyone have a script that can clear the buffers and stop hobbit from paging me? Can I modify the script to only alert when REAL memory is at 100% or higher? Or do I have to reboot my server ever morning to resolve this alert? I currently have the alerts disabled, but I am concerned that I could miss a critical error
Assuming that you're using the Hobbit client on the Linux box, you can just configure the client to only go red if the actual memory usage goes above a certain threshold. In your hobbit-clients.cfg, you would have
HOST=linux.foo.com MEMPHYS 100 101 MEMACT 80 95 MEMSWAP 40 70
Then it will stay green as long as the "actual" memory usage is below 80%, go yellow when it's between 80-95%, and go red when it is at 95% or higher.
Regards, Henrik
participants (5)
-
david＠stenhouseconsulting.com
-
henrik＠hswn.dk
-
ralphmitchell＠gmail.com
-
Schrittenlocher＠rz.uni-frankfurt.de
-
stef.coene＠docum.org