out-of-inodes hobbit conn error for all 300 systems
Hi All
I am running solaris 9. My hobbit is running off of /export/home/hobbit
dir. The df -i shows
(root)@hobbit:/export/home/hobbit/data$ /usr/local/bin/df -i Filesystem Inodes IUsed IFree IUse% Mounted on /dev/md/dsk/d0 494592 7665 486927 2% / /dev/md/dsk/d40 504000 63200 440800 13% /usr /dev/md/dsk/d20 494592 8052 486540 2% /var swap 98024 112 97912 1% /var/run swap 98024 112 97912 1% /tmp /dev/md/dsk/d50 504000 16322 487678 4% /opt /dev/md/dsk/d30 1005696 1005005 691 100% /export/home
But still have enough disk space per df -k
(root)@bigbrother:/export/home/hobbit/data$ /usr/local/bin/df -k Filesystem 1K-blocks Used Available Use% Mounted on /dev/md/dsk/d0 986131 114371 812593 13% / /dev/md/dsk/d40 2033103 1281660 690450 65% /usr /dev/md/dsk/d20 986131 153015 773949 17% /var swap 2705592 24 2705568 1% /var/run swap 2706504 936 2705568 1% /tmp /dev/md/dsk/d50 2033103 850773 1121337 44% /opt /dev/md/dsk/d30 8261393 4437625 3741155 55% /export/home
Wish I had the filesystem built optimized for lots of inodes with small files instead. But it is too late. As far as resources on this system I also have 674M actual and 2642M swap memory left.
Either I think of using these resources, like may be creating some RAM
disks (Henrik had a tip on that, cant find it) or get rid of hist' folder which has about 4000 files in it. I really like to keep the hist' files for report and going back to check for old events.
Anyone can think of any suggestion to overcome this predicament?
Thanks
-- Asif Iqbal PGP Key: 0xE62693C5 KeyServer: pgp.mit.edu "..there are two kinds of people: those who work and those who take the credit...try to be in the first group;...less competition there." - Indira Gandhi
On Fri, Dec 16, 2005 at 10:11:18PM -0500, Asif Iqbal wrote:
Hi All
I am running solaris 9. My hobbit is running off of /export/home/hobbit dir. The
df -ishows /dev/md/dsk/d30 1005696 1005005 691 100% /export/home [snip] Either I think of using these resources, like may be creating some RAM disks (Henrik had a tip on that, cant find it) or get rid ofhist' folder which has about 4000 files in it. I really like to keep thehist' files for report and going back to check for old events.
Keep the "hist" folder - there's only one file per status you have in Hobbit, and there won't be more.
What you should have a look at is the "histlogs" folder. This holds a copy of each status log whenever a status change occurred - and it gets *really* large over time. These historical status logs are only used when you look at the history of a status, and click on the red button off the "Jun 16 2003 14:21:58" event to see the detailed status log.
What I do on my production systems is to archive the history logs once a month, and then deleting those history logs that are older than 3 months. So you can get the full, detailed status of any event that happened within the past 3 months; if you need to go further back than that, it is still visible in the history view, but you'll need to extract the detailed status log from the tar file to see all of the details.
Feel free to use this snippet from my monthly cron job - it requires the GNU date utility. Right now, you will of course need to clean up your histlogs directory by hand, the snippet below only processes the histlog-files one month at a time.
cd /var/lib/hobbit/data
Archive histlogs generated in the past month
FNPATTERN=date --date="today -1 month" +"???_%b_*_%Y"
ARCFN=date --date="today -1 month" +"histlogs_%Y_%m.cpio.gz"
find histlogs -name "$FNPATTERN" | cpio -o --format=newc | gzip >archive/$ARCFN
Delete histlogs from 4 months ago (so we always have the last 3 months online)
FNPATTERN=date --date="today -4 months" +"???_%b_*_%Y"
find histlogs -name "$FNPATTERN" | xargs rm
Regards, Henrik
On Sat, Dec 17, 2005 at 08:43:31AM, Henrik Stoerner wrote:
On Fri, Dec 16, 2005 at 10:11:18PM -0500, Asif Iqbal wrote:
Hi All
I am running solaris 9. My hobbit is running off of /export/home/hobbit dir. The
df -ishows /dev/md/dsk/d30 1005696 1005005 691 100% /export/home [snip] Either I think of using these resources, like may be creating some RAM disks (Henrik had a tip on that, cant find it) or get rid ofhist' folder which has about 4000 files in it. I really like to keep thehist' files for report and going back to check for old events.Keep the "hist" folder - there's only one file per status you have in Hobbit, and there won't be more.
What you should have a look at is the "histlogs" folder. This holds a copy of each status log whenever a status change occurred - and it gets *really* large over time. These historical status logs are only used when you look at the history of a status, and click on the red button off the "Jun 16 2003 14:21:58" event to see the detailed status log.
What I do on my production systems is to archive the history logs once a month, and then deleting those history logs that are older than 3 months. So you can get the full, detailed status of any event that happened within the past 3 months; if you need to go further back than that, it is still visible in the history view, but you'll need to extract the detailed status log from the tar file to see all of the details.
Feel free to use this snippet from my monthly cron job - it requires the GNU date utility. Right now, you will of course need to clean up your histlogs directory by hand, the snippet below only processes the histlog-files one month at a time.
cd /var/lib/hobbit/data
Archive histlogs generated in the past month
FNPATTERN=
date --date="today -1 month" +"???_%b_*_%Y"ARCFN=date --date="today -1 month" +"histlogs_%Y_%m.cpio.gz"find histlogs -name "$FNPATTERN" | cpio -o --format=newc | gzip >archive/$ARCFNDelete histlogs from 4 months ago (so we always have the last 3 months online)
FNPATTERN=
date --date="today -4 months" +"???_%b_*_%Y"find histlogs -name "$FNPATTERN" | xargs rm
Excellent. I am slowly moving old logs. I wonder if that will skew my monthly and yearly reports.
snippet of the yearly report that gets called by bbcmd.
YEAR=$1
DATE=/usr/local/bin/date
MKDIR=/usr/local/bin/mkdir
YEAR=${YEAR:=$DATE +%Y}
START=$DATE +%s --date="01 Jan $YEAR 00:00:00"
END=$DATE +%s --date="31 Dec $YEAR 23:59:59"
$MKDIR -p /usr/local/apache/htdocs/reports/yearly/$YEAR
BBWEB=/reports/yearly/$YEAR
$BBHOME/bin/bbgen --reportopts=$START:$END:1:crit
--ignorecolumns=ssh,bbgen,bbtest,hobbitd,info,trends,msgs,scanmsg,http
--subpagecolumns=2
/usr/local/apache/htdocs/reports/yearly/$YEAR
I noticed I have about 70000 files under histlogs. Wow!!. I guess would be nice to use database instead to store histlogs and make report still work. I know there are suggestion out there regarding mysql use to store logs.
(hobbit)@bigbrother:~/data$ find histlogs -name "???_Nov_*_2005" | wc -l 67912
Do you have any plan to have a DB plugin for that in the future? Or may be a suggestion, on storing log in database, that you think best work with hobbit?
Thanks
Regards, Henrik
To unsubscribe from the hobbit list, send an e-mail to hobbit-unsubscribe at hswn.dk
-- Asif Iqbal PGP Key: 0xE62693C5 KeyServer: pgp.mit.edu "..there are two kinds of people: those who work and those who take the credit...try to be in the first group;...less competition there." - Indira Gandhi
On Sat, Dec 17, 2005 at 11:31:42AM -0500, Asif Iqbal wrote:
On Sat, Dec 17, 2005 at 08:43:31AM, Henrik Stoerner wrote:
On Fri, Dec 16, 2005 at 10:11:18PM -0500, Asif Iqbal wrote:
Hi All
I am running solaris 9. My hobbit is running off of /export/home/hobbit dir. The
df -ishows /dev/md/dsk/d30 1005696 1005005 691 100% /export/homeDelete histlogs from 4 months ago (so we always have the last 3 months online)
FNPATTERN=
date --date="today -4 months" +"???_%b_*_%Y"find histlogs -name "$FNPATTERN" | xargs rmExcellent. I am slowly moving old logs. I wonder if that will skew my monthly and yearly reports.
It should not affect those. The reports and their percentages are generated exclusively from the contents of the hist/* files, and just links to the histlogs files for the detailed status.
I noticed I have about 70000 files under histlogs. Wow!!. I guess would be nice to use database instead to store histlogs and make report still work. I know there are suggestion out there regarding mysql use to store logs.
I dont have any immediate plans for this, but it would be a nice little project if someone wants to dip into the Hobbit code (hint!). You could easily turn off saving the file-based status logs (there's a hobbitserver.cfg setting for that), and a module for storing the status logs in a DB would be fairly trivial to implement as a Hobbit worker module hanging off the "stachg" (status change) channel. Apart from that, there's a simple change to pick the status log from the DB when viewing it, and that's all.
Regards, Henrik
I dont have any immediate plans for this, but it would be a nice
little project if someone wants to dip into the Hobbit code (hint!). You
could easily turn off saving the file-based status logs (there's a hobbitserver.cfg setting for that), and a module for storing the
status logs in a DB would be fairly trivial to implement as a Hobbit worker module hanging off the "stachg" (status change) channel. Apart from that, there's a simple change to pick the status log from the DB when viewing it, and that's all.
Ahhh the BB/Hobbit, "How about a DB backend question ;)"
I've always been amazed that BB and Hobbit could use a filesystem as
a database and not kill a server much *sooner*. hobbits use of
memory for bbvar/logs is a huge win . . . .
Ideally the backend for most, if not all, of the state/history
information for events would be a database. Problem is, that
introduces a dependency of the product (hobbit) an another tool
(Database). This can very easily become a nightmare from a support/
integration point of view. Not to mention you raise the bar for
entry level admins ability to 'tinker' with the product.
Unless the entire hobbit tool were re-architected to have DB back-
end, I don't think it would be worth the overall effort for keeping
the histlogs area small.
That's *not* to say it isn't a good idea or it *wouldn't* be a great
coding project for someone.
But I think it would be a better use of resources and benefit more of
the user community if hobbit could internally, archive and retrieve
this data. The history data is *extremely* useful and having it
around is a wonderful asset for problem history/determination.
Henrik, I was thinking along the lines of every month or so archive
the histlogs into a single large file, or one file per host, and then
introduce logic into the show history that could determine if it
needed to expand the archive to get it.
scott
On Sun, Dec 18, 2005 at 10:28:17AM -0500, Scott Walters wrote:
Ahhh the BB/Hobbit, "How about a DB backend question ;)"
I've always been amazed that BB and Hobbit could use a filesystem as
a database and not kill a server much *sooner*. hobbits use of
memory for bbvar/logs is a huge win . . . .Ideally the backend for most, if not all, of the state/history
information for events would be a database. Problem is, that
introduces a dependency of the product (hobbit) an another tool
(Database). This can very easily become a nightmare from a support/ integration point of view. Not to mention you raise the bar for
entry level admins ability to 'tinker' with the product.Unless the entire hobbit tool were re-architected to have DB back- end, I don't think it would be worth the overall effort for keeping
the histlogs area small.
That in itself would not gain you much, I agree. There has to be some other benefit besides the space-win to justify moving to a DB backend. E.g. if there were some fancy backend pulling data from Hobbit which could benefit from having the Hobbit data in a DB.
That's *not* to say it isn't a good idea or it *wouldn't* be a great
coding project for someone.
The nice thing about it is that this can be implemented as a separate module, without affecting the core Hobbit programs at all. So if someone desperately needs a DB backend for the Hobbit history-data, I'm sure that they will figure out how to write one and offer me the sources for inclusion :-)
But I think it would be a better use of resources and benefit more of
the user community if hobbit could internally, archive and retrieve
this data. The history data is *extremely* useful and having it
around is a wonderful asset for problem history/determination.Henrik, I was thinking along the lines of every month or so archive
the histlogs into a single large file, or one file per host, and then
introduce logic into the show history that could determine if it
needed to expand the archive to get it.
That idea had crossed my mind. The archiving bit is something I already do with a fairly simple shell script once a month; this can easily be turned into a general solution. Adding a bit of code to try and extract the history logfile from an archive if the file is not immediately available is also rather trivial. So maybe I'll try and do it.
Henrik
On Mon, 2005-12-19 at 00:08 +0100, Henrik Stoerner wrote:
On Sun, Dec 18, 2005 at 10:28:17AM -0500, Scott Walters wrote:
Ahhh the BB/Hobbit, "How about a DB backend question ;)"
I've always been amazed that BB and Hobbit could use a filesystem as
a database and not kill a server much *sooner*. hobbits use of
memory for bbvar/logs is a huge win . . . .Ideally the backend for most, if not all, of the state/history
information for events would be a database. Problem is, that
introduces a dependency of the product (hobbit) an another tool
(Database). This can very easily become a nightmare from a support/ integration point of view. Not to mention you raise the bar for
entry level admins ability to 'tinker' with the product.Unless the entire hobbit tool were re-architected to have DB back- end, I don't think it would be worth the overall effort for keeping
the histlogs area small.That in itself would not gain you much, I agree. There has to be some other benefit besides the space-win to justify moving to a DB backend. E.g. if there were some fancy backend pulling data from Hobbit which could benefit from having the Hobbit data in a DB.
Well, the other option is to simply use a filesystem that doesn't incur a penalty for having large numbers of files in a directory, or on the filesystem. My favourite is of course ReiserFS, where multiple files can share the same block (no space wastage from allocating whole blocks to a single file), as well as no limited inodes for the filesystem etc...
On the other hand, for the interested individual, I've been told that there is some way to basically embed mysql into a product, thereby allowing you to 'contain' the entire system, rather than relying on the administrator to configure mysql/etc properly. In fact, this might be helpful initially, and for the larger sites, then they would simply reconfig hobbit to point to a external mysql server that they can customise themselves.
Regards, Adam
On Sat, Dec 17, 2005 at 08:43:31AM, Henrik Stoerner wrote:
On Fri, Dec 16, 2005 at 10:11:18PM -0500, Asif Iqbal wrote:
Hi All
I am running solaris 9. My hobbit is running off of /export/home/hobbit dir. The
df -ishows /dev/md/dsk/d30 1005696 1005005 691 100% /export/home [snip] Either I think of using these resources, like may be creating some RAM disks (Henrik had a tip on that, cant find it) or get rid ofhist' folder which has about 4000 files in it. I really like to keep thehist' files for report and going back to check for old events.Keep the "hist" folder - there's only one file per status you have in Hobbit, and there won't be more.
[...]
What I do on my production systems is to archive the history logs once a month, and then deleting those history logs that are older than 3 months. So you can get the full, detailed status of any event that happened within the past 3 months; if you need to go further back than that, it is still visible in the history view, but you'll need to extract the detailed status log from the tar file to see all of the details.
Feel free to use this snippet from my monthly cron job - it requires the GNU date utility. Right now, you will of course need to clean up your histlogs directory by hand, the snippet below only processes the histlog-files one month at a time.
cd /var/lib/hobbit/data
Archive histlogs generated in the past month
FNPATTERN=
date --date="today -1 month" +"???_%b_*_%Y"ARCFN=date --date="today -1 month" +"histlogs_%Y_%m.cpio.gz"find histlogs -name "$FNPATTERN" | cpio -o --format=newc | gzip >archive/$ARCFNDelete histlogs from 4 months ago (so we always have the last 3 months online)
FNPATTERN=
date --date="today -4 months" +"???_%b_*_%Y"find histlogs -name "$FNPATTERN" | xargs rm
Excellent. I am slowly moving old logs. I wonder if that will skew my monthly and yearly reports.
snippet of the yearly report that gets called by bbcmd.
YEAR=$1
DATE=/usr/local/bin/date
MKDIR=/usr/local/bin/mkdir
YEAR=${YEAR:=$DATE +%Y}
START=$DATE +%s --date="01 Jan $YEAR 00:00:00"
END=$DATE +%s --date="31 Dec $YEAR 23:59:59"
$MKDIR -p /usr/local/apache/htdocs/reports/yearly/$YEAR
BBWEB=/reports/yearly/$YEAR
$BBHOME/bin/bbgen --reportopts=$START:$END:1:crit
--ignorecolumns=ssh,bbgen,bbtest,hobbitd,info,trends,msgs,scanmsg,http
--subpagecolumns=2
/usr/local/apache/htdocs/reports/yearly/$YEAR
I noticed I have about 70000 files under histlogs. Wow!!. I guess would be nice to use database instead to store histlogs and make report still work. I know there are suggestion out there regarding mysql use to store logs.
(hobbit)@bigbrother:~/data$ find histlogs -name "???_Nov_*_2005" | wc -l 67912
Do you have any plan to have a DB plugin for that in the future? Or may be a suggestion, on storing log in database, that you think best work with hobbit?
-- Asif Iqbal PGP Key: 0xE62693C5 KeyServer: pgp.mit.edu "..there are two kinds of people: those who work and those who take the credit...try to be in the first group;...less competition there." - Indira Gandhi
On Sat, Dec 17, 2005 at 08:43:31AM, Henrik Stoerner wrote:
On Fri, Dec 16, 2005 at 10:11:18PM -0500, Asif Iqbal wrote:
Hi All
I am running solaris 9. My hobbit is running off of /export/home/hobbit dir. The
df -ishows /dev/md/dsk/d30 1005696 1005005 691 100% /export/home [snip] Either I think of using these resources, like may be creating some RAM disks (Henrik had a tip on that, cant find it) or get rid ofhist' folder which has about 4000 files in it. I really like to keep thehist' files for report and going back to check for old events.Keep the "hist" folder - there's only one file per status you have in Hobbit, and there won't be more.
What you should have a look at is the "histlogs" folder. This holds a copy of each status log whenever a status change occurred - and it gets *really* large over time. These historical status logs are only used when you look at the history of a status, and click on the red button off the "Jun 16 2003 14:21:58" event to see the detailed status log.
What I do on my production systems is to archive the history logs once a month, and then deleting those history logs that are older than 3 months. So you can get the full, detailed status of any event that happened within the past 3 months; if you need to go further back than that, it is still visible in the history view, but you'll need to extract the detailed status log from the tar file to see all of the details.
Feel free to use this snippet from my monthly cron job - it requires the GNU date utility. Right now, you will of course need to clean up your histlogs directory by hand, the snippet below only processes the histlog-files one month at a time.
cd /var/lib/hobbit/data
Archive histlogs generated in the past month
FNPATTERN=
date --date="today -1 month" +"???_%b_*_%Y"ARCFN=date --date="today -1 month" +"histlogs_%Y_%m.cpio.gz"find histlogs -name "$FNPATTERN" | cpio -o --format=newc | gzip >archive/$ARCFN
My manager wants to see some old data for one of the host. What is the best way to restore it? I tried to restore one of the archive file and it does not expand the same way as active histlogs dir and hence finding data about a host seems difficult.
Any suggestion on this?
Thanks
Delete histlogs from 4 months ago (so we always have the last 3 months online)
FNPATTERN=
date --date="today -4 months" +"???_%b_*_%Y"find histlogs -name "$FNPATTERN" | xargs rmRegards, Henrik
To unsubscribe from the hobbit list, send an e-mail to hobbit-unsubscribe at hswn.dk
-- Asif Iqbal PGP Key: 0xE62693C5 KeyServer: pgp.mit.edu "..there are two kinds of people: those who work and those who take the credit...try to be in the first group;...less competition there." - Indira Gandhi
participants (5)
-
henrik＠hswn.dk
-
iqbala-hobbit＠qwestip.net
-
iqbala＠qwestip.net
-
mailinglists＠websitemanagers.com.au
-
scott＠PacketPushers.com