[hobbit] Server-side extension scripts: shell vs. C programming
I remember reading somewhere in the Hobbit documentation that when an extension script starts to do a lot of things, it should be coded in a compiled language such as C, instead of as a shell script. I have a custom script that takes a lot of data and converts it into NCV graphs, and I believe it is at the point where I should consider rewriting it in C.
Before I get to far into it, is the hobbitd_sample.c something that I should look at for this? I'm not sure if I'm reading the documentation on it correctly, if it is a good example for an external script. Has anyone else had experience in needing to convert a shell script to C/C++/etc for Hobbit? I'm just trying to get a rough idea of how much effort this will require.
Before embarking on this, you might want to write a wrapper that runs your real script with a "time" command and then log the results. The reason for rewriting in C is to make it more efficient, but if what you are doing is not very bad, there is no reason for the heartache unless you need to learn C, or are into self-abuse.
And you can probably modify the Hobbit RRD module (realizing that you will have to maintain it forever) to add your additional test parsing to what is already provided.
You also might consider converting your shell script into a Perl script -- Perl is a good intermediate between shell and C, and is pretty fast for longer running tasks. The only reason Perl doesn't win for all tasks is the need to compile the script before running it -- a slight performance hit for repetitive short tasks.
GLH
From: Gary Baluha [mailto:gumby3203 at gmail.com]
Sent: Thursday, June 28, 2007 1:13 PM
To: hobbit at hswn.dk
Subject: [hobbit] Server-side extension scripts: shell vs. Cprogramming
I remember reading somewhere in the Hobbit documentation thatwhen an extension script starts to do a lot of things, it should be coded in a compiled language such as C, instead of as a shell script. I have a custom script that takes a lot of data and converts it into NCV graphs, and I believe it is at the point where I should consider rewriting it in C.
Before I get to far into it, is the hobbitd_sample.c somethingthat I should look at for this? I'm not sure if I'm reading the documentation on it correctly, if it is a good example for an external script. Has anyone else had experience in needing to convert a shell script to C/C++/etc for Hobbit? I'm just trying to get a rough idea of how much effort this will require.
Are you familiar with C programming at all? Your question "...is the hobbitd_sample.c something that I should look at for this?" almost sounds like you may not be. You could be for a long haul if your intent is to learn a programming language while converting your existing Hobbit shell script into something more efficient. It's not going to be a simple cut-paste-compile scenerio, that's for sure. Pardon me if I've interpreted you posting incorrectly.
Personally, I use PERL for my Hobbit server scripts, except the ones that are the simplest-of-the-simple, and for those I do use just plain inefficent shell. I know C as well, but find PERL to be amost as efficient for what I do, and a heckuva lot quicker to program in.
From: Gary Baluha [mailto:gumby3203 at gmail.com] Sent: Thursday, June 28, 2007 12:13 PM To: hobbit at hswn.dk Subject: [hobbit] Server-side extension scripts: shell vs. C programming
I remember reading somewhere in the Hobbit documentation that when an extension script starts to do a lot of things, it should be coded in a compiled language such as C, instead of as a shell script. I have a custom script that takes a lot of data and converts it into NCV graphs, and I believe it is at the point where I should consider rewriting it in C.
Before I get to far into it, is the hobbitd_sample.c something that I should look at for this? I'm not sure if I'm reading the documentation on it correctly, if it is a good example for an external script. Has anyone else had experience in needing to convert a shell script to C/C++/etc for Hobbit? I'm just trying to get a rough idea of how much effort this will require.
Are you familiar with C programming at all? Your question "...is the hobbitd_sample.c something that I should look at for this?" almost sounds like you may not be. You could be for a long haul if your intent is to learn a programming language while converting your existing Hobbit shell script into something more efficient. It's not going to be a simple cut-paste-compile scenerio, that's for sure. Pardon me if I've interpreted you posting incorrectly.
I was asking that to determine if is an appropriate model specifically for an extension script. But yes, I am familiar with C programming.
Personally, I use PERL for my Hobbit server scripts, except the ones that
are the simplest-of-the-simple, and for those I do use just plain inefficent shell. I know C as well, but find PERL to be amost as efficient for what I do, and a heckuva lot quicker to program in.
I am considering C mostly because I'm starting to out-grow the capabilities of simple shell programming, rather than purely efficiency reasons. Perl would probably also work.
*From:* Gary Baluha [mailto:gumby3203 at gmail.com] *Sent:* Thursday, June 28, 2007 12:13 PM *To:* hobbit at hswn.dk *Subject:* [hobbit] Server-side extension scripts: shell vs. C programming
I remember reading somewhere in the Hobbit documentation that when an extension script starts to do a lot of things, it should be coded in a compiled language such as C, instead of as a shell script. I have a custom script that takes a lot of data and converts it into NCV graphs, and I believe it is at the point where I should consider rewriting it in C.
Before I get to far into it, is the hobbitd_sample.c something that I should look at for this? I'm not sure if I'm reading the documentation on it correctly, if it is a good example for an external script. Has anyone else had experience in needing to convert a shell script to C/C++/etc for Hobbit? I'm just trying to get a rough idea of how much effort this will require.
Too bad you didn't write it in perl, then you could have just used perlcompile to spit out C code for actual compiling.
About shell and other interpreted languages, what gets expensive is looping, since for each loop iteration you must interpret, compile, and run each lines in that loop. With shell script, add to that forking, which is expensive too.
On modern computers, the time saved might not be worth dealing with recoding your script in C. All the "magic" stuff like back ticks is not very fun to implement in C. Plus you have to deal with a strictly typed language, no automatic memory management, etc
you'd be better off recoding it in perl. then you'll have an easy way to get it in C and see if that really saves you time.
Daniel Bourque Systems/Network Administrator Weather Data Inc
Office (316) 266-8013 Office (316) 265-9127 ext. 3013 Mobile (316) 640-1024
Gary Baluha wrote:
I remember reading somewhere in the Hobbit documentation that when an extension script starts to do a lot of things, it should be coded in a compiled language such as C, instead of as a shell script. I have a custom script that takes a lot of data and converts it into NCV graphs, and I believe it is at the point where I should consider rewriting it in C.
Before I get to far into it, is the hobbitd_sample.c something that I should look at for this? I'm not sure if I'm reading the documentation on it correctly, if it is a good example for an external script. Has anyone else had experience in needing to convert a shell script to C/C++/etc for Hobbit? I'm just trying to get a rough idea of how much effort this will require.
On 6/28/07, Daniel Bourque <dbourque at weatherdata.com> wrote:
Too bad you didn't write it in perl, then you could have just used perlcompile to spit out C code for actual compiling.
About shell and other interpreted languages, what gets expensive is looping, since for each loop iteration you must interpret, compile, and run each lines in that loop. With shell script, add to that forking, which is expensive too.
On modern computers, the time saved might not be worth dealing with recoding your script in C. All the "magic" stuff like back ticks is not very fun to implement in C. Plus you have to deal with a strictly typed language, no automatic memory management, etc
you'd be better off recoding it in perl. then you'll have an easy way to get it in C and see if that really saves you time.
The main reason I need something other than shell is because I need to do some work with arrays that I can't do in shell. I may end up just using perl instead of C.
Daniel Bourque
Systems/Network Administrator Weather Data Inc
Office (316) 266-8013 Office (316) 265-9127 ext. 3013 Mobile (316) 640-1024
Gary Baluha wrote:
I remember reading somewhere in the Hobbit documentation that when an extension script starts to do a lot of things, it should be coded in a compiled language such as C, instead of as a shell script. I have a custom script that takes a lot of data and converts it into NCV graphs, and I believe it is at the point where I should consider rewriting it in C.
Before I get to far into it, is the hobbitd_sample.c something that I should look at for this? I'm not sure if I'm reading the documentation on it correctly, if it is a good example for an external script. Has anyone else had experience in needing to convert a shell script to C/C++/etc for Hobbit? I'm just trying to get a rough idea of how much effort this will require.
On Thu, Jun 28, 2007 at 02:12:33PM -0400, Gary Baluha wrote:
I remember reading somewhere in the Hobbit documentation that when an extension script starts to do a lot of things, it should be coded in a compiled language such as C, instead of as a shell script. I have a custom script that takes a lot of data and converts it into NCV graphs, and I believe it is at the point where I should consider rewriting it in C.
It really depends on how much work it does, and how often it runs: Do you have 10 hosts calling into Hobbit with this kind of data, or 1000 ? Your system load will be much less with a compiled C program than with a shellscript or Perl program, but it should be considered against the amount of work needed to re-implement your script.
Before I get to far into it, is the hobbitd_sample.c something that I should look at for this? I'm not sure if I'm reading the documentation on it correctly, if it is a good example for an external script.
hobbitd_sample shows how your program is fed data from Hobbit. I'm not sure quite how your program currently works - does it run via the "--extra-script" option to hobbitd_rrd ? In that case it might be easier for you to just convert your current script into a C program and still run it via the extra-script option.
If you do decide that the extra-script method is too heavy, then modifying the hobbitd_rrd module is the easiest way to go. You'll have to create a new routine to parse the data - start with a simple one like hobbitd/rrd/do_bbtest.c, which just finds a single value in the "bbtest" status message, and updates and RRD file with it. Then you must add two lines to hobbitd/do_rrd.c: An "#include rrd/myfile.c" to pick up the parser routine you just wrote, and a line near the bottom of the file where you invoke your parser whenever the correct "status" or "data" message arrives.
Regards, Henrik
On Thu, Jun 28, 2007 at 02:12:33PM -0400, Gary Baluha wrote:
I remember reading somewhere in the Hobbit documentation that when an extension script starts to do a lot of things, it should be coded in a compiled language such as C, instead of as a shell script. I have a custom script that takes a lot of data and converts it into NCV graphs, and I believe it is at the point where I should consider rewriting it in C.
It really depends on how much work it does, and how often it runs: Do you have 10 hosts calling into Hobbit with this kind of data, or 1000 ? Your system load will be much less with a compiled C program than with a shellscript or Perl program, but it should be considered against the amount of work needed to re-implement your script.
I actually wasn't expecting to get this much [helpful] feedback, so maybe I should explain a little more what my script is currently doing.
I have about 20 or 30 html monitors (which are going after various non-port-80 pages). These web pages are output from a weblogic program which reports various statistics about the managed servers' health. My script takes this data and creates several RRD trend graphs. There are about 10 graphs per web page, and each graph has anywhere from 3-15 data points. The main reason I'm rewriting it from shell script is that the current code is starting to get difficult to add new data points to, and since I have to clean it up anyway, I'd like to take advantage of some of the features perl or C have.
Before I get to far into it, is the hobbitd_sample.c something that I should
look at for this? I'm not sure if I'm reading the documentation on it correctly, if it is a good example for an external script.
hobbitd_sample shows how your program is fed data from Hobbit. I'm not sure quite how your program currently works - does it run via the "--extra-script" option to hobbitd_rrd ? In that case it might be easier for you to just convert your current script into a C program and still run it via the extra-script option.
I'm running the script from the hobbitlaunch.cfg config file.
If you do decide that the extra-script method is too heavy, then
modifying the hobbitd_rrd module is the easiest way to go. You'll have to create a new routine to parse the data - start with a simple one like hobbitd/rrd/do_bbtest.c, which just finds a single value in the "bbtest" status message, and updates and RRD file with it. Then you must add two lines to hobbitd/do_rrd.c: An "#include rrd/myfile.c" to pick up the parser routine you just wrote, and a line near the bottom of the file where you invoke your parser whenever the correct "status" or "data" message arrives.
participants (5)
-
dbourque＠weatherdata.com
-
greg.hubbard＠eds.com
-
gumby3203＠gmail.com
-
haertig＠avaya.com
-
henrik＠hswn.dk