Thanks Gatis

That was not exactly that need, but I have found a lot of useful info.

Thanks Ricardas

"Gatis A." <gatis.anee at gmail.com> wrote on 22/05/2009 11:42:16:

If login is enough, then you can do something like this in bb-hosts:

0.0.0.0 testhost # https://testuser:testpass at testhost.lt:8443

see http://www.hswn.dk/hobbit/help/manpages/man5/bb-hosts.5.html and search for "HTTP TESTS"

On Fri, May 22, 2009 at 10:57 AM, Ricardas Vaitkus <Ricardas.Vaitkus at seb.lt> wrote:

...

Hi all

I want to make some http tests, but not only access web applications. I want to do some login and/or simulate some actions in applications. Has any one done some thing like that or know HOW TO :) using some

external

...

scripts and/or applications.

Thanks in advance Ricardas

To unsubscribe from the hobbit list, send an e-mail to hobbit-unsubscribe at hswn.dk

To unsubscribe from the hobbit list, send an e-mail to hobbit-unsubscribe at hswn.dk

Hi Ralph

I tried to find your message in mail archive, but I couldn't. Can you tell me the title of your message, because by date I was not able to find it.

Thanks Ricardas

Ralph Mitchell <ralphmitchell at gmail.com> wrote on 22/05/2009 19:48:51:

I did a whole lot of that for my last employer.  As far as I know they still use my scripts to run about 2500 checks on roughly 300 servers.

See my message on Jun 7, 2007 at 12:08 AM in the mail archive for details and a sample script.

Ralph Mitchell

On Mon, May 25, 2009 at 10:23 AM, Ralph Mitchell <ralphmitchell at gmail.com>wrote:

It was titled "Hobbit content check script". Here's the body of the message, and the attachments:

Sorry I didn't get back to you earlier - I've been fighting with perl & SOAP, trying to get some status messages out of a bunch of load balancers. I think I finally got it now....

Attached (hopefully :) are a couple of files I tried to post to the list some time last year. There's a shell script that does a login to a Siteminder site, and a perl script evolved from something Daniel Stenberg wrote - he's the original author of curl. Daniel's very active on the curl mailing list, by the way, along with a bunch of other helpful folks, if it starts going weird on you.

The Siteminder login probably won't be much use to you as is, but it demonstrates a few things I've worked out over the years. Like, the standard options string to pass to curl, collecting messages along the way, and loading up the message string with a Manual Check url at the end for ops to try the check for themselves before waking people up...

You can put almost *anything* into the message string that gets sent to Hobbit. Right now I have scripts that check car rentals at travel sites for a particular company and emit a table of the current prices. Another script hits a cruise monitor and puts out a list using the &red and &green dots to mark cruise lines as reachable or not. You can push out font color and size changes, urls to saved pages, etc.

As far as cookies go, I generally just have curl save them whenever it finishes:

curl -b cookies -c cookies .....

If you do that you can go look at the cookies and see what came in. Other things become possible too, such as editing cookies in between curl calls. If you do keep cookies, though, the *first* thing you should do in the script is delete the cookie file. That's like clearing the browser cookie cache, and can avoid a lot of mysterious errors.

Finally, if you think you might spend some time on web site scripts, there's a Firefox extension that is absolutely invaluable - LiveHTTPheaders. It shows you *exactly* what the browser sends to the server, as well as what it gets back - not the page content, just the headers, but that's all you need to recreate the sequence. It even works twith secure sites. If you see something going out that you can't account for, the usual suspect is javascript. I've seen javascript that assembles a url on the fly from form elements, fiddles with some of the form elements, then posts the form. That kind of thing can be difficult to nail down, and almost impossible to replicate...

Ralph Mitchell

The attachments may not have gone through, so here they are again:

========== bb-siteminder.sh ========== #!/bin/sh

In case of disaster, uncomment the next line and watch the output

set -x

This script runs out of cron. Tell it where to find Big Brother.

BBHOME=/home/bb/bb18b3 export BBHOME

TESTHOST=system.domain.com URL="http://$TESTHOST/" ORIGURL="$URL" USERNAME=bbusername PASSWORD=bbsecretword

Where to stash the downloaded html files. These get patched into the BB

report so that the monitoring guys can click through and see what we got.

BBDIR="$BBHOME/www/siteminder"

Gonna need a cookie jar. Start off empty, so each checkout starts clean.

COOKIES="$BBDIR/$TESTHOST.cookies" rm -f $COOKIES

Some standard curl flags. See the curl man page for details

STDOPTS="-w %{url_effective} -L --connect-timeout 60 --max-time 120 -s -S -b $COOKIES -c $COOKIES -A Mozilla/4.79"

This is just a tag for identifying environment dumps.

BBPROG=bb-siteminder.sh; export BBPROG

The next block came from the BB ext example script. If BBHOME is not set

there's no point continuing. If BBTMP is not set, load up the standard

environment from bbdef.sh.

if test "$BBHOME" = "" then echo "BBHOME is not set... exiting" exit 1 fi

if test ! "$BBTMP" # GET DEFINITIONS IF NEEDED then # echo "*** LOADING BBDEF ***" . $BBHOME/etc/bbdef.sh # INCLUDE STANDARD DEFINITIONS fi

This is used to label the result column

TEST="login"

Make sure bail out code is set.

bailout=0

========== step 1 ==========

Grab the first page

STEP=step1

Name the file for the host, test and step, to make it unique

FILENAME=$BBDIR/$TESTHOST.$TEST.$STEP.html

remove any previous copy to avoid misunderstandings

rm -f $FILENAME

Grab the first page. The effective url is recorded for patching

into the action url is necessary

effect=$CURL $STDOPTS \ -o $FILENAME \ $URL

Save the return code, it might be important

ret=$?

code 52 means "the server sent nothing"

if [ "$ret" -eq "52" ]; then

empty reply, try it again

MESSAGE="$STEP: failed with curl error code: 52 - Empty reply from server. Url: <a href=$URL>$URL</a>" MESSAGE="$MESSAGE $STEP: retrying..." sleep 2 effect=$CURL $STDOPTS \ -o /dev/null \ $URL ret=$? fi

if [ -s $FILENAME ]; then

file exists and has size greater than zero. look for something to

validate TESTSTR="Something from the Login Page" check=$GREP -ic "$TESTSTR" $FILENAME if [ "$check" -gt "0" ]; then # Found the test string, we're good to continue MESSAGE="$MESSAGE $STEP: Got Login page - found <b>$TESTSTR</b>" else # Didn't find the test string, bail out of subsequent steps MESSAGE="$MESSAGE $STEP: failed to find <b>$TESTSTR</b>. curl error code: $ret. Url: <a href=$URL>$URL</a>" bailout=1 fi else

Got nothing useful from the server, give up

MESSAGE-"$MESSAGE $STEP: failed with empty file. curl error code: $ret. Url: <a href=$URL>$URL</a>" bailout=1 fi

If previous step completed OK, perform next step

if [ "$bailout" -eq "0" ]; then

formextract.pl strips out the form elements and formats them like this:

-d username=

-d password=

http://server.domain.com/path/to/form/processor

On some pages, the action url is not fully qualified.

FORM=cat $FILENAME | $HOME/bin/formextract.pl

Pull out the form variable lines and patch in the userid & password

FORMVARS=echo "$FORM" | $GREP -- '-d' | $SED \ -e "s/txtUsername=/txtUsername=$USERNAME/" \ -e "s/Password=/Password=$PASSWORD/"

extract the action url. Adjustments can be to the url in the pipeline

ACTION=echo "$FORM" | $TAIL -1

Break up the effective url based on the /'s

OFS="$IFS" IFS="/" set $effect IFS="$OFS"

Assemble the proper action url. It should be one of these forms:

Complete URL: http://system.domain.com/path/to/form/processor

URL="$ACTION"

Absolute path, no hostname: /path/to/form/processor

URL="$1//$3$ACTION"

Relative path, no hostname: path/to/form/processor

URL="$1//$3/$ACTION"

pick up $1 (protocol) & $3 (hostname), because the // in the effective

url makes $2=null. URL="$1//$3$ACTION"

STEP=step2 FILENAME=$BBDIR/$TESTHOST.$TEST.$STEP.html rm -f $FILENAME

Post the form back to the action url. The returned file should

contain the page a browser would show after logging in.

effect=$CURL $STDOPTS \ -w "%{url_effective}" \ -o $FILENAME \ $FORMVARS \ $URL ret=$?

As before, make sure the file is not empty, and that it contains

something we recognise from a successful login. You can also look

for signs of login failure, such as a repeat of the login page, or

some message such as "login failed".

if [ -s $FILENAME ]; then if [ "$ret" -eq "0" ]; then TESTSTR="Big Brother Test Page" check=$GREP -ic "$TESTSTR" $FILENAME if [ "$check" -gt "0" ]; then MESSAGE="$MESSAGE $STEP: Found <b>$TESTSTR</b>. Looking good" fi else MESSAGE="$MESSAGE $STEP: failed to find <b>$TESTSTR</b>. curl error code: $ret. Url: <a href=$URL>$URL</a>" bailout=1 fi else MESSAGE-"$MESSAGE $STEP: failed with empty file. curl error code: $ret. Url: <a href=$URL>$URL</a>" bailout=1 fi fi

If bailout is non-zero, something unexpected happened.

if [ "$bailout" -eq "1" ]; then COLOR="red" elif [ "$bailout" -eq "2" ]; then

This is useful for non-critical-but-interesting conditions

that may be worth looking for. There are none in this example.

COLOR="yellow" else

WooHoo!!

COLOR="green" fi

MESSAGE="$MESSAGE <P> See the saved files in <a href=$BBWEBHOST/bb/siteminder/>the checkout directory</a> <P> <a href=$ORIGURL>Manual Check</a>"

The script runs from cron, with output directed to a file in /tmp.

This echo is not strictly necessary, but it can be useful to validate

that it ran correctly

echo "$TESTHOST: $TEST: $COLOR"

convert the dots to commas in the hostname

MACHINE=echo $TESTHOST | $SED -e 's/\./,/g'

Assemble the status message as per normal

LINE="status $MACHINE.$TEST $COLOR date $MESSAGE"

fire it off to BB/Hobbit

$BB $BBDISP "$LINE" # SEND IT TO BBDISPLAY

========== formextract.pl ==========

#!/usr/bin/env perl

$Id: formfind,v 1.5 2003/04/28 13:48:16 bagder Exp $

formfind.pl

This script gets a HTML page on stdin and presents form information on

stdout.

Author: Daniel Stenberg <daniel at haxx.se>

Version: 0.2 Nov 18, 2002

HISTORY

0.1 - Nov 12 1998 - Created now!

0.2 - Nov 18 2002 - Enhanced. Removed URL support, use only stdin.

sometime - echo the first form in a format suitable for dropping

into a curl command line. - Ralph Mitchell

$in="";

if($ARGV[0] eq "-h") { print "Usage: $0 < HTML\n"; exit; }

sub namevalue { my ($tag)=@_; my $name=$tag; if($name =~ /name *=/i) { if($name =~ /name *= *([^\"\']([^ \">]*))/i) { $name = $1; } elsif($name =~ /name *= *(\"|\')([^\"\']*)(\"|\')/i) { $name=$2; } else { # there is a tag but we didn't find the contents $name="[weird]"; }

}
else {
    # no name given
    $name="";
}
# get value tag
my $value= $tag;
if($value =~ /[^\.a-zA-Z0-9]value *=/i) {
    if($value =~ /[^\.a-zA-Z0-9]value *= *([^\"\']([^ \">]*))/i) {
        $value = $1;
    }
    elsif($value =~ /[^\.a-zA-Z0-9]value *= *(\"|\')([^\"\']*)(\"|\')/i)

{ $value=$2; } else { # there is a tag but we didn't find the contents $value="[weird]"; } } else { $value=""; } return ($name, $value); }

while(<STDIN>) { $line = $_; push @indoc, $line; $line=~ s/\n//g; $line=~ s/\r//g; $in=$in.$line; }

while($in =~ /[^<]*(<[^>]+>)/g ) { # we have a tag in $1 $tag = $1; if($tag =~ /^<!--/) { # this is a comment tag, ignore it } else { if(!$form && ($tag =~ /^< *form/i )) { $method= $tag; if($method =~ /method *=/i) { $method=~ s/.*method *= *(\"|)([^ \">]*).*/$2/gi; } else { $method="get"; # default method } $action= $tag; $action=~ s/.*action *= *(\'|\"|)([^ \"\'>]*).*/$2/gi;

        $method=uc($method);

        $enctype=$tag;
        if ($enctype =~ /enctype *=/) {
            $enctype=~ s/.*enctype *= *(\'|\"|)([^ \"\'>]*).*/$2/gi;

            if($enctype eq "multipart/form-data") {
                $enctype="multipart form upload [use -F]"
}
            $enctype = "\n--- type: $enctype";
        }
        else {
            $enctype="";
        }

        # print "URL=\"$action\"$enctype\n";
        $form=1;
    }
    elsif($form &&
          ($tag =~ /< *\/form/i )) {

        # print "--- end of FORM\n";
        print "$action\n";
        $form=0;
        if( 0 ) {
            print "*** Fill in all or any of these: (default assigns may

be shown)\n"; for(@vars) { $var = $_; $def = $value{$var}; print "$var=$def\n"; } print "*** Pick one of these:\n"; for(@alts) { print "$_\n"; } } undef @vars; undef @alts; } elsif($form && ($tag =~ /^< *(input|select)/i)) { $mtag = $1;

        ($name, $value)=namevalue($tag);
        if($mtag =~ /select/i) {
            print "Select: NAME=\"$name\"\n";
            push @vars, "$name";
            $select = 1;
        }
        else {
            $type=$tag;
            if($type =~ /type *=/i) {
                $type =~ s/.*type *= *(\'|\"|)([^ \"\'>]*).*/$2/gi;
            }
            else {
                $type="text"; # default type
            }
            $type=uc($type);
            if(lc($type) eq "reset") {
                # reset types are for UI only, ignore.
            }
            elsif($name eq "") {
                # let's read the value parameter

                print "Button: \"$value\" ($type)\n";
                push @alts, "$value";
            }
            else {
                $xname = s/\ /\+/g;
                print "  -d $name=";
                if($value ne "") {
                    # print "\"$value\"";
                    print "$value";
                }
                print " \n";
                push @vars, "$name";
                # store default value:
                $value{$name}=$value;
            }
        }
    }
    elsif($form &&
          ($tag =~ /^< *textarea/i)) {
        my ($name, $value)=namevalue($tag);

        print "Textarea: NAME=\"$name\"\n";
    }
    elsif($select) {
        if($tag =~ /^< *\/ *select/i) {
            print "[end of select]\n";
            $select = 0;
        }
        elsif($tag =~ /[^\/] *option/i ) {
            my ($name, $value)=namevalue($tag);
            my $s;
            if($tag =~ /selected/i) {
                $s= " (SELECTED)";
            }
            print "  Option VALUE=\"$value\"$s\n";
        }
    }
}

}

====================