Would you share the bourne script. I need something quickly for only 1 site at the moment, so anything will do.

Regards, Thomas

         "Ralph Mitchell"                                              
         <ralphmitchell at gm                                             
         ail.com>                                                   To 
                                   hobbit at hswn.dk                      
         29-04-2008 22:04                                           cc 
                                                                       
                                                               Subject 
         Please respond to         Re: [hobbit] URLplus                
          hobbit at hswn.dk                                               
                                                                       
                                                                       
                                                                       
                                                                       
                                                                       

On Tue, Apr 29, 2008 at 2:28 PM, Thomas Laurids Pedersen <tlp at mach.com> wrote:

  Can this new extension do the post like the builtin http test can do
  ?

  Regards, Thomas

I don't think so, as it currently stands. This line:

$ret=qx(curl -m $tout -s -k -S $pstr "$url" 2>&1);

in showUrl is the bit that actually fetches the web page. To do a POST using curl, you would need to change that to:

$ret=qx(curl -m $tout -s -k -S -d $postArgs $pstr "$url" 2>&1);

with matching additions elsewhere to fill the $postArgs variable with whatever you want to post. If you're wanting to implement something that performs a login, you'd probably want to track cookies as well.

I'm not much of a perl programmer, so I'm not confident about making such changes. On the other hand, I've done quite a lot of this in Bourne shell...

Ralph Mitchell

Hi Gary,

is there any new URLPlus version? I'm currently using v1.21 ($Id: url-plus.pl,v 1.21 2008/04/30 13:56:07).

Thanks.

Regards.

Massimo Morsiani Information Technology Dept.

Gilbarco S.r.l. via de' Cattani, 220/G 50145 Firenze, Italy tel: +39-055-30941 fax: +39-055-318603 email: massimo.morsiani at gilbarco.com web: http://www.gilbarco.it <http://www.gilbarco.it/>

From: Gary Baluha [mailto:gumby3203 at gmail.com] Sent: mercoledì 30 aprile 2008 16.13 To: hobbit at hswn.dk Subject: Re: [hobbit] URLplus

I have an updated version of URLPLus which supports single-form submissions. I haven't gotten around to being able to submit directly to The Shire, so I'm attaching it here.

I've also updated this to do multi-page checking (i.e. full "end-to-end" transaction monitoring), but consider that feature mostly experimental at this time.

On Wed, Apr 30, 2008 at 4:16 AM, Thomas Laurids Pedersen <tlp at mach.com> wrote:

Would you share the bourne script. I need something quickly for only 1 site at the moment, so anything will do.

Regards, Thomas

 "Ralph Mitchell" <ralphmitchell at gmail.com>




			"Ralph Mitchell" <ralphmitchell at gmail.com> 

			29-04-2008 22:04 


Please respond to

hobbit at hswn.dk

To

hobbit at hswn.dk

cc

Subject

Re: [hobbit] URLplus

On Tue, Apr 29, 2008 at 2:28 PM, Thomas Laurids Pedersen <tlp at mach.com <mailto:tlp at mach.com> > wrote: 

		Can this new extension do the post like the builtin http test can do ?
		
		Regards, Thomas


I don't think so, as it currently stands. This line:

$ret=qx(curl -m $tout -s -k -S $pstr "$url" 2>&1);

in showUrl is the bit that actually fetches the web page. To do a POST using curl, you would need to change that to:

$ret=qx(curl -m $tout -s -k -S -d $postArgs $pstr "$url" 2>&1);

with matching additions elsewhere to fill the $postArgs variable with whatever you want to post. If you're wanting to implement something that performs a login, you'd probably want to track cookies as well.

I'm not much of a perl programmer, so I'm not confident about making such changes. On the other hand, I've done quite a lot of this in Bourne shell...

Ralph Mitchell

Please be advised that this email may contain confidential information. If you are not the intended recipient, please do not read, copy or re-transmit this email. If you have received this email in error, please notify us by email by replying to the sender and by telephone (call us collect at +1 202-828-0850) and delete this message and any attachments. Thank you in advance for your cooperation and assistance.

In addition, Danaher and its subsidiaries disclaim that the content of this email constitutes an offer to enter into, or the acceptance of, any contract or agreement or any amendment thereto; provided that the foregoing disclaimer does not invalidate the binding effect of any digital or other electronic reproduction of a manual signature that is included in any attachment to this email.

On Tue, Jun 23, 2009 at 10:19:44AM -0400, Gary Baluha wrote:

I haven't checked The Shire in a while, but I think it still has an older version. I personally haven't been working on URLPlus for a few months (actually, I left the company I was using the script at), but a former coworker of mine has taken the task of updating the code. I will check with him to see where he's at. Also, I do plan on getting back to developing it, hopefully in the near future (sometime later this year).

Massimo:

What exactly are you looking for out of urlplus?

I am working on changing urlplus to use libcurl instead of the using the curl command line tool. Some website monitoring I needed involved using shared session data (in a nefarious cross-site scripting way, but that is what our app was doing) and hitting multiple links on a website with the same initial login authentication session data. You can only do the shared cookies/session stuff with libcurl.

I am using a bare bones perl/libcurl script to do the aforementioned monitoring now and have not yet plugged it into Gary's urlplus. Furthermore, given the complexity of what I am trying to do with shared session login the urlplus configuration file layout has to be looked at again.

We'll keep the list posted on any progress.

Gary has a 1.22 version laying around but I am not sure what's changed and it's a year old already.

Regards, -dkw

On Mon, Jun 22, 2009 at 4:59 AM, Morsiani, Massimo <[1]massimo.morsiani at gilbarco.com> wrote:

Hi Gary,

is there any new URLPlus version? I'm currently using v1.21 ($Id: url-plus.pl,v 1.21 2008/04/30 13:56:07).

Thanks.

Regards.

Massimo Morsiani Information Technology Dept.

Gilbarco S.r.l. via de' Cattani, 220/G 50145 Firenze, Italy tel: +39-055-30941 fax: +39-055-318603 email: [2]massimo.morsiani at gilbarco.com web: [3]http://www.gilbarco.it _________________________________________________________________

From: Gary Baluha [mailto:[4]gumby3203 at gmail.com] Sent: mercoled� 30 aprile 2008 16.13 To: [5]hobbit at hswn.dk Subject: Re: [hobbit] URLplus I have an updated version of URLPLus which supports single-form submissions. I haven't gotten around to being able to submit directly to The Shire, so I'm attaching it here. I've also updated this to do multi-page checking (i.e. full "end-to-end" transaction monitoring), but consider that feature mostly experimental at this time. On Wed, Apr 30, 2008 at 4:16 AM, Thomas Laurids Pedersen <[6]tlp at mach.com> wrote:

Would you share the bourne script. I need something quickly for only 1 site at the moment, so anything will do. Regards, Thomas Inactive hide details for "Ralph Mitchell" <ralphmitchell at gmail.com> "Ralph Mitchell" <[7]ralphmitchell at gmail.com>

"Ralph Mitchell" <[8]ralphmitchell at gmail.com> 29-04-2008 22:04

                         Please respond to
                         [9]hobbit at hswn.dk

                                                                   To

[10]hobbit at hswn.dk

                                                                   cc

                                                              Subject

Re: [hobbit] URLplus On Tue, Apr 29, 2008 at 2:28 PM, Thomas Laurids Pedersen <[11]tlp at mach.com> wrote:

Can this new extension do the post like the builtin http test can do ? Regards, Thomas

I don't think so, as it currently stands. This line: $ret=qx(curl -m $tout -s -k -S $pstr "$url" 2>&1); in showUrl is the bit that actually fetches the web page. To do a POST using curl, you would need to change that to: $ret=qx(curl -m $tout -s -k -S -d $postArgs $pstr "$url" 2>&1); with matching additions elsewhere to fill the $postArgs variable with whatever you want to post. If you're wanting to implement something that performs a login, you'd probably want to track cookies as well. I'm not much of a perl programmer, so I'm not confident about making such changes. On the other hand, I've done quite a lot of this in Bourne shell... Ralph Mitchell

Please be advised that this email may contain confidential information. If you are not the intended recipient, please do not read, copy or re-transmit this email. If you have received this email in error, please notify us by email by replying to the sender and by telephone (call us collect at +1 202-828-0850) and delete this message and any attachments. Thank you in advance for your cooperation and assistance. In addition, Danaher and its subsidiaries disclaim that the content of this email constitutes an offer to enter into, or the acceptance of, any contract or agreement or any amendment thereto; provided that the foregoing disclaimer does not invalidate the binding effect of any digital or other electronic reproduction of a manual signature that is included in any attachment to this email.

References

1. mailto:massimo.morsiani at gilbarco.com
2. mailto:massimo.morsiani at gilbarco.com
3. http://www.gilbarco.it/
4. mailto:gumby3203 at gmail.com
5. mailto:hobbit at hswn.dk
6. mailto:tlp at mach.com
7. mailto:ralphmitchell at gmail.com
8. mailto:ralphmitchell at gmail.com
9. mailto:hobbit at hswn.dk
10. mailto:hobbit at hswn.dk
11. mailto:tlp at mach.com

Unfortunately it's not just one script. I've found many variations in login pages, so I've had to customize the scripts for each different target. Some logins require a form to be posted back, others are handled via WWW-Authenticate pop-up boxes and sometimes redirects have to be followed. That's not hard if the site uses Location headers, because curl can handle those, but I've also found meta-refresh tags and several types of javascript "go-elsewhere" redirects that have to be individually scripted.

I posted something a year or so ago that might get you started. If I can find it, I'll post it again.

Ralph Mitchell

On Wed, Apr 30, 2008 at 3:16 AM, Thomas Laurids Pedersen <tlp at mach.com> wrote:

Would you share the bourne script. I need something quickly for only 1 site at the moment, so anything will do.

Regards, Thomas

[image: Inactive hide details for "Ralph Mitchell" <ralphmitchell at gmail.com>]"Ralph Mitchell" <ralphmitchell at gmail.com>

*"Ralph Mitchell" <ralphmitchell at gmail.com>*

        29-04-2008 22:04
        Please respond to
        hobbit at hswn.dk

To

hobbit at hswn.dk cc

Subject

Re: [hobbit] URLplus

On Tue, Apr 29, 2008 at 2:28 PM, Thomas Laurids Pedersen <*tlp at mach.com*<tlp at mach.com>> wrote:

Can this new extension do the post like the builtin http test can do ?

  Regards, Thomas

I don't think so, as it currently stands. This line:

$ret=qx(curl -m $tout -s -k -S $pstr "$url" 2>&1);

in showUrl is the bit that actually fetches the web page. To do a POST using curl, you would need to change that to:

$ret=qx(curl -m $tout -s -k -S -d $postArgs $pstr "$url" 2>&1);

with matching additions elsewhere to fill the $postArgs variable with whatever you want to post. If you're wanting to implement something that performs a login, you'd probably want to track cookies as well.

I'm not much of a perl programmer, so I'm not confident about making such changes. On the other hand, I've done quite a lot of this in Bourne shell...

Ralph Mitchell

On Wed, Apr 30, 2008 at 12:35 PM, Gary Baluha <gumby3203 at gmail.com> wrote:

On Wed, Apr 30, 2008 at 1:28 PM, Ralph Mitchell <ralphmitchell at gmail.com> wrote:

...

Unfortunately it's not just one script. I've found many variations in login pages, so I've had to customize the scripts for each different target. Some logins require a form to be posted back, others are handled via WWW-Authenticate pop-up boxes and sometimes redirects have to be followed. That's not hard if the site uses Location headers, because curl can handle those, but I've also found meta-refresh tags and several types of javascript "go-elsewhere" redirects that have to be individually scripted.

And these are some of the same issues I've run in to while writing the URLPlus script. I've been trying to slowly morph it into something that can configured solely through the configuration file without having to change any code, but that's easier said than done. Especially since the original reason I was writing the script got removed, and is supposed to be converted to yet another monitoring system (which so far hasn't progressed, probably because of the fact that some of the monitors are non-trivial).

Seems like we have a "get-rid-of-Hobbit" project about every 2 years or so.

I think our National Tools Team has some kind of click-through tools that records page requests, but I'm pretty sure it doesn't do too well with forms that include some variable(s) sent back by the server. For example, one thing I've seen is a form variable called __VIEWSTATE that seems to get a new, longer value with each successive page loaded. I've no idea what the value is as it's encoded some way, but I suppose it tracks where you are in the page.

I don't think the script recorder can handle variable inputs either. I'm watching a number of web pages where I have to enter a date that's days or weeks ahead of today's date. The recorder will just see the post containing the future date and faithfully re-post it every time, right up until it becomes a past date and the monitor breaks...

I won't be too surprised if in the end I get to keep my scripts, but I'll probably have to send traps to Unicenter as well as sending Hobbit reports. That way the "best shore" eyes-on-glass folks in Brazil or Singapore or wherever won't have to deal with Hobbit, while the 2nd-level folks at home still get the detail they're used to.

In the meantime, attached are the two files I sent previously, on 6/23/06. Here's what I wrote then:

bb-siteminder.sh is an example script to demonstrate how to login

formextract.pl is a slightly modified copy of Daniel Stenberg's formfind.pl, which can be found here:

http://curl.haxx.se/docs/programs.html

I'm not a perl programmer, so any screwups in formextract are mine, not Daniel's... :)

It's doubtful that the example script will work out of the box. I've found that I generally have to take it in easy stages - fetch the first page, eyeball it, adjust the script, lather, rinse, repeat.

LiveHTTPHeaders is a really useful Firefox extension for tracking what happens when navigating web pages.

Ralph Mitchell