URLPlus as it is now does support proxy servers. I think the version on The Shire (I still call it by that; I keep forgetting the new name) is an older version prior to the proxy support I added.

On Fri, Jul 31, 2009 at 2:15 AM, Ralph Mitchell <ralphmitchell at gmail.com>wrote:

Heh - that looks like exactly what I was doing for the last 8 years. Do you have any vacancies?? :) URLPlus uses curl, so it should be able to do what your existing scripts do. This is the line in the showUrl subroutine:

$ret=qx(curl -m $tout -s -k -S $pstr "$url" 2>&1);

I guess that's one place some changes could be made - adding options for user authentication, proxy url and proxy authentication.

Ralph Mitchell

On Thu, Jul 30, 2009 at 9:14 PM, Everett, Vernon < Vernon.Everett at woodside.com.au> wrote:

...

Hi Gary

I have seen URLPlus, but never delved into it. Instead, I have been messing about with scripts using curl to try and get web sites checked. It gets a bit tricky here, because we have internal and external web sites we need to check, some need to go through our proxy, some not. And then we also need to check external "service" web sites that we require to be up. And it's all a nice mix of http and https. Does URLPlus handle proxy servers OK?

Not sure what it needs, since I haven't used it yet, but I would encourage you to continue working on it.

We have a major project deadline we are working towards right now, so time is a premium, but in the near future, I will give URLPlus a go, and let you know what I think.

Regards Vernon

I could really have used something like your feature request about 6 years ago. Instead I spent a lot of time handcrafting bash scripts to login to web pages.

Don't get me started on the sites that hit you with 5 different types of redirects before reaching the front page, or the sites where each input field is held in it's own personal form. and the submit button executes javascript to copy the values into form full of hidden fields for the actual submittal.

Gary, I'll help out where I can, but I'm not very good with perl...

I'm pretty sure you're going to need to break out the url stuff into some kind of config file and just leave an "urlplus" flag in bb-hosts.

Ralph Mitchell

On Fri, Jul 31, 2009 at 2:44 AM, Morsiani, Massimo < massimo.morsiani at gilbarco.com> wrote:

Hi Gary,

nice to hear that you're in the loop again.

I'm currently using v1.21 ($Id: url-plus.pl,v 1.21 2008/04/30 13:56:07) and I'm happy with it.

My main reason to use it is:

• per host customizable response time

My feature request is the capability to do something like the following:

1. go to a web page
2. insert username/password to login
3. surf the web site following specific links
4. grab some data from a specific web page
5. check the grabbed data (ie if GRABBED_DATA < MY_THRESHOLD then alert)

Regards.

Massimo Morsiani Information Technology Dept.

Gilbarco S.r.l. via de' Cattani, 220/G 50145 Firenze, Italy tel: +39-055-30941 fax: +39-055-318603 email: massimo.morsiani at gilbarco.com web: http://www.gilbarco.it

---

*From:* Gary Baluha [mailto:gumby3203 at gmail.com] *Sent:* mercoledì 29 luglio 2009 18.35 *To:* hobbit at hswn.dk *Subject:* [hobbit] [Hobbit] URLPlus interest - looking for feedback

For those of you who have been around here for a while, you will have noticed that I've been somewhat absent of late. The short story is, I have changed jobs and just don't get as many opportunities to check in as I used to. This also means I have left my reasons for working on URLPlus behind, which is why development of it has stopped.

The purpose of this post is because I have gotten a few requests from time to time about URLPlus, and I'm curious what the interest in me getting back to developing it would be. I've been out of the loop as far as new features in the latest version of Hobbit being worked on, but from what I understand, while there are more web page monitoring features built-in, there are still some things where URLPlus could be useful.

So, I would be interested in the following feedback:

1. Is there still demand for more robust website content checking features than what Hobbit supports out of the box?
2. If so, what are the features missing in Hobbit that people are looking for?
3. For those who have chosen to use URLPlus, what are the reasons for that, versus using some other Hobbit plugin (or for that matter, some other monitoring tool)?
4. Finally, for those that are using URLPlus, what additional features would you like to see it be capable of doing?

-Gary

Please be advised that this email may contain confidential information. If you are not the intended recipient, please do not read, copy or re-transmit this email. If you have received this email in error, please notify us by email by replying to the sender and by telephone (call us collect at +1 202-828-0850) and delete this message and any attachments. Thank you in advance for your cooperation and assistance.

In addition, Danaher and its subsidiaries disclaim that the content of this email constitutes an offer to enter into, or the acceptance of, any contract or agreement or any amendment thereto; provided that the foregoing disclaimer does not invalidate the binding effect of any digital or other electronic reproduction of a manual signature that is included in any attachment to this email.

On Fri, Jul 31, 2009 at 11:34 AM, Gary Baluha <gumby3203 at gmail.com> wrote:

On Fri, Jul 31, 2009 at 4:57 AM, Ralph Mitchell <ralphmitchell at gmail.com>wrote:

...

I could really have used something like your feature request about 6 years ago. Instead I spent a lot of time handcrafting bash scripts to login to web pages.

Yep, that's kind of how URLPlus got started in the first place ;-)

...

Don't get me started on the sites that hit you with 5 different types of redirects before reaching the front page, or the sites where each input field is held in it's own personal form. and the submit button executes javascript to copy the values into form full of hidden fields for the actual submittal.

The redirect issue actually isn't too difficult to work around. I have been working on a perl program that is capable of more in-depth session management than URLPlus is currently capable of, and the solution I'm using now seems to work pretty well. My goal is to eventually convert URLPlus from using a command-line curl solution, to my current one. This new method deals with multi-page redirects better.

It's not so much the multi-page redirects using the standard "302: page is now elsewhere" format, as the other weird ways redirects are sometimes done. The one that irritated me the most did all of these, in no particular order:

1. meta-refresh with zero time delay and a new url

2. self-submitting form - i.e. a preloaded form with "form.submit();" at the end of the html, between script tags

3. self-submitting form - another preloaded form, but with "onLoad=form.submit();" in the html BODY tag

4. in script tags, change the page location via: top.location="newurl"

5. as above, but use "top.href", or "page.href" or something similar.

I'm not knocking your efforts - you've already done more than I ever did towards a generic webpage check. I just think that the above are going to be tricky to handle in an automated way without replicating a large fraction of a web browser. But, now at least they're documented in the mailing list for anyone interested in doing their own web checks... :)

As for the javascript part, that is a bit more difficult.

Especially when the page you just downloaded creates the form POST url on-the-fly from some of the form elements filled in by the user. Yep, saw that happen too... Another weird page ran a java function to generate a random character string to include in the url - luckily the function wasn't too hard to extract and shove through the spidermonkey javascript interpreter... :)

Ralph Mitchell