It's not so much the multi-page redirects using the standard "302: page is now elsewhere" format, as the other weird ways redirects are sometimes done. The one that irritated me the most did all of these, in no particular order:
1) meta-refresh with zero time delay and a new url 2) self-submitting form - i.e. a preloaded form with "form.submit();" at the end of the html, between script tags
3) self-submitting form - another preloaded form, but with "onLoad=form.submit();" in the html BODY tag 4) in script tags, change the page location via: top.location="newurl"
5) as above, but use "top.href", or "page.href" or something similar. I'm not knocking your efforts - you've already done more than I ever did towards a generic webpage check. I just think that the above are going to be tricky to handle in an automated way without replicating a large fraction of a web browser. But, now at least they're documented in the mailing list for anyone interested in doing their own web checks... :)
1: I think my new method would handle that, though I'll have to double-check. 2-5: Wow...
As for the javascript part, that is a bit more difficult.
Especially when the page you just downloaded creates the form POST url on-the-fly from some of the form elements filled in by the user. Yep, saw that happen too... Another weird page ran a java function to generate a random character string to include in the url - luckily the function wasn't too hard to extract and shove through the spidermonkey javascript interpreter... :)
There are some really weird websites out there...