I have a script which tries to verify the existence of a BugID by
making a http::get request to a bug tracking url.

http://tracking.edgetrade.com/default.asp?pg=pgEditBug&command=view&ixBug=9999

typing it in manually returns the expected page (of course) but when i
do it in perl i get the default page only
(http://tracking.edgetrade.com)

what am i doing wrong?


#!/usr/bin/perl
use LWP;
print "verifying BugID...\n";
$logFile = shift @ARGV;
open(LOGFILE, $logFile) or die "Can't verify BugID in CVS commit log
file: $logFile: $!\n";
while ($line = <LOGFILE>) {
   print "line: $line\n";
   if ($line =~ /\s*Bug[zs]*\s*IDs*\s*[#:; ]+((\d+[ ,:;#]*)+)/i)
    {
       $bugIDstring .= " " . $1;
    }
}
print "BugID's: $bugIDstring\n";
@bugIDlist = split("[ ,:;#]+", $bugIDstring);
$ixBug = 0;
foreach (@bugIDlist)
{
   if (/\d+/)
   {
      $ixBug = int($_);
      print "Attempting to verify bug info for Bug ID #$ixBug...\n";
      $ua = LWP::UserAgent->new();
      $url =
"http://tracking.edgetrade.com/default.asp?pg=pgEditBug&command=view&ixBug=$ixBug";
      $webdoc = $ua->request(HTTP::Request->new(GET => $url));
      $content = $webdoc->content;
      print "$content\n";
      if ($content =~ /Case #$ixBug does not exist or has been
deleted./)
      {
         die "BugID: $ixBug does not exist in FogBugz\n";
      }
   }
}

Mike Starkie wrote:
> I have a script which tries to verify the existence of a BugID by
> making a http::get request to a bug tracking url.
> 
> http://tracking.edgetrade.com/default.asp?pg=pgEditBug&command=view&ixBug=9999
> 
> typing it in manually returns the expected page (of course) but when i

Really?  When I "type it in manually", I get:

[1] 83664
http://tracking.edgetrade.com/default.asp?pg=pgEditBug: No match.
[2] 83665
ixBug=9999: Command not found.
command=view: Command not found.
[1]  - Exit 1 
http://tracking.edgetrade.com/default.asp?pg=pgEditBug

:-)

> do it in perl i get the default page only
> (http://tracking.edgetrade.com)
> 
> what am i doing wrong?

First, you should provide code can be executed without modification.  We 
have no idea what's in $logFile. e.g. (based on your code..)

#!/usr/local/bin/perl
use strict;
use LWP;

my $ixBug=999;
print "Attempting to verify bug info for Bug ID #$ixBug...\n";
my $ua = LWP::UserAgent->new();
my $url = 
"http://tracking.edgetrade.com/default.asp?pg=pgEditBug&command=view&ixBug=$ixBug";
my $webdoc = $ua->request(HTTP::Request->new(GET => $url));
my $content = $webdoc->content;
print "$content\n";


Second, look at what's returned.  Isn't the text "Not Logged On" 
helpful? It's asking for authentication, you need to log on to the site, 
before you can get to that URL.

Finally, take a look at WWW::Mechanize. It should help you submit your 
information to log on, and then to view the bug.

Well it certainly is due to the fact that I'm not logged in.  How do I
know what the form expects for authentication (it's a POST)?  It seems
to me I would have to sniff the wifre to see the post right?

The form expects two inputs a 'User' and 'Password".  i wrote this test
script that fails:

#!/usr/bin/perl
use LWP;
my $browser = LWP::UserAgent->new;
my $url = 'http://tracking.edgetrade.com/default.asp?';
my $res = $browser->post($url, [User => "me", Password =>
"mypasswrod"]);
my $content = $res->content;
if($content =~ /Welcome to FogBUGZ/)
{
	print "logged on";
}

Mike Starkie wrote:
> Well it certainly is due to the fact that I'm not logged in.  How do I
> know what the form expects for authentication (it's a POST)?  It seems
> to me I would have to sniff the wifre to see the post right?

No, simply looking at the correct HTML should be enough.

Ya need to go to the right URL. Selecting "Log On", in the browser
takes me to:

http://tracking.edgetrade.com/default.asp?pg=pgLogon&dest=

View the source to get the names of the fields, and be sure to set the 
various hidden variables, you'll also need to store at least one cookie.

Again, do yourself and us a favor and at least read about 
WWW::Mechanize.  There are examples, and some very nice methods to make 
this much easier.

J. Gleixner <glex_nospam@qwest.invalid> wrote:
> Mike Starkie wrote:
>> Well it certainly is due to the fact that I'm not logged in.  How do I
>> know what the form expects for authentication (it's a POST)?  It seems
>> to me I would have to sniff the wifre to see the post right?
> 
> No, simply looking at the correct HTML should be enough.
> 
> Ya need to go to the right URL. Selecting "Log On", in the browser
> takes me to:
> 
> http://tracking.edgetrade.com/default.asp?pg=pgLogon&dest=
> 
> View the source to get the names of the fields, and be sure to set the 
> various hidden variables, you'll also need to store at least one cookie.
> 
> Again, do yourself and us a favor and at least read about 
> WWW::Mechanize.  There are examples, and some very nice methods to make 
> this much easier.


See also

   Web Scraping Proxy

      http://www.research.att.com/~hpk/wsp/

Which will write the Perl code for you.


-- 
    Tad McClellan                          SGML consulting
    tadmc@augustmail.com                   Perl programming
    Fort Worth, Texas

In article <1115402404.130396.85040@f14g2000cwb.googlegroups.com>,
Mike Starkie <skijor@gmail.com> wrote:
>I have a script which tries to verify the existence of a BugID by
> ...
>      $webdoc = $ua->request(HTTP::Request->new(GET => $url));
>      $content = $webdoc->content;
>      print "$content\n";
>      if ($content =~ /Case #$ixBug does not exist or has been
>deleted./)
>      {
>         die "BugID: $ixBug does not exist in FogBugz\n";
>      }
>   }
>}
>
In addition to the other answers, you'll normally want to 
glean more from the response:  

$webdoc = $ua->request(...);
if ($webdoc->is_success) { 
  print $webdoc->content; } 
else   { 
  print "failed: ", $webdoc->status_line; }

hth,
--
Charles DeRykus
#! rnews 1193
Xref: xyzzy comp.unix.programmer:136921
Newsgroups: comp.unix.programmer
Path: xyzzy!nntp
From: "Fred L. Kleinschmidt" <fred.l.kleinschmidt@nospam_boeing.com>
Subject: Re: Folder contents
X-Nntp-Posting-Host: xpc-ps-53.nw.nos.boeing.com
Content-Type: text/plain; charset=us-ascii
Message-ID: <427BE8E4.71D456CB@nospam_boeing.com>
Sender: nntp@news.boeing.com (Boeing NNTP News Access)
Content-Transfer-Encoding: 7bit
Organization: Boeing
X-Accept-Language: en
References: <1115395160.999855.129060@g14g2000cwa.googlegroups.com>
Mime-Version: 1.0
Date: Fri, 6 May 2005 22:00:04 GMT
X-Mailer: Mozilla 4.79 [en]C-CCK-MCD Boeing Kit  (Windows NT 5.0; U)



yorkyboy wrote:
> 
> Guys im very new to Unix and scripting as a whole, but can anyone tell
> me if its posible to create a script that looks at say 4 folders, then
> checks which has the least amount of files in, then copy a file to that
> folder?.

First, get used to calling them "directories", instead of "folders". 

Look at the 'man' pages for ls, wc, and cp (i.e., type "man ls" or "man
wc" etc.)
-- 
Fred L. Kleinschmidt
Boeing Associate Technical Fellow
Technical Architect, Common User Interface Services
M/S 2R-94  (206)544-5225

For some reason, this tool does not generate any perl for me.  The
website uses asp and the results of using wsp are a few files (w0 - w6)
which contain no perl only html and javascript

In article <1115663471.848260.112780@f14g2000cwb.googlegroups.com>,
Mike Starkie <skijor@gmail.com> wrote:

> For some reason, this tool does not generate any perl for me.  The
> website uses asp and the results of using wsp are a few files (w0 - w6)
> which contain no perl only html and javascript

HTTP::Request does not generate Perl code. It encapsulates an HTTP/GET
request and the response from an HTTP server for some URL. The response
should contain HTML, which can contain javascript code, but that is up
to the server, not your program. Why do you think it should contain
Perl code?


----== Posted via Newsfeeds.Com - Unlimited-Uncensored-Secure Usenet News==----
http://www.newsfeeds.com The #1 Newsgroup Service in the World! >100,000 Newsgroups
---= East/West-Coast Server Farms - Total Privacy via Encryption =---

Jim Gibson wrote:
> In article <1115663471.848260.112780@f14g2000cwb.googlegroups.com>,
> Mike Starkie <skijor@gmail.com> wrote:
> 
> 
>>For some reason, this tool does not generate any perl for me.  The
>>website uses asp and the results of using wsp are a few files (w0 - w6)
>>which contain no perl only html and javascript
> 
> 
> HTTP::Request does not generate Perl code. It encapsulates an HTTP/GET
> request and the response from an HTTP server for some URL. The response
> should contain HTML, which can contain javascript code, but that is up
> to the server, not your program. Why do you think it should contain
> Perl code?

Mike neglected to include information from the post he was referring to 
that mentioned "wsp".

Tad McClellan wrote:
 > See also
 >
 >    Web Scraping Proxy
 >
 >       http://www.research.att.com/~hpk/wsp/
 >
 > Which will write the Perl code for you.

As for wsp.pl writing code for you, it took a little looking around, but 
I found this:

   http://hacks.oreilly.com/pub/h/955#code

Very nice!

WooHoo.......sweeeeeet.

>./wsp.pl -v | ./translate.pl > test.pl

>./test.pl

works.....problem solved !!!

for the annals of time:

1: I didn't realize the perl output of wsp is on STDOUT

2: Use Both  wsp and translate together to save a ton of time since you
don't have to encode the session by trial and error and piece together
the perl

4: my complaints should include more context so that new joiners on the
thread can understand them.

Mike Starkie <skijor@gmail.com> wrote:

> For some reason, this tool does not generate any perl for me.


Which tool are you speaking of?

HTTP::Request is a "module", not a "tool", and it isn't _supposed_
to generate any perl.

What made you think that it would generate perl (or Perl, more likely)?


-- 
    Tad McClellan                          SGML consulting
    tadmc@augustmail.com                   Perl programming
    Fort Worth, Texas

these two 'tools' taken together will generate perl:

../wsp.pl -v | ./translate.pl > test.pl
../test.pl

you're the second person that mis-understood me.....sorry.

see links above for tools:  wsp and translate

Mike Starkie <skijor@gmail.com> wrote:

> you're 


Who is?

Please quote some context in followups like everybody else does.


> the second person that mis-understood me.....sorry.


That's because you are not quoting context.

And you are *still* not quoting context.

I'm doubting the sincerity of your "sorry" there.

If you were truly sorry you would stop the behavior that led
to the misunderstanding, else it will just happen again.


-- 
    Tad McClellan                          SGML consulting
    tadmc@augustmail.com                   Perl programming
    Fort Worth, Texas

Mike Starkie <skijor@gmail.com> wrote:


> see links above for tools:  wsp and translate


There are no "links above", so once again we don't know what 
you're talking about.


-- 
    Tad McClellan                          SGML consulting
    tadmc@augustmail.com                   Perl programming
    Fort Worth, Texas

of course you could always read the entire thread instead in order to
clear up your confusion instead of doubting my sincerity.

Mike Starkie <skijor@gmail.com> wrote:

> of course you could always read the entire thread 


Reading the entire thread would not be necessary if you quoted
some context like everybody else does.

You appear to not know much about the propogation of Usenet
articles. It is entirely possible that a followup arrives
*before* the article that it is followup up to.

The original article may even *never* show up. NNTP is not guaranteed.

So, many people don't *have* and entire thread, and they're left
wondering what you're talking about.


> instead in order to
> clear up your confusion instead of doubting my sincerity.


There is no longer any confusion, you've just confirmed
that your apology was indeed insincere.

*plonk*


-- 
    Tad McClellan                          SGML consulting
    tadmc@augustmail.com                   Perl programming
    Fort Worth, Texas

Actually you are right.  There's no need for me to aplogize for you not
understanding my reference to the tool that you suggested in the first
place. My aplogies to others though and to redeem my sincerity I
include some quoted text here:

You wrote:

>See also
>   Web Scraping Proxy
>
>      http://www.research.att.com/~hpk/wsp/
>
>Which will write the Perl code for you.

then I wrote:

>WooHoo.......sweeeeeet.

>./wsp.pl -v | ./translate.pl > test.pl
>./test.pl

>works.....problem solved !!!

then you wrote:

>Which tool are you speaking of?

then I wrote:

>these two 'tools' taken together will generate perl:

>./wsp.pl -v | ./translate.pl > test.pl
>./test.pl

skijor@gmail.com <skijor@gmail.com> wrote:

> Actually you are right. 


It is rare, but it has been known to happen.


> my reference to the tool that you suggested in the first
> place.


I answer many many questions each day.

I don't remember who I said what to, they all meld together
after a bit of time has passed.


-- 
    Tad McClellan                          SGML consulting
    tadmc@augustmail.com                   Perl programming
    Fort Worth, Texas