Hi guys,

I'm trying to use Mechanize with Ruby to scrape data from this site
(http://www.tse.or.jp/tseHpFront/HPLCDS0101E.do). The first part of my
task is trying to automate an initial search form, I can do this fine
using the following code.


require 'rubygems'
require 'mechanize'

agent = Mechanize.new
page = agent.get('http://www.tse.or.jp/tseHpFront/HPLCDS0101E.do')

search_form = page.form('HPLCDS0101Form')
search_form.callJorEFlg = 1
search_form['method'] = 'search'
search_form.exeKind = 'HPLCDS0101E'
search_form.searchListCount = 2500
search_form.checkbox_with(:value => '001').check
search_form.checkbox_with(:value => '002').check
search_form.checkbox_with(:value => '004').check

page = agent.submit(search_form)
stockform = page.form('HPLCDS0301Form')

stock = stockform.button_with(:value => 'Display of stock price')

pp stock


When I submit the form, the data I get returned contains over 2300 links
of the type below. The onclick="chart('1378')" with the number is the
stock ticker symbol that determines which stock is going to be opened
when you click that particular link.

<input type="button"  property="chart_button"  value="Display of stock
price" onclick="chart('1378')" class="negativeButton" />

Basically, I want to be able to click all the links of the above type,
and save the returned data in a file.

Can anyone help nudge me along here? Really appreciate it, thanks!
-- 
Posted via http://www.ruby-forum.com/.

Mechanize can return a Nokogiri::HTML document when you do:

doc =3D page.parser

Using Nokogiri you can do something like:

doc.css('input[@class=3D"negativeButton"]')

to retrieve an array of the input tags with a class of
"negativeButton". You can fine-tune the returned values as needed
using select() or reject() or by switching from .css() to .xpath().

You'll need to look at the contents of the chart() Javascript function
which is being called by the onclick() handler to figure out what the
add the parameter to, and that will give you the full URL to retrieve.

On Mar 28, 3:29=A0pm, Robert Molloy <r.mol...@gmail.com> wrote:
> Hi guys,
>
> I'm trying to use Mechanize with Ruby to scrape data from this site
> (http://www.tse.or.jp/tseHpFront/HPLCDS0101E.do). The first part of my
> task is trying to automate an initial search form, I can do this fine
> using the following code.
>
> require 'rubygems'
> require 'mechanize'
>
> agent =3D Mechanize.new
> page =3D agent.get('http://www.tse.or.jp/tseHpFront/HPLCDS0101E.do')
>
> search_form =3D page.form('HPLCDS0101Form')
> search_form.callJorEFlg =3D 1
> search_form['method'] =3D 'search'
> search_form.exeKind =3D 'HPLCDS0101E'
> search_form.searchListCount =3D 2500
> search_form.checkbox_with(:value =3D> '001').check
> search_form.checkbox_with(:value =3D> '002').check
> search_form.checkbox_with(:value =3D> '004').check
>
> page =3D agent.submit(search_form)
> stockform =3D page.form('HPLCDS0301Form')
>
> stock =3D stockform.button_with(:value =3D> 'Display of stock price')
>
> pp stock
>
> When I submit the form, the data I get returned contains over 2300 links
> of the type below. The onclick=3D"chart('1378')" with the number is the
> stock ticker symbol that determines which stock is going to be opened
> when you click that particular link.
>
> <input type=3D"button" =A0property=3D"chart_button" =A0value=3D"Display o=
f stock
> price" onclick=3D"chart('1378')" class=3D"negativeButton" />
>
> Basically, I want to be able to click all the links of the above type,
> and save the returned data in a file.
>
> Can anyone help nudge me along here? Really appreciate it, thanks!
> --
> Posted viahttp://www.ruby-forum.com/.

Oh, BTW, these will give you some more info...

http://stackoverflow.com/questions/2114695/extract-single-string-from-html-using-ruby-mechanize-and-nokogiri
http://stackoverflow.com/questions/2120012/how-to-use-nokogiri-methods-xpath-at-xpath

On Mar 28, 2010, at 15:29 , Robert Molloy wrote:

> <input type=3D"button"  property=3D"chart_button"  value=3D"Display of =
stock
> price" onclick=3D"chart('1378')" class=3D"negativeButton" />
>=20
> Basically, I want to be able to click all the links of the above type,
> and save the returned data in a file.

You're prolly looking for something like:

  search_form.buttons_with(:onclick =3D> /^chart/).each do |button|
    button.click
  end

this is untested, but a good portion was glommed from mechanize's tests. =
Do this:

gem unpack mechanize
cd mechanize*
grep -r buttons_with .

and such. I find lots of stuff that way.