mechanize 0.8.4 → 0.8.5

data/EXAMPLES.txt

@@ -117,7 +117,7 @@ This example also demonstrates subclassing Mechanize.
       search_form.words = 'WWW'
       submit search_form
 
-      page.links.with.href( %r{/projects/} ).each do |link|
+      page.links_with(:href => %r{/projects/} ).each do |link|
         next if link.href =~ %r{/projects/support/}
 
         puts 'Loading %-30s %s' % [link.href, link.text]

data/GUIDE.txt

@@ -36,17 +36,16 @@ link to click on.  Lets say we wanted to click the link whose text is 'News'.
 Normally, we would have to do this:
  page = agent.click page.links.find { |l| l.text == 'News' }
 But Mechanize gives us a shortcut.  Instead we can say this:
- page = agent.click page.links.text('News')
+ page = agent.click page.link_with(:text => 'News')
 That shortcut says "find all links with the name 'News'".  You're probably
 thinking "there could be multiple links with that text!", and you would be
-correct!  If you pass a list of links to the "click" method, Mechanize will
-click on the first one.  If you wanted to click on the second news link, you
-could do this:
- agent.click page.links.text('News')[1]
+correct!  If you use the plural form, you can access the list.
+If you wanted to click on the second news link, you could do this:
+ agent.click page.links_with(:text => 'News')[1]
 We can even find a link with a certain href like so:
- page.links.href('/something')
+ page.links_with(:href => '/something')
 Or chain them together to find a link with certain text and certain href:
- page.links.text('News').href('/something')
+ page.links_with(:text => 'News', :href => '/something')
 
 These shortcuts that mechanize provides are available on any list that you
 can fetch like frames, iframes, or forms.  Now that we know how to find and
@@ -102,24 +101,24 @@ have many options associated with them.  If you select one option, mechanize
 will deselect the other options (unless it is a multi select!).
 
 For example, lets select an option on a list:
- form.fields.name('list').options[0].select
+ form.field_with(:name => 'list').options[0].select
 
 Now lets take a look at checkboxes and radio buttons.  To select a checkbox,
 just check it like this:
- form.checkboxes.name('box').check
+ form.checkbox_with(:name => 'box').check
 Radio buttons are very similar to checkboxes, but they know how to uncheck
 other radio buttons of the same name.  Just check a radio button like you
 would a checkbox:
- form.radiobuttons.name('box')[1].check
+ form.radiobuttons_with(:name => 'box')[1].check
 Mechanize also makes file uploads easy!  Just find the file upload field, and
 tell it what file name you want to upload:
-  form.file_uploads.file_name = "somefile.jpg"
+  form.file_uploads.first.file_name = "somefile.jpg"
 
 == Scraping Data
 Mechanize uses hpricot[http://code.whytheluckystiff.net/hpricot/] to parse
 html.  What does this mean for you?  You can treat a mechanize page like
 an hpricot object.  After you have used Mechanize to navigate to the page
 that you need to scrape, then scrape it using hpricot methods:
-  agent.get('http://someurl.com/').search("//p[@class='posted']")
+  agent.get('http://someurl.com/').search(".//p[@class='posted']")
 For more information on this powerful scraper, take a look at
 HpricotBasics[http://code.whytheluckystiff.net/hpricot/wiki/HpricotBasics]

data/History.txt

@@ -1,5 +1,29 @@
 = Mechanize CHANGELOG
 
+=== 0.8.5
+
+* Deprecations
+  * WWW::Mechanize::List will be deprecated in 0.9.0, and warnings have
+    been added to help you upgrade.
+
+* Bug Fixes:
+  * Stopped raising EOF exceptions on HEAD requests. ありがとう：HIRAKU Kuroda
+  * Fixed exceptions when a logger is set and file:// requests are made.
+  * Made Mechanize 1.9 compatible
+  * Not setting the port in the host header for SSL sites.
+  * Following refresh headers.  Thanks Tim Connor!
+  * Cookie Jar handles cookie domains containing ports, like
+    'mydomain.com:443' (Thanks Michal Ochman!)
+  * Fixing strange uri escaping problems [#22604]
+  * Making content-type determintation more robust.  (thanks Han Holl!)
+  * Dealing with links that are query string only.  [#22402]
+  * Nokogiri may be dropped in as a replacement.
+      WWW::Mechanize.html_parser = Nokogiri::HTML
+  * Making sure the correct page is added to the history on meta refresh.
+    [#22708]
+  * Mechanize#get requests no longer send a referer unless they are relative
+    requests.
+
 === 0.8.4
 
 * Bug Fixes:

data/Manifest.txt

@@ -4,7 +4,6 @@ GUIDE.txt
 History.txt
 LICENSE.txt
 Manifest.txt
-NOTES.txt
 README.txt
 Rakefile
 examples/flickr_upload.rb
@@ -58,6 +57,7 @@ lib/www/mechanize/page/link.rb
 lib/www/mechanize/page/meta.rb
 lib/www/mechanize/pluggable_parsers.rb
 lib/www/mechanize/redirect_limit_reached_error.rb
+lib/www/mechanize/redirect_not_get_or_head_error.rb
 lib/www/mechanize/response_code_error.rb
 lib/www/mechanize/unsupported_scheme_error.rb
 lib/www/mechanize/util.rb
@@ -151,6 +151,7 @@ test/test_post_form.rb
 test/test_pretty_print.rb
 test/test_radiobutton.rb
 test/test_redirect_limit_reached.rb
+test/test_redirect_verb_handling.rb
 test/test_referer.rb
 test/test_relative_links.rb
 test/test_response_code.rb

data/README.txt

@@ -29,7 +29,7 @@ Copyright (c) 2005 by Michael Neumann (mneumann@ntecs.de)
 Copyright (c) 2006-2008:
 
 * {Aaron Patterson}[http://tenderlovemaking.com] (aaronp@rubyforge.org)
-* Mike Dalessio (mike@csa.net)
+* {Mike Dalessio}[http://mike.daless.io] (mike@csa.net)
 
 This library comes with a shameless plug for employing me
 (Aaron[http://tenderlovemaking.com/]) programming

data/Rakefile

@@ -6,8 +6,8 @@ require 'mechanize'
 
 HOE = Hoe.new('mechanize', WWW::Mechanize::VERSION) do |p|
   p.rubyforge_name  = 'mechanize'
-  p.author          = 'Aaron Patterson'
-  p.email           = 'aaronp@rubyforge.org'
+  p.developer('Aaron Patterson','aaronp@rubyforge.org')
+  p.developer('Mike Dalessio','mike.dalessio@gmail.com')
   p.summary         = "Mechanize provides automated web-browsing"
   p.extra_deps      = [['hpricot', '>= 0.5.0']]
 end

data/lib/www/mechanize.rb

@@ -14,6 +14,7 @@ require 'www/mechanize/content_type_error'
 require 'www/mechanize/response_code_error'
 require 'www/mechanize/unsupported_scheme_error'
 require 'www/mechanize/redirect_limit_reached_error'
+require 'www/mechanize/redirect_not_get_or_head_error'
 require 'www/mechanize/cookie'
 require 'www/mechanize/cookie_jar'
 require 'www/mechanize/history'
@@ -39,14 +40,14 @@ module WWW
   #  agent = WWW::Mechanize.new { |a| a.log = Logger.new("mech.log") }
   #  agent.user_agent_alias = 'Mac Safari'
   #  page = agent.get("http://www.google.com/")
-  #  search_form = page.forms.name("f").first
-  #  search_form.fields.name("q").value = "Hello"
+  #  search_form = page.form_with(:name => "f")
+  #  search_form.field_with(:name => "q").value = "Hello"
   #  search_results = agent.submit(search_form)
   #  puts search_results.body
   class Mechanize
     ##
     # The version of Mechanize you are using.
-    VERSION = '0.8.4'
+    VERSION = '0.8.5'
   
     ##
     # User Agent aliases
@@ -203,7 +204,13 @@ module WWW
         headers = options[:headers]
       end
 
-      referer ||= current_page || Page.new(nil, {'content-type'=>'text/html'})
+      unless referer
+        if url =~ /^http/
+          referer = Page.new(nil, {'content-type'=>'text/html'})
+        else
+          referer = current_page || Page.new(nil, {'content-type'=>'text/html'})
+        end
+      end
 
       # FIXME: Huge hack so that using a URI as a referer works.  I need to
       # refactor everything to pass around URIs but still support
@@ -440,8 +447,8 @@ module WWW
       http_obj      = options[:connection]
 
       # Add If-Modified-Since if page is in history
-      if( (page = visited_page(uri)) && cur_page.response['Last-Modified'] )
-        request['If-Modified-Since'] = cur_page.response['Last-Modified']
+      if( (page = visited_page(uri)) && page.response['Last-Modified'] )
+        request['If-Modified-Since'] = page.response['Last-Modified']
       end if(@conditional_requests)
 
       # Specify timeouts if given
@@ -458,9 +465,9 @@ module WWW
       # Send the request
       attempts = 0
       begin
-        response = http_obj.request(request, *request_data) { |response|
+        response = http_obj.request(request, *request_data) { |r|
           connection_chain = Chain.new([
-            Chain::ResponseReader.new(response),
+            Chain::ResponseReader.new(r),
             Chain::BodyDecodingHandler.new,
           ])
           connection_chain.handle(options)
@@ -488,9 +495,32 @@ module WWW
 
       log.info("status: #{ page.code }") if log
   
-      if follow_meta_refresh && page.respond_to?(:meta) &&
-        (redirect = page.meta.first)
-        return redirect.click
+      if follow_meta_refresh
+        redirect_uri = nil
+        if (page.respond_to?(:meta) && (redirect = page.meta.first))
+          redirect_uri = redirect.uri.to_s
+        elsif refresh = response['refresh']
+          parsed_refresh = refresh.match(/^\s*(\d+\.?\d*);\s*(url|URL)=(\S*)\s*$/)
+          raise StandardError, "Invalid refresh http header" unless parsed_refresh
+          delay = parsed_refresh[1]
+          location = parsed_refresh[3]
+          location = "http://#{uri.host}#{location}" unless location.include?("http")
+          if redirects + 1 > redirection_limit
+            raise RedirectLimitReachedError.new(page, redirects)
+          end
+          sleep delay.to_i
+          redirect_uri = location
+        end
+        if redirect_uri
+          @history.push(page, page.uri)
+          return fetch_page(
+            :uri        => redirect_uri,
+            :referer    => page,
+            :params     => [],
+            :verb       => :get,
+            :redirects  => redirects + 1
+          )
+        end
       end
   
       return page if res_klass <= Net::HTTPSuccess
@@ -503,9 +533,11 @@ module WWW
         log.info("follow redirect to: #{ response['Location'] }") if log
         from_uri  = page.uri
         raise RedirectLimitReachedError.new(page, redirects) if redirects + 1 > redirection_limit
+        redirect_verb = options[:verb] == :head ? :head : :get
         page = fetch_page(  :uri => response['Location'].to_s,
                             :referer => page,
                             :params  => [],
+                            :verb => redirect_verb,
                             :redirects => redirects + 1
                          )
         @history.push(page, from_uri)

data/lib/www/mechanize/chain/header_resolver.rb

@@ -23,7 +23,7 @@ module WWW
           end
           request['Accept-Encoding'] = 'gzip,identity'
           request['Accept-Language'] = 'en-us,en;q=0.5'
-          host = "#{uri.host}#{uri.port.to_i == 80 ? '' : ':' + uri.port.to_s}"
+          host = "#{uri.host}#{[80, 443].include?(uri.port.to_i) ? '' : ':' + uri.port.to_s}"
           request['Host'] = host
           request['Accept-Charset'] = 'ISO-8859-1,utf-8;q=0.7,*;q=0.7'
   

data/lib/www/mechanize/chain/request_resolver.rb

@@ -16,6 +16,10 @@ module WWW
             class << o
               def add_field(*args); end
               alias :[]= :add_field
+              def path
+                uri.path
+              end
+              def each_header; end
             end
             params[:request] ||= o
           end

data/lib/www/mechanize/chain/response_body_parser.rb

@@ -17,7 +17,7 @@ module WWW
           content_type = nil
           unless response['Content-Type'].nil?
             data = response['Content-Type'].match(/^([^;]*)/)
-            content_type = data[1].downcase unless data.nil?
+            content_type = data[1].downcase.split(',')[0] unless data.nil?
           end
 
           # Find our pluggable parser

data/lib/www/mechanize/chain/response_reader.rb

@@ -24,7 +24,7 @@ module WWW
 
           # Net::HTTP ignores EOFError if Content-length is given, so we emulate it here.
           unless res_klass <= Net::HTTPRedirection
-            raise EOFError if @response.content_length() && @response.content_length() != total
+            raise EOFError if (!params[:request].is_a?(Net::HTTP::Head)) && @response.content_length() && @response.content_length() != total
           end
   
           @response.each_header { |k,v|

data/lib/www/mechanize/chain/ssl_resolver.rb

@@ -15,7 +15,7 @@ module WWW
         def handle(ctx, params)
           uri       = params[:uri]
           http_obj  = params[:connection]
-          if uri.scheme == 'https' && ! http_obj.started?
+          if uri.scheme == 'https' && ! http_obj.started? && ! http_obj.frozen?
             http_obj.use_ssl = true
             http_obj.verify_mode = OpenSSL::SSL::VERIFY_NONE
             if @ca_file

data/lib/www/mechanize/chain/uri_resolver.rb

@@ -17,19 +17,31 @@ module WWW
               sprintf('%%%X', match.unpack($KCODE == 'UTF8' ? 'U' : 'c')[0])
             }
 
-            uri = URI.parse(
-                    Util.html_unescape(
-                      uri.split(/(?:%[0-9A-Fa-f]{2})+|#/).zip(
-                        uri.scan(/(?:%[0-9A-Fa-f]{2})+|#/)
-                      ).map { |x,y|
-                        "#{URI.escape(x)}#{y}"
-                      }.join('')
-                    )
-                  )
+            escaped_uri = Util.html_unescape(
+              uri.split(/(?:%[0-9A-Fa-f]{2})+|#/).zip(
+                uri.scan(/(?:%[0-9A-Fa-f]{2})+|#/)
+              ).map { |x,y|
+                "#{URI.escape(x)}#{y}"
+              }.join('')
+            )
+
+            begin
+              uri = URI.parse(escaped_uri)
+            rescue
+              uri = URI.parse(URI.escape(escaped_uri))
+            end
+
           end
           uri = @scheme_handlers[
             uri.relative? ? 'relative' : uri.scheme.downcase
           ].call(uri, params[:referer])
+
+          if params[:referer] && params[:referer].uri
+            if uri.path.length == 0 && uri.relative?
+              uri.path = params[:referer].uri.path
+            end
+          end
+
           uri.path = '/' if uri.path.length == 0
 
           if uri.relative?

data/lib/www/mechanize/cookie_jar.rb

@@ -13,7 +13,7 @@ module WWW
     
       # Add a cookie to the Jar.
       def add(uri, cookie)
-        return unless uri.host =~ /#{cookie.domain}$/i
+        return unless uri.host =~ /#{CookieJar.strip_port(cookie.domain)}$/i
         normal_domain = cookie.domain.downcase
         unless @jar.has_key?(normal_domain)
           @jar[normal_domain] = Hash.new
@@ -30,7 +30,7 @@ module WWW
         cookies = []
         url.path = '/' if url.path.empty?
         @jar.each_key do |domain|
-          if url.host =~ /#{domain}$/i
+          if url.host =~ /#{CookieJar.strip_port(domain)}$/i
             @jar[domain].each_key do |name|
               if url.path =~ /^#{@jar[domain][name].path}/
                 if @jar[domain][name].expires.nil?
@@ -68,9 +68,9 @@ module WWW
       def save_as(file, format = :yaml)
         ::File.open(file, "w") { |f|
           case format
-          when :yaml:
+          when :yaml then
             YAML::dump(@jar, f)
-          when :cookiestxt:
+          when :cookiestxt then
             dump_cookiestxt(f)
           else
             raise "Unknown cookie jar file format"
@@ -86,9 +86,9 @@ module WWW
       def load(file, format = :yaml)
         @jar = ::File.open(file) { |f|
           case format
-          when :yaml:
+          when :yaml then
             YAML::load(f)
-          when :cookiestxt:
+          when :cookiestxt then
             load_cookiestxt(f)
           else
             raise "Unknown cookie jar file format"
@@ -181,6 +181,11 @@ module WWW
           end
         end
       end
+
+      def self.strip_port(host)
+        host.gsub(/:[0-9]+$/,'')
+      end
+
     end
   end
 end

data/lib/www/mechanize/file_response.rb

@@ -28,6 +28,8 @@ module WWW
         ::File.exists?(@file_path) ? ::File.stat(@file_path).size : 0
       end
 
+      def each_header; end
+
       def [](key)
         return nil unless key.downcase == 'content-type'
         return 'text/html' if directory?

data/lib/www/mechanize/form.rb

@@ -61,11 +61,6 @@ module WWW
 
       def values; fields.map { |f| f.value }; end
 
-      # Fetch the first field whose name is equal to +field_name+
-      def field(field_name)
-        fields.find { |f| f.name.eql? field_name }
-      end
-
       # Add a field with +field_name+ and +value+
       def add_field!(field_name, value = nil)
         fields << Field.new(field_name, value)
@@ -84,16 +79,16 @@ module WWW
           case v
           when Hash
             v.each do |index, value|
-              self.fields.name(k.to_s).[](index).value = value
+              self.fields_with(:name => k.to_s).[](index).value = value
             end
           else
             value = nil
             index = 0
-            v.each do |val|
+            [v].flatten.each do |val|
               index = val.to_i unless value.nil?
               value = val if value.nil?
             end
-            self.fields.name(k.to_s).[](index).value = value
+            self.fields_with(:name => k.to_s).[](index).value = value
           end
         end
       end
@@ -209,7 +204,32 @@ module WWW
       def delete_field!(field_name)
         @fields.delete_if{ |f| f.name == field_name}
       end
-          
+
+      { :field        => :fields,
+        :button       => :buttons,
+        :file_upload  => :file_uploads,
+        :radiobutton  => :radiobuttons,
+        :checkbox     => :checkboxes,
+      }.each do |singular,plural|
+        eval(<<-eomethod)
+          def #{plural}_with criteria = {}
+            criteria = {:name => criteria} if String === criteria
+            f = #{plural}.find_all do |thing|
+              criteria.all? { |k,v| v === thing.send(k) }
+            end
+            yield f if block_given?
+            f
+          end
+
+          def #{singular}_with criteria = {}
+            f = #{plural}_with(criteria).first
+            yield f if block_given?
+            f
+          end
+          alias :#{singular} :#{singular}_with
+        eomethod
+      end
+ 
       private
       def parse
         @fields       = WWW::Mechanize::List.new
@@ -219,7 +239,7 @@ module WWW
         @checkboxes   = WWW::Mechanize::List.new
     
         # Find all input tags
-        form_node.search('//input').each do |node|
+        form_node.search('input').each do |node|
           type = (node['type'] || 'text').downcase
           name = node['name']
           next if name.nil? && !(type == 'submit' || type =='button')
@@ -242,13 +262,13 @@ module WWW
         end
 
         # Find all textarea tags
-        form_node.search('//textarea').each do |node|
+        form_node.search('textarea').each do |node|
           next if node['name'].nil?
           @fields << Field.new(node['name'], node.inner_text)
         end
 
         # Find all select tags
-        form_node.search('//select').each do |node|
+        form_node.search('select').each do |node|
           next if node['name'].nil?
           if node.has_attribute? 'multiple'
             @fields << MultiSelectList.new(node['name'], node)
@@ -259,7 +279,7 @@ module WWW
 
         # Find all submit button tags
         # FIXME: what can I do with the reset buttons?
-        form_node.search('//button').each do |node|
+        form_node.search('button').each do |node|
           type = (node['type'] || 'submit').downcase
           next if type == 'reset'
           @buttons << Button.new(node['name'], node['value'])