mechanize 2.4 → 2.5

data/CHANGELOG.rdoc

@@ -1,12 +1,38 @@
 = Mechanize CHANGELOG
 
+=== 2.5
+
+* Minor enhancement
+  * Added Mechanize#ignore_bad_chunking for working around servers that don't
+    terminate chunked transfer-encoding properly.  Enabling this may cause
+    data loss.  Issue #116
+  * Removed content-type check from Mechanize::Page allowing forced parsing
+    of incorrect or missing content-types.  Issue #221 by GarthSnyder
+* Bug fixes
+  * Fixed typos in EXAMPLES and GUIDES.  Pull Request #213 by Erkan Yilmaz.
+  * Fixed handling of a quoted content-disposition size.  Pull Request #220 by
+    Jason Rust
+  * Mechanize now ignores a missing gzip footer like browsers do.  Issue #224
+    by afhbl
+  * Mechanize handles saving of files with the same name better now.  Pull
+    Request #223 by Godfrey Chan, Issue #219 by Jon Hart
+  * Mechanize now sends headers across redirects.  Issue #215 by Chris Gahan
+  * Mechanize now raises Mechanize::ResponseReadError when the server does not
+    terminate chunked transfer-encoding properly.  Issue #116
+  * Mechanize no longer raises an exception when multiple identical
+    radiobuttons are checked.  Issue #214 by Matthias Guenther
+  * Fixed documentation for pre_connect_hooks and post_connect_hooks.  Issue
+    #226 by Robert Poor
+  * Worked around ruby 1.8 run with -Ku and ISO-8859-1 encoded characters in
+    URIs.  Issue #228 by Stanislav O.Pogrebnyak
+
 === 2.4
 
 * Security fix:
 
   Mechanize#auth and Mechanize#basic_auth allowed disclosure of passwords to
-  malicious servers and have been removed.
-  
+  malicious servers and have been deprecated.
+
   In prior versions of mechanize only one set of HTTP authentication
   credentials were allowed for all connections.  If a mechanize instance
   connected to more than one server then a malicious server detecting
@@ -14,8 +40,8 @@
   username and password intended only for one server.
 
   Mechanize#auth and Mechanize#basic_auth now warn when used.
-  
-  To fix the warning switch to Mechanize#add_auth which requires at the URI
+
+  To fix the warning switch to Mechanize#add_auth which requires the URI
   the credentials are intended for, the username and the password.
   Optionally an HTTP authentication realm or NTLM domain may be provided.
 

data/EXAMPLES.rdoc

@@ -83,7 +83,7 @@ Upload a file to flickr.
 
 == Pluggable Parsers
 
-Lets say you want HTML pages to automatically be parsed with Rubyful Soup.
+Let's say you want HTML pages to automatically be parsed with Rubyful Soup.
 This example shows you how:
 
   require 'rubygems'
@@ -119,7 +119,7 @@ Beautiful Soup for that page.
 
 Mechanize#transact runs the given block and then resets the page history. I.e.
 after the block has been executed, you're back at the original page; no need
-count how many times to call the back method at the end of a loop (while
+to count how many times to call the back method at the end of a loop (while
 accounting for possible exceptions).
 
 This example also demonstrates subclassing Mechanize.

data/GUIDE.rdoc

@@ -26,7 +26,7 @@ Mechanize stored any cookies that were set, and followed any redirects that
 google may have sent.  The agent gave us back a page that we can use to
 scrape data, find links to click, or find forms to fill out.
 
-Next, lets try finding some links to click.
+Next, let's try finding some links to click.
 
 == Finding Links
 
@@ -34,14 +34,14 @@ Mechanize returns a page object whenever you get a page, post, or submit a
 form.  When a page is fetched, the agent will parse the page and put a list
 of links on the page object.
 
-Now that we've fetched google's homepage, lets try listing all of the links:
+Now that we've fetched google's homepage, let's try listing all of the links:
 
   page.links.each do |link|
     puts link.text
   end
 
 We can list the links, but Mechanize gives a few shortcuts to help us find a
-link to click on.  Lets say we wanted to click the link whose text is 'News'.
+link to click on.  Let's say we wanted to click the link whose text is 'News'.
 Normally, we would have to do this:
 
   page = agent.page.links.find { |l| l.text == 'News' }.click
@@ -65,13 +65,13 @@ Or chain them together to find a link with certain text and certain href:
 
   page.link_with(:text => 'News', :href => '/something')
 
-These shortcuts that mechanize provides are available on any list that you
+These shortcuts that Mechanize provides are available on any list that you
 can fetch like frames, iframes, or forms.  Now that we know how to find and
-click links, lets try something more complicated like filling out a form.
+click links, let's try something more complicated like filling out a form.
 
 == Filling Out Forms
 
-Lets continue with our google example.  Here's the code we have so far:
+Let's continue with our google example.  Here's the code we have so far:
   require 'rubygems'
   require 'mechanize'
 
@@ -83,17 +83,17 @@ that has a couple buttons and a few fields:
 
   pp page
 
-Now that we know the name of the form, lets fetch it off the page:
+Now that we know the name of the form, let's fetch it off the page:
 
  google_form = page.form('f')
 
 Mechanize lets you access form input fields in a few different ways, but the
 most convenient is that you can access input fields as accessors on the
-object.  So lets set the form field named 'q' on the form to 'ruby mechanize':
+object.  So let's set the form field named 'q' on the form to 'ruby mechanize':
 
   google_form.q = 'ruby mechanize'
 
-To make sure that we set the value, lets pretty print the form, and you should
+To make sure that we set the value, let's pretty print the form, and you should
 see a line similar to this:
 
   #<Mechanize::Field:0x1403488 @name="q", @value="ruby mechanize">
@@ -109,7 +109,7 @@ clicking the 'Google Search' button.  If we had submitted the form without
 a button, it would be like typing in the text field and hitting the return
 button.
 
-Lets take a look at the code all together:
+Let's take a look at the code all together:
 
   require 'rubygems'
   require 'mechanize'
@@ -121,7 +121,7 @@ Lets take a look at the code all together:
   page = agent.submit(google_form)
   pp page
 
-Before we go on to screen scraping, lets take a look at forms a little more
+Before we go on to screen scraping, let's take a look at forms a little more
 in depth.  Unless you want to skip ahead!
 
 == Advanced Form Techniques
@@ -132,11 +132,11 @@ text input fields.  Select fields are very similar to text fields, but they
 have many options associated with them.  If you select one option, mechanize
 will de-select the other options (unless it is a multi select!).
 
-For example, lets select an option on a list:
+For example, let's select an option on a list:
 
   form.field_with(:name => 'list').options[0].select
 
-Now lets take a look at checkboxes and radio buttons.  To select a checkbox,
+Now let's take a look at checkboxes and radio buttons.  To select a checkbox,
 just check it like this:
 
   form.checkbox_with(:name => 'box').check

data/Manifest.txt

@@ -13,6 +13,7 @@ examples/rubyforge.rb
 examples/spider.rb
 examples/wikipedia_links_to_philosophy.rb
 lib/mechanize.rb
+lib/mechanize/chunked_termination_error.rb
 lib/mechanize/content_type_error.rb
 lib/mechanize/cookie.rb
 lib/mechanize/cookie_jar.rb
@@ -66,6 +67,34 @@ lib/mechanize/response_code_error.rb
 lib/mechanize/response_read_error.rb
 lib/mechanize/robots_disallowed_error.rb
 lib/mechanize/test_case.rb
+lib/mechanize/test_case/.document
+lib/mechanize/test_case/bad_chunking_servlet.rb
+lib/mechanize/test_case/basic_auth_servlet.rb
+lib/mechanize/test_case/content_type_servlet.rb
+lib/mechanize/test_case/digest_auth_servlet.rb
+lib/mechanize/test_case/file_upload_servlet.rb
+lib/mechanize/test_case/form_servlet.rb
+lib/mechanize/test_case/gzip_servlet.rb
+lib/mechanize/test_case/header_servlet.rb
+lib/mechanize/test_case/http_refresh_servlet.rb
+lib/mechanize/test_case/infinite_redirect_servlet.rb
+lib/mechanize/test_case/infinite_refresh_servlet.rb
+lib/mechanize/test_case/many_cookies_as_string_servlet.rb
+lib/mechanize/test_case/many_cookies_servlet.rb
+lib/mechanize/test_case/modified_since_servlet.rb
+lib/mechanize/test_case/ntlm_servlet.rb
+lib/mechanize/test_case/one_cookie_no_spaces_servlet.rb
+lib/mechanize/test_case/one_cookie_servlet.rb
+lib/mechanize/test_case/quoted_value_cookie_servlet.rb
+lib/mechanize/test_case/redirect_servlet.rb
+lib/mechanize/test_case/referer_servlet.rb
+lib/mechanize/test_case/refresh_with_empty_url.rb
+lib/mechanize/test_case/refresh_without_url.rb
+lib/mechanize/test_case/response_code_servlet.rb
+lib/mechanize/test_case/send_cookies_servlet.rb
+lib/mechanize/test_case/server.rb
+lib/mechanize/test_case/servlets.rb
+lib/mechanize/test_case/verb_servlet.rb
 lib/mechanize/unauthorized_error.rb
 lib/mechanize/unsupported_scheme_error.rb
 lib/mechanize/util.rb
@@ -128,6 +157,7 @@ test/test_mechanize_download.rb
 test/test_mechanize_file.rb
 test/test_mechanize_file_connection.rb
 test/test_mechanize_file_request.rb
+test/test_mechanize_file_response.rb
 test/test_mechanize_file_saver.rb
 test/test_mechanize_form.rb
 test/test_mechanize_form_check_box.rb

data/README.rdoc

@@ -28,8 +28,25 @@ The bug tracker is available here:
 
 == Examples
 
-If you are just starting, check out the GUIDE.
-Also, check out the EXAMPLES file.
+If you are just starting, check out the GUIDE.  Also, check out the EXAMPLES
+file.
+
+== Developers
+
+To run the tests for the first time:
+
+  gem install hoe rake
+  rake newb
+
+This will install all the required dependencies for running the tests.  For
+subsequent test runs:
+
+  rake test
+
+You can also use +autotest+ from the ZenTest gem to run tests.
+
+See also Mechanize::TestCase to read about the built-in testing
+infrastructure.
 
 == Authors
 

data/lib/mechanize.rb

@@ -73,7 +73,7 @@ class Mechanize
   ##
   # The version of Mechanize you are using.
 
-  VERSION = '2.4'
+  VERSION = '2.5'
 
   ##
   # Base mechanize error class
@@ -263,16 +263,16 @@ class Mechanize
   attr_accessor :history_added
 
   ##
-  # A list of hooks to call after retrieving a response.  Hooks are called with
-  # the agent and the response returned.
+  # A list of hooks to call after retrieving a response. Hooks are called with
+  # the agent, the URI, the response, and the response body.
 
   def post_connect_hooks
     @agent.post_connect_hooks
   end
 
   ##
-  # A list of hooks to call before making a request.  Hooks are called with
-  # the agent and the request to be performed.
+  # A list of hooks to call before retrieving a response. Hooks are called
+  # with the agent, the URI, the response, and the response body.
 
   def pre_connect_hooks
     @agent.pre_connect_hooks
@@ -752,6 +752,28 @@ Use of #auth and #basic_auth are deprecated due to a security vulnerability.
     @agent.idle_timeout = idle_timeout
   end
 
+  ##
+  # When set to true mechanize will ignore an EOF during chunked transfer
+  # encoding so long as at least one byte was received.  Be careful when
+  # enabling this as it may cause data loss.
+  #
+  # Net::HTTP does not inform mechanize of where in the chunked stream the EOF
+  # occurred.  Usually it is after the last-chunk but before the terminating
+  # CRLF (invalid termination) but it may occur earlier.  In the second case
+  # your response body may be incomplete.
+
+  def ignore_bad_chunking
+    @agent.ignore_bad_chunking
+  end
+
+  ##
+  # When set to true mechanize will ignore an EOF during chunked transfer
+  # encoding.  See ignore_bad_chunking for further details
+
+  def ignore_bad_chunking= ignore_bad_chunking
+    @agent.ignore_bad_chunking = ignore_bad_chunking
+  end
+
   ##
   # Are HTTP/1.1 keep-alive connections enabled?
 
@@ -1219,6 +1241,8 @@ Use of #auth and #basic_auth are deprecated due to a security vulnerability.
 
 end
 
+require 'mechanize/response_read_error'
+require 'mechanize/chunked_termination_error'
 require 'mechanize/content_type_error'
 require 'mechanize/cookie'
 require 'mechanize/cookie_jar'
@@ -1244,9 +1268,8 @@ require 'mechanize/pluggable_parsers'
 require 'mechanize/redirect_limit_reached_error'
 require 'mechanize/redirect_not_get_or_head_error'
 require 'mechanize/response_code_error'
-require 'mechanize/unauthorized_error'
-require 'mechanize/response_read_error'
 require 'mechanize/robots_disallowed_error'
+require 'mechanize/unauthorized_error'
 require 'mechanize/unsupported_scheme_error'
 require 'mechanize/util'
 

data/lib/mechanize/chunked_termination_error.rb

@@ -0,0 +1,7 @@
+##
+# Raised when Mechanize detects the chunked transfer-encoding may be
+# incorrectly terminated.
+
+class Mechanize::ChunkedTerminationError < Mechanize::ResponseReadError
+end
+

data/lib/mechanize/form.rb

@@ -231,12 +231,14 @@ class Mechanize::Form
     radio_groups.each_value do |g|
       checked = g.select {|f| f.checked}
 
-      if checked.size == 1
-        f = checked.first
-        successful_controls << f
-      elsif checked.size > 1
+      if checked.uniq.size > 1 then
+        values = checked.map { |button| button.value }.join(', ').inspect
+        name = checked.first.name.inspect
         raise Mechanize::Error,
-              "multiple radiobuttons are checked in the same group!"
+              "radiobuttons #{values} are checked in the #{name} group, " \
+              "only one is allowed"
+      else
+        successful_controls << checked.first unless checked.empty?
       end
     end
 

data/lib/mechanize/form/radio_button.rb

@@ -4,6 +4,7 @@
 
 class Mechanize::Form::RadioButton < Mechanize::Form::Field
   attr_accessor :checked
+  attr_reader :form
 
   def initialize node, form
     @checked = !!node['checked']
@@ -11,6 +12,15 @@ class Mechanize::Form::RadioButton < Mechanize::Form::Field
     super(node)
   end
 
+  def == other # :nodoc:
+    self.class === other and
+      other.form  == @form and
+      other.name  == @name and
+      other.value == @value
+  end
+
+  alias eql? == # :nodoc:
+
   def check
     uncheck_peers
     @checked = true
@@ -26,6 +36,10 @@ class Mechanize::Form::RadioButton < Mechanize::Form::Field
     checked ? uncheck : check
   end
 
+  def hash # :nodoc:
+    @form.hash ^ @name.hash ^ @value.hash
+  end
+
   def label
     (id = self['id']) && @form.page.labels_hash[id] || nil
   end

data/lib/mechanize/http/agent.rb

@@ -106,6 +106,11 @@ class Mechanize::HTTP::Agent
 
   attr_reader :http # :nodoc:
 
+  # When set to true mechanize will ignore an EOF during chunked transfer
+  # encoding so long as at least one byte was received.  Be careful when
+  # enabling this as it may cause data loss.
+  attr_accessor :ignore_bad_chunking
+
   # Handlers for various URI schemes
   attr_accessor :scheme_handlers
 
@@ -123,6 +128,7 @@ class Mechanize::HTTP::Agent
     @follow_meta_refresh_self = false
     @gzip_enabled             = true
     @history                  = Mechanize::History.new
+    @ignore_bad_chunking      = false
     @keep_alive               = true
     @max_file_buffer          = 100_000 # 5MB for response bodies
     @open_timeout             = nil
@@ -248,13 +254,20 @@ class Mechanize::HTTP::Agent
     response_body_io = nil
 
     # Send the request
-    response = connection.request(uri, request) { |res|
-      response_log res
+    begin
+      response = connection.request(uri, request) { |res|
+        response_log res
 
-      response_body_io = response_read res, request, uri
+        response_body_io = response_read res, request, uri
 
-      res
-    }
+        res
+      }
+    rescue Mechanize::ChunkedTerminationError => e
+      raise unless @ignore_bad_chunking
+
+      response = e.response
+      response_body_io = e.body_io
+    end
 
     hook_content_encoding response, uri, response_body_io
 
@@ -283,7 +296,7 @@ class Mechanize::HTTP::Agent
       log.debug("Got cached page") if log
       visited_page(uri) || page
     when Net::HTTPRedirection
-      response_redirect response, method, page, redirects, referer
+      response_redirect response, method, page, redirects, headers, referer
     when Net::HTTPUnauthorized
       response_authenticate(response, page, uri, request, headers, params,
                             referer)
@@ -402,20 +415,23 @@ class Mechanize::HTTP::Agent
     zio.finish
 
     return out_io
-  rescue Zlib::Error
-    log.error('unable to gunzip response, trying raw inflate') if log
+  rescue Zlib::Error => gz_error
+    log.warn "unable to gunzip response: #{gz_error} (#{gz_error.class})" if
+      log
 
     body_io.rewind
     body_io.read 10
 
     begin
+      log.warn "trying raw inflate on response" if log
       return inflate body_io, -Zlib::MAX_WBITS
     rescue Zlib::Error => e
-      log.error("unable to gunzip response: #{e}") if log
+      log.error "unable to inflate response: #{e} (#{e.class})" if log
       raise
     end
   ensure
-    zio.close if zio and not zio.closed?
+    # do not close a second time if we failed the first time
+    zio.close if zio and not (zio.closed? or gz_error)
     body_io.close unless body_io.closed?
   end
 
@@ -583,7 +599,13 @@ class Mechanize::HTTP::Agent
         if RUBY_VERSION >= "1.9.0"
           Mechanize::Util.uri_escape(match)
         else
-          sprintf('%%%X', match.unpack($KCODE == 'UTF8' ? 'U' : 'C')[0])
+          begin
+            sprintf('%%%X', match.unpack($KCODE == 'UTF8' ? 'U' : 'C').first)
+          rescue ArgumentError
+            # workaround for ruby 1.8 with -Ku but ISO-8859-1 characters in
+            # URIs.  See #227.  I can't wait to drop 1.8 support
+            sprintf('%%%X', match.unpack('C').first)
+          end
         end
       }
 
@@ -845,9 +867,7 @@ class Mechanize::HTTP::Agent
     content_length = response.content_length
 
     if use_tempfile? content_length then
-      body_io = Tempfile.new 'mechanize-raw'
-      body_io.unlink
-      body_io.binmode if defined? body_io.binmode
+      body_io = make_tempfile 'mechanize-raw'
     else
       body_io = StringIO.new
     end
@@ -860,9 +880,7 @@ class Mechanize::HTTP::Agent
         total += part.length
 
         if StringIO === body_io and use_tempfile? total then
-          new_io = Tempfile.new 'mechanize-raw'
-          new_io.unlink
-          new_io.binmode
+          new_io = make_tempfile 'mechanize-raw'
 
           new_io.write body_io.string
 
@@ -872,6 +890,13 @@ class Mechanize::HTTP::Agent
         body_io.write(part)
         log.debug("Read #{part.length} bytes (#{total} total)") if log
       }
+    rescue EOFError => e
+      # terminating CRLF might be missing, let the user check the document
+      raise unless response.chunked? and total.nonzero?
+
+      body_io.rewind
+      raise Mechanize::ChunkedTerminationError.new(e, response, body_io, uri,
+                                                   @context)
     rescue Net::HTTP::Persistent::Error => e
       body_io.rewind
       raise Mechanize::ResponseReadError.new(e, response, body_io, uri,
@@ -895,7 +920,8 @@ class Mechanize::HTTP::Agent
     body_io
   end
 
-  def response_redirect response, method, page, redirects, referer = current_page
+  def response_redirect(response, method, page, redirects, headers,
+                        referer = current_page)
     case @redirect_ok
     when true, :all
       # shortcut
@@ -915,7 +941,7 @@ class Mechanize::HTTP::Agent
     @history.push(page, page.uri)
     new_uri = resolve response['Location'].to_s, page
 
-    fetch new_uri, redirect_method, {}, [], referer, redirects + 1
+    fetch new_uri, redirect_method, headers, [], referer, redirects + 1
   end
 
   # :section: Robots
@@ -1089,10 +1115,7 @@ class Mechanize::HTTP::Agent
 
     until input_io.eof? do
       if StringIO === out_io and use_tempfile? out_io.size then
-        new_io = Tempfile.new name
-        new_io.unlink
-        new_io.binmode
-
+        new_io = make_tempfile name
         new_io.write out_io.string
         out_io = new_io
       end
@@ -1115,7 +1138,7 @@ class Mechanize::HTTP::Agent
       inflate.inflate chunk
     end
 
-    out_io.write inflate.finish
+    inflate.finish
 
     out_io
   ensure
@@ -1158,6 +1181,13 @@ class Mechanize::HTTP::Agent
     @http.proxy = proxy_uri
   end
 
+  def make_tempfile name
+    io = Tempfile.new name
+    io.unlink
+    io.binmode if io.respond_to? :binmode
+    io
+  end
+
   def use_tempfile? size
     return false unless @max_file_buffer
     return false unless size