mechanize 2.0.1 → 2.1.pre.1

data/CHANGELOG.rdoc

@@ -1,5 +1,87 @@
 = Mechanize CHANGELOG
 
+=== 2.1 / ??
+
+* Deprecations
+  * Mechanize#get no longer accepts an options hash.
+  * Mechanize::Util::to_native_charset has been removed.
+
+* Minor enhancements
+  * Mechanize now depends on net-http-persistent 2.3+.  This new version
+    brings idle timeouts to help with the dreaded "too many connection resets"
+    issue when POSTing to a closed connection.  Also, SSL connections will be
+    verified against the system certificate store by default.  Issue #123
+  * Added Mechanize#retry_change_requests to allow mechanize to retry POST and
+    other non-idempotent requests when you know it is safe to do so.  Issue
+    #123
+  * Mechanize can now stream files directly to disk without loading them into
+    memory first through Mechanize::Download, a pluggable parser for
+    downloading files.
+
+    All responses larger than Mechanize#max_file_buffer are downloaded to a
+    Tempfile.  For backwards compatibility Mechanize::File subclasses still
+    load the response body into memory.
+
+    To force all unknown content types to download to disk instead of memory
+    set:
+
+      agent.pluggable_parser.default = Mechanize::Download
+  * Added Mechanize#content_encoding_hooks which allow handling of
+    non-standard content encodings like "agzip".  Patch #125 by kitamomonga
+  * Added dom_class to elements and the element matcher like dom_id.  Patch
+    #156 by Dan Hansen.
+  * Added support for the HTML5 keygen form element.  See
+    http://dev.w3.org/html5/spec/Overview.html#the-keygen-element  Patch #157
+    by Victor Costan.
+  * Mechanize no longer follows meta refreshes that have no "url=" in the
+    content attribute to avoid infinite loops.  To follow a meta refresh to
+    the same page set Mechanize#follow_meta_refresh_self to true.  Issue #134
+    by Jo Hund.
+  * Updated 'Mac Safari' User-Agent alias to Safari 5.1.1.  'Mac Safari 4' can
+    be used for the old 'Mac Safari' alias.
+  * When given multiple HTTP authentication options mechanize can better pick
+    the strongest method.
+  * Improvements to HTTP authorization:
+    * mechanize raises Mechanize::UnathorizedError for 401 responses which is
+      a sublcass of Mechanize::ResponseCodeError.
+    * Added support for NTLM authentication, but this has not been tested.
+  * Mechanize::Cookie.new accepts attributes in a hash.
+  * Mechanize::CookieJar#<<(cookie) (alias: add!) is added.  Issue #139
+  * Different mechanize instances may now have different loggers.  Issue #122
+  * Mechanize now accepts a proxy port as a service name or number string.
+    Issue #167
+
+* Bug fixes
+  * Mechanize takes more care to avoid saving files with certain unsafe names.
+    You should still take care not to use mechanize to save files into your
+    home directory.  Issue #163.
+  * Mechanize#cookie_jar= works again.  Issue #126
+  * Mechanize now handles cookies just as most modern browsers do,
+    roughly based on RFC 6265.
+    * domain=.example.com (which is invalid) is considered identical to
+      domain=example.com.
+    * A cookie with domain=example.com is sent to host.sub.example.com
+      as well as host.example.com and example.com.
+    * A cookie with domain=TLD (no dots) is accepted and sent if the
+      host name is TLD, and rejected otherwise.  To retain compatibility
+      and convention, host/domain names starting with "local" are exempt
+      from this rule.
+    * A cookie with no domain attribute is only sent to the original
+      host.
+    * A cookie with an Effective TLD is rejected based on the public
+      suffix list. (cf. http://publicsuffix.org/)
+    * "Secure" cookies are not sent via non-https connection.
+    * Subdomain match is not performed against an IP address.
+    * It is recommended that you clear out existing cookie jars for
+      regeneration because previously saved cookies may not have been
+      parsed correctly.
+  * The original Referer value persists on redirection.  Issue #150
+  * Do not send a referer on a Refresh header based redirection.
+  * Fixed encoding error in tests when LANG=C.  Patch #142 by jinschoi.
+  * The order of items in a form submission now match the DOM order.  Patch
+    #129 by kitamomonga
+  * Fixed proxy example in EXAMPLE.  Issue #146 by NielsKSchjoedt
+
 === 2.0.1 / 2011-06-28
 
 Mechanize now uses minitest to avoid 1.9 vs 1.8 assertion availability in

data/EXAMPLES.rdoc

@@ -109,7 +109,7 @@ Beautiful Soup for that page.
   require 'mechanize'
 
   agent = Mechanize.new
-  agent.set_proxy('localhost', '8000')
+  agent.set_proxy 'localhost', 8000
   page = agent.get(ARGV[0])
   puts page.body
 

data/FAQ.rdoc

@@ -1,11 +1,11 @@
-Q: I keep getting an EOFError:
-  protocol.rb:133:in `sysread': end of file reached (EOFError)
-
-A:  Some people have experienced an EOFError during normal mechanize usage.
-    Most of the time this occurs because the remote website claims to support
-    keep alives, but does not implement them correctly.  Try turning off
-    keep alives on your mechanize object:
-
-      mech.keep_alive = false
+Q:: Why do I keep getting an EOFError?
 
+A:: For older versions of mechanize turning off keep_alive could help with the
+    problem, but mechanize now has more robust handling of persistent
+    connections.
 
+    Older versions of mechanize would raise an EOFError when a chunked body was
+    not terminated properly, a common bug of IIS servers.  Since 2.0
+    Mechanize::ResponseReadError is raised containing the original response and
+    body read so far so if the server is broken you can still retrieve the
+    entire content.

data/Manifest.txt

@@ -16,6 +16,7 @@ lib/mechanize.rb
 lib/mechanize/content_type_error.rb
 lib/mechanize/cookie.rb
 lib/mechanize/cookie_jar.rb
+lib/mechanize/download.rb
 lib/mechanize/element_matcher.rb
 lib/mechanize/file.rb
 lib/mechanize/file_connection.rb
@@ -27,16 +28,25 @@ lib/mechanize/form/button.rb
 lib/mechanize/form/check_box.rb
 lib/mechanize/form/field.rb
 lib/mechanize/form/file_upload.rb
+lib/mechanize/form/hidden.rb
 lib/mechanize/form/image_button.rb
+lib/mechanize/form/keygen.rb
 lib/mechanize/form/multi_select_list.rb
 lib/mechanize/form/option.rb
 lib/mechanize/form/radio_button.rb
+lib/mechanize/form/reset.rb
 lib/mechanize/form/select_list.rb
+lib/mechanize/form/submit.rb
+lib/mechanize/form/text.rb
+lib/mechanize/form/textarea.rb
 lib/mechanize/headers.rb
 lib/mechanize/history.rb
 lib/mechanize/http.rb
 lib/mechanize/http/agent.rb
-lib/mechanize/inspect.rb
+lib/mechanize/http/auth_challenge.rb
+lib/mechanize/http/auth_realm.rb
+lib/mechanize/http/content_disposition_parser.rb
+lib/mechanize/http/www_authenticate_parser.rb
 lib/mechanize/monkey_patch.rb
 lib/mechanize/page.rb
 lib/mechanize/page/base.rb
@@ -45,12 +55,15 @@ lib/mechanize/page/image.rb
 lib/mechanize/page/label.rb
 lib/mechanize/page/link.rb
 lib/mechanize/page/meta_refresh.rb
+lib/mechanize/parser.rb
 lib/mechanize/pluggable_parsers.rb
 lib/mechanize/redirect_limit_reached_error.rb
 lib/mechanize/redirect_not_get_or_head_error.rb
 lib/mechanize/response_code_error.rb
 lib/mechanize/response_read_error.rb
 lib/mechanize/robots_disallowed_error.rb
+lib/mechanize/test_case.rb
+lib/mechanize/unauthorized_error.rb
 lib/mechanize/unsupported_scheme_error.rb
 lib/mechanize/util.rb
 test/data/htpasswd
@@ -58,7 +71,6 @@ test/data/server.crt
 test/data/server.csr
 test/data/server.key
 test/data/server.pem
-test/helper.rb
 test/htdocs/alt_text.html
 test/htdocs/bad_form_test.html
 test/htdocs/button.jpg
@@ -71,31 +83,24 @@ test/htdocs/form_multi_select.html
 test/htdocs/form_multival.html
 test/htdocs/form_no_action.html
 test/htdocs/form_no_input_name.html
+test/htdocs/form_order_test.html
 test/htdocs/form_select.html
-test/htdocs/form_select_all.html
-test/htdocs/form_select_none.html
-test/htdocs/form_select_noopts.html
 test/htdocs/form_set_fields.html
 test/htdocs/form_test.html
 test/htdocs/frame_referer_test.html
 test/htdocs/frame_test.html
 test/htdocs/google.html
-test/htdocs/iframe_test.html
 test/htdocs/index.html
 test/htdocs/link with space.html
 test/htdocs/meta_cookie.html
 test/htdocs/no_title_test.html
-test/htdocs/nofollow.html
 test/htdocs/noindex.html
-test/htdocs/norobots.html
 test/htdocs/rails_3_encoding_hack_form_test.html
-test/htdocs/rel_nofollow.html
 test/htdocs/relative/tc_relative_links.html
 test/htdocs/robots.html
 test/htdocs/robots.txt
 test/htdocs/tc_bad_charset.html
 test/htdocs/tc_bad_links.html
-test/htdocs/tc_base_images.html
 test/htdocs/tc_base_link.html
 test/htdocs/tc_blank_form.html
 test/htdocs/tc_charset.html
@@ -104,69 +109,51 @@ test/htdocs/tc_encoded_links.html
 test/htdocs/tc_field_precedence.html
 test/htdocs/tc_follow_meta.html
 test/htdocs/tc_form_action.html
-test/htdocs/tc_images.html
 test/htdocs/tc_links.html
 test/htdocs/tc_meta_in_body.html
-test/htdocs/tc_no_attributes.html
 test/htdocs/tc_pretty_print.html
-test/htdocs/tc_radiobuttons.html
 test/htdocs/tc_referer.html
 test/htdocs/tc_relative_links.html
 test/htdocs/tc_textarea.html
-test/htdocs/test_bad_encoding.html
 test/htdocs/test_click.html
 test/htdocs/unusual______.html
-test/servlets.rb
-test/ssl_server.rb
-test/test_cookies.rb
-test/test_form_action.rb
-test/test_form_as_hash.rb
-test/test_form_button.rb
-test/test_frames.rb
-test/test_headers.rb
-test/test_history.rb
-test/test_history_added.rb
-test/test_html_unscape_forms.rb
-test/test_if_modified_since.rb
-test/test_images.rb
 test/test_mechanize.rb
 test/test_mechanize_cookie.rb
 test/test_mechanize_cookie_jar.rb
+test/test_mechanize_download.rb
 test/test_mechanize_file.rb
+test/test_mechanize_file_connection.rb
 test/test_mechanize_file_request.rb
-test/test_mechanize_file_response.rb
+test/test_mechanize_file_saver.rb
 test/test_mechanize_form.rb
 test/test_mechanize_form_check_box.rb
 test/test_mechanize_form_encoding.rb
 test/test_mechanize_form_field.rb
+test/test_mechanize_form_file_upload.rb
 test/test_mechanize_form_image_button.rb
+test/test_mechanize_form_keygen.rb
+test/test_mechanize_form_multi_select_list.rb
+test/test_mechanize_form_option.rb
+test/test_mechanize_form_radio_button.rb
+test/test_mechanize_form_select_list.rb
 test/test_mechanize_form_textarea.rb
+test/test_mechanize_headers.rb
+test/test_mechanize_history.rb
 test/test_mechanize_http_agent.rb
+test/test_mechanize_http_auth_challenge.rb
+test/test_mechanize_http_auth_realm.rb
+test/test_mechanize_http_content_disposition_parser.rb
+test/test_mechanize_http_www_authenticate_parser.rb
 test/test_mechanize_link.rb
+test/test_mechanize_page.rb
 test/test_mechanize_page_encoding.rb
+test/test_mechanize_page_frame.rb
 test/test_mechanize_page_link.rb
 test/test_mechanize_page_meta_refresh.rb
+test/test_mechanize_parser.rb
+test/test_mechanize_pluggable_parser.rb
+test/test_mechanize_redirect_limit_reached_error.rb
 test/test_mechanize_redirect_not_get_or_head_error.rb
 test/test_mechanize_subclass.rb
 test/test_mechanize_util.rb
 test/test_multi_select.rb
-test/test_no_attributes.rb
-test/test_option.rb
-test/test_pluggable_parser.rb
-test/test_post_form.rb
-test/test_pretty_print.rb
-test/test_radiobutton.rb
-test/test_redirect_limit_reached.rb
-test/test_referer.rb
-test/test_relative_links.rb
-test/test_request.rb
-test/test_response_code.rb
-test/test_robots.rb
-test/test_save_file.rb
-test/test_scheme.rb
-test/test_select.rb
-test/test_select_all.rb
-test/test_select_none.rb
-test/test_select_noopts.rb
-test/test_set_fields.rb
-test/test_ssl_server.rb

data/README.rdoc

@@ -43,6 +43,7 @@ Copyright (c) 2006-2011:
 Copyright (c) 2011:
 
 * {Eric Hodel}[http://blog.segment7.net] (drbrain@segment7.net)
+* {Akinori MUSHA}[http://blog.akinori.org] (knu@idaemons.org)
 
 This library comes with a shameless plug for employing me
 (Aaron[http://tenderlovemaking.com/]) programming Ruby, my favorite language!
@@ -51,7 +52,7 @@ This library comes with a shameless plug for employing me
 
 This library was heavily influenced by its namesake in the perl world.  A big
 thanks goes to Andy Lester (andy@petdance.com), the author of the original
-perl Mechanize which is available here[http://search.cpan.org/~petdance/WWW-Mechanize-1.20/].  Ruby Mechanize would not be around without you!
+perl Mechanize which is available here[http://search.cpan.org/~petdance/WWW-Mechanize/].  Ruby Mechanize would not be around without you!
 
 Thank you to Michael Neumann for starting the Ruby version.  Thanks to everyone
 who's helped out in various ways.  Finally, thank you to the people using this

data/Rakefile

@@ -4,10 +4,11 @@ require 'hoe'
 Hoe.plugin :git
 Hoe.plugin :minitest
 
-Hoe.spec 'mechanize' do
+hoe = Hoe.spec 'mechanize' do
   developer 'Eric Hodel',      'drbrain@segment7.net'
   developer 'Aaron Patterson', 'aaronp@rubyforge.org'
   developer 'Mike Dalessio',   'mike.dalessio@gmail.com'
+  developer 'Akinori MUSHA',   'knu@idaemons.org'
 
   self.readme_file      = 'README.rdoc'
   self.history_file     = 'CHANGELOG.rdoc'
@@ -15,10 +16,12 @@ Hoe.spec 'mechanize' do
 
   rdoc_locations << 'drbrain@rubyforge.org:/var/www/gforge-projects/mechanize/'
 
-  self.extra_deps << ['nokogiri',             '~> 1.4']
-  self.extra_deps << ['net-http-persistent',  '~> 1.8']
   self.extra_deps << ['net-http-digest_auth', '~> 1.1', '>= 1.1.1']
+  self.extra_deps << ['net-http-persistent',  '~> 2.3', '>= 2.3.2']
+  self.extra_deps << ['nokogiri',             '~> 1.4']
+  self.extra_deps << ['ntlm-http',            '~> 0.1', '>= 0.1.1']
   self.extra_deps << ['webrobots',            '~> 0.0', '>= 0.0.9']
+  self.extra_deps << ['domain_name',          '~> 0.5', '>= 0.5.1']
 
   self.spec_extras[:required_ruby_version] = '>= 1.8.7'
 end
@@ -34,3 +37,13 @@ task('ssl_cert') do |p|
   sh "mv server.key server.csr server.crt server.pem test/data/"
   sh "rm server.key.org"
 end
+
+desc 'Install deps for travis to work around Hoe/RubyGems bug'
+task 'travis_deps' do
+  hoe.spec.dependencies.each do |dep|
+    first_requirement = dep.requirement.requirements.first.join ' '
+    system('gem', 'install', dep.name, '-v', first_requirement,
+           '--no-rdoc', '--no-ri')
+  end
+end
+

data/lib/mechanize.rb

@@ -7,33 +7,42 @@ require 'net/http/persistent'
 require 'nkf'
 require 'nokogiri'
 require 'openssl'
+require 'pp'
 require 'stringio'
 require 'uri'
 require 'webrick/httputils'
 require 'zlib'
 
-# = Synopsis
-# The Mechanize library is used for automating interaction with a website.  It
-# can follow links, and submit forms.  Form fields can be populated and
+##
+# The Mechanize library is used for automating interactions with a website.  It
+# can follow links and submit forms.  Form fields can be populated and
 # submitted.  A history of URL's is maintained and can be queried.
 #
 # == Example
-#  require 'rubygems'
-#  require 'mechanize'
-#  require 'logger'
 #
-#  agent = Mechanize.new { |a| a.log = Logger.new("mech.log") }
-#  agent.user_agent_alias = 'Mac Safari'
-#  page = agent.get("http://www.google.com/")
-#  search_form = page.form_with(:name => "f")
-#  search_form.field_with(:name => "q").value = "Hello"
-#  search_results = agent.submit(search_form)
-#  puts search_results.body
+#   require 'mechanize'
+#   require 'logger'
+#
+#   agent = Mechanize.new
+#   agent.log = Logger.new "mech.log"
+#   agent.user_agent_alias = 'Mac Safari'
+#
+#   page = agent.get "http://www.google.com/"
+#   search_form = page.form_with :name => "f"
+#   search_form.field_with(:name => "q").value = "Hello"
+#
+#   search_results = agent.submit search_form
+#   puts search_results.body
+
 class Mechanize
 
   ##
   # The version of Mechanize you are using.
-  VERSION = '2.0.1'
+
+  VERSION = '2.1'
+
+  ##
+  # Base mechanize error class
 
   class Error < RuntimeError
   end
@@ -44,517 +53,916 @@ class Mechanize
                    "#{RUBY_VERSION}dev#{RUBY_REVISION}"
                  end
 
-  # HTTP/1.1 keep-alives are always active.  This does nothing.
-  attr_accessor :keep_alive
+  ##
+  # Supported User-Agent aliases for use with user_agent_alias=.  The
+  # description in parenthesis is for informative purposes and is not part of
+  # the alias name.
+  #
+  # * Linux Firefox (3.6.1)
+  # * Linux Konqueror (3)
+  # * Linux Mozilla
+  # * Mac Firefox (3.6)
+  # * Mac Mozilla
+  # * Mac Safari (5)
+  # * Mac Safari 4
+  # * Mechanize (default)
+  # * Windows IE 6
+  # * Windows IE 7
+  # * Windows IE 8
+  # * Windows IE 9
+  # * Windows Mozilla
+  # * iPhone (3.0)
+  #
+  # Example:
+  #
+  #   agent = Mechanize.new
+  #   agent.user_agent_alias = 'Mac Safari'
+
+  AGENT_ALIASES = {
+    'Mechanize' => "Mechanize/#{VERSION} Ruby/#{ruby_version} (http://github.com/tenderlove/mechanize/)",
+    'Linux Firefox' => 'Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.2.1) Gecko/20100122 firefox/3.6.1',
+    'Linux Konqueror' => 'Mozilla/5.0 (compatible; Konqueror/3; Linux)',
+    'Linux Mozilla' => 'Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.4) Gecko/20030624',
+    'Mac FireFox' => 'Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.6; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6',
+    'Mac Mozilla' => 'Mozilla/5.0 (Macintosh; U; PPC Mac OS X Mach-O; en-US; rv:1.4a) Gecko/20030401',
+    'Mac Safari 4' => 'Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_6_2; de-at) AppleWebKit/531.21.8 (KHTML, like Gecko) Version/4.0.4 Safari/531.21.10',
+    'Mac Safari' => 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_2) AppleWebKit/534.51.22 (KHTML, like Gecko) Version/5.1.1 Safari/534.51.22',
+    'Windows IE 6' => 'Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)',
+    'Windows IE 7' => 'Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; .NET CLR 1.1.4322; .NET CLR 2.0.50727)',
+    'Windows IE 8' => 'Mozilla/5.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; .NET CLR 1.1.4322; .NET CLR 2.0.50727)',
+    'Windows IE 9' => 'Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0)',
+    'Windows Mozilla' => 'Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.4b) Gecko/20030516 Mozilla Firebird/0.6',
+    'iPhone' => 'Mozilla/5.0 (iPhone; U; CPU like Mac OS X; en) AppleWebKit/420+ (KHTML, like Gecko) Version/3.0 Mobile/1C28 Safari/419.3',
+  }
+
+  def self.inherited(child) # :nodoc:
+    child.html_parser ||= html_parser
+    child.log ||= log
+    super
+  end
+
+  ##
+  # Creates a new mechanize instance.  If a block is given, the created
+  # instance is yielded to the block for setting up pre-connection state such
+  # as SSL parameters or proxies:
+  #
+  #   agent = Mechanize.new do |a|
+  #     a.proxy_host = 'proxy.example'
+  #     a.proxy_port = 8080
+  #   end
+
+  def initialize
+    @agent = Mechanize::HTTP::Agent.new
+    @agent.context = self
+    @log = nil
+
+    # attr_accessors
+    @agent.user_agent = AGENT_ALIASES['Mechanize']
+    @watch_for_set    = nil
+    @history_added    = nil
+
+    # attr_readers
+    @pluggable_parser = PluggableParser.new
+
+    @keep_alive_time  = 0
+
+    # Proxy
+    @proxy_addr = nil
+    @proxy_port = nil
+    @proxy_user = nil
+    @proxy_pass = nil
+
+    @html_parser = self.class.html_parser
+
+    @default_encoding = nil
+    @force_default_encoding = false
+
+    yield self if block_given?
+
+    @agent.set_proxy @proxy_addr, @proxy_port, @proxy_user, @proxy_pass
+    @agent.set_http
+  end
+
+  # :section: History
+  #
+  # Methods for navigating and controlling history
+
+  ##
+  # Equivalent to the browser back button.  Returns the previous page visited.
+
+  def back
+    @agent.history.pop
+  end
+
+  ##
+  # Returns the latest page loaded by Mechanize
+
+  def current_page
+    @agent.current_page
+  end
+
+  alias page current_page
+
+  ##
+  # The history of this mechanize run
+
+  def history
+    @agent.history
+  end
+
+  ##
+  # Maximum number of items allowed in the history.
+
+  def max_history
+    @agent.history.max_size
+  end
+
+  ##
+  # Sets the maximum number of items allowed in the history to +length+.
+
+  def max_history= length
+    @agent.history.max_size = length
+  end
+
+  ##
+  # Returns a visited page for the +url+ passed in, otherwise nil
+
+  def visited? url
+    url = url.href if url.respond_to? :href
+
+    @agent.visited_page url
+  end
+
+  ##
+  # Returns whether or not a url has been visited
+
+  alias visited_page visited?
+
+  # :section: Hooks
+  #
+  # Hooks into the operation of mechanize
+
+  ##
+  # A list of hooks to call before reading response header 'content-encoding'.
+  #
+  # The hook is called with the agent making the request, the URI of the
+  # request, the response an IO containing the response body.
+
+  def content_encoding_hooks
+    @agent.content_encoding_hooks
+  end
+
+  ##
+  # Callback which is invoked with the page that was added to history.
+
+  attr_accessor :history_added
+
+  ##
+  # A list of hooks to call after retrieving a response.  Hooks are called with
+  # the agent and the response returned.
+
+  def post_connect_hooks
+    @agent.post_connect_hooks
+  end
+
+  ##
+  # A list of hooks to call before making a request.  Hooks are called with
+  # the agent and the request to be performed.
+
+  def pre_connect_hooks
+    @agent.pre_connect_hooks
+  end
+
+  # :section: Requests
+  #
+  # Methods for making HTTP requests
+
+  ##
+  # If the parameter is a string, finds the button or link with the
+  # value of the string on the current page and clicks it.  Otherwise, clicks
+  # the Mechanize::Page::Link object passed in.  Returns the page fetched.
+
+  def click link
+    case link
+    when Page::Link then
+      referer = link.page || current_page()
+      if @agent.robots
+        if (referer.is_a?(Page) and referer.parser.nofollow?) or
+           link.rel?('nofollow') then
+          raise RobotsDisallowedError.new(link.href)
+        end
+      end
+      if link.rel?('noreferrer')
+        href = @agent.resolve(link.href, link.page || current_page)
+        referer = Page.new(nil, {'content-type'=>'text/html'})
+      else
+        href = link.href
+      end
+      get href, [], referer
+    when String, Regexp then
+      if real_link = page.link_with(:text => link)
+        click real_link
+      else
+        button = nil
+        form = page.forms.find do |f|
+          button = f.button_with(:value => link)
+          button.is_a? Form::Submit
+        end
+        submit form, button if form
+      end
+    else
+      referer = current_page()
+      href = link.respond_to?(:href) ? link.href :
+        (link['href'] || link['src'])
+      get href, [], referer
+    end
+  end
+
+  ##
+  # DELETE +uri+ with +query_params+, and setting +headers+:
+  #
+  #   delete('http://example/', {'q' => 'foo'}, {})
+
+  def delete(uri, query_params = {}, headers = {})
+    page = @agent.fetch(uri, :delete, headers, query_params)
+    add_to_history(page)
+    page
+  end
+
+  ##
+  # GET the +uri+ with the given request +parameters+, +referer+ and
+  # +headers+.
+  #
+  # The +referer+ may be a URI or a page.
+
+  def get(uri, parameters = [], referer = nil, headers = {})
+    method = :get
+
+    referer ||=
+      if uri.to_s =~ %r{\Ahttps?://}
+        Page.new(nil, {'content-type'=>'text/html'})
+      else
+        current_page || Page.new(nil, {'content-type'=>'text/html'})
+      end
+
+    # FIXME: Huge hack so that using a URI as a referer works.  I need to
+    # refactor everything to pass around URIs but still support
+    # Mechanize::Page#base
+    unless referer.is_a?(Mechanize::File)
+      referer = referer.is_a?(String) ?
+      Page.new(URI.parse(referer), {'content-type' => 'text/html'}) :
+        Page.new(referer, {'content-type' => 'text/html'})
+    end
+
+    # fetch the page
+    headers ||= {}
+    page = @agent.fetch uri, method, headers, parameters, referer
+    add_to_history(page)
+    yield page if block_given?
+    page
+  end
+
+  ##
+  # GET +url+ and return only its contents
+
+  def get_file(url)
+    get(url).body
+  end
+
+  ##
+  # HEAD +uri+ with +query_params+, and setting +headers+:
+  #
+  #   head('http://example/', {'q' => 'foo'}, {})
+
+  def head(uri, query_params = {}, headers = {})
+    # fetch the page
+    page = @agent.fetch(uri, :head, headers, query_params)
+    yield page if block_given?
+    page
+  end
+
+  ##
+  # POST to the given +uri+ with the given +query+.  The query is specified by
+  # either a string, or a list of key-value pairs represented by a hash or an
+  # array of arrays.
+  #
+  # Examples:
+  #   agent.post 'http://example.com/', "foo" => "bar"
+  #
+  #   agent.post 'http://example.com/', [%w[foo bar]]
+  #
+  #   agent.post('http://example.com/', "<message>hello</message>",
+  #              'Content-Type' => 'application/xml')
+
+  def post(uri, query={}, headers={})
+    return request_with_entity(:post, uri, query, headers) if String === query
+
+    node = {}
+    # Create a fake form
+    class << node
+      def search(*args); []; end
+    end
+    node['method'] = 'POST'
+    node['enctype'] = 'application/x-www-form-urlencoded'
+
+    form = Form.new(node)
+
+    query.each { |k, v|
+      if v.is_a?(IO)
+        form.enctype = 'multipart/form-data'
+        ul = Form::FileUpload.new({'name' => k.to_s},::File.basename(v.path))
+        ul.file_data = v.read
+        form.file_uploads << ul
+      else
+        form.fields << Form::Field.new({'name' => k.to_s},v)
+      end
+    }
+    post_form(uri, form, headers)
+  end
+
+  ##
+  # PUT to +uri+ with +entity+, and setting +headers+:
+  #
+  #   put('http://example/', 'new content', {'Content-Type' => 'text/plain'})
+
+  def put(uri, entity, headers = {})
+    request_with_entity(:put, uri, entity, headers)
+  end
+
+  ##
+  # Makes an HTTP request to +url+ using HTTP method +verb+.  +entity+ is used
+  # as the request body, if allowed.
+
+  def request_with_entity(verb, uri, entity, headers = {})
+    cur_page = current_page || Page.new(nil, {'content-type'=>'text/html'})
+
+    headers = {
+      'Content-Type' => 'application/octet-stream',
+      'Content-Length' => entity.size.to_s,
+    }.update headers
+
+    page = @agent.fetch uri, verb, headers, [entity], cur_page
+    add_to_history(page)
+    page
+  end
+
+  ##
+  # Submits +form+ with an optional +button+.
+  #
+  # Without a button:
+  #
+  #   page = agent.get('http://example.com')
+  #   agent.submit(page.forms.first)
+  #
+  # With a button:
+  #
+  #   agent.submit(page.forms.first, page.forms.first.buttons.first)
+
+  def submit(form, button=nil, headers={})
+    form.add_button_to_query(button) if button
+
+    case form.method.upcase
+    when 'POST'
+      post_form(form.action, form, headers)
+    when 'GET'
+      get(form.action.gsub(/\?[^\?]*$/, ''),
+          form.build_query,
+          form.page,
+          headers)
+    else
+      raise ArgumentError, "unsupported method: #{form.method.upcase}"
+    end
+  end
+
+  ##
+  # Runs given block, then resets the page history as it was before. self is
+  # given as a parameter to the block.  Returns the value of the block.
+
+  def transact
+    history_backup = @agent.history.dup
+    begin
+      yield self
+    ensure
+      @agent.history = history_backup
+    end
+  end
+
+  # :section: Settings
+  #
+  # Settings that adjust how mechanize makes HTTP requests including timeouts,
+  # keep-alives, compression, redirects and headers.
+
+  @html_parser = Nokogiri::HTML
+
+  class << self
+
+    ##
+    # Default HTML parser for all mechanize instances
+    #
+    #   Mechanize.html_parser = Nokogiri::XML
+
+    attr_accessor :html_parser
+
+    ##
+    # Default logger for all mechanize instances
+    #
+    #   Mechanize.log = Logger.new $stderr
+
+    attr_accessor :log
+
+  end
+
+  ##
+  # A default encoding name used when parsing HTML parsing.  When set it is
+  # used after any other encoding.  The default is nil.
+
+  attr_accessor :default_encoding
+
+  ##
+  # Overrides the encodings given by the HTTP server and the HTML page with
+  # the default_encoding when set to true.
+
+  attr_accessor :force_default_encoding
+
+  ##
+  # The HTML parser to be used when parsing documents
+
+  attr_accessor :html_parser
 
+  ##
   # HTTP/1.0 keep-alive time.  This is no longer supported by mechanize as it
   # now uses net-http-persistent which only supports HTTP/1.1 persistent
   # connections
+
   attr_accessor :keep_alive_time
 
   ##
-  # User Agent aliases
+  # The HTTP proxy address
 
-  AGENT_ALIASES = {
-    'Windows IE 6' => 'Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)',
-    'Windows IE 7' => 'Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; .NET CLR 1.1.4322; .NET CLR 2.0.50727)',
-    'Windows Mozilla' => 'Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.4b) Gecko/20030516 Mozilla Firebird/0.6',
-    'Mac Safari' => 'Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_6_2; de-at) AppleWebKit/531.21.8 (KHTML, like Gecko) Version/4.0.4 Safari/531.21.10',
-    'Mac FireFox' => 'Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.6; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6',
-    'Mac Mozilla' => 'Mozilla/5.0 (Macintosh; U; PPC Mac OS X Mach-O; en-US; rv:1.4a) Gecko/20030401',
-    'Linux Mozilla' => 'Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.4) Gecko/20030624',
-    'Linux Firefox' => 'Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.2.1) Gecko/20100122 firefox/3.6.1',
-    'Linux Konqueror' => 'Mozilla/5.0 (compatible; Konqueror/3; Linux)',
-    'iPhone' => 'Mozilla/5.0 (iPhone; U; CPU like Mac OS X; en) AppleWebKit/420+ (KHTML, like Gecko) Version/3.0 Mobile/1C28 Safari/419.3',
-    'Mechanize' => "Mechanize/#{VERSION} Ruby/#{ruby_version} (http://github.com/tenderlove/mechanize/)"
-  }
+  attr_reader :proxy_addr
 
-  # A Mechanize::CookieJar which stores cookies
+  ##
+  # The HTTP proxy password
 
-  def cookie_jar
-    @agent.cookie_jar
-  end
+  attr_reader :proxy_pass
 
-  def cookie_jar= cookie_jar
-    @agent.cookie_jar = cookie_jar
-  end
+  ##
+  # The HTTP proxy port
 
-  # Length of time to wait until a connection is opened in seconds
-  def open_timeout
-    @agent.open_timeout
-  end
+  attr_reader :proxy_port
 
-  def open_timeout= open_timeout
-    @agent.open_timeout = open_timeout
-  end
+  ##
+  # The HTTP proxy username
 
-  # Length of time to attempt to read data from the server
-  def read_timeout
-    @agent.read_timeout
-  end
+  attr_reader :proxy_user
 
-  def read_timeout= read_timeout
-    @agent.read_timeout = read_timeout
-  end
+  ##
+  # Sets the user and password to be used for HTTP authentication.
 
-  # The identification string for the client initiating a web request
-  def user_agent
-    @agent.user_agent
+  def auth(user, password)
+    @agent.user     = user
+    @agent.password = password
   end
 
-  # The value of watch_for_set is passed to pluggable parsers for retrieved
-  # content
-  attr_accessor :watch_for_set
+  alias basic_auth auth
 
-  # Path to an OpenSSL server certificate file
-  def ca_file
-    @agent.ca_file
-  end
+  ##
+  # Are If-Modified-Since conditional requests enabled?
 
-  def ca_file= ca_file
-    @agent.ca_file = ca_file
+  def conditional_requests
+    @agent.conditional_requests
   end
 
-  def certificate
-    @agent.certificate
-  end
+  ##
+  # Disables If-Modified-Since conditional requests (enabled by default)
 
-  # An OpenSSL private key or the path to a private key
-  def key
-    @agent.key
+  def conditional_requests= enabled
+    @agent.conditional_requests = enabled
   end
 
-  def key= key
-    @agent.key = key
+  ##
+  # A Mechanize::CookieJar which stores cookies
+
+  def cookie_jar
+    @agent.cookie_jar
   end
 
-  # An OpenSSL client certificate or the path to a certificate file.
-  def cert
-    @agent.cert
+  ##
+  # Replaces the cookie jar with +cookie_jar+
+
+  def cookie_jar= cookie_jar
+    @agent.cookie_jar = cookie_jar
   end
 
-  def cert= cert
-    @agent.cert = cert
+  ##
+  # Returns a list of cookies stored in the cookie jar.
+
+  def cookies
+    @agent.cookie_jar.to_a
   end
 
-  # OpenSSL key password
-  def pass
-    @agent.pass
+  ##
+  # Follow HTML meta refresh and HTTP Refresh headers.  If set to +:anywhere+
+  # meta refresh tags outside of the head element will be followed.
+
+  def follow_meta_refresh
+    @agent.follow_meta_refresh
   end
 
-  def pass= pass
-    @agent.pass = pass
+  ##
+  # Controls following of HTML meta refresh and HTTP Refresh headers in
+  # responses.
+
+  def follow_meta_refresh= follow
+    @agent.follow_meta_refresh = follow
   end
 
-  # Controls how this agent deals with redirects.  The following values are
-  # allowed:
+  ##
+  # Follow an HTML meta refresh and HTTP Refresh headers that have no "url="
+  # in the content attribute.
   #
-  # :all, true:: All 3xx redirects are followed (default)
-  # :permanent:: Only 301 Moved Permanantly redirects are followed
-  # false:: No redirects are followed
+  # Defaults to false to prevent infinite refresh loops.
 
-  def redirect_ok
-    @agent.redirect_ok
+  def follow_meta_refresh_self
+    @agent.follow_meta_refresh_self
   end
 
-  def redirect_ok= follow
-    @agent.redirect_ok = follow
+  ##
+  # Alters the following of HTML meta refresh and HTTP Refresh headers that
+  # point to the same page.
+
+  def follow_meta_refresh_self= follow
+    @agent.follow_meta_refresh_self = follow
   end
 
+  ##
+  # Is gzip compression of responses enabled?
+
   def gzip_enabled
     @agent.gzip_enabled
   end
 
+  ##
   # Disables HTTP/1.1 gzip compression (enabled by default)
+
   def gzip_enabled=enabled
     @agent.gzip_enabled = enabled
   end
 
-  def conditional_requests
-    @agent.conditional_requests
-  end
+  ##
+  # Connections that have not been used in this many seconds will be reset.
 
-  # Disables If-Modified-Since conditional requests (enabled by default)
-  def conditional_requests= enabled
-    @agent.conditional_requests = enabled
+  def idle_timeout
+    @agent.idle_timeout
   end
 
-  # Follow HTML meta refresh.  If set to +:anywhere+ meta refresh tags outside
-  # of the head element will be followed.
-  def follow_meta_refresh
-    @agent.follow_meta_refresh
+  # Sets the idle timeout to +idle_timeout+.  The default timeout is 5
+  # seconds.  If you experience "too many connection resets", reducing this
+  # value may help.
+
+  def idle_timeout= idle_timeout
+    @agent.idle_timeout = idle_timeout
   end
 
-  def follow_meta_refresh= follow
-    @agent.follow_meta_refresh = follow
+  ##
+  # Are HTTP/1.1 keep-alive connections enabled?
+
+  def keep_alive
+    @agent.keep_alive
   end
 
-  # A callback for additional certificate verification.  See
-  # OpenSSL::SSL::SSLContext#verify_callback
+  ##
+  # Disable HTTP/1.1 keep-alive connections if +enable+ is set to false.  If
+  # you are experiencing "too many connection resets" errors setting this to
+  # false will eliminate them.
   #
-  # The callback can be used for debugging or to ignore errors by always
-  # returning +true+.  Specifying nil uses the default method that was valid
-  # when the SSLContext was created
-  def verify_callback
-    @agent.verify_callback
+  # You should first investigate reducing idle_timeout.
+
+  def keep_alive= enable
+    @agent.keep_alive = enable
   end
 
-  def verify_callback= verify_callback
-    @agent.verify_callback = verify_callback
+  ##
+  # The current logger.  If no logger has been set Mechanize.log is used.
+
+  def log
+    @log || Mechanize.log
   end
 
-  attr_accessor :history_added
+  ##
+  # Sets the +logger+ used by this instance of mechanize
 
-  def redirection_limit
-    @agent.redirection_limit
+  def log= logger
+    @log = logger
   end
 
-  def redirection_limit= limit
-    @agent.redirection_limit = limit
+  ##
+  # Responses larger than this will be written to a Tempfile instead of stored
+  # in memory.  The default is 10240 bytes
+
+  def max_file_buffer
+    @agent.max_file_buffer
   end
 
-  def scheme_handlers
-    @agent.scheme_handlers
+  ##
+  # Sets the maximum size of a response body that will be stored in memory to
+  # +bytes+
+
+  def max_file_buffer= bytes
+    @agent.max_file_buffer = bytes
   end
 
-  def scheme_handlers= scheme_handlers
-    @agent.scheme_handlers = scheme_handlers
+  ##
+  # Length of time to wait until a connection is opened in seconds
+
+  def open_timeout
+    @agent.open_timeout
   end
 
-  # A hash of custom request headers
-  def request_headers
-    @agent.request_headers
+  ##
+  # Sets the connection open timeout to +open_timeout+
+
+  def open_timeout= open_timeout
+    @agent.open_timeout = open_timeout
   end
 
-  def request_headers= request_headers
-    @agent.request_headers = request_headers
+  ##
+  # Length of time to wait for data from the server
+
+  def read_timeout
+    @agent.read_timeout
   end
 
-  # Proxy settings
-  attr_reader :proxy_addr
-  attr_reader :proxy_pass
-  attr_reader :proxy_port
-  attr_reader :proxy_user
+  ##
+  # Sets the timeout for each chunk of data read from the server to
+  # +read_timeout+.  A single request may read many chunks of data.
 
-  # The HTML parser to be used when parsing documents
-  attr_accessor :html_parser
+  def read_timeout= read_timeout
+    @agent.read_timeout = read_timeout
+  end
 
-  attr_reader :agent # :nodoc:
+  ##
+  # Controls how mechanize deals with redirects.  The following values are
+  # allowed:
+  #
+  # :all, true:: All 3xx redirects are followed (default)
+  # :permanent:: Only 301 Moved Permanantly redirects are followed
+  # false:: No redirects are followed
 
-  def history
-    @agent.history
+  def redirect_ok
+    @agent.redirect_ok
   end
 
-  attr_reader :pluggable_parser
+  alias follow_redirect? redirect_ok
 
-  # A list of hooks to call after retrieving a response.  Hooks are called with
-  # the agent and the response returned.
+  ##
+  # Sets the mechanize redirect handling policy.  See redirect_ok for allowed
+  # values
 
-  def post_connect_hooks
-    @agent.post_connect_hooks
+  def redirect_ok= follow
+    @agent.redirect_ok = follow
   end
 
-  # A list of hooks to call before making a request.  Hooks are called with
-  # the agent and the request to be performed.
+  ##
+  # Maximum number of redirections to follow
 
-  def pre_connect_hooks
-    @agent.pre_connect_hooks
+  def redirection_limit
+    @agent.redirection_limit
   end
 
-  alias follow_redirect? redirect_ok
+  ##
+  # Sets the maximum number of redirections to follow to +limit+
 
-  @html_parser = Nokogiri::HTML
-  class << self
-    attr_accessor :html_parser, :log
+  def redirection_limit= limit
+    @agent.redirection_limit = limit
+  end
 
-    def inherited(child)
-      child.html_parser ||= html_parser
-      child.log ||= log
-      super
-    end
+  ##
+  # A hash of custom request headers that will be sent on every request
+
+  def request_headers
+    @agent.request_headers
   end
 
-  # A default encoding name used when parsing HTML parsing.  When set it is
-  # used after any other encoding.  The default is nil.
+  ##
+  # Replaces the custom request headers that will be sent on every request
+  # with +request_headers+
 
-  attr_accessor :default_encoding
+  def request_headers= request_headers
+    @agent.request_headers = request_headers
+  end
 
-  # Overrides the encodings given by the HTTP server and the HTML page with
-  # the default_encoding when set to true.
-  attr_accessor :force_default_encoding
+  ##
+  # Retry POST and other non-idempotent requests.  See RFC 2616 9.1.2.
 
-  def initialize
-    @agent = Mechanize::HTTP::Agent.new
-    @agent.context = self
+  def retry_change_requests
+    @agent.retry_change_requests
+  end
 
-    # attr_accessors
-    @agent.user_agent = AGENT_ALIASES['Mechanize']
-    @watch_for_set    = nil
-    @history_added    = nil
+  ##
+  # When setting +retry_change_requests+ to true you are stating that, for all
+  # the URLs you access with mechanize, making POST and other non-idempotent
+  # requests is safe and will not cause data duplication or other harmful
+  # results.
+  #
+  # If you are experiencing "too many connection resets" errors you should
+  # instead investigate reducing the idle_timeout or disabling keep_alive
+  # connections.
 
-    # attr_readers
-    @pluggable_parser = PluggableParser.new
+  def retry_change_requests= retry_change_requests
+    @agent.retry_change_requests = retry_change_requests
+  end
 
-    @keep_alive       = true
-    @keep_alive_time  = 0
+  ##
+  # Will <code>/robots.txt</code> files be obeyed?
 
-    # Proxy
-    @proxy_addr = nil
-    @proxy_port = nil
-    @proxy_user = nil
-    @proxy_pass = nil
+  def robots
+    @agent.robots
+  end
 
-    @html_parser = self.class.html_parser
+  ##
+  # When +enabled+ mechanize will retrieve and obey <code>robots.txt</code>
+  # files
 
-    @default_encoding = nil
-    @force_default_encoding = false
+  def robots= enabled
+    @agent.robots = enabled
+  end
 
-    yield self if block_given?
+  ##
+  # The handlers for HTTP and other URI protocols.
 
-    @agent.set_proxy @proxy_addr, @proxy_port, @proxy_user, @proxy_pass
-    @agent.set_http
+  def scheme_handlers
+    @agent.scheme_handlers
   end
 
-  def max_history
-    @agent.history.max_size
+  ##
+  # Replaces the URI scheme handler table with +scheme_handlers+
+
+  def scheme_handlers= scheme_handlers
+    @agent.scheme_handlers = scheme_handlers
   end
 
-  def max_history= length
-    @agent.history.max_size = length
+  ##
+  # The identification string for the client initiating a web request
+
+  def user_agent
+    @agent.user_agent
   end
 
-  def log=(l); self.class.log = l end
-  def log; self.class.log end
+  ##
+  # Sets the User-Agent used by mechanize to +user_agent+.  See also
+  # user_agent_alias
 
   def user_agent= user_agent
     @agent.user_agent = user_agent
   end
 
-  # Set the user agent for the Mechanize object.  See AGENT_ALIASES
-  def user_agent_alias=(al)
-    self.user_agent = AGENT_ALIASES[al] ||
-      raise(ArgumentError, "unknown agent alias #{al.inspect}")
-  end
+  ##
+  # Set the user agent for the Mechanize object based on the given +name+.
+  #
+  # See also AGENT_ALIASES
 
-  # Returns a list of cookies stored in the cookie jar.
-  def cookies
-    @agent.cookie_jar.to_a
+  def user_agent_alias= name
+    self.user_agent = AGENT_ALIASES[name] ||
+      raise(ArgumentError, "unknown agent alias #{name.inspect}")
   end
 
-  # Sets the user and password to be used for authentication.
-  def auth(user, password)
-    @agent.user     = user
-    @agent.password = password
+  ##
+  # The value of watch_for_set is passed to pluggable parsers for retrieved
+  # content
+
+  attr_accessor :watch_for_set
+
+  # :section: SSL
+  #
+  # SSL settings for mechanize.  These must be set in the block given to
+  # Mechanize.new
+
+  ##
+  # Path to an OpenSSL server certificate file
+
+  def ca_file
+    @agent.ca_file
   end
 
-  alias :basic_auth :auth
+  ##
+  # Sets the certificate file used for SSL connections
 
-  # Fetches the URL passed in and returns a page.
-  def get(uri, parameters = [], referer = nil, headers = {})
-    method = :get
+  def ca_file= ca_file
+    @agent.ca_file = ca_file
+  end
 
-    if Hash === uri then
-      options = uri
-      location = Gem.location_of_caller.join ':'
-      warn "#{location}: Mechanize#get with options hash is deprecated and will be removed October 2011"
+  ##
+  # An OpenSSL client certificate or the path to a certificate file.
 
-      raise ArgumentError, "url must be specified" unless uri = options[:url]
-      parameters = options[:params] || []
-      referer    = options[:referer]
-      headers    = options[:headers]
-      method     = options[:verb] || method
-    end
+  def cert
+    @agent.cert
+  end
 
-    referer ||=
-      if uri.to_s =~ %r{\Ahttps?://}
-        Page.new(nil, {'content-type'=>'text/html'})
-      else
-        current_page || Page.new(nil, {'content-type'=>'text/html'})
-      end
+  ##
+  # Sets the OpenSSL client certificate +cert+ to the given path or
+  # certificate instance
 
-    # FIXME: Huge hack so that using a URI as a referer works.  I need to
-    # refactor everything to pass around URIs but still support
-    # Mechanize::Page#base
-    unless referer.is_a?(Mechanize::File)
-      referer = referer.is_a?(String) ?
-      Page.new(URI.parse(referer), {'content-type' => 'text/html'}) :
-        Page.new(referer, {'content-type' => 'text/html'})
-    end
+  def cert= cert
+    @agent.cert = cert
+  end
 
-    # fetch the page
-    headers ||= {}
-    page = @agent.fetch uri, method, headers, parameters, referer
-    add_to_history(page)
-    yield page if block_given?
-    page
+  ##
+  # An OpenSSL certificate store for verifying server certificates.  This
+  # defaults to the default certificate store.
+
+  def cert_store
+    @agent.cert_store
   end
 
   ##
-  # PUT to +url+ with +entity+, and setting +headers+:
-  #
-  #   put('http://example/', 'new content', {'Content-Type' => 'text/plain'})
-  #
-  def put(url, entity, headers = {})
-    request_with_entity(:put, url, entity, headers)
+  # Sets the OpenSSL certificate store to +store+.
+
+  def cert_store= cert_store
+    @agent.cert_store = cert_store
   end
 
   ##
-  # DELETE to +url+ with +query_params+, and setting +headers+:
-  #
-  #   delete('http://example/', {'q' => 'foo'}, {})
+  # What is this?
   #
-  def delete(uri, query_params = {}, headers = {})
-    page = @agent.fetch(uri, :delete, headers, query_params)
-    add_to_history(page)
-    page
+  # Why is it different from #cert?
+
+  def certificate # :nodoc:
+    @agent.certificate
   end
 
   ##
-  # HEAD to +url+ with +query_params+, and setting +headers+:
-  #
-  #   head('http://example/', {'q' => 'foo'}, {})
-  #
-  def head(uri, query_params = {}, headers = {})
-    # fetch the page
-    page = @agent.fetch(uri, :head, headers, query_params)
-    yield page if block_given?
-    page
+  # An OpenSSL private key or the path to a private key
+
+  def key
+    @agent.key
   end
 
-  # Fetch a file and return the contents of the file.
-  def get_file(url)
-    get(url).body
+  ##
+  # Sets the OpenSSL client +key+ to the given path or key instance
+
+  def key= key
+    @agent.key = key
   end
 
-  # If the parameter is a string, finds the button or link with the
-  # value of the string and clicks it. Otherwise, clicks the
-  # Mechanize::Page::Link object passed in. Returns the page fetched.
-  def click(link)
-    case link
-    when Page::Link
-      referer = link.page || current_page()
-      if @agent.robots
-        if (referer.is_a?(Page) && referer.parser.nofollow?) || link.rel?('nofollow')
-          raise RobotsDisallowedError.new(link.href)
-        end
-      end
-      if link.rel?('noreferrer')
-        href = @agent.resolve(link.href, link.page || current_page)
-        referer = Page.new(nil, {'content-type'=>'text/html'})
-      else
-        href = link.href
-      end
-      get href, [], referer
-    when String, Regexp
-      if real_link = page.link_with(:text => link)
-        click real_link
-      else
-        button = nil
-        form = page.forms.find do |f|
-          button = f.button_with(:value => link)
-          button.is_a? Form::Submit
-        end
-        submit form, button if form
-      end
-    else
-      referer = current_page()
-      href = link.respond_to?(:href) ? link.href :
-        (link['href'] || link['src'])
-      get href, [], referer
-    end
+  ##
+  # OpenSSL client key password
+
+  def pass
+    @agent.pass
   end
 
-  # Equivalent to the browser back button.  Returns the most recent page
-  # visited.
-  def back
-    @agent.history.pop
+  ##
+  # Sets the client key password to +pass+
+
+  def pass= pass
+    @agent.pass = pass
   end
 
-  # Posts to the given URL with the request entity.  The request
-  # entity is specified by either a string, or a list of key-value
-  # pairs represented by a hash or an array of arrays.
-  #
-  # Examples:
-  #  agent.post('http://example.com/', "foo" => "bar")
-  #
-  #  agent.post('http://example.com/', [ ["foo", "bar"] ])
+  ##
+  # A callback for additional certificate verification.  See
+  # OpenSSL::SSL::SSLContext#verify_callback
   #
-  #  agent.post('http://example.com/', "<message>hello</message>", 'Content-Type' => 'application/xml')
-  def post(url, query={}, headers={})
-    if query.is_a?(String)
-      return request_with_entity(:post, url, query, headers)
-    end
-    node = {}
-    # Create a fake form
-    class << node
-      def search(*args); []; end
-    end
-    node['method'] = 'POST'
-    node['enctype'] = 'application/x-www-form-urlencoded'
-
-    form = Form.new(node)
+  # The callback can be used for debugging or to ignore errors by always
+  # returning +true+.  Specifying nil uses the default method that was valid
+  # when the SSLContext was created
 
-    query.each { |k, v|
-      if v.is_a?(IO)
-        form.enctype = 'multipart/form-data'
-        ul = Form::FileUpload.new({'name' => k.to_s},::File.basename(v.path))
-        ul.file_data = v.read
-        form.file_uploads << ul
-      else
-        form.fields << Form::Field.new({'name' => k.to_s},v)
-      end
-    }
-    post_form(url, form, headers)
+  def verify_callback
+    @agent.verify_callback
   end
 
-  # Submit a form with an optional button.
-  # Without a button:
-  #  page = agent.get('http://example.com')
-  #  agent.submit(page.forms.first)
-  # With a button
-  #  agent.submit(page.forms.first, page.forms.first.buttons.first)
-  def submit(form, button=nil, headers={})
-    form.add_button_to_query(button) if button
-    case form.method.upcase
-    when 'POST'
-      post_form(form.action, form, headers)
-    when 'GET'
-      get(form.action.gsub(/\?[^\?]*$/, ''),
-          form.build_query,
-          form.page,
-          headers)
-    else
-      raise ArgumentError, "unsupported method: #{form.method.upcase}"
-    end
-  end
+  ##
+  # Sets the OpenSSL certificate verification callback
 
-  def request_with_entity(verb, uri, entity, headers = {})
-    cur_page = current_page || Page.new(nil, {'content-type'=>'text/html'})
+  def verify_callback= verify_callback
+    @agent.verify_callback = verify_callback
+  end
 
-    headers = {
-      'Content-Type' => 'application/octet-stream',
-      'Content-Length' => entity.size.to_s,
-    }.update headers
+  ##
+  # the OpenSSL server certificate verification method.  The default is
+  # OpenSSL::SSL::VERIFY_PEER and certificate verification uses the default
+  # system certificates.  See also cert_store
 
-    page = @agent.fetch uri, verb, headers, [entity], cur_page
-    add_to_history(page)
-    page
+  def verify_mode
+    @agent.verify_mode
   end
 
-  # Returns the current page loaded by Mechanize
-  def current_page
-    @agent.current_page
+  ##
+  # Sets the OpenSSL server certificate verification method.
+
+  def verify_mode= verify_mode
+    @agent.verify_mode = verify_mode
   end
 
-  # Returns a visited page for the url passed in, otherwise nil
-  def visited_page(url)
-    url = url.href if url.respond_to? :href
+  # :section: Utilities
 
-    @agent.visited_page url
-  end
+  attr_reader :agent # :nodoc:
 
-  # Returns whether or not a url has been visited
-  alias visited? visited_page
+  attr_reader :pluggable_parser # :nodoc:
+
+  ##
+  # Parses the +body+ of the +response+ from +uri+ using the pluggable parser
+  # that matches its content type
 
   def parse uri, response, body
     content_type = nil
@@ -567,6 +975,15 @@ class Mechanize
     # Find our pluggable parser
     parser_klass = @pluggable_parser.parser content_type
 
+    unless Mechanize::Download === parser_klass then
+      body = case body
+             when IO, Tempfile, StringIO then
+               body.read
+             else
+               body
+             end
+    end
+
     parser_klass.new uri, response, body, response.code do |parser|
       parser.mech = self if parser.respond_to? :mech=
 
@@ -575,6 +992,15 @@ class Mechanize
     end
   end
 
+  def pretty_print(q) # :nodoc:
+    q.object_group(self) {
+      q.breakable
+      q.pp cookie_jar
+      q.breakable
+      q.pp current_page
+    }
+  end
+
   ##
   # Sets the proxy +address+ at +port+ with an optional +user+ and +password+
 
@@ -588,29 +1014,11 @@ class Mechanize
     @agent.set_http
   end
 
-  # Runs given block, then resets the page history as it was before. self is
-  # given as a parameter to the block. Returns the value of the block.
-  def transact
-    history_backup = @agent.history.dup
-    begin
-      yield self
-    ensure
-      @agent.history = history_backup
-    end
-  end
-
-  def robots
-    @agent.robots
-  end
-
-  def robots= enabled
-    @agent.robots = enabled
-  end
-
-  alias :page :current_page
-
   private
 
+  ##
+  # Posts +form+ to +uri+
+
   def post_form(uri, form, headers = {})
     cur_page = form.page || current_page ||
       Page.new(nil, {'content-type'=>'text/html'})
@@ -630,6 +1038,9 @@ class Mechanize
     page
   end
 
+  ##
+  # Adds +page+ to the history
+
   def add_to_history(page)
     @agent.history.push(page, @agent.resolve(page.uri))
     @history_added.call(page) if @history_added
@@ -640,6 +1051,8 @@ end
 require 'mechanize/content_type_error'
 require 'mechanize/cookie'
 require 'mechanize/cookie_jar'
+require 'mechanize/parser'
+require 'mechanize/download'
 require 'mechanize/file'
 require 'mechanize/file_connection'
 require 'mechanize/file_request'
@@ -648,13 +1061,17 @@ require 'mechanize/form'
 require 'mechanize/history'
 require 'mechanize/http'
 require 'mechanize/http/agent'
+require 'mechanize/http/auth_challenge'
+require 'mechanize/http/auth_realm'
+require 'mechanize/http/content_disposition_parser'
+require 'mechanize/http/www_authenticate_parser'
 require 'mechanize/page'
-require 'mechanize/inspect'
 require 'mechanize/monkey_patch'
 require 'mechanize/pluggable_parsers'
 require 'mechanize/redirect_limit_reached_error'
 require 'mechanize/redirect_not_get_or_head_error'
 require 'mechanize/response_code_error'
+require 'mechanize/unauthorized_error'
 require 'mechanize/response_read_error'
 require 'mechanize/robots_disallowed_error'
 require 'mechanize/unsupported_scheme_error'