mechanize 2.1.1 → 2.2

data/CHANGELOG.rdoc

@@ -1,6 +1,41 @@
 = Mechanize CHANGELOG
 
-=== 2.1.1 / 2010-02-03
+=== 2.2 / 2012-02-12
+
+* API changes
+  * MetaRefresh#href is not normalized to an absolute URL, but set to the
+    original value and resolved later.  It is even set to nil when the
+    Refresh URL is unspecified or empty.
+
+* Minor enhancements
+  * Expose ssl_version from net-http-persistent.  Patch by astera.
+  * SSL parameters and proxy may now be set at any time.  Issue #194 by
+    dsisnero.
+  * Improved Mechanize::Page with #image_with and #images_with and
+    Mechanize::Page::Image various img element attribute accessors, #caption,
+    #extname, #mime_type and #fetch.  Pull request #173 by kitamomonga
+  * Added MIME type parsing for content-types in Mechanize::PluggableParser
+    for fine-grained parser choices.  Parsers will be chosen based on exact
+    match, simplified type or media type in that order.  See
+    Mechanize::PluggableParser#[]=.
+  * Added Mechanize#download which downloads a response body to an IO-like or
+    filename.
+  * Added Mechanize::DirectorySaver which saves responses in a single
+    directory.  Issue #187 by yoshie902a.
+  * Added Mechanize::Page::Link#noreferrer?
+  * The documentation for Mechanize::Page#search and #at now show that both
+    XPath and CSS expressions are allowed.  Issue #199 by Shane Becker.
+
+* Bug fixes
+  * Fixed handling of a HEAD request with Accept-Encoding: gzip.  Issue #198
+    by Oleg Dashevskii
+  * Use #resolve for resolving a Location header value.  fixes #197
+  * A Refresh value can have whitespaces around the semicolon and equal sign.
+  * MetaRefresh#click no longer sends out Referer.
+  * A link with an empty href is now resolved correctly where previously the
+    query part was dropped.
+
+=== 2.1.1 / 2012-02-03
 
 * Bug fixes
   * Set missing idle_timeout default.  Issue #196

data/EXAMPLES.rdoc

@@ -1,8 +1,9 @@
 = Mechanize examples
 
 Note: Several examples show methods chained to the end of do/end blocks.
-Do...end is the same as curly braces ({...}). For example, do ... end.submit
-is the same as { ... }.submit.
+<code>do...end</code> is the same as curly braces (<code>{...}</code>).  For
+example, <code>do ... end.submit</code> is the same as <code>{ ...
+}.submit</code>.
 
 == Google
 
@@ -81,7 +82,8 @@ Upload a file to flickr.
   end
 
 == Pluggable Parsers
-Lets say you want html pages to automatically be parsed with Rubyful Soup.
+
+Lets say you want HTML pages to automatically be parsed with Rubyful Soup.
 This example shows you how:
 
   require 'rubygems'
@@ -115,10 +117,10 @@ Beautiful Soup for that page.
 
 == The transact method
 
-transact runs the given block and then resets the page history. I.e. after the
-block has been executed, you're back at the original page; no need count how
-many times to call the back method at the end of a loop (while accounting for
-possible exceptions).
+Mechanize#transact runs the given block and then resets the page history. I.e.
+after the block has been executed, you're back at the original page; no need
+count how many times to call the back method at the end of a loop (while
+accounting for possible exceptions).
 
 This example also demonstrates subclassing Mechanize.
 
@@ -154,17 +156,12 @@ This example also demonstrates subclassing Mechanize.
 
 == Client Certificate Authentication (Mutual Auth)
 
-In most cases a client certificate is created as an additional layer of security
-for certain websites.  The specific case that this was initially tested on was
-for automating the download of archived images from a banks (Wachovia) lockbox
-system.  Once the certificate is installed into your browser you will have to
-export it and split the certificate and private key into separate files.
-Exported files are usually in .p12 format (IE 7 & Firefox 2.0) which stands for
-PKCS #12.  You can convert them from p12 to pem format by using the following
-commands:
-
-openssl.exe pkcs12 -in input_file.p12 -clcerts -out example.key -nocerts -nodes
-openssl.exe pkcs12 -in input_file.p12 -clcerts -out example.cer -nokeys
+In most cases a client certificate is created as an additional layer of
+security for certain websites.  The specific case that this was initially
+tested on was for automating the download of archived images from a banks
+(Wachovia) lockbox system.  Once the certificate is installed into your
+browser you will have to export it and split the certificate and private key
+into separate files.
 
   require 'rubygems'
   require 'mechanize'
@@ -185,3 +182,11 @@ openssl.exe pkcs12 -in input_file.p12 -clcerts -out example.cer -nokeys
 
   # submit login form
   agent.submit(login_form, login_form.buttons.first)
+
+Exported files are usually in .p12 format (IE 7 & Firefox 2.0) which stands
+for PKCS #12.  You can convert them from p12 to pem format by using the
+following commands:
+
+  openssl pkcs12 -in input_file.p12 -clcerts -out example.key -nocerts -nodes
+  openssl pkcs12 -in input_file.p12 -clcerts -out example.cer -nokeys
+

data/GUIDE.rdoc

@@ -130,7 +130,7 @@ In this section, I want to touch on using the different types in input fields
 possible with a form.  Password and textarea fields can be treated just like
 text input fields.  Select fields are very similar to text fields, but they
 have many options associated with them.  If you select one option, mechanize
-will deselect the other options (unless it is a multi select!).
+will de-select the other options (unless it is a multi select!).
 
 For example, lets select an option on a list:
 
@@ -154,10 +154,15 @@ tell it what file name you want to upload:
 
 == Scraping Data
 
-Mechanize uses nokogiri[http://nokogiri.org/] to parse
-html.  What does this mean for you?  You can treat a mechanize page like
-an nokogiri object.  After you have used Mechanize to navigate to the page
-that you need to scrape, then scrape it using nokogiri methods:
+Mechanize uses nokogiri[http://nokogiri.org/] to parse HTML.  What does this
+mean for you?  You can treat a mechanize page like an nokogiri object.  After
+you have used Mechanize to navigate to the page that you need to scrape, then
+scrape it using nokogiri methods:
+
+  agent.get('http://someurl.com/').search("p.posted")
+
+The expression given to Mechanize::Page#search may be a CSS expression or an
+XPath expression:
 
   agent.get('http://someurl.com/').search(".//p[@class='posted']")
 

data/Manifest.txt

@@ -47,6 +47,7 @@ lib/mechanize/http/auth_challenge.rb
 lib/mechanize/http/auth_realm.rb
 lib/mechanize/http/content_disposition_parser.rb
 lib/mechanize/http/www_authenticate_parser.rb
+lib/mechanize/image.rb
 lib/mechanize/monkey_patch.rb
 lib/mechanize/page.rb
 lib/mechanize/page/base.rb
@@ -144,16 +145,19 @@ test/test_mechanize_http_auth_challenge.rb
 test/test_mechanize_http_auth_realm.rb
 test/test_mechanize_http_content_disposition_parser.rb
 test/test_mechanize_http_www_authenticate_parser.rb
+test/test_mechanize_image.rb
 test/test_mechanize_link.rb
 test/test_mechanize_page.rb
 test/test_mechanize_page_encoding.rb
 test/test_mechanize_page_frame.rb
+test/test_mechanize_page_image.rb
 test/test_mechanize_page_link.rb
 test/test_mechanize_page_meta_refresh.rb
 test/test_mechanize_parser.rb
 test/test_mechanize_pluggable_parser.rb
 test/test_mechanize_redirect_limit_reached_error.rb
 test/test_mechanize_redirect_not_get_or_head_error.rb
+test/test_mechanize_response_read_error.rb
 test/test_mechanize_subclass.rb
 test/test_mechanize_util.rb
 test/test_multi_select.rb

data/Rakefile

@@ -17,7 +17,8 @@ hoe = Hoe.spec 'mechanize' do
   rdoc_locations << 'drbrain@rubyforge.org:/var/www/gforge-projects/mechanize/'
 
   self.extra_deps << ['net-http-digest_auth', '~> 1.1', '>= 1.1.1']
-  self.extra_deps << ['net-http-persistent',  '~> 2.4', '>= 2.4.1']
+  self.extra_deps << ['net-http-persistent',  '~> 2.5', '>= 2.5.2']
+  self.extra_deps << ['mime-types',           '~> 1.17', '>= 1.17.2']
   self.extra_deps << ['nokogiri',             '~> 1.4']
   self.extra_deps << ['ntlm-http',            '~> 0.1', '>= 0.1.1']
   self.extra_deps << ['webrobots',            '~> 0.0', '>= 0.0.9']

data/lib/mechanize.rb

@@ -1,6 +1,7 @@
 require 'fileutils'
 require 'forwardable'
 require 'iconv' if RUBY_VERSION < '1.9.2'
+require 'mime/types'
 require 'mutex_m'
 require 'net/http/digest_auth'
 require 'net/http/persistent'
@@ -72,7 +73,7 @@ class Mechanize
   ##
   # The version of Mechanize you are using.
 
-  VERSION = '2.1.1'
+  VERSION = '2.2'
 
   ##
   # Base mechanize error class
@@ -176,7 +177,6 @@ class Mechanize
     yield self if block_given?
 
     @agent.set_proxy @proxy_addr, @proxy_port, @proxy_user, @proxy_pass
-    @agent.set_http
   end
 
   # :section: History
@@ -293,7 +293,7 @@ class Mechanize
           raise RobotsDisallowedError.new(link.href)
         end
       end
-      if link.rel?('noreferrer')
+      if link.noreferrer?
         href = @agent.resolve(link.href, link.page || current_page)
         referer = Page.new(nil, {'content-type'=>'text/html'})
       else
@@ -319,6 +319,47 @@ class Mechanize
     end
   end
 
+  ##
+  # GETs +uri+ and writes it to +io_or_filename+ without recording the request
+  # in the history.  If +io_or_filename+ does not respond to #write it will be
+  # used as a file name.  +parameters+, +referer+ and +headers+ are used as in
+  # #get.
+  #
+  # By default, if the Content-type of the response matches a Mechanize::File
+  # or Mechanize::Page parser, the response body will be loaded into memory
+  # before being saved.  See #pluggable_parser for details on changing this
+  # default.
+  #
+  # For alternate ways of downloading files see Mechanize::FileSaver and
+  # Mechanize::DirectorySaver.
+
+  def download uri, io_or_filename, parameters = [], referer = nil, headers = {}
+    page = transact do
+      get uri, parameters, referer, headers
+    end
+
+    io = if io_or_filename.respond_to? :write then
+           io_or_filename
+         else
+           open io_or_filename, 'wb'
+         end
+
+    case page
+    when Mechanize::File then
+      io.write page.body
+    else
+      body_io = page.body_io
+
+      until body_io.eof? do
+        io.write body_io.read 16384
+      end
+    end
+
+    page
+  ensure
+    io.close if io and not io_or_filename.respond_to? :write
+  end
+
   ##
   # DELETE +uri+ with +query_params+, and setting +headers+:
   #
@@ -341,18 +382,20 @@ class Mechanize
 
     referer ||=
       if uri.to_s =~ %r{\Ahttps?://}
-        Page.new(nil, {'content-type'=>'text/html'})
+        Page.new(nil, 'content-type' => 'text/html')
       else
-        current_page || Page.new(nil, {'content-type'=>'text/html'})
+        current_page || Page.new(nil, 'content-type' => 'text/html')
       end
 
     # FIXME: Huge hack so that using a URI as a referer works.  I need to
     # refactor everything to pass around URIs but still support
     # Mechanize::Page#base
     unless Mechanize::Parser === referer then
-      referer = referer.is_a?(String) ?
-      Page.new(URI.parse(referer), {'content-type' => 'text/html'}) :
-        Page.new(referer, {'content-type' => 'text/html'})
+      referer = if referer.is_a?(String) then
+                  Page.new URI(referer), 'content-type' => 'text/html'
+                else
+                  Page.new referer, 'content-type' => 'text/html'
+                end
     end
 
     # fetch the page
@@ -371,14 +414,15 @@ class Mechanize
   end
 
   ##
-  # HEAD +uri+ with +query_params+, and setting +headers+:
+  # HEAD +uri+ with +query_params+ and +headers+:
   #
   #   head('http://example/', {'q' => 'foo'}, {})
 
   def head(uri, query_params = {}, headers = {})
-    # fetch the page
-    page = @agent.fetch(uri, :head, headers, query_params)
+    page = @agent.fetch uri, :head, headers, query_params
+
     yield page if block_given?
+
     page
   end
 
@@ -537,6 +581,18 @@ class Mechanize
 
   attr_accessor :keep_alive_time
 
+  ##
+  # The pluggable parser maps a response Content-Type to a parser class.  The
+  # registered Content-Type may be either a full content type like 'image/png'
+  # or a media type 'text'.  See Mechanize::PluggableParser for further
+  # details.
+  #
+  # Example:
+  #
+  #   agent.pluggable_parser['application/octet-stream'] = Mechanize::Download
+
+  attr_reader :pluggable_parser
+
   ##
   # The HTTP proxy address
 
@@ -907,7 +963,7 @@ class Mechanize
   # certificate instance
 
   def cert= cert
-    @agent.cert = cert
+    @agent.certificate = cert
   end
 
   ##
@@ -962,10 +1018,11 @@ class Mechanize
   end
 
   ##
-  # Sets the OpenSSL client +key+ to the given path or key instance
+  # Sets the OpenSSL client +key+ to the given path or key instance.  If a
+  # path is given, the path must contain an RSA key file.
 
   def key= key
-    @agent.key = key
+    @agent.private_key = key
   end
 
   ##
@@ -982,6 +1039,21 @@ class Mechanize
     @agent.pass = pass
   end
 
+  ##
+  # SSL version to use.  Ruby 1.9 and newer only.
+
+  def ssl_version
+    @agent.ssl_version
+  end if RUBY_VERSION > '1.9'
+
+  ##
+  # Sets the SSL version to use to +version+ without client/server
+  # negotiation.  Ruby 1.9 and newer only.
+
+  def ssl_version= ssl_version
+    @agent.ssl_version = ssl_version
+  end if RUBY_VERSION > '1.9'
+
   ##
   # A callback for additional certificate verification.  See
   # OpenSSL::SSL::SSLContext#verify_callback
@@ -1021,8 +1093,6 @@ class Mechanize
 
   attr_reader :agent # :nodoc:
 
-  attr_reader :pluggable_parser # :nodoc:
-
   ##
   # Parses the +body+ of the +response+ from +uri+ using the pluggable parser
   # that matches its content type
@@ -1035,7 +1105,6 @@ class Mechanize
       content_type, = data.downcase.split ',', 2 unless data.nil?
     end
 
-    # Find our pluggable parser
     parser_klass = @pluggable_parser.parser content_type
 
     unless parser_klass <= Mechanize::Download then
@@ -1074,7 +1143,6 @@ class Mechanize
     @proxy_pass = password
 
     @agent.set_proxy address, port, user, password
-    @agent.set_http
   end
 
   private
@@ -1116,6 +1184,7 @@ require 'mechanize/cookie'
 require 'mechanize/cookie_jar'
 require 'mechanize/parser'
 require 'mechanize/download'
+require 'mechanize/directory_saver'
 require 'mechanize/file'
 require 'mechanize/file_connection'
 require 'mechanize/file_request'
@@ -1128,6 +1197,7 @@ require 'mechanize/http/auth_challenge'
 require 'mechanize/http/auth_realm'
 require 'mechanize/http/content_disposition_parser'
 require 'mechanize/http/www_authenticate_parser'
+require 'mechanize/image'
 require 'mechanize/page'
 require 'mechanize/monkey_patch'
 require 'mechanize/pluggable_parsers'

data/lib/mechanize/file_request.rb

@@ -22,5 +22,9 @@ class Mechanize::FileRequest
   def each_header
   end
 
+  def response_body_permitted?
+    true
+  end
+
 end
 

data/lib/mechanize/file_saver.rb

@@ -1,7 +1,7 @@
 ##
 # This is a pluggable parser that automatically saves every file it
-# encounters.  It saves the files as a tree, reflecting the host and file
-# path.
+# encounters.  Unlike Mechanize::DirectorySaver, the file saver saves the
+# responses as a tree, reflecting the host and file path.
 #
 # == Example
 #
@@ -11,7 +11,7 @@
 #
 #   agent = Mechanize.new
 #   agent.pluggable_parser.pdf = Mechanize::FileSaver
-#   agent.get('http://example.com/foo.pdf')
+#   agent.get 'http://example.com/foo.pdf'
 #
 #   Dir['example.com/*'] # => foo.pdf
 

data/lib/mechanize/http/agent.rb

@@ -78,37 +78,11 @@ class Mechanize::HTTP::Agent
 
   # :section: SSL
 
-  # Path to an OpenSSL server certificate file
-  attr_accessor :ca_file
-
-  # An OpenSSL private key or the path to a private key
-  attr_accessor :key
-
-  # An OpenSSL client certificate or the path to a certificate file.
-  attr_accessor :cert
-
-  # An SSL certificate store
-  attr_accessor :cert_store
-
   # OpenSSL key password
   attr_accessor :pass
 
-  # A callback for additional certificate verification.  See
-  # OpenSSL::SSL::SSLContext#verify_callback
-  #
-  # The callback can be used for debugging or to ignore errors by always
-  # returning +true+.  Specifying nil uses the default method that was valid
-  # when the SSLContext was created
-  attr_accessor :verify_callback
-
-  # How to verify SSL connections.  Defaults to VERIFY_PEER
-  attr_accessor :verify_mode
-
   # :section: Timeouts
 
-  # Reset connections that have not been used in this many seconds
-  attr_reader   :idle_timeout
-
   # Set to false to disable HTTP/1.1 keep-alive requests
   attr_accessor :keep_alive
 
@@ -123,12 +97,6 @@ class Mechanize::HTTP::Agent
   # The cookies for this agent
   attr_accessor :cookie_jar
 
-  # URI for a proxy connection
-  attr_reader :proxy_uri
-
-  # Retry non-idempotent requests?
-  attr_reader :retry_change_requests
-
   # Responses larger than this will be written to a Tempfile instead of stored
   # in memory.
   attr_accessor :max_file_buffer
@@ -157,19 +125,15 @@ class Mechanize::HTTP::Agent
     @follow_meta_refresh_self = false
     @gzip_enabled             = true
     @history                  = Mechanize::History.new
-    @idle_timeout             = 5
     @keep_alive               = true
-    @keep_alive_time          = 300
     @max_file_buffer          = 10240
     @open_timeout             = nil
     @post_connect_hooks       = []
     @pre_connect_hooks        = []
-    @proxy_uri                = nil
     @read_timeout             = nil
     @redirect_ok              = true
     @redirection_limit        = 20
     @request_headers          = {}
-    @retry_change_requests    = false
     @robots                   = false
     @user_agent               = nil
     @webrobots                = nil
@@ -188,13 +152,7 @@ class Mechanize::HTTP::Agent
     @domain               = nil # NTLM HTTP domain
 
     # SSL
-    @ca_file         = nil
-    @cert            = nil
-    @cert_store      = nil
-    @key             = nil
-    @pass            = nil
-    @verify_callback = nil
-    @verify_mode     = nil
+    @pass = nil
 
     @scheme_handlers = Hash.new { |h, scheme|
       h[scheme] = lambda { |link, page|
@@ -206,6 +164,10 @@ class Mechanize::HTTP::Agent
     @scheme_handlers['https']     = @scheme_handlers['http']
     @scheme_handlers['relative']  = @scheme_handlers['http']
     @scheme_handlers['file']      = @scheme_handlers['http']
+
+    @http = Net::HTTP::Persistent.new 'mechanize'
+    @http.idle_timeout = 5
+    @http.keep_alive   = 300
   end
 
   # Retrieves +uri+ and parses it into a page or other object according to
@@ -273,7 +235,8 @@ class Mechanize::HTTP::Agent
 
     hook_content_encoding response, uri, response_body_io
 
-    response_body_io = response_content_encoding response, response_body_io
+    response_body_io = response_content_encoding response, response_body_io if
+      request.response_body_permitted?
 
     post_connect uri, response, response_body_io
 
@@ -306,11 +269,21 @@ class Mechanize::HTTP::Agent
     end
   end
 
+  # URI for a proxy connection
+
+  def proxy_uri
+    @http.proxy_uri
+  end
+
+  # Retry non-idempotent requests?
+  def retry_change_requests
+    @http.retry_change_requests
+  end
+
   # Retry non-idempotent requests
 
   def retry_change_requests= retri
-    @retry_change_requests = retri
-    @http.retry_change_requests = retri if @http
+    @http.retry_change_requests = retri
   end
 
   # :section: Headers
@@ -568,10 +541,24 @@ class Mechanize::HTTP::Agent
   end
 
   def resolve(uri, referer = current_page)
-    uri = uri.dup if uri.is_a?(URI)
+    referer_uri = referer && referer.uri
+    if uri.is_a?(URI)
+      uri = uri.dup
+    elsif uri.nil?
+      if referer_uri
+        return referer_uri
+      end
+      raise ArgumentError, "absolute URL needed (not nil)"
+    else
+      url = uri.to_s.strip
+      if url.empty?
+        if referer_uri
+          return referer_uri.dup.tap { |u| u.fragment = nil }
+        end
+        raise ArgumentError, "absolute URL needed (not #{uri.inspect})"
+      end
 
-    unless uri.is_a?(URI)
-      uri = uri.to_s.strip.gsub(/[^#{0.chr}-#{126.chr}]/o) { |match|
+      url.gsub!(/[^#{0.chr}-#{126.chr}]/o) { |match|
         if RUBY_VERSION >= "1.9.0"
           Mechanize::Util.uri_escape(match)
         else
@@ -579,28 +566,25 @@ class Mechanize::HTTP::Agent
         end
       }
 
-      unescaped = uri.split(/(?:%[0-9A-Fa-f]{2})+|#/)
-      escaped   = uri.scan(/(?:%[0-9A-Fa-f]{2})+|#/)
-
-      escaped_uri = Mechanize::Util.html_unescape(
-        unescaped.zip(escaped).map { |x,y|
+      escaped_url = Mechanize::Util.html_unescape(
+        url.split(/((?:%[0-9A-Fa-f]{2})+|#)/).each_slice(2).map { |x, y|
           "#{WEBrick::HTTPUtils.escape(x)}#{y}"
         }.join('')
       )
 
       begin
-        uri = URI.parse(escaped_uri)
+        uri = URI.parse(escaped_url)
       rescue
-        uri = URI.parse(WEBrick::HTTPUtils.escape(escaped_uri))
+        uri = URI.parse(WEBrick::HTTPUtils.escape(escaped_url))
       end
     end
 
     scheme = uri.relative? ? 'relative' : uri.scheme.downcase
     uri = @scheme_handlers[scheme].call(uri, referer)
 
-    if referer && referer.uri
+    if referer_uri
       if uri.path.length == 0 && uri.relative?
-        uri.path = referer.uri.path
+        uri.path = referer_uri.path
       end
     end
 
@@ -608,17 +592,16 @@ class Mechanize::HTTP::Agent
 
     if uri.relative?
       raise ArgumentError, "absolute URL needed (not #{uri})" unless
-        referer && referer.uri
+        referer_uri
 
-      base = nil
-      if referer.respond_to?(:bases) && referer.parser
-        base = referer.bases.last
+      if referer.respond_to?(:bases) && referer.parser &&
+          (lbase = referer.bases.last) && lbase.uri && lbase.uri.absolute?
+        base = lbase
+      else
+        base = nil
       end
 
-      uri = ((base && base.uri && base.uri.absolute?) ?
-             base.uri :
-             referer.uri) + uri
-      uri = referer.uri + uri
+      uri = referer_uri + (base ? base.uri : referer_uri) + uri
       # Strip initial "/.." bits from the path
       uri.path.sub!(/^(\/\.\.)+(?=\/)/, '')
     end
@@ -791,7 +774,8 @@ class Mechanize::HTTP::Agent
 
   def response_follow_meta_refresh response, uri, page, redirects
     delay, new_url = get_meta_refresh(response, uri, page)
-    return nil unless new_url
+    return nil unless delay
+    new_url = new_url ? resolve(new_url, page) : uri
 
     raise Mechanize::RedirectLimitReachedError.new(page, redirects) if
       redirects + 1 > @redirection_limit
@@ -888,9 +872,8 @@ class Mechanize::HTTP::Agent
 
     redirect_method = method == :head ? :head : :get
 
-    from_uri = page.uri
-    @history.push(page, from_uri)
-    new_uri = from_uri + response['Location'].to_s
+    @history.push(page, page.uri)
+    new_uri = resolve response['Location'].to_s, page
 
     fetch new_uri, redirect_method, {}, [], referer, redirects + 1
   end
@@ -949,16 +932,104 @@ class Mechanize::HTTP::Agent
 
   # :section: SSL
 
+  # Path to an OpenSSL CA certificate file
+  def ca_file
+    @http.ca_file
+  end
+
+  # Sets the path to an OpenSSL CA certificate file
+  def ca_file= ca_file
+    @http.ca_file = ca_file
+  end
+
+  # The SSL certificate store used for validating connections
+  def cert_store
+    @http.cert_store
+  end
+
+  # Sets the SSL certificate store used for validating connections
+  def cert_store= cert_store
+    @http.cert_store = cert_store
+  end
+
+  # The client X509 certificate
   def certificate
     @http.certificate
   end
 
+  # Sets the client certificate to given X509 certificate.  If a path is given
+  # the certificate will be loaded and set.
+  def certificate= certificate
+    certificate = if OpenSSL::X509::Certificate === certificate then
+                    certificate
+                  else
+                    OpenSSL::X509::Certificate.new File.read certificate
+                  end
+
+    @http.certificate = certificate
+  end
+
+  # An OpenSSL private key or the path to a private key
+  def private_key
+    @http.private_key
+  end
+
+  # Sets the client's private key
+  def private_key= private_key
+    private_key = if OpenSSL::PKey::PKey === private_key then
+                    private_key
+                  else
+                    OpenSSL::PKey::RSA.new File.read(private_key), @pass
+                  end
+
+    @http.private_key = private_key
+  end
+
+  # SSL version to use
+  def ssl_version
+    @http.ssl_version
+  end if RUBY_VERSION > '1.9'
+
+  # Sets the SSL version to use
+  def ssl_version= ssl_version
+    @http.ssl_version = ssl_version
+  end if RUBY_VERSION > '1.9'
+
+  # A callback for additional certificate verification.  See
+  # OpenSSL::SSL::SSLContext#verify_callback
+  #
+  # The callback can be used for debugging or to ignore errors by always
+  # returning +true+.  Specifying nil uses the default method that was valid
+  # when the SSLContext was created
+  def verify_callback
+    @http.verify_callback
+  end
+
+  # Sets the certificate verify callback
+  def verify_callback= verify_callback
+    @http.verify_callback = verify_callback
+  end
+
+  # How to verify SSL connections.  Defaults to VERIFY_PEER
+  def verify_mode
+    @http.verify_mode
+  end
+
+  # Sets the mode for verifying SSL connections
+  def verify_mode= verify_mode
+    @http.verify_mode = verify_mode
+  end
+
   # :section: Timeouts
 
-  # Sets the conection idle timeout for persistent connections
+  # Reset connections that have not been used in this many seconds
+  def idle_timeout
+    @http.idle_timeout
+  end
+
+  # Sets the connection idle timeout for persistent connections
   def idle_timeout= timeout
-    @idle_timeout = timeout
-    @http.idle_timeout = timeout if @http
+    @http.idle_timeout = timeout
   end
 
   # :section: Utility
@@ -980,47 +1051,17 @@ class Mechanize::HTTP::Agent
     @context.log
   end
 
-  def set_http
-    @http = Net::HTTP::Persistent.new 'mechanize', @proxy_uri
-
-    @http.keep_alive = @keep_alive_time
-    @http.idle_timeout = @idle_timeout if @idle_timeout
-    @http.retry_change_requests = @retry_change_requests
-
-    @http.ca_file         = @ca_file
-    @http.cert_store      = @cert_store if @cert_store
-    @http.verify_callback = @verify_callback
-    @http.verify_mode     = @verify_mode if @verify_mode
-
-    # update our cached value
-    @verify_mode = @http.verify_mode
-    @cert_store  = @http.cert_store
-
-    if @cert and @key then
-      cert = if OpenSSL::X509::Certificate === @cert then
-               @cert
-             else
-               OpenSSL::X509::Certificate.new ::File.read @cert
-             end
-
-      key = if OpenSSL::PKey::PKey === @key then
-              @key
-            else
-              OpenSSL::PKey::RSA.new ::File.read(@key), @pass
-            end
-
-      @http.certificate = cert
-      @http.private_key = key
-    end
-  end
-
   ##
   # Sets the proxy address, port, user, and password +addr+ should be a host,
   # with no "http://", +port+ may be a port number, service name or port
   # number string.
 
-  def set_proxy(addr, port, user = nil, pass = nil)
-    return unless addr and port
+  def set_proxy addr, port, user = nil, pass = nil
+    unless addr and port then
+      @http.proxy = nil
+
+      return
+    end
 
     unless Integer === port then
       begin
@@ -1034,12 +1075,12 @@ class Mechanize::HTTP::Agent
       end
     end
 
-    @proxy_uri = URI "http://#{addr}"
-    @proxy_uri.port = port
-    @proxy_uri.user     = user if user
-    @proxy_uri.password = pass if pass
+    proxy_uri = URI "http://#{addr}"
+    proxy_uri.port = port
+    proxy_uri.user     = user if user
+    proxy_uri.password = pass if pass
 
-    @proxy_uri
+    @http.proxy = proxy_uri
   end
 
 end