bterlson-httpclient 2.1.4

data/README.txt

@@ -0,0 +1,368 @@
+httpclient - HTTP accessing library.
+Copyright (C) 2000-2009  NAKAMURA, Hiroshi  <nahi@ruby-lang.org>.
+
+'httpclient' gives something like the functionality of libwww-perl (LWP) in
+Ruby.  'httpclient' formerly known as 'http-access2'.
+
+See HTTPClient for documentation.
+
+
+== Features
+
+* methods like GET/HEAD/POST/* via HTTP/1.1.
+* HTTPS(SSL), Cookies, proxy, authentication(Digest, NTLM, Basic), etc.
+* asynchronous HTTP request, streaming HTTP request.
+
+* by contrast with net/http in standard distribution;
+  * Cookies support
+  * MT-safe
+  * streaming POST (POST with File/IO)
+  * Digest auth
+  * Negotiate/NTLM auth for WWW-Authenticate (requires net/htlm module)
+  * NTLM auth for Proxy-Authenticate (requires win32/sspi module)
+  * extensible with filter interface
+  * you don't have to care HTTP/1.1 persistent connection
+    (httpclient cares instead of you)
+
+* Not supported now
+  * Cache
+  * Rather advanced HTTP/1.1 usage such as Range, deflate, etc.
+    (of course you can set it in header by yourself)
+
+
+== Author
+
+Name:: NAKAMURA, Hiroshi
+E-mail:: nahi@ruby-lang.org
+Project web site:: http://dev.ctor.org/httpclient/
+
+
+== License
+
+This program is copyrighted free software by NAKAMURA, Hiroshi.  You can
+redistribute it and/or modify it under the same terms of Ruby's license;
+either the dual license version in 2003, or any later version.
+
+httpclient/session.rb is based on http-access.rb in http-access/0.0.4.
+Some part of code in http-access.rb was recycled in http-access2.rb.
+Those part is copyrighted by Maehashi-san who made and distributed
+http-access/0.0.4. Many thanks to Maehashi-san.
+
+
+== Install
+
+  $ ruby install.rb
+
+It will install lib/* to your site_ruby directory such as
+/usr/local/lib/ruby/site_ruby/1.8/.
+
+
+== Uninstall
+
+Delete installed files from your site_ruby directory.
+
+
+== Usage
+
+See HTTPClient for documentation.
+You can also check sample/howto.rb how to use APIs.
+
+
+== Download
+
+* Stable: http://dev.ctor.org/download/httpclient-2.1.4.tar.gz (tar + gzip)
+* Stable: http://dev.ctor.org/download/httpclient-2.1.4.zip (ZIP) 
+
+* Older versions: http://dev.ctor.org/download/archive/ 
+
+* Gem repository for stable version
+  * (at default remove source at rubyforge.org) 
+* Gem repository for development version
+  * http://dev.ctor.org/download/ 
+
+* svn: http://dev.ctor.org/svn/http-access2/trunk/ 
+
+=== Gem
+
+You can install httpclient with rubygems.
+
+  % gem install httpclient --source http://dev.ctor.org/download/
+
+
+== Bug report or Feature request
+
+Please file a ticket at the project web site.
+
+1. find a similar ticket from http://dev.ctor.org/http-access2/search?ticket=on
+2. login as an anonymous user from http://dev.ctor.org/http-access2/login
+3. create a new ticket from http://dev.ctor.org/http-access2/newticket 
+
+Thanks in advance. 
+
+
+== Changes
+
+  Feb 13, 2009 - version 2.1.4
+
+    * Bug fixes
+      * When we hit some site through http-proxy we get a response without
+        Content-Length header.  httpclient/2.1.3 drops response body for such
+        case. fixed. (#199)
+      * Avoid duplicated 'Date' header in request. Fixed. (#194)
+      * Avoid to add port number to 'Host' header.  Some servers like GFE/1.3
+        dislike it. Thanks to anonymous user for investigating the behavior.
+        (#195)
+      * httpclient/2.1.3 does not work when you fork a process after requiring
+        httpclient module (Passenger). Thanks to Akira Yamada for tracing down
+        this bug. (#197)
+      * httpclient/2.1.3 cannot handle Cookie header with 'expires=' and
+        'expires=""'.  Empty String for Time.parse returns Time.now unlike
+        ParseDate.parsedate. Thanks to Mark for the patch. (#200) 
+
+  Jan 8, 2009 - version 2.1.3.1
+
+    * Security fix introduced at 2.1.3.
+      * get_content/post_content of httpclient/2.1.3 may send secure cookies
+        for a https site to non-secure (non-https) site when the https site
+        redirects the request to a non-https site.  httpclient/2.1.3 caches
+        request object and reuses it for redirection.  It should not be cached
+        and recreated for each time as httpclient <= 2.1.2 and http-access2.
+      * I realized this bug when I was reading open-uri story on
+        [ruby-core:21205].  Ruby users should use open-uri rather than using
+        net/http directly wherever possible.
+
+  Dec 29, 2008 - version 2.1.3
+
+    * Features
+      * Proxy Authentication for SSL.
+      * Performance improvements.
+      * Full RDoc. Please tell me any English problem. Thanks in advance.
+      * Do multipart file upload when a given body includes a File. You don't
+        need to set 'Content-Type' and boundary String any more.
+      * Added propfind and proppatch methods. 
+
+    * Changes
+      * Avoid unnecessary memory consuming for get_content/post_content with
+        block.  get_content returns nil when you call it with a block.
+      * post_content with IO did not work when redirect/auth cycle is required.
+        (CAUTION: post_content now correctly follows redirection and posts the
+        given content)
+      * Exception handling cleanups.
+        * Raises HTTPClient::ConfigurationError? for environment problem.
+          (trying to do SSL without openssl installed for example)
+        * Raises HTTPClient::BadResponse? for HTTP response problem.  You can
+          get the response HTTPMessage returned via $!.res.
+        * Raises SocketError? for connection problem (as same as before). 
+
+    * Bug fixes
+      * Avoid unnecessary negotiation cycle for Negotiate(NTLM) authentication.
+        Thanks Rishav for great support for debugging Negotiate authentication.
+      * get_content/post_content with block yielded unexpected message body
+        during redirect/auth cycle.
+      * Relative URI redirection should be allowed from 2.1.2 but it did not
+        work... fixed.
+      * Avoid unnecessary timeout waiting when no message body returned such as
+        '204 No Content' for DAV.
+      * Avoid blocking on socket closing when the socket is already closed by
+        foreign host and the client runs under MT-condition. 
+
+  Sep 22, 2007 - version 2.1.2
+
+    * HTTP
+      * implemented Negotiate authentication with a support from exterior
+        modules. 'rubyntlm' module is required for Negotiate auth with IIS.
+        'win32/sspi' module is required for Negotiate auth with ISA.
+      * a workaround for Ubuntu + SonicWALL timeout problem. try to send HTTP
+        request in one chunk.
+
+    * SSL
+      * create new self-signing dist-cert which has serial number 0x01 and
+        embed it in httpclient.rb.
+      * update cacert.p7s. certificates are imported from cacerts in JRE 6
+        Update 2. 1 expired CA certificate
+        'C=US, O=GTE Corporation, CN=GTE CyberTrust Root' is removed.
+
+    * Bug fix
+      * [BUG] SSL + debug_dev didn't work under version 2.1.1.
+      * [BUG] Reason-Phrase of HTTP response status line can be empty according
+      * to RFC2616.
+
+  Aug 28, 2007 - version 2.1.1
+
+    * bug fix
+      * domain_match should be case insensitive. thanks to Brian for the patch.
+      * before calling SSLSocket#post_connection_check, check if
+        RUBY_VERSION > "1.8.4" for CN based wildcard certificate. when
+        RUBY_VERSION <= "1.8.4",  it fallbacks to the post_connection_check
+        method in HTTPClient so httpclient should run on 1.8.4 fine as before.
+
+    * misc
+      * added HTTPClient#test_loopback_http_response which accepts test
+        loopback response which contains HTTP header. 
+
+  Jul 14, 2007 - version 2.1.0
+
+    * program/project renamed from 'http-access2' to 'httpclient'.
+      there's compatibility layer included so existing programs for
+      http-access2 which uses HTTPAccess2::Client should work with
+      httpclient/2.1.0 correctly.
+
+    * misc
+      * install.rb did not install cacerts.p7s.  Thanks to knu.
+      * now HTTPClient loads http_proxy/HTTP_PROXY and no_proxy/NO_PROXY
+        environment variable at initialization time. bear in mind that it
+        doesn't load http_proxy/HTTP_PROXY when a library is considered to be
+        running under CGI environment (checked by ENVREQUEST_METHOD existence.
+        cgi_http_proxy/CGI_HTTP_PROXY is loaded instead.
+
+  Jul 4, 2007 - version 2.0.9
+
+    * bug fix
+      * fix the BasicAuth regression problem in 2.0.8.  A server may return
+        "BASIC" as an authenticate scheme label instead of "Basic".  It must be
+        treated as a case-insensitive token according to RFC2617 section 1.2.
+        Thanks to mwedeme for contributing the patch. (#159)
+
+  Jun 30, 2007 - version 2.0.8
+
+    * HTTP
+      * added request/response filter interface and implemented DigestAuth
+        based on the filter interface.  DigestAuth calc engine is based on
+        http://tools.assembla.com/breakout/wiki/DigestForSoap
+        Thanks to sromano. (#155)
+      * re-implemented BasicAuth based on the filter interface.  send BasicAuth
+        header only if it's needed. (#31)
+      * handle a response which has 2XX status code as a successfull response
+        while retry check.  applied the patch from Micah Wedemeyer.
+        Thanks! (#158)
+
+    * Connection
+      * show more friendly error message for unconnectable URL. (#156)
+
+    * bug fixes
+      * to avoid MIME format incompatibility, add empty epilogue chunk
+        explicitly.  Thanks to the anonymous user who reported #154 (#154)
+      * rescue EPIPE for keep-alive reconnecting.  Thanks to anonymous user
+        who posted a patch at #124. (#124)
+
+  May 13, 2007 - version 2.0.7
+
+    * HTTP
+      * added proxyauth support. (#6)
+      * let developer allow to rescue a redirect with relative URI. (#28)
+      * changed last-chunk condition statement to allow "0000\r\n" marker from
+        WebLogic Server 7.0 SP5 instead of "0\r\n". (#30)
+      * fixed multipart form submit. (#29, #116)
+      * use http_date format as a date in a request header. (#35)
+      * avoid duplicated Date header when running under mod_ruby. (#127)
+      * reason phrase in Message#reason contains \r. (#122)
+      * trim "\n"s in base64 encoded BasicAuth value for interoperability.
+        (#149)
+      * let retry_connect return a Message not a content. (#119)
+      * rescue SocketError and dump a message when a wrong address given. (#152)
+
+    * HTTP-Cookies
+      * changed "domain" parameter matching condition statement to allow
+        followings; (#24, #32, #118, #147)
+        * [host, domain] = [rubyforge.com, .rubyforge.com]
+        * [host, domain] = [reddit.com, reddit.com]
+
+    * SSL
+      * bundles CA certificates as trust anchors.
+      * allow user to get peer_cert. (#117, #123)
+      * added wildcard certificate support. (#151)
+      * SSL + HTTP keep-alive + long wait causes uncaught exception.  fixed.
+        (#120)
+
+    * Connection
+      * fixed a loop condition bug that caused intermittent empty response.
+        (#150, #26, #125)
+
+  September 16, 2005 - version 2.0.6
+
+    * HTTP
+      * allows redirects from a "POST" request.  imported a patch from sveit.
+        Thanks! (#7)
+      * add 'content-type: application/application/x-www-form-urlencoded' when
+        a request contains message-body. (#11)
+      * HTTP/0.9 support.  (#15)
+      * allows submitting multipart forms.  imported a patch from sveit.
+        Thanks! (#7)
+
+    * HTTP-Cookies
+      * avoid NameError when a cookie value is nil. (#10)
+      * added netscape_rule property to CookieManager (false by default).  You
+        can turn on the domain attribute test of Netscape rule with the
+        property.  cf. http://wp.netscape.com/newsref/std/cookie_spec.html
+      * added HTTPClient#cookie_manager property for accessing its properties.
+        (#13)
+      * added save_all_cookies method to save unused and discarded cookies as
+        well.  The patch is from Christian Lademann.  Thanks!  (#21)
+      * allow to set cookie_manager.  raise an error when set_cookie_store
+        called and cookie_store has already been set. (#20)
+
+    * SSL
+      * allows SSL connection debugging when debug_dev != nil. (#14)
+      * skip post_connection_check when
+        verify_mode == OpenSSL::SSL::VERIFY_NONE.  Thanks to kdraper. (#12)
+      * post_connection_check: support a certificate with a wildcard in the
+        hostname. (#18)
+      * avoid NameError when no peer_cert and VERIFY_FAIL_IF_NO_PEER_CERT
+        given.  Thanks to Christian Lademann.
+
+    * Connection
+      * insert a connecting host and port to an exception message when
+        connecting failed. (#5)
+      * added socket_sync property to HTTPClient(HTTPAccess2::Client) that
+        controls socket's sync property.  the default value is true.  CAUTION:
+        if your ruby is older than 2005-09-06 and you want to use SSL
+        connection, do not set socket_sync = false to avoid a blocking bug of
+        openssl/buffering.rb.
+
+  December 24, 2004 - version 2.0.5
+    This is a minor bug fix release.
+    - Connect/Send/Receive timeout cannot be configured. fixed.
+    - IPSocket#addr caused SocketError? on Mac OS X 10.3.6 + ruby-1.8.1 GA.
+      fixed.
+    - There is a server which does not like 'foo.bar.com:80' style Host header.
+      The server for http://rubyforge.org/export/rss_sfnews.php seems to
+      dislike HTTP/1.1 Host header "Host: rubyforge.net:80".  It returns
+      HTTP 302: Found and redirects to the page again, causes
+      HTTPAccess2::Client to raise "retry count exceeded".  Keat found that the
+      server likes "Host: rubyforge.net" (not with port number).
+
+  February 11, 2004 - version 2.0.4
+    - add Client#redirect_uri_callback interface.
+    - refactorings and bug fixes found during negative test.
+    - add SSL test.
+
+  December 16, 2003 - version 2.0.3
+    - no_proxy was broken in 2.0.2.
+    - do not dump 'Host' header under protocol_version == 'HTTP/1.0'
+
+  December ?, 2003 - version 2.0.2
+    - do not trust HTTP_PROXY environment variable. set proxy server manually.
+      http://ftp.ics.uci.edu/pub/websoft/libwww-perl/archive/2001h1/0072.html
+      http://ftp.ics.uci.edu/pub/websoft/libwww-perl/archive/2001h1/0241.html
+      http://curl.haxx.se/mail/archive-2001-12/0034.html
+    - follow ossl2 change.
+
+  October 4, 2003 - version 2.0.1
+    Query was not escaped when query was given as an Array or a Hash.  Fixed.
+    Do not use http_proxy defined by ENV['http_proxy'] or ENV['HTTP_PROXY'] if
+      the destination host is 'localhost'.
+    Hosts which matches ENV['no_proxy'] or ENV['NO_PROXY'] won't be proxyed.
+      [,:] separated. ("ruby-lang.org:rubyist.net")
+      No regexp. (give "ruby-lang.org", not "*.ruby-lang.org")
+      If you want specify hot by IP address, give full address.
+	("192.168.1.1, 192.168.1.2")
+
+  September 10, 2003 - version 2.0
+    CamelCase to non_camel_case.
+    SSL support (requires Ruby/OpenSSL).
+    Cookies support.  lib/http-access2/cookie.rb is redistributed file which is
+      originally included in Webagent by TAKAHASHI `Maki' Masayoshi.  You can
+      download the entire package from http://www.rubycolor.org/arc/.
+
+  January 11, 2003 - version J
+    ruby/1.8 support.

data/VERSION.yml

@@ -0,0 +1,4 @@
+--- 
+:major: 2
+:minor: 1
+:patch: 4

data/lib/http-access2.rb

@@ -0,0 +1,53 @@
+# HTTPAccess2 - HTTP accessing library.
+# Copyright (C) 2000-2007  NAKAMURA, Hiroshi  <nakahiro@sarion.co.jp>.
+
+# This program is copyrighted free software by NAKAMURA, Hiroshi.  You can
+# redistribute it and/or modify it under the same terms of Ruby's license;
+# either the dual license version in 2003, or any later version.
+
+# http-access2.rb is based on http-access.rb in http-access/0.0.4.  Some part
+# of code in http-access.rb was recycled in http-access2.rb.  Those part is
+# copyrighted by Maehashi-san.
+
+
+require 'httpclient'
+
+
+module HTTPAccess2
+  VERSION = ::HTTPClient::VERSION
+  RUBY_VERSION_STRING = ::HTTPClient::RUBY_VERSION_STRING
+  SSLEnabled = ::HTTPClient::SSLEnabled
+  SSPIEnabled = ::HTTPClient::SSPIEnabled
+  DEBUG_SSL = true
+
+  Util = ::HTTPClient::Util
+
+  class Client < ::HTTPClient
+    class RetryableResponse < StandardError
+    end
+  end
+
+  SSLConfig = ::HTTPClient::SSLConfig
+  BasicAuth = ::HTTPClient::BasicAuth
+  DigestAuth = ::HTTPClient::DigestAuth
+  NegotiateAuth = ::HTTPClient::NegotiateAuth
+  AuthFilterBase = ::HTTPClient::AuthFilterBase
+  WWWAuth = ::HTTPClient::WWWAuth
+  ProxyAuth = ::HTTPClient::ProxyAuth
+  Site = ::HTTPClient::Site
+  Connection = ::HTTPClient::Connection
+  SessionManager = ::HTTPClient::SessionManager
+  SSLSocketWrap = ::HTTPClient::SSLSocketWrap
+  DebugSocket = ::HTTPClient::DebugSocket
+
+  class Session < ::HTTPClient::Session
+    class Error < StandardError
+    end
+    class InvalidState < Error
+    end
+    class BadResponse < Error
+    end
+    class KeepAliveDisconnected < Error
+    end
+  end
+end

data/lib/http-access2/cookie.rb

@@ -0,0 +1 @@
+require 'httpclient/cookie'

data/lib/http-access2/http.rb

@@ -0,0 +1 @@
+require 'httpclient/http'

data/lib/httpclient.rb

@@ -0,0 +1,1016 @@
+# HTTPClient - HTTP client library.
+# Copyright (C) 2000-2009  NAKAMURA, Hiroshi  <nahi@ruby-lang.org>.
+#
+# This program is copyrighted free software by NAKAMURA, Hiroshi.  You can
+# redistribute it and/or modify it under the same terms of Ruby's license;
+# either the dual license version in 2003, or any later version.
+
+$:.unshift(File.dirname(__FILE__))
+
+require 'uri'
+require 'stringio'
+require 'digest/sha1'
+
+# Extra library
+require 'httpclient/util'
+require 'httpclient/ssl_config'
+require 'httpclient/connection'
+require 'httpclient/session'
+require 'httpclient/http'
+require 'httpclient/auth'
+require 'httpclient/cookie'
+
+
+# The HTTPClient class provides several methods for accessing Web resources
+# via HTTP.
+#
+# HTTPClient instance is designed to be MT-safe.  You can call a HTTPClient
+# instance from several threads without synchronization after setting up an
+# instance.
+#
+#   clnt = HTTPClient.new
+#   clnt.set_cookie_store('/home/nahi/cookie.dat')
+#   urls.each do |url|
+#     Thread.new(url) do |u|
+#       p clnt.head(u).status
+#     end
+#   end
+#
+# == How to use
+#
+# At first, how to create your client.  See initialize for more detail.
+#
+# 1. Create simple client.
+#
+#     clnt = HTTPClient.new
+#
+# 2. Accessing resources through HTTP proxy.  You can use environment
+#    variable 'http_proxy' or 'HTTP_PROXY' instead.
+#
+#     clnt = HTTPClient.new('http://myproxy:8080')
+#
+# === How to retrieve web resources
+#
+# See get_content.
+#
+# 1. Get content of specified URL.  It returns a String of whole result.
+#
+#     puts clnt.get_content('http://dev.ctor.org/')
+#
+# 2. Get content as chunks of String.  It yields chunks of String.
+#
+#     clnt.get_content('http://dev.ctor.org/') do |chunk|
+#       puts chunk
+#     end
+#
+# === Invoking other HTTP methods
+#
+# See head, get, post, put, delete, options, propfind, proppatch and trace.  
+# It returns a HTTP::Message instance as a response.
+#
+# 1. Do HEAD request.
+#
+#     res = clnt.head(uri)
+#     p res.header['Last-Modified'][0]
+#
+# 2. Do GET request with query.
+#
+#     query = { 'keyword' => 'ruby', 'lang' => 'en' }
+#     res = clnt.get(uri, query)
+#     p res.status
+#     p res.contenttype
+#     p res.header['X-Custom']
+#     puts res.content
+#
+# === How to POST
+#
+# See post.
+#
+# 1. Do POST a form data.
+#
+#     body = { 'keyword' => 'ruby', 'lang' => 'en' }
+#     res = clnt.post(uri, body)
+#
+# 2. Do multipart file upload with POST.  No need to set extra header by
+#    yourself from httpclient/2.1.4.
+#
+#     File.open('/tmp/post_data') do |file|
+#       body = { 'upload' => file, 'user' => 'nahi' }
+#       res = clnt.post(uri, body)
+#     end
+#
+# === Accessing via SSL
+#
+# Ruby needs to be compiled with OpenSSL.
+#
+# 1. Get content of specified URL via SSL.
+#    Just pass an URL which starts with 'https://'.
+#
+#     https_url = 'https://www.rsa.com'
+#     clnt.get_content(https_url)
+#
+# 2. Getting peer certificate from response.
+#
+#     res = clnt.get(https_url)
+#     p res.peer_cert #=> returns OpenSSL::X509::Certificate
+#
+# 3. Configuring OpenSSL options.  See HTTPClient::SSLConfig for more details.
+#
+#     user_cert_file = 'cert.pem'
+#     user_key_file = 'privkey.pem'
+#     clnt.ssl_config.set_client_cert_file(user_cert_file, user_key_file)
+#     clnt.get_content(https_url)
+#
+# === Handling Cookies
+#
+# 1. Using volatile Cookies.  Nothing to do.  HTTPClient handles Cookies.
+#
+#     clnt = HTTPClient.new
+#     clnt.get_content(url1) # receives Cookies.
+#     clnt.get_content(url2) # sends Cookies if needed.
+#
+# 2. Saving non volatile Cookies to a specified file.  Need to set a file at
+#    first and invoke save method at last.
+#
+#     clnt = HTTPClient.new
+#     clnt.set_cookie_store('/home/nahi/cookie.dat')
+#     clnt.get_content(url)
+#     ...
+#     clnt.save_cookie_store
+#
+# 3. Disabling Cookies.
+#
+#     clnt = HTTPClient.new
+#     clnt.cookie_manager = nil
+#
+# === Configuring authentication credentials
+#
+# 1. Authentication with Web server.  Supports BasicAuth, DigestAuth, and
+#    Negotiate/NTLM (requires ruby/ntlm module).
+#
+#     clnt = HTTPClient.new
+#     domain = 'http://dev.ctor.org/http-access2/'
+#     user = 'user'
+#     password = 'user'
+#     clnt.set_auth(domain, user, password)
+#     p clnt.get_content('http://dev.ctor.org/http-access2/login').status
+#
+# 2. Authentication with Proxy server.  Supports BasicAuth and NTLM
+#    (requires win32/sspi)
+#
+#     clnt = HTTPClient.new(proxy)
+#     user = 'proxy'
+#     password = 'proxy'
+#     clnt.set_proxy_auth(user, password)
+#     p clnt.get_content(url)
+#
+# === Invoking HTTP methods with custom header
+#
+# Pass a Hash or an Array for extheader argument.
+#
+#     extheader = { 'Accept' => '*/*' }
+#     clnt.get_content(uri, query, extheader)
+#
+#     extheader = [['Accept', 'image/jpeg'], ['Accept', 'image/png']]
+#     clnt.get_content(uri, query, extheader)
+#
+# === Invoking HTTP methods asynchronously
+#
+# See head_async, get_async, post_async, put_async, delete_async,
+# options_async, propfind_async, proppatch_async, and trace_async.
+# It immediately returns a HTTPClient::Connection instance as a returning value.
+#
+#     connection = clnt.post_async(url, body)
+#     print 'posting.'
+#     while true
+#       break if connection.finished?
+#       print '.'
+#       sleep 1
+#     end
+#     puts '.'
+#     res = connection.pop
+#     p res.status
+#     p res.content.read # res.content is an IO for the res of async method.
+#
+# === Shortcut methods
+#
+# You can invoke get_content, get, etc. without creating HTTPClient instance.
+#
+#   ruby -rhttpclient -e 'puts HTTPClient.get_content(ARGV.shift)' http://dev.ctor.org/
+#   ruby -rhttpclient -e 'p HTTPClient.head(ARGV.shift).header["last-modified"]' http://dev.ctor.org/
+#
+class HTTPClient
+  VERSION = '2.1.4.1'
+  RUBY_VERSION_STRING = "ruby #{RUBY_VERSION} (#{RUBY_RELEASE_DATE}) [#{RUBY_PLATFORM}]"
+  /: (\S+) (\S+)/ =~ %q$Id$
+  LIB_NAME = "(#{$1}/#{$2}, #{RUBY_VERSION_STRING})"
+
+  include Util
+
+  # Raised for indicating running environment configuration error for example
+  # accessing via SSL under the ruby which is not compiled with OpenSSL.
+  class ConfigurationError < StandardError
+  end
+
+  # Raised for indicating HTTP response error.
+  class BadResponseError < RuntimeError
+    # HTTP::Message:: a response
+    attr_reader :res
+
+    def initialize(msg, res = nil) # :nodoc:
+      super(msg)
+      @res = res
+    end
+  end
+
+  # Raised for indicating a timeout error.
+  class TimeoutError < RuntimeError
+  end
+
+  # Raised for indicating a connection timeout error.
+  # You can configure connection timeout via HTTPClient#connect_timeout=.
+  class ConnectTimeoutError < TimeoutError
+  end
+
+  # Raised for indicating a request sending timeout error.
+  # You can configure request sending timeout via HTTPClient#send_timeout=.
+  class SendTimeoutError < TimeoutError
+  end
+
+  # Raised for indicating a response receiving timeout error.
+  # You can configure response receiving timeout via
+  # HTTPClient#receive_timeout=.
+  class ReceiveTimeoutError < TimeoutError
+  end
+
+  # Deprecated.  just for backward compatibility
+  class Session
+    BadResponse = ::HTTPClient::BadResponseError
+  end
+
+  class << self
+    %w(get_content post_content head get post put delete options propfind proppatch trace).each do |name|
+      eval <<-EOD
+        def #{name}(*arg)
+          new.#{name}(*arg)
+        end
+      EOD
+    end
+
+  private
+
+    def attr_proxy(symbol, assignable = false)
+      name = symbol.to_s
+      define_method(name) {
+        @session_manager.__send__(name)
+      }
+      if assignable
+        aname = name + '='
+        define_method(aname) { |rhs|
+          reset_all
+          @session_manager.__send__(aname, rhs)
+        }
+      end
+    end
+  end
+
+  # HTTPClient::SSLConfig:: SSL configurator.
+  attr_reader :ssl_config
+  # WebAgent::CookieManager:: Cookies configurator.
+  attr_accessor :cookie_manager
+  # An array of response HTTP message body String which is used for loop-back
+  # test.  See test/* to see how to use it.  If you want to do loop-back test
+  # of HTTP header, use test_loopback_http_response instead.
+  attr_reader :test_loopback_response
+  # An array of request filter which can trap HTTP request/response.
+  # See HTTPClient::WWWAuth to see how to use it.
+  attr_reader :request_filter
+  # HTTPClient::ProxyAuth:: Proxy authentication handler.
+  attr_reader :proxy_auth
+  # HTTPClient::WWWAuth:: WWW authentication handler.
+  attr_reader :www_auth
+  # How many times get_content and post_content follows HTTP redirect.
+  # 10 by default.
+  attr_accessor :follow_redirect_count
+
+  # Set HTTP version as a String:: 'HTTP/1.0' or 'HTTP/1.1'
+  attr_proxy(:protocol_version, true)
+  # Connect timeout in sec.
+  attr_proxy(:connect_timeout, true)
+  # Request sending timeout in sec.
+  attr_proxy(:send_timeout, true)
+  # Response receiving timeout in sec.
+  attr_proxy(:receive_timeout, true)
+  # Negotiation retry count for authentication.  5 by default.
+  attr_proxy(:protocol_retry_count, true)
+  # if your ruby is older than 2005-09-06, do not set socket_sync = false to
+  # avoid an SSL socket blocking bug in openssl/buffering.rb.
+  attr_proxy(:socket_sync, true)
+  # User-Agent header in HTTP request.
+  attr_proxy(:agent_name, true)
+  # From header in HTTP request.
+  attr_proxy(:from, true)
+  # An array of response HTTP String (not a HTTP message body) which is used
+  # for loopback test.  See test/* to see how to use it.
+  attr_proxy(:test_loopback_http_response)
+
+  # Default extheader for PROPFIND request.
+  PROPFIND_DEFAULT_EXTHEADER = { 'Depth' => '0' }
+
+  # Creates a HTTPClient instance which manages sessions, cookies, etc.
+  #
+  # HTTPClient.new takes 3 optional arguments for proxy url string,
+  # User-Agent String and From header String.  User-Agent and From are embedded
+  # in HTTP request Header if given.  No User-Agent and From header added
+  # without setting it explicitly.
+  #
+  #   proxy = 'http://myproxy:8080'
+  #   agent_name = 'MyAgent/0.1'
+  #   from = 'from@example.com'
+  #   HTTPClient.new(proxy, agent_name, from)
+  #
+  # You can use a keyword argument style Hash.  Keys are :proxy, :agent_name
+  # and :from.
+  #
+  #   HTTPClient.new(:agent_name = 'MyAgent/0.1')
+  def initialize(*args)
+    proxy, agent_name, from = keyword_argument(args, :proxy, :agent_name, :from)
+    @proxy = nil        # assigned later.
+    @no_proxy = nil
+    @www_auth = WWWAuth.new
+    @proxy_auth = ProxyAuth.new
+    @request_filter = [@proxy_auth, @www_auth]
+    @debug_dev = nil
+    @redirect_uri_callback = method(:default_redirect_uri_callback)
+    @test_loopback_response = []
+    @session_manager = SessionManager.new(self)
+    @session_manager.agent_name = agent_name
+    @session_manager.from = from
+    @session_manager.ssl_config = @ssl_config = SSLConfig.new(self)
+    @cookie_manager = WebAgent::CookieManager.new
+    @follow_redirect_count = 10
+    load_environment
+    self.proxy = proxy if proxy
+  end
+
+  # Returns debug device if exists.  See debug_dev=.
+  def debug_dev
+    @debug_dev
+  end
+
+  # Sets debug device.  Once debug device is set, all HTTP requests and
+  # responses are dumped to given device.  dev must respond to << for dump.
+  #
+  # Calling this method resets all existing sessions.
+  def debug_dev=(dev)
+    @debug_dev = dev
+    reset_all
+    @session_manager.debug_dev = dev
+  end
+
+  # Returns URI object of HTTP proxy if exists.
+  def proxy
+    @proxy
+  end
+
+  # Sets HTTP proxy used for HTTP connection.  Given proxy can be an URI,
+  # a String or nil.  You can set user/password for proxy authentication like
+  # HTTPClient#proxy = 'http://user:passwd@myproxy:8080'
+  #
+  # You can use environment variable 'http_proxy' or 'HTTP_PROXY' for it.
+  # You need to use 'cgi_http_proxy' or 'CGI_HTTP_PROXY' instead if you run
+  # HTTPClient from CGI environment from security reason. (HTTPClient checks
+  # 'REQUEST_METHOD' environment variable whether it's CGI or not)
+  #
+  # Calling this method resets all existing sessions.
+  def proxy=(proxy)
+    if proxy.nil?
+      @proxy = nil
+      @proxy_auth.reset_challenge
+    else
+      @proxy = urify(proxy)
+      if @proxy.scheme == nil or @proxy.scheme.downcase != 'http' or
+          @proxy.host == nil or @proxy.port == nil
+        raise ArgumentError.new("unsupported proxy #{proxy}")
+      end
+      @proxy_auth.reset_challenge
+      if @proxy.user || @proxy.password
+        @proxy_auth.set_auth(@proxy.user, @proxy.password)
+      end
+    end
+    reset_all
+    @session_manager.proxy = @proxy
+    @proxy
+  end
+
+  # Returns NO_PROXY setting String if given.
+  def no_proxy
+    @no_proxy
+  end
+
+  # Sets NO_PROXY setting String.  no_proxy must be a comma separated String.
+  # Each entry must be 'host' or 'host:port' such as;
+  # HTTPClient#no_proxy = 'example.com,example.co.jp:443'
+  #
+  # 'localhost' is treated as a no_proxy site regardless of explicitly listed.
+  # HTTPClient checks given URI objects before accessing it.
+  # 'host' is tail string match.  No IP-addr conversion.
+  #
+  # You can use environment variable 'no_proxy' or 'NO_PROXY' for it.
+  #
+  # Calling this method resets all existing sessions.
+  def no_proxy=(no_proxy)
+    @no_proxy = no_proxy
+    reset_all
+  end
+
+  # Sets credential for Web server authentication.
+  # domain:: a String or an URI to specify where HTTPClient should use this
+  #       credential.  If you set uri to nil, HTTPClient uses this credential
+  #       wherever a server requires it.
+  # user:: username String.
+  # passwd:: password String.
+  #
+  # You can set multiple credentials for each uri.
+  #
+  #   clnt.set_auth('http://www.example.com/foo/', 'foo_user', 'passwd')
+  #   clnt.set_auth('http://www.example.com/bar/', 'bar_user', 'passwd')
+  #
+  # Calling this method resets all existing sessions.
+  def set_auth(domain, user, passwd)
+    uri = urify(domain)
+    @www_auth.set_auth(uri, user, passwd)
+    reset_all
+  end
+
+  # Deprecated.  Use set_auth instead.
+  def set_basic_auth(domain, user, passwd)
+    uri = urify(domain)
+    @www_auth.basic_auth.set(uri, user, passwd)
+    reset_all
+  end
+
+  # Sets credential for Proxy authentication.
+  # user:: username String.
+  # passwd:: password String.
+  #
+  # Calling this method resets all existing sessions.
+  def set_proxy_auth(user, passwd)
+    @proxy_auth.set_auth(user, passwd)
+    reset_all
+  end
+
+  # Sets the filename where non-volatile Cookies be saved by calling
+  # save_cookie_store.
+  # This method tries to load and managing Cookies from the specified file.
+  #
+  # Calling this method resets all existing sessions.
+  def set_cookie_store(filename)
+    @cookie_manager.cookies_file = filename
+    @cookie_manager.load_cookies if filename
+    reset_all
+  end
+
+  # Try to save Cookies to the file specified in set_cookie_store.  Unexpected
+  # error will be raised if you don't call set_cookie_store first.
+  # (interface mismatch between WebAgent::CookieManager implementation)
+  def save_cookie_store
+    @cookie_manager.save_cookies
+  end
+
+  # Sets callback proc when HTTP redirect status is returned for get_content
+  # and post_content.  default_redirect_uri_callback is used by default.
+  #
+  # If you need strict implementation which does not allow relative URI
+  # redirection, set strict_redirect_uri_callback instead.
+  #
+  #   clnt.redirect_uri_callback = clnt.method(:strict_redirect_uri_callback)
+  #
+  def redirect_uri_callback=(redirect_uri_callback)
+    @redirect_uri_callback = redirect_uri_callback
+  end
+
+  # Retrieves a web resource.
+  #
+  # uri:: a String or an URI object which represents an URL of web resource.
+  # query:: a Hash or an Array of query part of URL.
+  #         e.g. { "a" => "b" } => 'http://host/part?a=b'.
+  #         Give an array to pass multiple value like
+  #         [["a", "b"], ["a", "c"]] => 'http://host/part?a=b&a=c'.
+  # extheader:: a Hash or an Array of extra headers.  e.g.
+  #             { 'Accept' => '*/*' } or
+  #             [['Accept', 'image/jpeg'], ['Accept', 'image/png']].
+  # &block:: Give a block to get chunked message-body of response like
+  #          get_content(uri) { |chunked_body| ... }.
+  #          Size of each chunk may not be the same.
+  #
+  # get_content follows HTTP redirect status (see HTTP::Status.redirect?)
+  # internally and try to retrieve content from redirected URL.  See
+  # redirect_uri_callback= how HTTP redirection is handled.
+  #
+  # If you need to get full HTTP response including HTTP status and headers,
+  # use get method.  get returns HTTP::Message as a response and you need to
+  # follow HTTP redirect by yourself if you need.
+  def get_content(uri, query = nil, extheader = {}, &block)
+    follow_redirect(:get, uri, query, nil, extheader, &block).content
+  end
+
+  # Posts a content.
+  #
+  # uri:: a String or an URI object which represents an URL of web resource.
+  # body:: a Hash or an Array of body part.
+  #        e.g. { "a" => "b" } => 'a=b'.
+  #        Give an array to pass multiple value like
+  #        [["a", "b"], ["a", "c"]] => 'a=b&a=c'.
+  #        When you pass a File as a value, it will be posted as a
+  #        multipart/form-data.  e.g. { 'upload' => file }
+  # extheader:: a Hash or an Array of extra headers.  e.g.
+  #             { 'Accept' => '*/*' } or
+  #             [['Accept', 'image/jpeg'], ['Accept', 'image/png']].
+  # &block:: Give a block to get chunked message-body of response like
+  #          post_content(uri) { |chunked_body| ... }.
+  #          Size of each chunk may not be the same.
+  #
+  # post_content follows HTTP redirect status (see HTTP::Status.redirect?)
+  # internally and try to post the content to redirected URL.  See
+  # redirect_uri_callback= how HTTP redirection is handled.
+  #
+  # If you need to get full HTTP response including HTTP status and headers,
+  # use post method.
+  def post_content(uri, body = nil, extheader = {}, &block)
+    follow_redirect(:post, uri, nil, body, extheader, &block).content
+  end
+
+  # A method for redirect uri callback.  How to use:
+  #   clnt.redirect_uri_callback = clnt.method(:strict_redirect_uri_callback)
+  # This callback does not allow relative redirect such as
+  #   Location: ../foo/
+  # in HTTP header. (raises BadResponseError instead)
+  def strict_redirect_uri_callback(uri, res)
+    newuri = URI.parse(res.header['location'][0])
+    if https?(uri) && !https?(newuri)
+      raise BadResponseError.new("redirecting to non-https resource")
+    end
+    unless newuri.is_a?(URI::HTTP)
+      raise BadResponseError.new("unexpected location: #{newuri}", res)
+    end
+    puts "redirect to: #{newuri}" if $DEBUG
+    newuri
+  end
+
+  # A default method for redirect uri callback.  This method is used by
+  # HTTPClient instance by default.
+  # This callback allows relative redirect such as
+  #   Location: ../foo/
+  # in HTTP header.
+  def default_redirect_uri_callback(uri, res)
+    newuri = URI.parse(res.header['location'][0])
+    if https?(uri) && !https?(newuri)
+      raise BadResponseError.new("redirecting to non-https resource")
+    end
+    unless newuri.is_a?(URI::HTTP)
+      newuri = uri + newuri
+      STDERR.puts("could be a relative URI in location header which is not recommended")
+      STDERR.puts("'The field value consists of a single absolute URI' in HTTP spec")
+    end
+    puts "redirect to: #{newuri}" if $DEBUG
+    newuri
+  end
+
+  # Sends HEAD request to the specified URL.  See request for arguments.
+  def head(uri, query = nil, extheader = {})
+    request(:head, uri, query, nil, extheader)
+  end
+
+  # Sends GET request to the specified URL.  See request for arguments.
+  def get(uri, query = nil, extheader = {}, &block)
+    request(:get, uri, query, nil, extheader, &block)
+  end
+
+  # Sends POST request to the specified URL.  See request for arguments.
+  def post(uri, body = nil, extheader = {}, &block)
+    request(:post, uri, nil, body, extheader, &block)
+  end
+
+  # Sends PUT request to the specified URL.  See request for arguments.
+  def put(uri, body = nil, extheader = {}, &block)
+    request(:put, uri, nil, body, extheader, &block)
+  end
+
+  # Sends DELETE request to the specified URL.  See request for arguments.
+  def delete(uri, extheader = {}, &block)
+    request(:delete, uri, nil, nil, extheader, &block)
+  end
+
+  # Sends OPTIONS request to the specified URL.  See request for arguments.
+  def options(uri, extheader = {}, &block)
+    request(:options, uri, nil, nil, extheader, &block)
+  end
+
+  # Sends PROPFIND request to the specified URL.  See request for arguments.
+  def propfind(uri, extheader = PROPFIND_DEFAULT_EXTHEADER, &block)
+    request(:propfind, uri, nil, nil, extheader, &block)
+  end
+  
+  # Sends PROPPATCH request to the specified URL.  See request for arguments.
+  def proppatch(uri, body = nil, extheader = {}, &block)
+    request(:proppatch, uri, nil, body, extheader, &block)
+  end
+  
+  # Sends TRACE request to the specified URL.  See request for arguments.
+  def trace(uri, query = nil, body = nil, extheader = {}, &block)
+    request('TRACE', uri, query, body, extheader, &block)
+  end
+
+  # Sends a request to the specified URL.
+  #
+  # method:: HTTP method to be sent.  method.to_s.upcase is used.
+  # uri:: a String or an URI object which represents an URL of web resource.
+  # query:: a Hash or an Array of query part of URL.
+  #         e.g. { "a" => "b" } => 'http://host/part?a=b'
+  #         Give an array to pass multiple value like
+  #         [["a", "b"], ["a", "c"]] => 'http://host/part?a=b&a=c'
+  # body:: a Hash or an Array of body part.
+  #        e.g. { "a" => "b" } => 'a=b'.
+  #        Give an array to pass multiple value like
+  #        [["a", "b"], ["a", "c"]] => 'a=b&a=c'.
+  #        When the given method is 'POST' and the given body contains a file
+  #        as a value, it will be posted as a multipart/form-data.
+  #        e.g. { 'upload' => file }
+  #        See HTTP::Message.file? for actual condition of 'a file'.
+  # extheader:: a Hash or an Array of extra headers.  e.g.
+  #             { 'Accept' => '*/*' } or
+  #             [['Accept', 'image/jpeg'], ['Accept', 'image/png']].
+  # &block:: Give a block to get chunked message-body of response like
+  #          get(uri) { |chunked_body| ... }.
+  #          Size of each chunk may not be the same.
+  #
+  # You can also pass a String as a body.  HTTPClient just sends a String as
+  # a HTTP request message body.
+  #
+  # When you pass an IO as a body, HTTPClient sends it as a HTTP request with
+  # chunked encoding (Transfer-Encoding: chunked in HTTP header).  Bear in mind
+  # that some server application does not support chunked request.  At least
+  # cgi.rb does not support it.
+  def request(method, uri, query = nil, body = nil, extheader = {}, &block)
+    uri = urify(uri)
+    proxy = no_proxy?(uri) ? nil : @proxy
+    req = create_request(method, uri, query, body, extheader)
+    if block
+      filtered_block = proc { |res, str|
+        block.call(str)
+      }
+    end
+    do_request(req, proxy, &filtered_block)
+  end
+
+  # Sends HEAD request in async style.  See request_async for arguments.
+  # It immediately returns a HTTPClient::Connection instance as a result.
+  def head_async(uri, query = nil, extheader = {})
+    request_async(:head, uri, query, nil, extheader)
+  end
+
+  # Sends GET request in async style.  See request_async for arguments.
+  # It immediately returns a HTTPClient::Connection instance as a result.
+  def get_async(uri, query = nil, extheader = {})
+    request_async(:get, uri, query, nil, extheader)
+  end
+
+  # Sends POST request in async style.  See request_async for arguments.
+  # It immediately returns a HTTPClient::Connection instance as a result.
+  def post_async(uri, body = nil, extheader = {})
+    request_async(:post, uri, nil, body, extheader)
+  end
+
+  # Sends PUT request in async style.  See request_async for arguments.
+  # It immediately returns a HTTPClient::Connection instance as a result.
+  def put_async(uri, body = nil, extheader = {})
+    request_async(:put, uri, nil, body, extheader)
+  end
+
+  # Sends DELETE request in async style.  See request_async for arguments.
+  # It immediately returns a HTTPClient::Connection instance as a result.
+  def delete_async(uri, extheader = {})
+    request_async(:delete, uri, nil, nil, extheader)
+  end
+
+  # Sends OPTIONS request in async style.  See request_async for arguments.
+  # It immediately returns a HTTPClient::Connection instance as a result.
+  def options_async(uri, extheader = {})
+    request_async(:options, uri, nil, nil, extheader)
+  end
+
+  # Sends PROPFIND request in async style.  See request_async for arguments.
+  # It immediately returns a HTTPClient::Connection instance as a result.
+  def propfind_async(uri, extheader = PROPFIND_DEFAULT_EXTHEADER)
+    request_async(:propfind, uri, nil, nil, extheader)
+  end
+  
+  # Sends PROPPATCH request in async style.  See request_async for arguments.
+  # It immediately returns a HTTPClient::Connection instance as a result.
+  def proppatch_async(uri, body = nil, extheader = {})
+    request_async(:proppatch, uri, nil, body, extheader)
+  end
+  
+  # Sends TRACE request in async style.  See request_async for arguments.
+  # It immediately returns a HTTPClient::Connection instance as a result.
+  def trace_async(uri, query = nil, body = nil, extheader = {})
+    request_async(:trace, uri, query, body, extheader)
+  end
+
+  # Sends a request in async style.  request method creates new Thread for
+  # HTTP connection and returns a HTTPClient::Connection instance immediately.
+  #
+  # Arguments definition is the same as request.
+  def request_async(method, uri, query = nil, body = nil, extheader = {})
+    uri = urify(uri)
+    proxy = no_proxy?(uri) ? nil : @proxy
+    req = create_request(method, uri, query, body, extheader)
+    do_request_async(req, proxy)
+  end
+
+  # Resets internal session for the given URL.  Keep-alive connection for the
+  # site (host-port pair) is disconnected if exists.
+  def reset(uri)
+    uri = urify(uri)
+    @session_manager.reset(uri)
+  end
+
+  # Resets all of internal sessions.  Keep-alive connections are disconnected.
+  def reset_all
+    @session_manager.reset_all
+  end
+
+private
+
+  class RetryableResponse < StandardError # :nodoc:
+  end
+
+  class KeepAliveDisconnected < StandardError # :nodoc:
+  end
+
+  def do_request(req, proxy, &block)
+    conn = Connection.new
+    res = nil
+    retry_count = @session_manager.protocol_retry_count
+    while retry_count > 0
+      begin
+        protect_keep_alive_disconnected do
+          do_get_block(req, proxy, conn, &block)
+        end
+        res = conn.pop
+        break
+      rescue RetryableResponse
+        res = conn.pop
+        retry_count -= 1
+      end
+    end
+    res
+  end
+
+  def do_request_async(req, proxy)
+    conn = Connection.new
+    t = Thread.new(conn) { |tconn|
+      retry_count = @session_manager.protocol_retry_count
+      while retry_count > 0
+        begin
+          protect_keep_alive_disconnected do
+            do_get_stream(req, proxy, tconn)
+          end
+          break
+        rescue RetryableResponse
+          retry_count -= 1
+        end
+      end
+    }
+    conn.async_thread = t
+    conn
+  end
+
+  def load_environment
+    # http_proxy
+    if getenv('REQUEST_METHOD')
+      # HTTP_PROXY conflicts with the environment variable usage in CGI where
+      # HTTP_* is used for HTTP header information.  Unlike open-uri, we
+      # simply ignore http_proxy in CGI env and use cgi_http_proxy instead.
+      self.proxy = getenv('cgi_http_proxy')
+    else
+      self.proxy = getenv('http_proxy')
+    end
+    # no_proxy
+    self.no_proxy = getenv('no_proxy')
+  end
+
+  def getenv(name)
+    ENV[name.downcase] || ENV[name.upcase]
+  end
+
+  def follow_redirect(method, uri, query, body, extheader, &block)
+    uri = urify(uri)
+    if block
+      filtered_block = proc { |r, str|
+        block.call(str) if HTTP::Status.successful?(r.status)
+      }
+    end
+    if HTTP::Message.file?(body)
+      pos = body.pos rescue nil
+    end
+    retry_number = 0
+    while retry_number < @follow_redirect_count
+      body.pos = pos if pos
+      req = create_request(method, uri, query, body, extheader)
+      proxy = no_proxy?(req.header.request_uri) ? nil : @proxy
+      res = do_request(req, proxy, &filtered_block)
+      if HTTP::Status.successful?(res.status)
+        return res
+      elsif HTTP::Status.redirect?(res.status)
+        uri = urify(@redirect_uri_callback.call(req.header.request_uri, res))
+        retry_number += 1
+      else
+        raise BadResponseError.new("unexpected response: #{res.header.inspect}", res)
+      end
+    end
+    raise BadResponseError.new("retry count exceeded", res)
+  end
+
+  def protect_keep_alive_disconnected
+    begin
+      yield
+    rescue KeepAliveDisconnected
+      yield
+    end
+  end
+
+  def create_request(method, uri, query, body, extheader)
+    method = method.to_s.upcase
+    if extheader.is_a?(Hash)
+      extheader = extheader.to_a
+    else
+      extheader = extheader.dup
+    end
+    boundary = nil
+    if body
+      dummy, content_type = extheader.find { |key, value|
+        key.downcase == 'content-type'
+      }
+      if content_type
+        if /\Amultipart/ =~ content_type
+          if content_type =~ /boundary=(.+)\z/
+            boundary = $1
+          else
+            boundary = create_boundary
+            content_type = "#{content_type}; boundary=#{boundary}"
+            extheader = override_header(extheader, 'Content-Type', content_type)
+          end
+        end
+      elsif method == 'POST'
+        if file_in_form_data?(body)
+          boundary = create_boundary
+          content_type = "multipart/form-data; boundary=#{boundary}"
+        else
+          content_type = 'application/x-www-form-urlencoded'
+        end
+        extheader << ['Content-Type', content_type]
+      end
+    end
+    req = HTTP::Message.new_request(method, uri, query, body, boundary)
+    extheader.each do |key, value|
+      req.header.add(key, value)
+    end
+    if @cookie_manager && cookie = @cookie_manager.find(uri)
+      req.header.add('Cookie', cookie)
+    end
+    req
+  end
+
+  def create_boundary
+    Digest::SHA1.hexdigest(Time.now.to_s)
+  end
+
+  def file_in_form_data?(body)
+    HTTP::Message.multiparam_query?(body) &&
+      body.any? { |k, v| HTTP::Message.file?(v) }
+  end
+
+  def override_header(extheader, key, value)
+    result = []
+    extheader.each do |k, v|
+      if k.downcase == key.downcase
+        result << [key, value]
+      else
+        result << [k, v]
+      end
+    end
+    result
+  end
+
+  NO_PROXY_HOSTS = ['localhost']
+
+  def no_proxy?(uri)
+    if !@proxy or NO_PROXY_HOSTS.include?(uri.host)
+      return true
+    end
+    unless @no_proxy
+      return false
+    end
+    @no_proxy.scan(/([^:,]+)(?::(\d+))?/) do |host, port|
+      if /(\A|\.)#{Regexp.quote(host)}\z/i =~ uri.host &&
+          (!port || uri.port == port.to_i)
+        return true
+      end
+    end
+    false
+  end
+
+  def https?(uri)
+    uri.scheme.downcase == 'https'
+  end
+
+  # !! CAUTION !!
+  #   Method 'do_get*' runs under MT conditon. Be careful to change.
+  def do_get_block(req, proxy, conn, &block)
+    @request_filter.each do |filter|
+      filter.filter_request(req)
+    end
+    if str = @test_loopback_response.shift
+      dump_dummy_request_response(req.body.dump, str) if @debug_dev
+      conn.push(HTTP::Message.new_response(str))
+      return
+    end
+    content = block ? nil : ''
+    res = HTTP::Message.new_response(content)
+    @debug_dev << "= Request\n\n" if @debug_dev
+    sess = @session_manager.query(req, proxy)
+    res.peer_cert = sess.ssl_peer_cert
+    @debug_dev << "\n\n= Response\n\n" if @debug_dev
+    do_get_header(req, res, sess)
+    conn.push(res)
+    sess.get_body do |part|
+      if block
+        block.call(res, part)
+      else
+        content << part
+      end
+    end
+    @session_manager.keep(sess) unless sess.closed?
+    commands = @request_filter.collect { |filter|
+      filter.filter_response(req, res)
+    }
+    if commands.find { |command| command == :retry }
+      raise RetryableResponse.new
+    end
+  end
+
+  def do_get_stream(req, proxy, conn)
+    @request_filter.each do |filter|
+      filter.filter_request(req)
+    end
+    if str = @test_loopback_response.shift
+      dump_dummy_request_response(req.body.dump, str) if @debug_dev
+      conn.push(HTTP::Message.new_response(StringIO.new(str)))
+      return
+    end
+    piper, pipew = IO.pipe
+    res = HTTP::Message.new_response(piper)
+    @debug_dev << "= Request\n\n" if @debug_dev
+    sess = @session_manager.query(req, proxy)
+    res.peer_cert = sess.ssl_peer_cert
+    @debug_dev << "\n\n= Response\n\n" if @debug_dev
+    do_get_header(req, res, sess)
+    conn.push(res)
+    sess.get_body do |part|
+      pipew.syswrite(part)
+    end
+    pipew.close
+    @session_manager.keep(sess) unless sess.closed?
+    commands = @request_filter.collect { |filter|
+      filter.filter_response(req, res)
+    }
+    # ignore commands (not retryable in async mode)
+  end
+
+  def do_get_header(req, res, sess)
+    res.version, res.status, res.reason, headers = sess.get_header
+    headers.each do |key, value|
+      res.header.add(key, value)
+    end
+    if @cookie_manager
+      res.header['set-cookie'].each do |cookie|
+        @cookie_manager.parse(cookie, req.header.request_uri)
+      end
+      
+      # Set the cookie header in the current request so cookies are present in the event we have to retry the
+      # request, such as when responding to an authorization challenge.
+      if (cookie = @cookie_manager.find(req.header.request_uri)) && req.header['Cookie'].empty?
+        req.header.add('Cookie', cookie)
+      end
+    end
+  end
+
+  def dump_dummy_request_response(req, res)
+    @debug_dev << "= Dummy Request\n\n"
+    @debug_dev << req
+    @debug_dev << "\n\n= Dummy Response\n\n"
+    @debug_dev << res
+  end
+end