typhoeus 0.4.2 → 0.5.0.alpha

data/CHANGELOG.md

@@ -1,5 +1,76 @@
-0.4.0
------
+# Changelog
+
+## 0.5.0.pre
+
+[Full Changelog](http://github.com/typhoeus/typhoeus/compare/v0.4.2...master)
+
+Major Changes:
+
+* Ethon integration
+  * Params are url params and a body is always a body for every request type
+  * Request parameter and body are properly encoded (only POST multiform body is not)
+  * No more header sanitizing
+
+      Before: `:headers => { 'user_agent' => 'Custom' }` was modified to
+        `:headers => { 'User-Agent' => 'Custom' }`
+
+  * The options you can set might have a slightly other names, as Ethon sticks to
+    libcurl names. See
+    [Easy.new](http://rubydoc.info/github/typhoeus/ethon/Ethon/Easy#initialize-instance_method)
+    for a description.
+  * The following classes were deleted because they do not seemed to be uesed at all. If that
+    turns out to be wrong, they will be restored: `Typhoeus::Filter`, `Typhoeus::Remote`, `Typhoeus::RemoteMethod`, `Typhoeus::RemoteProxyObject`
+  * `Typhoeus::Easy` and `Typhoeus::Multi` are now `Ethon::Easy` and `Ethon::Multi`
+
+* Request shortcuts: `Typhoeus.get("www.google.de")`
+* Global configuration:
+```ruby
+Typhoeus.configure do |config|
+  config.verbose = true
+  config.memoize = true
+end
+```
+* No more Response#headers_hash, instead response#header returning the last
+  header and response#redirections returning the responses with headers
+  generated through redirections
+* Instead of defining the same callbacks on every request, you can define global callbacks:
+
+```ruby
+Typhoeus.on_complete { p "yay" }
+```
+
+* The stubbing interface changed slightly. You now have the same syntax as for requests:
+
+```ruby
+Typhoeus.stub(url, options).and_return(response)
+```
+
+Enhancements:
+
+* Documentation
+  ( [Alex P](https://github.com/ifesdjeen), [\#188](https://github.com/typhoeus/typhoeus/issues/188) )
+* Request#on\_complete can hold multiple blocks.
+* Request#eql? recognizes when header/params/body has a different order, but still same keys and values
+  ( [Alex P](https://github.com/ifesdjeen), [\#194](https://github.com/typhoeus/typhoeus/issues/194) )
+
+Bug Fixes:
+
+* Zero bytes in strings are escaped for libcurl
+* Add support for socks5 hostname proxy type
+  ( [eweathers](https://github.com/eweathers), [\#183](https://github.com/typhoeus/typhoeus/issues/183) )
+* Post body is encoded
+  ( [Rohan Deshpande](https://github.com/rdeshpande), [\#143](https://github.com/typhoeus/typhoeus/issues/143) )
+* Set default user agent
+  ( [Steven Shingler](https://github.com/sshingler), [\#176](https://github.com/typhoeus/typhoeus/issues/176) )
+
+## 0.4.2
+* A header hotfix
+
+## 0.4.1
+* Fix verifypeer and verifyhost options
+* Fix header sending
+
+## 0.4.0
 * Make a GET even when a body is given
 * Deprecated User Agent setter removed
 * Allow cache key basis overwrite (John Crepezzi, #147)
@@ -7,14 +78,12 @@
 * Refactor upload code (Marnen Laibow-Koser, #152)
 * Fix travis-ci build (Ezekiel Templin, #160)
 
-0.3.3
------
+## 0.3.3
 * Make sure to call the Easy::failure callback on all non-success http response codes, even invalid ones. [balexis]
 * Use bytesize instead of length to determine Content-Length [dlamacchia]
 * Added SSL version option to Easy/Request [michelbarbosa/dbalatero]
 
-0.3.2
------
+## 0.3.2
 * Fix array params to be consistent with HTTP spec [gridaphobe]
 * traversal\_to\_params\_hash should use the escape option [itsmeduncan]
 * Fix > 1024 open file descriptors [mschulkind]
@@ -35,20 +104,16 @@
 * Fix HTTP status edge-case [balexis]
 * Expose primary\_ip to easy object [balexis]
 
-0.2.4
------
+## 0.2.4
 * Fix form POSTs to only use multipart for file uploads, otherwise use application/x-www-form-urlencoded [dbalatero]
 
-0.2.3
------
+## 0.2.3
 * Code duplication in Typhoeus::Form led to nested URL param errors on POST only. Fixed [dbalatero]
 
-0.2.2
------
+## 0.2.2
 * Fixed a problem with nested URL params encoding incorrectly [dbalatero]
 
-0.2.1
------
+## 0.2.1
 * Added extended proxy support [Zapotek, GH-46]
 * eliminated compile time warnings by using proper type declarations [skaes, GH-54]
 * fixed broken calls to rb\_raise [skaes, GH-54]
@@ -60,34 +125,27 @@
 * added abort to Hydra to prematurely stop a hydra.run [Zapotek]
 * added file upload support for POST requests [jtarchie, GH-59]
 
-0.2.0
-------
+## 0.2.0
 * Fix warning in Request#headers from attr\_accessor
-* Params with array values were not parsing into the format that rack expects
-[GH-39, smartocci]
+* Params with array values were not parsing into the format that rack expects [GH-39, smartocci]
 * Removed Rack as a dependency [GH-45]
 * Added integration hooks for VCR!
 
-0.1.31
-------
+## 0.1.31
 * Fixed bug in setting compression encoding [morhekil]
 * Exposed authentication control methods through Request interface [morhekil]
 
-0.1.30
------------
+## 0.1.30
 * Exposed CURLOPT\_CONNECTTIMEOUT\_MS to Requests [balexis]
 
-0.1.29
-------
+## 0.1.29
 * Fixed a memory corruption with using CURLOPT\_POSTFIELDS [gravis,
 32531d0821aecc4]
 
-0.1.28
-----------------
+## 0.1.28
 * Added SSL cert options for Typhoeus::Easy [GH-25, gravis]
 * Ported SSL cert options to Typhoeus::Request interface [gravis]
 * Added support for any HTTP method (purge for Varnish) [ryana]
 
-0.1.27
-------
+## 0.1.27
 * Added rack as dependency, added dev dependencies to Rakefile [GH-21]

data/Gemfile

@@ -1,3 +1,19 @@
 source :rubygems
-
 gemspec
+
+gem "rake"
+
+group :development, :test do
+  gem "rspec", "~> 2.11"
+
+  gem "sinatra", "~> 1.3"
+  gem "json"
+
+  if RUBY_PLATFORM == "java"
+    gem "spoon"
+  end
+
+  unless ENV["CI"]
+    gem "guard-rspec", "~> 0.7"
+  end
+end

data/README.md

@@ -1,438 +1,36 @@
-#  Typhoeus [![Build Status](https://secure.travis-ci.org/typhoeus/typhoeus.png?branch=master)](http://travis-ci.org/typhoeus/typhoeus)
+# Typhoeus [![Build Status](https://secure.travis-ci.org/typhoeus/typhoeus.png)](http://travis-ci.org/typhoeus/typhoeus)
 
-[the mailing list](http://groups.google.com/group/typhoeus)
+Like a modern code version of the mythical beast with 100 serpent heads, Typhoeus runs HTTP requests in parallel while cleanly encapsulating handling logic.
 
-##  Summary
+## Example
 
-Like a modern code version of the mythical beast with 100 serpent heads,
-Typhoeus runs HTTP requests in parallel while cleanly encapsulating handling
-logic. To be a little more specific, it’s a library for accessing web services
-in Ruby. It’s specifically designed for building RESTful service oriented
-architectures in Ruby that need to be fast enough to process calls to multiple
-services within the client’s HTTP request/response life cycle.
+Single request:
 
-Some of the awesome features are parallel request execution, memoization of
-request responses (so you don’t make the same request multiple times in a
-single group), built in support for caching responses to memcached (or
-whatever), and mocking capability baked in. It uses libcurl and libcurl-multi
-to work this speedy magic. I wrote the bindings myself so it’s yet another
-Ruby libcurl library, but with some extra awesomeness added in. FFI is used to
-interface with the library so it works with any Ruby implementation.
+```ruby
+Typhoeus.get("www.example.com")
+```
 
-##  Installation
+Parallel requests:
 
-Typhoeus requires you to have a current version of libcurl installed. The
-easiest solution is to use your system’s package manager to install it. If
-that doesn’t work, you can grab a package off of [the curl
-website](http://curl.haxx.se/download.html) and manually install it following
-the instructions given there. Typhoeus will work with version 7.19.4 or higher
-(earlier versions might work but no guarantees are provided).
+```ruby
+hydra = Typhoeus::Hydra.new
+10.times.map{ hydra.queue(Typhoeus::Request.new("www.example.com")) }
+hydra.run
+```
 
-To install Typhoeus, simply run:
+## Project Tracking
 
-    gem install typhoeus
+* [Documentation](http://rubydoc.info/github/typhoeus/typhoeus)
+* [Website](http://typhoeus.github.com/)
+* [Mailinglist](http://groups.google.com/group/typhoeus)
 
-If you’re on Debian or Ubuntu and getting errors while trying to install, it
-could be because you don’t have the latest version of libcurl installed. Do
-this to fix:
-
-    sudo apt-get install libcurl4-gnutls-dev
-
-If you’re still having issues, please let me know on [the mailing
-list](http://groups.google.com/group/typhoeus).
-
-There’s one other thing you should know. The Easy object (which is just a
-libcurl thing) allows you to set timeout values in milliseconds. However, for
-this to work you need to build libcurl with c-ares support built in.
-
-##  Windows Support
-
-Typhoeus runs perfectly on Windows. The tricky part is knowing how to install
-libcurl in the absence of a package manager.
-
-To install libcurl, simply grab [the latest libcurl
-package](http://curl.haxx.se/download.html#Win32) off of the curl website,
-extract the bin directory, and then add the path to the bin directory into the
-PATH environment variable. Ruby with then be able to find libcurl properly and
-everything will just work.
-
-##  Usage
-
-The primary interface for Typhoeus is comprised of three classes: Request,
-Response, and Hydra. Request represents an HTTP request object, response
-represents an HTTP response, and Hydra manages making parallel HTTP
-connections.
-
-    require 'rubygems'
-    require 'typhoeus'
-    require 'json'
-
-    # the request object
-    request = Typhoeus::Request.new("http://www.pauldix.net",
-                                    :body          => "this is a request body",
-                                    :method        => :post,
-                                    :headers       => {:Accept => "text/html"},
-                                    :timeout       => 100, # milliseconds
-                                    :cache_timeout => 60, # seconds
-                                    :params        => {:field1 => "a field"})
-    # we can see from this that the first argument is the url. the second is a set of options.
-    # the options are all optional. The default for :method is :get. Timeout is measured in milliseconds.
-    # cache_timeout is measured in seconds.
-
-    # Run the request via Hydra.
-    hydra = Typhoeus::Hydra.new
-    hydra.queue(request)
-    hydra.run
-
-    # the response object will be set after the request is run
-    response = request.response
-    response.code    # http status code
-    response.time    # time in seconds the request took
-    response.headers # the http headers
-    response.headers_hash # http headers put into a hash
-    response.body    # the response body
-
-**Making Quick Requests**
-
-The request object has some convenience methods for performing single HTTP
-requests. The arguments are the same as those you pass into the request
-constructor.
-
-    response = Typhoeus::Request.get("http://www.pauldix.net")
-    response = Typhoeus::Request.head("http://www.pauldix.net")
-    response = Typhoeus::Request.put("http://localhost:3000/posts/1", :body => "whoo, a body")
-    response = Typhoeus::Request.post("http://localhost:3000/posts", :params => {:title => "test post", :content => "this is my test"})
-    response = Typhoeus::Request.delete("http://localhost:3000/posts/1")
-
-**Handling HTTP errors**
-
-You can query the response object to figure out if you had a successful
-request or not. Here’s some example code that you might use to handle errors.
-
-    request.on_complete do |response|
-      if response.success?
-        # hell yeah
-      elsif response.timed_out?
-        # aw hell no
-        log("got a time out")
-      elsif response.code == 0
-        # Could not get an http response, something's wrong.
-        log(response.curl_error_message)
-      else
-        # Received a non-successful http response.
-        log("HTTP request failed: " + response.code.to_s)
-      end
-    end
-
-This also works with serial (blocking) requests in the same fashion. Both
-serial and parallel requests return a Response object.
-
-**Handling file uploads**
-
-A File object can be passed as a param for a POST request to handle uploading
-files to the server. Typhoeus will upload the file as the original file name
-and use Mime::Types to set the content type.
-
-    response = Typhoeus::Request.post("http://localhost:3000/posts",
-      :params => {
-        :title => "test post", :content => "this is my test",
-        :file => File.open("thesis.txt","r")
-      }
-    )
-
-**Making Parallel Requests**
-
-    # Generally, you should be running requests through hydra. Here is how that looks
-    hydra = Typhoeus::Hydra.new
-
-    first_request = Typhoeus::Request.new("http://localhost:3000/posts/1.json")
-    first_request.on_complete do |response|
-      post = JSON.parse(response.body)
-      third_request = Typhoeus::Request.new(post.links.first) # get the first url in the post
-      third_request.on_complete do |response|
-        # do something with that
-      end
-      hydra.queue third_request
-      return post
-    end
-    second_request = Typhoeus::Request.new("http://localhost:3000/users/1.json")
-    second_request.on_complete do |response|
-      JSON.parse(response.body)
-    end
-    hydra.queue first_request
-    hydra.queue second_request
-    hydra.run # this is a blocking call that returns once all requests are complete
-
-    first_request.handled_response # the value returned from the on_complete block
-    second_request.handled_response # the value returned from the on_complete block (parsed JSON)
-
-The execution of that code goes something like this. The first and second
-requests are built and queued. When hydra is run the first and second requests
-run in parallel. When the first request completes, the third request is then
-built and queued up. The moment it is queued Hydra starts executing it.
-Meanwhile the second request would continue to run (or it could have completed
-before the first). Once the third request is done, hydra.run returns.
-
-**Specifying Max Concurrency**
-
-Hydra will also handle how many requests you can make in parallel. Things will
-get flakey if you try to make too many requests at the same time. The built in
-limit is 200. When more requests than that are queued up, hydra will save them
-for later and start the requests as others are finished. You can raise or
-lower the concurrency limit through the Hydra constructor.
-
-    hydra = Typhoeus::Hydra.new(:max_concurrency => 20) # keep from killing some servers
-
-**Memoization**
-
-Hydra memoizes requests within a single run call. You can also disable
-memoization.
-
-    hydra = Typhoeus::Hydra.new
-    2.times do
-      r = Typhoeus::Request.new("http://localhost/3000/users/1")
-      hydra.queue r
-    end
-    hydra.run # this will result in a single request being issued. However, the on_complete handlers of both will be called.
-    hydra.disable_memoization
-    2.times do
-      r = Typhoeus::Request.new("http://localhost/3000/users/1")
-      hydra.queue r
-    end
-    hydra.run # this will result in a two requests.
-
-**Caching**
-
-Hydra includes built in support for creating cache getters and setters. In the
-following example, if there is a cache hit, the cached object is passed to the
-on\_complete handler of the request object.
-
-    hydra = Typhoeus::Hydra.new
-    hydra.cache_setter do |request|
-      @cache.set(request.cache_key, request.response, request.cache_timeout)
-    end
-
-    hydra.cache_getter do |request|
-      @cache.get(request.cache_key) rescue nil
-    end
-
-**Direct Stubbing**
-
-Hydra allows you to stub out specific urls and patterns to avoid hitting
-remote servers while testing.
-
-    hydra = Typhoeus::Hydra.new
-    response = Response.new(:code => 200, :headers => "", :body => "{'name' : 'paul'}", :time => 0.3)
-    hydra.stub(:get, "http://localhost:3000/users/1").and_return(response)
-
-    request = Typhoeus::Request.new("http://localhost:3000/users/1")
-    request.on_complete do |response|
-      JSON.parse(response.body)
-    end
-    hydra.queue request
-    hydra.run
-
-The queued request will hit the stub. The on\_complete handler will be called
-and will be passed the response object. You can also specify a regex to match
-urls.
-
-    hydra.stub(:get, /http\:\/\/localhost\:3000\/users\/.*/).and_return(response)
-    # any requests for a user will be stubbed out with the pre built response.
-
-**The Singleton**
-
-All of the quick requests are done using the singleton hydra object. If you
-want to enable caching or stubbing on the quick requests, set those options on
-the singleton.
-
-    hydra = Typhoeus::Hydra.hydra
-    hydra.stub(:get, "http://localhost:3000/users")
-
-**Timeouts**
-
-No exceptions are raised on HTTP timeouts. You can check whether a request
-timed out with the following methods:
-
-    easy.timed_out?  # for a raw Easy handle
-    response.timed_out?  # for a Response handle
-
-**Following Redirections**
-
-Use `:follow_location => true`, eg:
-
-    Typhoeus::Request.new(“www.example.com”, :follow_location => true)
-
-**Basic Authentication**
-
-    response = Typhoeus::Request.get("http://twitter.com/statuses/followers.json",
-                                     :username => username, :password => password)
-
-**SSL**
-
-SSL comes built in to libcurl so it’s in Typhoeus as well. If you pass in a
-url with “https” it should just work assuming that you have your [cert
-bundle](http://curl.haxx.se/docs/caextract.html) in order and the server is
-verifiable. You must also have libcurl built with SSL support enabled. You can
-check that by doing this:
-
-    Typhoeus::Easy.new.curl_version # output should include OpenSSL/...
-
-Now, even if you have libcurl built with OpenSSL you may still have a messed
-up cert bundle or if you’re hitting a non-verifiable SSL server then you’ll
-have to disable peer verification to make SSL work. Like this:
-
-    Typhoeus::Request.get("https://mail.google.com/mail", :disable_ssl_peer_verification => true)
-
-If you are getting “SSL: certificate subject name does not match target host
-name” from curl (ex:- you are trying to access to b.c.host.com when the
-certificate subject is \*.host.com). You can disable host verification. Like
-this:
-
-    Typhoeus::Request.get("https://mail.google.com/mail", :disable_ssl_host_verification => true)
-
-**LibCurl**
-
-Typhoeus also has a more raw libcurl interface. These are the Easy and Multi
-objects. If you’re into accessing just the raw libcurl style, those are your
-best bet.
-
-However, by using this raw interface, you do not get access to Hydra-specific
-features, such as stubbing/mocking.
-
-SSL Certs can be provided to the Easy interface:
-
-    e = Typhoeus::Easy.new
-    e.url = "https://example.com/action"
-    s.ssl_cacert = "ca_file.cer"
-    e.ssl_cert = "acert.crt"
-    e.ssl_key = "akey.key"
-    [...]
-    e.perform
-
-or directly to a Typhoeus::Request :
-
-    e = Typhoeus::Request.get("https://example.com/action",
-      :ssl_cacert => "ca_file.cer",
-      :ssl_cert => "acert.crt",
-      :ssl_key => "akey.key",
-      [...]
-    end
-
-##  Advanced authentication
-
-Thanks for the authentication piece and this description go to Oleg Ivanov
-(morhekil). The major reason to start this fork was the need to perform NTLM
-authentication in Ruby, and other libcurl’s authentications method were made
-possible as a result. Now you can do it via Typhoeus::Easy interface using the
-following API.
-
-    e = Typhoeus::Easy.new
-    e.auth = {
-      :username => 'username',
-      :password => 'password',
-      :method => Typhoeus::Easy::AUTH_TYPES[:CURLAUTH_NTLM]
-    }
-    e.url = "http://example.com/auth_ntlm"
-    e.method = :get
-    e.perform
-
-**Other authentication types**
-
-The following authentication types are available:
-
-  * CURLAUTH\_BASIC
-  * CURLAUTH\_DIGEST
-  * CURLAUTH\_GSSNEGOTIATE
-  * CURLAUTH\_NTLM
-  * CURLAUTH\_DIGEST\_IE
-  * CURLAUTH\_AUTO
-
-The last one (CURLAUTH\_AUTO) is really a combination of all previous methods
-and is provided by Typhoeus for convenience. When you set authentication to
-auto, Typhoeus will retrieve the given URL first and examine it’s headers to
-confirm what auth types are supported by the server. The it will select the
-strongest of available auth methods and will send the second request using the
-selected authentication method.
-
-**Authentication via the quick request interface**
-
-There’s also an easy way to perform any kind of authentication via the quick
-request interface:
-
-    e = Typhoeus::Request.get("http://example.com",
-      :username => 'username',
-      :password => 'password',
-      :auth_method => :ntlm)
-
-All methods listed above is available in a shorter form – :basic, :digest,
-:gssnegotiate, :ntlm, :digest\_ie, :auto.
-
-**Query of available auth types**
-
-After the initial request you can get the authentication types available on
-the server via Typhoues::Easy#auth\_methods call. It will return a number
-
-that you’ll need to decode yourself, please refer to easy.rb source code to
-see the numeric values of different auth types.
-
-##  Verbose debug output
-
-Sometime it’s useful to see verbose output from curl. You may now enable it:
-
-    e = Typhoeus::Easy.new
-    e.verbose = 1
-
-or using the quick request:
-
-    e = Typhoeus::Request.get("http://example.com", :verbose => true)
-
-Just remember that libcurl prints it’s debug output to the console (to
-STDERR), so you’ll need to run your scripts from the console to see it.
-
-##  Benchmarks
-
-I set up a benchmark to test how the parallel performance works vs Ruby’s
-built in NET::HTTP. The setup was a local evented HTTP server that would take
-a request, sleep for 500 milliseconds and then issued a blank response. I set
-up the client to call this 20 times. Here are the results:
-
-      net::http  0.030000   0.010000   0.040000 ( 10.054327)
-      typhoeus   0.020000   0.070000   0.090000 (  0.508817)
-
-We can see from this that NET::HTTP performs as expected, taking 10 seconds to
-run 20 500ms requests. Typhoeus only takes 500ms (the time of the response
-that took the longest.) One other thing to note is that Typhoeus keeps a pool
-of libcurl Easy handles to use. For this benchmark I warmed the pool first. So
-if you test this out it may be a bit slower until the Easy handle pool has
-enough in it to run all the simultaneous requests. For some reason the easy
-handles can take quite some time to allocate.
-
-##  Running the specs
-
-Running the specs requires a couple of Sinatra servers to be booted. rake spec
-will do this for you, but if you’re needing to run the specs a lot, spinning
-up the servers manually and leaving them running should speed things up a bit.
-Do this:
-
-      # Start up the test servers (in another terminal)
-      rake start_test_servers
-
-      # Run the specs
-      rake spec
-
-
-##  Next Steps
-
-  * Add in ability to keep-alive requests and reuse them within hydra.
-  * Add support for automatic retry, exponential back-off, and queuing for later.
-
-##  LICENSE
+## LICENSE
 
 (The MIT License)
 
-Copyright © 2009-2010 Paul Dix
+Copyright © 2009-2010 [Paul Dix](http://www.pauldix.net/)
 
-Copyright © 2011 David Balatero
+Copyright © 2011 [David Balatero](https://github.com/dbalatero/)
 
 Copyright © 2012 [Hans Hasselberg](http://www.hans.io)