typhoeus 0.1.28 → 0.2.0

data/.gitignore

@@ -1,2 +1,3 @@
 .*.sw?
 *.gem
+.bundle

data/CHANGELOG.markdown

@@ -1,3 +1,25 @@
+0.2.0
+------------
+* Fix warning in Request#headers from attr_accessor
+* Params with array values were not parsing into the format that rack expects
+[GH-39, smartocci]
+* Removed Rack as a dependency [GH-45]
+* Added integration hooks for VCR!
+
+0.1.31
+------
+* Fixed bug in setting compression encoding [morhekil]
+* Exposed authentication control methods through Request interface [morhekil]
+
+0.1.30
+-----------
+* Exposed CURLOPT_CONNECTTIMEOUT_MS to Requests [balexis]
+
+0.1.29
+------
+* Fixed a memory corruption with using CURLOPT_POSTFIELDS [gravis,
+32531d0821aecc4]
+
 0.1.28
 ----------------
 * Added SSL cert options for Typhoeus::Easy [GH-25, gravis]

data/Gemfile

@@ -0,0 +1,9 @@
+source :rubygems
+
+group :test do
+  gem 'rspec', '1.3.1'
+  gem 'jeweler'
+  gem 'json'
+  gem 'sinatra'
+  gem 'diff-lcs'
+end

data/Gemfile.lock

@@ -0,0 +1,30 @@
+GEM
+  remote: http://rubygems.org/
+  specs:
+    diff-lcs (1.1.2)
+    gemcutter (0.6.1)
+    git (1.2.5)
+    jeweler (1.4.0)
+      gemcutter (>= 0.1.0)
+      git (>= 1.2.5)
+      rubyforge (>= 2.0.0)
+    json (1.4.6)
+    json_pure (1.4.6)
+    rack (1.2.1)
+    rspec (1.3.1)
+    rubyforge (2.0.4)
+      json_pure (>= 1.1.7)
+    sinatra (1.1.0)
+      rack (~> 1.1)
+      tilt (~> 1.1)
+    tilt (1.1)
+
+PLATFORMS
+  ruby
+
+DEPENDENCIES
+  diff-lcs
+  jeweler
+  json
+  rspec (= 1.3.1)
+  sinatra

data/README.textile

@@ -51,7 +51,7 @@ require 'json'
 request = Typhoeus::Request.new("http://www.pauldix.net",
                                 :body          => "this is a request body",
                                 :method        => :post,
-                                :headers       => {:Accepts => "text/html"},
+                                :headers       => {:Accept => "text/html"},
                                 :timeout       => 100, # milliseconds
                                 :cache_timeout => 60, # seconds
                                 :params        => {:field1 => "a field"})
@@ -103,8 +103,8 @@ hydra.queue first_request
 hydra.queue second_request
 hydra.run # this is a blocking call that returns once all requests are complete
 
-first_request.handled_resposne # the value returned from the on_complete block
-second_request.handled_resposne # the value returned from the on_complete block (parsed JSON)
+first_request.handled_response # the value returned from the on_complete block
+second_request.handled_response # the value returned from the on_complete block (parsed JSON)
 </pre>
 
 The execution of that code goes something like this. The first and second requests are built and queued. When hydra is run the first and second requests run in parallel. When the first request completes, the third request is then built and queued up. The moment it is queued Hydra starts executing it. Meanwhile the second request would continue to run (or it could have completed before the first). Once the third request is done, hydra.run returns.
@@ -183,9 +183,8 @@ hydra.stub(:get, "http://localhost:3000/users")
 *Basic Authentication*
 
 <pre>
-require 'base64'
 response = Typhoeus::Request.get("http://twitter.com/statuses/followers.json",
-                                 :headers => {"Authorization" => "Basic #{Base64.b64encode("#{username}:#{password}")}"})
+                                 :username => username, :password => password)
 </pre>
 
 *SSL*
@@ -227,9 +226,9 @@ e = Typhoeus::Request.get("https://example.com/action",
 end
 </pre>
 
-h2. NTLM authentication
+h2. Advanced authentication
 
-Thanks for the authentication piece and this description go to Oleg Ivanov (morhekil). The major reason to start this fork was the need to perform NTLM authentication in Ruby. Now you can do it via Typhoeus::Easy interface using the following API.
+Thanks for the authentication piece and this description go to Oleg Ivanov (morhekil). The major reason to start this fork was the need to perform NTLM authentication in Ruby, and other libcurl's authentications method were made possible as a result. Now you can do it via Typhoeus::Easy interface using the following API.
 
 <pre>
 e = Typhoeus::Easy.new
@@ -251,6 +250,22 @@ The following authentication types are available:
 * CURLAUTH_GSSNEGOTIATE
 * CURLAUTH_NTLM
 * CURLAUTH_DIGEST_IE
+* CURLAUTH_AUTO
+
+The last one (CURLAUTH_AUTO) is really a combination of all previous methods and is provided by Typhoeus for convenience. When you set authentication to auto, Typhoeus will retrieve the given URL first and examine it's headers to confirm what auth types are supported by the server. The it will select the strongest of available auth methods and will send the second request using the selected authentication method.
+
+*Authentication via the quick request interface*
+
+There's also an easy way to perform any kind of authentication via the quick request interface:
+
+<pre>
+e = Typhoeus::Request.get("http://example.com",
+  :username => 'username',
+  :password => 'password',
+  :method => :ntlm)
+</pre>
+
+All methods listed above is available in a shorter form - :basic, :digest, :gssnegotiate, :ntlm, :digest_ie, :auto.
 
 *Query of available auth types*
 
@@ -266,7 +281,13 @@ e = Typhoeus::Easy.new
 e.verbose = 1
 </pre>
 
-Please note that libcurl prints it's output to the console, so you'll need to run your scripts from the console to see the debug info.
+or using the quick request:
+
+<pre>
+e = Typhoeus::Request.get("http://example.com", :verbose => true)
+</pre>
+
+Just remember that libcurl prints it's debug output to the console (to STDERR), so you'll need to run your scripts from the console to see it.
 
 h2. Benchmarks
 

data/Rakefile

@@ -11,7 +11,6 @@ begin
     gemspec.email = "paul@pauldix.net"
     gemspec.homepage = "http://github.com/pauldix/typhoeus"
     gemspec.authors = ["Paul Dix"]
-    gemspec.add_dependency "rack"
     gemspec.add_development_dependency "rspec"
     gemspec.add_development_dependency "jeweler"
     gemspec.add_development_dependency "diff-lcs"

data/VERSION

@@ -1 +1 @@
-0.1.28
+0.2.0

data/examples/twitter.rb

@@ -1,5 +1,5 @@
-require File.dirname(__FILE__) + '/../lib/typhoeus.rb'
 require 'rubygems'
+require File.dirname(__FILE__) + '/../lib/typhoeus.rb'
 require 'json'
 
 class Twitter

data/ext/typhoeus/.gitignore

@@ -2,4 +2,5 @@ conftest.dSYM/*
 conftest.dSYM
 mkmf.log
 *.o
-native.bundle
+native.bundleMakefile
+*.bundle

data/lib/typhoeus/easy.rb

@@ -12,9 +12,14 @@ module Typhoeus
       :CURLOPT_UPLOAD         => 46,
       :CURLOPT_CUSTOMREQUEST  => 10036,
       :CURLOPT_POSTFIELDS     => 10015,
+      :CURLOPT_COPYPOSTFIELDS     => 10165,
       :CURLOPT_POSTFIELDSIZE  => 60,
       :CURLOPT_USERAGENT      => 10018,
       :CURLOPT_TIMEOUT_MS     => 155,
+      # Time-out connect operations after this amount of milliseconds.
+      # [Only works on unix-style/SIGALRM operating systems. IOW, does
+      # not work on Windows.
+      :CURLOPT_CONNECTTIMEOUT_MS  => 156,
       :CURLOPT_NOSIGNAL       => 99,
       :CURLOPT_HTTPHEADER     => 10023,
       :CURLOPT_FOLLOWLOCATION => 52,
@@ -25,7 +30,7 @@ module Typhoeus
       :CURLOPT_PROXY          => 10004,
       :CURLOPT_VERIFYPEER     => 64,
       :CURLOPT_NOBODY         => 44,
-      :CURLOPT_ENCODING       => 102,
+      :CURLOPT_ENCODING       => 10000 + 102,
       :CURLOPT_SSLCERT        => 10025,
       :CURLOPT_SSLCERTTYPE    => 10086,
       :CURLOPT_SSLKEY         => 10087,
@@ -45,15 +50,16 @@ module Typhoeus
       :CURLAUTH_DIGEST        => 2,
       :CURLAUTH_GSSNEGOTIATE  => 4,
       :CURLAUTH_NTLM          => 8,
-      :CURLAUTH_DIGEST_IE     => 16
+      :CURLAUTH_DIGEST_IE     => 16,
+      :CURLAUTH_AUTO          => 16 | 8 | 4 | 2 | 1
     }
 
     def initialize
       @method = :get
-      @post_dat_set = nil
       @headers = {}
 
-      set_option(OPTION_VALUES[:CURLOPT_ENCODING], 'zlib') if supports_zlib?
+      # Enable encoding/compression support
+      set_option(OPTION_VALUES[:CURLOPT_ENCODING], '')
     end
 
     def headers=(hash)
@@ -100,6 +106,12 @@ module Typhoeus
     def max_redirects=(redirects)
       set_option(OPTION_VALUES[:CURLOPT_MAXREDIRS], redirects)
     end
+    
+    def connect_timeout=(milliseconds)
+      @connect_timeout = milliseconds
+      set_option(OPTION_VALUES[:CURLOPT_NOSIGNAL], 1)
+      set_option(OPTION_VALUES[:CURLOPT_CONNECTTIMEOUT_MS], milliseconds)
+    end
 
     def timeout=(milliseconds)
       @timeout = milliseconds
@@ -158,8 +170,8 @@ module Typhoeus
 
     def post_data=(data)
       @post_data_set = true
-      set_option(OPTION_VALUES[:CURLOPT_POSTFIELDS], data)
       set_option(OPTION_VALUES[:CURLOPT_POSTFIELDSIZE], data.length)
+      set_option(OPTION_VALUES[:CURLOPT_COPYPOSTFIELDS], data)
     end
 
     def params=(params)
@@ -167,12 +179,12 @@ module Typhoeus
       params_string = params.keys.collect do |k|
         value = params[k]
         if value.is_a? Hash
-          value.keys.collect {|sk| Rack::Utils.escape("#{k}[#{sk}]") + "=" + Rack::Utils.escape(value[sk].to_s)}
+          value.keys.collect {|sk| Typhoeus::Utils.escape("#{k}[#{sk}]") + "=" + Typhoeus::Utils.escape(value[sk].to_s)}
         elsif value.is_a? Array
-          key = Rack::Utils.escape(k.to_s)
-          value.collect { |v| "#{key}=#{Rack::Utils.escape(v.to_s)}" }.join('&')
+          key = Typhoeus::Utils.escape(k.to_s)
+          value.collect { |v| "#{key}=#{Typhoeus::Utils.escape(v.to_s)}" }.join('&')
         else
-          "#{Rack::Utils.escape(k.to_s)}=#{Rack::Utils.escape(params[k].to_s)}"
+          "#{Typhoeus::Utils.escape(k.to_s)}=#{Typhoeus::Utils.escape(params[k].to_s)}"
         end
       end.flatten.join("&")
 

data/lib/typhoeus/hydra/callbacks.rb

@@ -0,0 +1,24 @@
+module Typhoeus
+  class Hydra
+    module Callbacks
+      def self.extended(base)
+        class << base
+          attr_accessor :global_hooks
+        end
+        base.global_hooks = Hash.new { |h, k| h[k] = [] }
+      end
+
+      def after_request_before_on_complete(&block)
+        global_hooks[:after_request_before_on_complete] << block
+      end
+
+      def run_global_hooks_for(name, request)
+        global_hooks[name].each { |hook| hook.call(request) }
+      end
+
+      def clear_global_hooks
+        global_hooks.clear
+      end
+    end
+  end
+end

data/lib/typhoeus/hydra/connect_options.rb

@@ -0,0 +1,45 @@
+module Typhoeus
+  class Hydra
+    class NetConnectNotAllowedError < StandardError; end
+
+    module ConnectOptions
+      def self.included(base)
+        base.extend(ClassMethods)
+      end
+
+      # This method checks to see if we should raise an error on
+      # a request.
+      #
+      # @raises NetConnectNotAllowedError
+      def check_allow_net_connect!(request)
+        if !Typhoeus::Hydra.allow_net_connect? and (!Typhoeus::Hydra.ignore_localhost? or !request.localhost?)
+          raise NetConnectNotAllowedError, "Real HTTP requests are not allowed. Unregistered request: #{request.inspect}"
+        end
+      end
+      private :check_allow_net_connect!
+
+      module ClassMethods
+        def self.extended(base)
+          class << base
+            attr_accessor :allow_net_connect
+            attr_accessor :ignore_localhost
+          end
+          base.allow_net_connect = true
+          base.ignore_localhost = false
+        end
+
+        # Returns whether we allow external HTTP connections.
+        # Useful for mocking/tests.
+        #
+        # @return [boolean] true/false
+        def allow_net_connect?
+          allow_net_connect
+        end
+
+        def ignore_localhost?
+          ignore_localhost
+        end
+      end
+    end
+  end
+end

data/lib/typhoeus/hydra/stubbing.rb

@@ -0,0 +1,52 @@
+module Typhoeus
+  class Hydra
+    module Stubbing
+      module SharedMethods
+        def stub(method, url, options = {})
+          stubs << HydraMock.new(url, method, options)
+          stubs.last
+        end
+
+        def clear_stubs
+          self.stubs = []
+        end
+
+        def find_stub_from_request(request)
+          stubs.detect { |stub| stub.matches?(request) }
+        end
+
+        def self.extended(base)
+          class << base
+            attr_accessor :stubs
+          end
+          base.stubs = []
+        end
+      end
+
+      def self.included(base)
+        base.extend(SharedMethods)
+        base.class_eval do
+          attr_accessor :stubs
+        end
+      end
+
+      def assign_to_stub(request)
+        m = find_stub_from_request(request)
+
+        # Fallback to global stubs.
+        m ||= self.class.find_stub_from_request(request)
+
+        if m
+          m.add_request(request)
+          @active_stubs << m
+          m
+        else
+          nil
+        end
+      end
+      private :assign_to_stub
+
+      include SharedMethods
+    end
+  end
+end

data/lib/typhoeus/hydra.rb

@@ -1,5 +1,13 @@
+require 'typhoeus/hydra/callbacks'
+require 'typhoeus/hydra/connect_options'
+require 'typhoeus/hydra/stubbing'
+
 module Typhoeus
   class Hydra
+    include ConnectOptions
+    include Stubbing
+    extend Callbacks
+
     def initialize(options = {})
       @memoize_requests = true
       @multi       = Multi.new
@@ -7,12 +15,13 @@ module Typhoeus
       initial_pool_size = options[:initial_pool_size] || 10
       @max_concurrency = options[:max_concurrency] || 200
       initial_pool_size.times { @easy_pool << Easy.new }
-      @stubs       = []
       @memoized_requests = {}
       @retrieved_from_cache = {}
       @queued_requests = []
       @running_requests = 0
-      @stubbed_request_count = 0
+
+      self.stubs = []
+      @active_stubs = []
     end
 
     def self.hydra
@@ -28,10 +37,6 @@ module Typhoeus
       @cache_getter = nil
     end
 
-    def clear_stubs
-      @stubs = []
-    end
-
     def fire_and_forget
       @queued_requests.each {|r| queue(r, false)}
       @multi.fire_and_forget
@@ -40,6 +45,10 @@ module Typhoeus
     def queue(request, obey_concurrency_limit = true)
       return if assign_to_stub(request)
 
+      # At this point, we are running over live HTTP. Make sure we haven't
+      # disabled live requests.
+      check_allow_net_connect!(request)
+
       if @running_requests >= @max_concurrency && obey_concurrency_limit
         @queued_requests << request
       else
@@ -62,14 +71,14 @@ module Typhoeus
     end
 
     def run
-      while @stubbed_request_count > 0
-        @stubs.each do |m|
-          while request = m.requests.shift
-            @stubbed_request_count -= 1
-            m.response.request = request
-            handle_request(request, m.response)
-          end
+      while !@active_stubs.empty?
+        m = @active_stubs.first
+        while request = m.requests.shift
+          response = m.response
+          response.request = request
+          handle_request(request, response)
         end
+        @active_stubs.delete(m)
       end
 
       @multi.perform
@@ -97,22 +106,6 @@ module Typhoeus
       @on_complete = proc
     end
 
-    def stub(method, url)
-      @stubs << HydraMock.new(url, method)
-      @stubs.last
-    end
-
-    def assign_to_stub(request)
-      m = @stubs.detect {|stub| stub.matches? request}
-      if m
-        m.add_request(request)
-        @stubbed_request_count += 1
-      else
-        nil
-      end
-    end
-    private :assign_to_stub
-
     def get_from_cache_or_queue(request)
       if @cache_getter
         val = @cache_getter.call(request)
@@ -132,12 +125,19 @@ module Typhoeus
       @running_requests += 1
 
       easy = @easy_pool.pop || Easy.new
+      easy.verbose          = request.verbose
+      if request.username || request.password
+        auth = { :username => request.username, :password => request.password }
+        auth[:method] = Typhoeus::Easy::AUTH_TYPES["CURLAUTH_#{request.auth_method.to_s.upcase}".to_sym] if request.auth_method
+        easy.auth = auth
+      end
       easy.url          = request.url
       easy.method       = request.method
       easy.params       = request.params  if request.method == :post && !request.params.nil?
       easy.headers      = request.headers if request.headers
       easy.request_body = request.body    if request.body
       easy.timeout      = request.timeout if request.timeout
+      easy.connect_timeout = request.connect_timeout if request.connect_timeout
       easy.follow_location = request.follow_location if request.follow_location
       easy.max_redirects = request.max_redirects if request.max_redirects
       easy.proxy = request.proxy if request.proxy
@@ -149,6 +149,7 @@ module Typhoeus
       easy.ssl_key_password = request.ssl_key_password
       easy.ssl_cacert       = request.ssl_cacert
       easy.ssl_capath       = request.ssl_capath
+      easy.verbose          = request.verbose
 
       easy.on_success do |easy|
         queue_next
@@ -180,6 +181,9 @@ module Typhoeus
     def handle_request(request, response, live_request = true)
       request.response = response
 
+      self.class.run_global_hooks_for(:after_request_before_on_complete,
+                                      request)
+
       if live_request && request.cache_timeout && @cache_setter
         @cache_setter.call(request)
       end
@@ -205,30 +209,4 @@ module Typhoeus
     end
     private :response_from_easy
   end
-
-  class HydraMock
-    attr_reader :url, :method, :response, :requests
-
-    def initialize(url, method)
-      @url      = url
-      @method   = method
-      @requests = []
-    end
-
-    def add_request(request)
-      @requests << request
-    end
-
-    def and_return(val)
-      @response = val
-    end
-
-    def matches?(request)
-      if url.kind_of?(String)
-        request.method == method && request.url == url
-      else
-        request.method == method && url =~ request.url
-      end
-    end
-  end
 end

data/lib/typhoeus/hydra_mock.rb

@@ -0,0 +1,131 @@
+module Typhoeus
+  class HydraMock
+    attr_reader :url, :method, :requests, :uri
+
+    def initialize(url, method, options = {})
+      @url      = url
+      @uri      = URI.parse(url) if url.kind_of?(String)
+      @method   = method
+      @requests = []
+      @options = options
+      if @options[:headers]
+        @options[:headers] = Typhoeus::NormalizedHeaderHash.new(@options[:headers])
+      end
+
+      @current_response_index = 0
+    end
+
+    def body
+      @options[:body]
+    end
+
+    def body?
+      @options.has_key?(:body)
+    end
+
+    def headers
+      @options[:headers]
+    end
+
+    def headers?
+      @options.has_key?(:headers)
+    end
+
+    def add_request(request)
+      @requests << request
+    end
+
+    def and_return(val)
+      if val.respond_to?(:each)
+        @responses = val
+      else
+        @responses = [val]
+      end
+
+      # make sure to mark them as a mock.
+      @responses.each { |r| r.mock = true }
+
+      val
+    end
+
+    def response
+      if @current_response_index == (@responses.length - 1)
+        @responses.last
+      else
+        value = @responses[@current_response_index]
+        @current_response_index += 1
+        value
+      end
+    end
+
+    def matches?(request)
+      if !method_matches?(request) or !url_matches?(request)
+        return false
+      end
+
+      if body?
+        return false unless body_matches?(request)
+      end
+
+      if headers?
+        return false unless headers_match?(request)
+      end
+
+      true
+    end
+
+  private
+    def method_matches?(request)
+      self.method == :any or self.method == request.method
+    end
+
+    def url_matches?(request)
+      if url.kind_of?(String)
+        request_uri = URI.parse(request.url)
+        request_uri == self.uri
+      else
+        self.url =~ request.url
+      end
+    end
+
+    def body_matches?(request)
+      !request.body.nil? && !request.body.empty? && request.body == self.body
+    end
+
+    def headers_match?(request)
+      request_headers = NormalizedHeaderHash.new(request.headers)
+
+      if empty_headers?(self.headers)
+        empty_headers?(request_headers)
+      else
+        return false if empty_headers?(request_headers)
+
+        headers.each do |key, value|
+          return false unless header_value_matches?(value, request_headers[key])
+        end
+
+        true
+      end
+    end
+
+    def header_value_matches?(mock_value, request_value)
+      mock_arr = mock_value.is_a?(Array) ? mock_value : [mock_value]
+      request_arr = request_value.is_a?(Array) ? request_value : [request_value]
+
+      return false unless mock_arr.size == request_arr.size
+      mock_arr.all? do |value|
+        request_arr.any? { |a| value === a }
+      end
+    end
+
+    def empty_headers?(headers)
+      # We consider the default User-Agent header to be empty since
+      # Typhoeus always adds that.
+      headers.nil? || headers.empty? || default_typhoeus_headers?(headers)
+    end
+
+    def default_typhoeus_headers?(headers)
+      headers.size == 1 && headers['User-Agent'] == Typhoeus::USER_AGENT
+    end
+  end
+end

data/lib/typhoeus/multi.rb

@@ -11,7 +11,7 @@ module Typhoeus
     end
 
     def add(easy)
-      easy.set_headers()
+      easy.set_headers() if easy.headers.empty?
       @easy_handles << easy
       multi_add_handle(easy)
     end