http_configuration 1.0.0

data/MIT-LICENSE

@@ -0,0 +1,20 @@
+Copyright (c) 2007 Brian Durand
+
+Permission is hereby granted, free of charge, to any person obtaining
+a copy of this software and associated documentation files (the
+"Software"), to deal in the Software without restriction, including
+without limitation the rights to use, copy, modify, merge, publish,
+distribute, sublicense, and/or sell copies of the Software, and to
+permit persons to whom the Software is furnished to do so, subject to
+the following conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
+LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
+OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
+WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

data/README

@@ -0,0 +1,56 @@
+= HTTP Configuration
+
+This gem provides a means of configuring the Net::HTTP package to suit your environment. It also fixes an issue with Net::HTTP where threads will exit when they timeout. Even if you don't need the configuration part of the code, the timeout fix may make this gem worth your while to install. This code can also be installed as a Rails plugin.
+
+== Consistent Configuration
+
+Configurations are defined by Net::HTTP::Configuration objects. You can set a global configuration or apply a configuration to just a block of code. By using the configuration objects, you can normalize how proxies and timeouts are set across your applications and even for code you didn't write.
+
+Options are passed in as a hash:
+
+* :read_timeout => the default read timeout value
+* :open_timeout => the default open timeout value
+* :proxy_host => the host name of the proxy server
+* :proxy_port => the port of the proxy server
+* :proxy_user => the user name for the proxy server
+* :proxy_password => the password for the proxy server
+* :no_proxy => either an array or a comma delimited list of host names not to proxy
+* :proxy => a shorthand method of setting the proxy information
+
+The :no_proxy values are case insensitive and only need to match the end of the host name. So, if you have a local network that uses *.local DNS names, you can set :no_proxy => '.local' to prevent the proxy server from being used for local hosts.
+
+The :proxy option can be used to set the proxy server information in a simplified manner. You can pass the proxy information as a string in the format [user:password@]host:port. In addition, you can pass the special value of :environment to automatically read the proxy information from the environment variables HTTP_PROXY or http_proxy and the :no_proxy value from the NO_PROXY or no_proxy variables. This lets you move your environment configuration entirely out of your ruby code if desired.
+
+When a configuration is applied to a block, you can also pass in some override values at the same time. In this code, the :read_timeout will be 5 seconds instead of the default 10:
+
+  config = Net::HTTP::Configuration.new(:proxy => :environment, :read_timeout => 10, :open_timeout => 5)
+	config.apply(:read_timeout => 5) do
+		Net::HTTP.get('http://example.com/')
+	end
+
+== Adapting Code To Your Environment
+
+If your code has to live in an environment that requires a proxy server, you may have run into problems with externally developed code where the author didn't have to worry about proxies. With this code installed, you don't have to worry about breaking open the code and hacking it up to add proxy support.
+
+Similarly, if you are building a high traffic web site that makes web service calls to external hosts, you may want to set HTTP timeouts to lower values so that if the external server is running slow, your server doesn't end up eating up all its threads waiting on that server. For example, suppose you have a site that handles 10 requests per second and 1% of those requests make a web service request to another server and to handle that traffic, you have 50 mongrel servers available. If the external service goes crazy for some reason and starts taking 60 seconds to serve a response that normally takes less than 1 second, you will start loosing a mongrel server every 10 seconds and your site will be completely down in about 10 minutes. You could significantly lower this risk by setting the open and read timeouts on HTTP to be a much smaller value. You'll still end up with errors, but the site won't go down.
+
+== Fix For Timeouts
+
+One potential issue you can have with the Net::HTTP is that it handles timeouts by throwing interrupts. This is very effective however, interrupts have the unfortunate side effect of killing the current thread. This can be quite a problem. Consider this code:
+
+	loop do
+	  begin
+			begin_really_important_transaction()
+	    Net::HTTP.get('http://www.example.com/really_important_request')
+		rescue => e
+		  logger.warn(e)
+			Notifications.new(:critical).send("The really important request failed; this must be fixed immediately.")
+		ensure
+			finish_really_important_transaction()
+		end
+		sleep(300)
+	end
+
+If the HTTP request times out, the code will not execute they way you'd think. It will receive an interrupt and immediately exit. The log warning and notification will not be sent. Further, the ensure block will not even be executed which could be really bad.
+
+This code simply sets the default exception to be thrown by timeouts in the Net module to Net::NetworkTimeoutError instead of TimeoutError. Since Net::NetworkTimeoutError doesn't extend from Interrupt, your thread can go on its merry way.

data/Rakefile

@@ -0,0 +1,42 @@
+require 'rubygems'
+require 'rake'
+require 'rake/testtask'
+require 'rake/rdoctask'
+require 'rake/gempackagetask'
+
+desc 'Default: run unit tests.'
+task :default => :test
+
+desc 'Test http_configuration.'
+Rake::TestTask.new(:test) do |t|
+  t.libs << 'lib'
+  t.pattern = 'test/**/*_test.rb'
+  t.verbose = true
+end
+
+desc 'Generate documentation for http_configuration.'
+Rake::RDocTask.new(:rdoc) do |rdoc|
+  rdoc.rdoc_dir = 'rdoc'
+  rdoc.title    = 'Http Configuration'
+  rdoc.options << '--line-numbers' << '--inline-source'
+  rdoc.rdoc_files.include('README')
+  rdoc.rdoc_files.include('lib/**/*.rb')
+end
+
+spec = Gem::Specification.new do |s| 
+  s.name = "http_configuration"
+  s.version = "1.0.0"
+  s.author = "Brian Durand"
+  s.platform = Gem::Platform::RUBY
+  s.summary = "Provide configuration options for Net::HTTP"
+  s.files = FileList["lib/*", "init.rb", "MIT-LICENSE", 'Rakefile'].to_a
+  s.require_path = "lib"
+  s.autorequire = "init.rb"
+  s.test_files = FileList["{test}/**/*_test.rb"].to_a
+  s.has_rdoc = true
+  s.extra_rdoc_files = ["README"]
+end
+ 
+Rake::GemPackageTask.new(spec) do |pkg| 
+  pkg.need_tar = true 
+end

data/init.rb

@@ -0,0 +1 @@
+require 'http_configuration'

data/lib/http_configuration.rb

@@ -0,0 +1,188 @@
+require 'net/protocol'
+require 'net/http'
+
+module Net
+  
+  class Protocol
+    
+    private
+    
+    # Default error type to a non-interrupt exception
+    def timeout (secs, errorType = NetworkTimeoutError)
+      super(secs, errorType)
+    end
+    
+  end
+  
+  class BufferedIO
+    
+    private
+    
+    # Default error type to a non-interrupt exception
+    def timeout (secs, errorType = NetworkTimeoutError)
+      super(secs, errorType)
+    end
+    
+  end
+  
+  # Error thrown by network timeouts
+  class NetworkTimeoutError < StandardError
+  end
+  
+  class HTTP
+    
+    class << self
+      def new_with_configuration (address, port = nil, p_addr = nil, p_port = nil, p_user = nil, p_pass = nil)
+        config_options = Configuration.current
+        if config_options
+          if Configuration.no_proxy?(address, config_options)
+            p_addr = nil
+            p_port = nil
+            p_user = nil
+            p_pass = nil
+          elsif p_addr.nil? and config_options[:proxy_host]
+            p_addr = config_options[:proxy_host]
+            p_port = config_options[:proxy_port].to_i
+            p_user = config_options[:proxy_user]
+            p_pass = config_options[:proxy_password]
+          end
+        end
+        
+        http = HTTP.new_without_configuration(address, port, p_addr, p_port, p_user, p_pass)
+      
+        if config_options
+          http.open_timeout = config_options[:open_timeout] if config_options[:open_timeout]
+          http.read_timeout = config_options[:read_timeout] if config_options[:read_timeout]
+        end
+      
+        return http
+      end
+      
+      alias_method :new_without_configuration, :new
+      alias_method :new, :new_with_configuration
+      
+    end
+  
+    # The Configuration class encapsulates a set of HTTP defaults. The configuration
+    # can made either global or all requests, or it can be applied only within a block.
+    # Configuration blocks can also set an additional set of options which take precedence
+    # over the initialization options.
+    #
+    # Available options are :proxy_host, :proxy_port, :proxy_user, :proxy_password, :no_proxy,
+    # :read_timeout, :open_timeout, and :proxy. This last value can either contain a proxy string
+    # or the symbol :none for no proxy or :environment to use the values in the HTTP_PROXY/http_proxy
+    # and NO_PROXY/no_proxy environment variables.
+    #
+    # If you specify a proxy, but don't want it to be used for certain hosts, specify the domain names
+    # in the :no_proxy option. This can either be an array or a comma delimited string. A request to a
+    # host name which ends with any of these values will not be proxied.
+    #
+    # The normal functionality for Net::HTTP is still available, so you can set proxies
+    # and timeouts manually if needed. Because of the way in which https calls are made, you cannot
+    # configure a special proxy just for https calls.
+    class Configuration
+    
+      def initialize (options = {})
+        @default_options = options.dup
+        expand_proxy_config!(@default_options)
+      end
+    
+      # Get the specified option for the configuration.
+      def [] (name)
+        @default_options[name]
+      end
+      
+      # Apply the configuration to the block. If any options are provided, they will override the default options
+      # for the configuration.
+      def apply (options = {})
+        options = @default_options.merge(options)
+        expand_proxy_config!(options)
+        Thread.current[:net_http_configuration] ||= []
+        Thread.current[:net_http_configuration].push(options)
+        begin
+          return yield
+        ensure
+          Thread.current[:net_http_configuration].pop
+        end
+      end
+      
+      def self.no_proxy? (host, options)
+        return false unless options[:no_proxy].kind_of?(Array)
+        
+        host = host.downcase
+        options[:no_proxy].each do |pattern|
+          pattern = pattern.downcase
+          return true if host[-pattern.length, pattern.length] == pattern
+        end
+        
+        return false
+      end
+      
+      # Set the options for a global configuration used for all HTTP requests. The global configuration can be cleared
+      # by setting nil
+      def self.set_global (options)
+        if options
+          @global = Configuration.new(options)
+        else
+          @global = nil
+        end
+      end
+      
+      def self.global
+        @global
+      end
+      
+      # Get the current configuration that is in scope.
+      def self.current
+        stack = Thread.current[:net_http_configuration]
+        config = stack.last if stack
+        config || global
+      end
+      
+      private
+      
+      def expand_proxy_config! (options)
+        proxy_config = options[:proxy]
+        if proxy_config
+          options.delete(:proxy)
+          if proxy_config == :environment
+            parse_proxy!(ENV['HTTP_PROXY'] || ENV['http_proxy'], options)
+            options[:no_proxy] = ENV['NO_PROXY'] || ENV['no_proxy']
+          elsif proxy_config == :none
+            options[:proxy_user] = nil
+            options[:proxy_password] = nil
+            options[:proxy_host] = nil
+            options[:proxy_port] = nil
+            options[:no_proxy] = nil
+          else
+            parse_proxy!(proxy_config, options)
+          end
+        end
+        parse_no_proxy!(options[:no_proxy], options)
+      end
+      
+      def parse_proxy! (proxy, options)
+        return unless proxy and proxy.length > 0
+        proxy_info = /(http:\/\/)?(([^:]+):([^@]+)@)?([^:]+)(:(\d+))?/i.match(proxy)
+        if proxy_info
+          options[:proxy_user] = proxy_info[3]
+          options[:proxy_password] = proxy_info[4]
+          options[:proxy_host] = proxy_info[5]
+          options[:proxy_port] = proxy_info[7].to_i if proxy_info[7]
+        end
+      end
+      
+      def parse_no_proxy! (no_proxy, options)
+        return unless no_proxy and no_proxy.length > 0
+        if no_proxy.kind_of?(Array)
+          options[:no_proxy] = no_proxy.dup
+        else
+          options[:no_proxy] = no_proxy.split(/\s*,\s*/)
+        end
+      end
+    end
+    
+  end
+  
+end
+      

data/test/http_configuration_test.rb

@@ -0,0 +1,169 @@
+require 'rubygems'
+require 'spec'
+require File.expand_path(File.join(File.dirname(__FILE__), '..', 'lib', 'http_configuration'))
+
+describe "Net::HTTP::Configuration" do
+  
+  it "should set the default configuration options" do
+    config = Net::HTTP::Configuration.new(:proxy_host => 'localhost', :proxy_port => 8080, :no_proxy => ['local1', 'local2'])
+    config[:proxy_host].should == 'localhost'
+    config[:proxy_port].should == 8080
+    config[:no_proxy].should == ['local1', 'local2']
+  end
+  
+  it "should be able to get the proxy from the environment" do
+    ENV.should_receive(:[]).with('HTTP_PROXY').and_return('localhost:80')
+    ENV.should_receive(:[]).with('NO_PROXY').and_return('.local1, .local2')
+    config = Net::HTTP::Configuration.new(:proxy => :environment)
+    config[:proxy_host].should == 'localhost'
+    config[:proxy_port].should == 80
+    config[:proxy_user].should == nil
+    config[:proxy_password].should == nil
+    config[:no_proxy].should == ['.local1', '.local2']
+  end
+  
+  it "should be able to parse the no_proxy option" do
+    Net::HTTP::Configuration.new(:no_proxy => 'host')[:no_proxy].should == ['host']
+    Net::HTTP::Configuration.new(:no_proxy => 'host1, host2')[:no_proxy].should == ['host1', 'host2']
+    Net::HTTP::Configuration.new(:no_proxy => ['host3', 'host4'])[:no_proxy].should == ['host3', 'host4']
+  end
+  
+  it "should be able to parse a proxy with user and password" do
+    config = Net::HTTP::Configuration.new(:proxy => 'http://user:password@proxy.local:9000', :no_proxy => '.local1, .local2')
+    config[:proxy_host].should == 'proxy.local'
+    config[:proxy_port].should == 9000
+    config[:proxy_user].should == 'user'
+    config[:proxy_password].should == 'password'
+    config[:no_proxy].should == ['.local1', '.local2']
+    config[:proxy].should == nil
+  end
+  
+  it "should be able to clear the proxy" do
+    config = Net::HTTP::Configuration.new(:proxy => :none)
+    config[:proxy_host].should == nil
+    config[:proxy_port].should == nil
+    config[:proxy_user].should == nil
+    config[:proxy_password].should == nil
+  end
+  
+  it "should be able to set a global configuration" do
+    Net::HTTP::Configuration.set_global(:proxy_host => 'localhost', :proxy_port => 8080, :read_timeout => 5)
+    Net::HTTP::Configuration.global[:proxy_host].should == 'localhost'
+    Net::HTTP::Configuration.global[:proxy_port].should == 8080
+    Net::HTTP::Configuration.global[:read_timeout].should == 5
+    Net::HTTP::Configuration.set_global(nil)
+    Net::HTTP::Configuration.global.should == nil
+  end
+  
+  it "should be able to apply a configuration to a block" do
+    Net::HTTP::Configuration.set_global(nil)
+    config = Net::HTTP::Configuration.new(:proxy_host => "proxy#{rand(1000)}", :read_timeout => 30)
+    retval = config.apply(:read_timeout => 5) do
+      Net::HTTP::Configuration.current[:proxy_host].should == config[:proxy_host]
+      Net::HTTP::Configuration.current[:read_timeout].should == 5
+      :done
+    end
+    retval.should == :done
+  end
+  
+  it "should be able to determine the current configuration" do
+    Net::HTTP::Configuration.set_global(:proxy_host => 'global')
+    config1 = Net::HTTP::Configuration.new(:proxy_host => 'config1')
+    config2 = Net::HTTP::Configuration.new(:proxy_host => 'config2')
+    
+    Net::HTTP::Configuration.current[:proxy_host].should == 'global'
+    config1.apply do
+      Net::HTTP::Configuration.current[:proxy_host].should == 'config1'
+      config2.apply do
+        Net::HTTP::Configuration.current[:proxy_host].should == 'config2'
+      end
+      Net::HTTP::Configuration.current[:proxy_host].should == 'config1'
+    end
+    Net::HTTP::Configuration.current[:proxy_host].should == 'global'
+    
+    Net::HTTP::Configuration.set_global(nil)
+    Net::HTTP::Configuration.current.should == nil
+  end
+  
+  it "should be able to determine if a host does not require a proxy" do
+    Net::HTTP::Configuration.no_proxy?('host.local', :no_proxy => ['host.local']).should == true
+    Net::HTTP::Configuration.no_proxy?('host.local', :no_proxy => ['not_this_one', '.local']).should == true
+    Net::HTTP::Configuration.no_proxy?('HOST.LOCAL', :no_proxy => ['.local']).should == true
+    Net::HTTP::Configuration.no_proxy?('host.local', :no_proxy => ['.LOCAL']).should == true
+    Net::HTTP::Configuration.no_proxy?('host.local', :no_proxy => ['other.host.local']).should == false
+    Net::HTTP::Configuration.no_proxy?('external.host', :no_proxy => ['.local']).should == false
+    Net::HTTP::Configuration.no_proxy?('external.host', :no_proxy => nil).should == false
+  end
+  
+end
+
+describe "Net::HTTP" do
+  
+  it "should work normally if no configuration has been set" do
+    Net::HTTP.should_receive(:new_without_configuration).with('localhost', 80, nil, nil, nil, nil)
+    Net::HTTP.new('localhost', 80)
+  end
+  
+  it "should use proxy settings if they have been set" do
+    config = Net::HTTP::Configuration.new(:proxy_host => 'proxy', :proxy_port => 8080, :proxy_user => 'user', :proxy_password => 'password')
+    Net::HTTP.should_receive(:new_without_configuration).with('localhost', 80, 'proxy', 8080, 'user', 'password')
+    config.apply do
+      Net::HTTP.new('localhost', 80)
+    end
+  end
+  
+  it "should honor no_proxy hosts" do
+    config = Net::HTTP::Configuration.new(:proxy_host => 'proxy', :proxy_port => 8080, :no_proxy => ['localhost'])
+    Net::HTTP.should_receive(:new_without_configuration).with('localhost', 80, nil, nil, nil, nil)
+    config.apply do
+      Net::HTTP.new('localhost', 80)
+    end
+  end
+  
+  it "should honor proxies explicitly passed" do
+    config = Net::HTTP::Configuration.new(:proxy_host => 'proxy', :proxy_port => 8080, :proxy_user => 'user', :proxy_password => 'password')
+    Net::HTTP.should_receive(:new_without_configuration).with('localhost', 80, 'other_proxy', 9000, nil, nil)
+    config.apply do
+      Net::HTTP.new('localhost', 80, 'other_proxy', 9000)
+    end
+  end
+  
+  it "should set read_timeout from the configuration" do
+    config = Net::HTTP::Configuration.new(:read_timeout => 7)
+    config.apply do
+      http = Net::HTTP.new('localhost', 80)
+      http.read_timeout.should == 7
+    end
+  end
+  
+  it "should set open_timeout from the configuration" do
+    config = Net::HTTP::Configuration.new(:open_timeout => 6)
+    config.apply do
+      http = Net::HTTP.new('localhost', 80)
+      http.open_timeout.should == 6
+    end
+  end
+  
+end
+
+describe "Net::BufferedIO" do
+  
+  it "should timeout without an interrupt" do
+    io = Net::BufferedIO.new(StringIO.new)
+    lambda do
+      io.send(:timeout, 1){sleep(2)}
+    end.should raise_error(Net::NetworkTimeoutError)
+  end
+
+end
+
+describe "Net::Protocol" do
+  
+  it "should timeout without an interrupt" do
+    http = Net::HTTP.new('localhost')
+    lambda do
+      http.send(:timeout, 1){sleep(2)}
+    end.should raise_error(Net::NetworkTimeoutError)
+  end
+  
+end

metadata

@@ -0,0 +1,50 @@
+--- !ruby/object:Gem::Specification 
+rubygems_version: 0.9.0
+specification_version: 1
+name: http_configuration
+version: !ruby/object:Gem::Version 
+  version: 1.0.0
+date: 2007-10-24 00:00:00 -05:00
+summary: Provide configuration options for Net::HTTP
+require_paths: 
+- lib
+email: 
+homepage: 
+rubyforge_project: 
+description: 
+autorequire: init.rb
+default_executable: 
+bindir: bin
+has_rdoc: true
+required_ruby_version: !ruby/object:Gem::Version::Requirement 
+  requirements: 
+  - - ">"
+    - !ruby/object:Gem::Version 
+      version: 0.0.0
+  version: 
+platform: ruby
+signing_key: 
+cert_chain: 
+post_install_message: 
+authors: 
+- Brian Durand
+files: 
+- lib/http_configuration.rb
+- init.rb
+- MIT-LICENSE
+- Rakefile
+- README
+test_files: 
+- test/http_configuration_test.rb
+rdoc_options: []
+
+extra_rdoc_files: 
+- README
+executables: []
+
+extensions: []
+
+requirements: []
+
+dependencies: []
+