http_configuration 1.0.0

Sign up to get free protection for your applications and to get access to all the features.
data/MIT-LICENSE ADDED
@@ -0,0 +1,20 @@
1
+ Copyright (c) 2007 Brian Durand
2
+
3
+ Permission is hereby granted, free of charge, to any person obtaining
4
+ a copy of this software and associated documentation files (the
5
+ "Software"), to deal in the Software without restriction, including
6
+ without limitation the rights to use, copy, modify, merge, publish,
7
+ distribute, sublicense, and/or sell copies of the Software, and to
8
+ permit persons to whom the Software is furnished to do so, subject to
9
+ the following conditions:
10
+
11
+ The above copyright notice and this permission notice shall be
12
+ included in all copies or substantial portions of the Software.
13
+
14
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
15
+ EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
16
+ MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
17
+ NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
18
+ LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
19
+ OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
20
+ WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
data/README ADDED
@@ -0,0 +1,56 @@
1
+ = HTTP Configuration
2
+
3
+ This gem provides a means of configuring the Net::HTTP package to suit your environment. It also fixes an issue with Net::HTTP where threads will exit when they timeout. Even if you don't need the configuration part of the code, the timeout fix may make this gem worth your while to install. This code can also be installed as a Rails plugin.
4
+
5
+ == Consistent Configuration
6
+
7
+ Configurations are defined by Net::HTTP::Configuration objects. You can set a global configuration or apply a configuration to just a block of code. By using the configuration objects, you can normalize how proxies and timeouts are set across your applications and even for code you didn't write.
8
+
9
+ Options are passed in as a hash:
10
+
11
+ * :read_timeout => the default read timeout value
12
+ * :open_timeout => the default open timeout value
13
+ * :proxy_host => the host name of the proxy server
14
+ * :proxy_port => the port of the proxy server
15
+ * :proxy_user => the user name for the proxy server
16
+ * :proxy_password => the password for the proxy server
17
+ * :no_proxy => either an array or a comma delimited list of host names not to proxy
18
+ * :proxy => a shorthand method of setting the proxy information
19
+
20
+ The :no_proxy values are case insensitive and only need to match the end of the host name. So, if you have a local network that uses *.local DNS names, you can set :no_proxy => '.local' to prevent the proxy server from being used for local hosts.
21
+
22
+ The :proxy option can be used to set the proxy server information in a simplified manner. You can pass the proxy information as a string in the format [user:password@]host:port. In addition, you can pass the special value of :environment to automatically read the proxy information from the environment variables HTTP_PROXY or http_proxy and the :no_proxy value from the NO_PROXY or no_proxy variables. This lets you move your environment configuration entirely out of your ruby code if desired.
23
+
24
+ When a configuration is applied to a block, you can also pass in some override values at the same time. In this code, the :read_timeout will be 5 seconds instead of the default 10:
25
+
26
+ config = Net::HTTP::Configuration.new(:proxy => :environment, :read_timeout => 10, :open_timeout => 5)
27
+ config.apply(:read_timeout => 5) do
28
+ Net::HTTP.get('http://example.com/')
29
+ end
30
+
31
+ == Adapting Code To Your Environment
32
+
33
+ If your code has to live in an environment that requires a proxy server, you may have run into problems with externally developed code where the author didn't have to worry about proxies. With this code installed, you don't have to worry about breaking open the code and hacking it up to add proxy support.
34
+
35
+ Similarly, if you are building a high traffic web site that makes web service calls to external hosts, you may want to set HTTP timeouts to lower values so that if the external server is running slow, your server doesn't end up eating up all its threads waiting on that server. For example, suppose you have a site that handles 10 requests per second and 1% of those requests make a web service request to another server and to handle that traffic, you have 50 mongrel servers available. If the external service goes crazy for some reason and starts taking 60 seconds to serve a response that normally takes less than 1 second, you will start loosing a mongrel server every 10 seconds and your site will be completely down in about 10 minutes. You could significantly lower this risk by setting the open and read timeouts on HTTP to be a much smaller value. You'll still end up with errors, but the site won't go down.
36
+
37
+ == Fix For Timeouts
38
+
39
+ One potential issue you can have with the Net::HTTP is that it handles timeouts by throwing interrupts. This is very effective however, interrupts have the unfortunate side effect of killing the current thread. This can be quite a problem. Consider this code:
40
+
41
+ loop do
42
+ begin
43
+ begin_really_important_transaction()
44
+ Net::HTTP.get('http://www.example.com/really_important_request')
45
+ rescue => e
46
+ logger.warn(e)
47
+ Notifications.new(:critical).send("The really important request failed; this must be fixed immediately.")
48
+ ensure
49
+ finish_really_important_transaction()
50
+ end
51
+ sleep(300)
52
+ end
53
+
54
+ If the HTTP request times out, the code will not execute they way you'd think. It will receive an interrupt and immediately exit. The log warning and notification will not be sent. Further, the ensure block will not even be executed which could be really bad.
55
+
56
+ This code simply sets the default exception to be thrown by timeouts in the Net module to Net::NetworkTimeoutError instead of TimeoutError. Since Net::NetworkTimeoutError doesn't extend from Interrupt, your thread can go on its merry way.
data/Rakefile ADDED
@@ -0,0 +1,42 @@
1
+ require 'rubygems'
2
+ require 'rake'
3
+ require 'rake/testtask'
4
+ require 'rake/rdoctask'
5
+ require 'rake/gempackagetask'
6
+
7
+ desc 'Default: run unit tests.'
8
+ task :default => :test
9
+
10
+ desc 'Test http_configuration.'
11
+ Rake::TestTask.new(:test) do |t|
12
+ t.libs << 'lib'
13
+ t.pattern = 'test/**/*_test.rb'
14
+ t.verbose = true
15
+ end
16
+
17
+ desc 'Generate documentation for http_configuration.'
18
+ Rake::RDocTask.new(:rdoc) do |rdoc|
19
+ rdoc.rdoc_dir = 'rdoc'
20
+ rdoc.title = 'Http Configuration'
21
+ rdoc.options << '--line-numbers' << '--inline-source'
22
+ rdoc.rdoc_files.include('README')
23
+ rdoc.rdoc_files.include('lib/**/*.rb')
24
+ end
25
+
26
+ spec = Gem::Specification.new do |s|
27
+ s.name = "http_configuration"
28
+ s.version = "1.0.0"
29
+ s.author = "Brian Durand"
30
+ s.platform = Gem::Platform::RUBY
31
+ s.summary = "Provide configuration options for Net::HTTP"
32
+ s.files = FileList["lib/*", "init.rb", "MIT-LICENSE", 'Rakefile'].to_a
33
+ s.require_path = "lib"
34
+ s.autorequire = "init.rb"
35
+ s.test_files = FileList["{test}/**/*_test.rb"].to_a
36
+ s.has_rdoc = true
37
+ s.extra_rdoc_files = ["README"]
38
+ end
39
+
40
+ Rake::GemPackageTask.new(spec) do |pkg|
41
+ pkg.need_tar = true
42
+ end
data/init.rb ADDED
@@ -0,0 +1 @@
1
+ require 'http_configuration'
@@ -0,0 +1,188 @@
1
+ require 'net/protocol'
2
+ require 'net/http'
3
+
4
+ module Net
5
+
6
+ class Protocol
7
+
8
+ private
9
+
10
+ # Default error type to a non-interrupt exception
11
+ def timeout (secs, errorType = NetworkTimeoutError)
12
+ super(secs, errorType)
13
+ end
14
+
15
+ end
16
+
17
+ class BufferedIO
18
+
19
+ private
20
+
21
+ # Default error type to a non-interrupt exception
22
+ def timeout (secs, errorType = NetworkTimeoutError)
23
+ super(secs, errorType)
24
+ end
25
+
26
+ end
27
+
28
+ # Error thrown by network timeouts
29
+ class NetworkTimeoutError < StandardError
30
+ end
31
+
32
+ class HTTP
33
+
34
+ class << self
35
+ def new_with_configuration (address, port = nil, p_addr = nil, p_port = nil, p_user = nil, p_pass = nil)
36
+ config_options = Configuration.current
37
+ if config_options
38
+ if Configuration.no_proxy?(address, config_options)
39
+ p_addr = nil
40
+ p_port = nil
41
+ p_user = nil
42
+ p_pass = nil
43
+ elsif p_addr.nil? and config_options[:proxy_host]
44
+ p_addr = config_options[:proxy_host]
45
+ p_port = config_options[:proxy_port].to_i
46
+ p_user = config_options[:proxy_user]
47
+ p_pass = config_options[:proxy_password]
48
+ end
49
+ end
50
+
51
+ http = HTTP.new_without_configuration(address, port, p_addr, p_port, p_user, p_pass)
52
+
53
+ if config_options
54
+ http.open_timeout = config_options[:open_timeout] if config_options[:open_timeout]
55
+ http.read_timeout = config_options[:read_timeout] if config_options[:read_timeout]
56
+ end
57
+
58
+ return http
59
+ end
60
+
61
+ alias_method :new_without_configuration, :new
62
+ alias_method :new, :new_with_configuration
63
+
64
+ end
65
+
66
+ # The Configuration class encapsulates a set of HTTP defaults. The configuration
67
+ # can made either global or all requests, or it can be applied only within a block.
68
+ # Configuration blocks can also set an additional set of options which take precedence
69
+ # over the initialization options.
70
+ #
71
+ # Available options are :proxy_host, :proxy_port, :proxy_user, :proxy_password, :no_proxy,
72
+ # :read_timeout, :open_timeout, and :proxy. This last value can either contain a proxy string
73
+ # or the symbol :none for no proxy or :environment to use the values in the HTTP_PROXY/http_proxy
74
+ # and NO_PROXY/no_proxy environment variables.
75
+ #
76
+ # If you specify a proxy, but don't want it to be used for certain hosts, specify the domain names
77
+ # in the :no_proxy option. This can either be an array or a comma delimited string. A request to a
78
+ # host name which ends with any of these values will not be proxied.
79
+ #
80
+ # The normal functionality for Net::HTTP is still available, so you can set proxies
81
+ # and timeouts manually if needed. Because of the way in which https calls are made, you cannot
82
+ # configure a special proxy just for https calls.
83
+ class Configuration
84
+
85
+ def initialize (options = {})
86
+ @default_options = options.dup
87
+ expand_proxy_config!(@default_options)
88
+ end
89
+
90
+ # Get the specified option for the configuration.
91
+ def [] (name)
92
+ @default_options[name]
93
+ end
94
+
95
+ # Apply the configuration to the block. If any options are provided, they will override the default options
96
+ # for the configuration.
97
+ def apply (options = {})
98
+ options = @default_options.merge(options)
99
+ expand_proxy_config!(options)
100
+ Thread.current[:net_http_configuration] ||= []
101
+ Thread.current[:net_http_configuration].push(options)
102
+ begin
103
+ return yield
104
+ ensure
105
+ Thread.current[:net_http_configuration].pop
106
+ end
107
+ end
108
+
109
+ def self.no_proxy? (host, options)
110
+ return false unless options[:no_proxy].kind_of?(Array)
111
+
112
+ host = host.downcase
113
+ options[:no_proxy].each do |pattern|
114
+ pattern = pattern.downcase
115
+ return true if host[-pattern.length, pattern.length] == pattern
116
+ end
117
+
118
+ return false
119
+ end
120
+
121
+ # Set the options for a global configuration used for all HTTP requests. The global configuration can be cleared
122
+ # by setting nil
123
+ def self.set_global (options)
124
+ if options
125
+ @global = Configuration.new(options)
126
+ else
127
+ @global = nil
128
+ end
129
+ end
130
+
131
+ def self.global
132
+ @global
133
+ end
134
+
135
+ # Get the current configuration that is in scope.
136
+ def self.current
137
+ stack = Thread.current[:net_http_configuration]
138
+ config = stack.last if stack
139
+ config || global
140
+ end
141
+
142
+ private
143
+
144
+ def expand_proxy_config! (options)
145
+ proxy_config = options[:proxy]
146
+ if proxy_config
147
+ options.delete(:proxy)
148
+ if proxy_config == :environment
149
+ parse_proxy!(ENV['HTTP_PROXY'] || ENV['http_proxy'], options)
150
+ options[:no_proxy] = ENV['NO_PROXY'] || ENV['no_proxy']
151
+ elsif proxy_config == :none
152
+ options[:proxy_user] = nil
153
+ options[:proxy_password] = nil
154
+ options[:proxy_host] = nil
155
+ options[:proxy_port] = nil
156
+ options[:no_proxy] = nil
157
+ else
158
+ parse_proxy!(proxy_config, options)
159
+ end
160
+ end
161
+ parse_no_proxy!(options[:no_proxy], options)
162
+ end
163
+
164
+ def parse_proxy! (proxy, options)
165
+ return unless proxy and proxy.length > 0
166
+ proxy_info = /(http:\/\/)?(([^:]+):([^@]+)@)?([^:]+)(:(\d+))?/i.match(proxy)
167
+ if proxy_info
168
+ options[:proxy_user] = proxy_info[3]
169
+ options[:proxy_password] = proxy_info[4]
170
+ options[:proxy_host] = proxy_info[5]
171
+ options[:proxy_port] = proxy_info[7].to_i if proxy_info[7]
172
+ end
173
+ end
174
+
175
+ def parse_no_proxy! (no_proxy, options)
176
+ return unless no_proxy and no_proxy.length > 0
177
+ if no_proxy.kind_of?(Array)
178
+ options[:no_proxy] = no_proxy.dup
179
+ else
180
+ options[:no_proxy] = no_proxy.split(/\s*,\s*/)
181
+ end
182
+ end
183
+ end
184
+
185
+ end
186
+
187
+ end
188
+
@@ -0,0 +1,169 @@
1
+ require 'rubygems'
2
+ require 'spec'
3
+ require File.expand_path(File.join(File.dirname(__FILE__), '..', 'lib', 'http_configuration'))
4
+
5
+ describe "Net::HTTP::Configuration" do
6
+
7
+ it "should set the default configuration options" do
8
+ config = Net::HTTP::Configuration.new(:proxy_host => 'localhost', :proxy_port => 8080, :no_proxy => ['local1', 'local2'])
9
+ config[:proxy_host].should == 'localhost'
10
+ config[:proxy_port].should == 8080
11
+ config[:no_proxy].should == ['local1', 'local2']
12
+ end
13
+
14
+ it "should be able to get the proxy from the environment" do
15
+ ENV.should_receive(:[]).with('HTTP_PROXY').and_return('localhost:80')
16
+ ENV.should_receive(:[]).with('NO_PROXY').and_return('.local1, .local2')
17
+ config = Net::HTTP::Configuration.new(:proxy => :environment)
18
+ config[:proxy_host].should == 'localhost'
19
+ config[:proxy_port].should == 80
20
+ config[:proxy_user].should == nil
21
+ config[:proxy_password].should == nil
22
+ config[:no_proxy].should == ['.local1', '.local2']
23
+ end
24
+
25
+ it "should be able to parse the no_proxy option" do
26
+ Net::HTTP::Configuration.new(:no_proxy => 'host')[:no_proxy].should == ['host']
27
+ Net::HTTP::Configuration.new(:no_proxy => 'host1, host2')[:no_proxy].should == ['host1', 'host2']
28
+ Net::HTTP::Configuration.new(:no_proxy => ['host3', 'host4'])[:no_proxy].should == ['host3', 'host4']
29
+ end
30
+
31
+ it "should be able to parse a proxy with user and password" do
32
+ config = Net::HTTP::Configuration.new(:proxy => 'http://user:password@proxy.local:9000', :no_proxy => '.local1, .local2')
33
+ config[:proxy_host].should == 'proxy.local'
34
+ config[:proxy_port].should == 9000
35
+ config[:proxy_user].should == 'user'
36
+ config[:proxy_password].should == 'password'
37
+ config[:no_proxy].should == ['.local1', '.local2']
38
+ config[:proxy].should == nil
39
+ end
40
+
41
+ it "should be able to clear the proxy" do
42
+ config = Net::HTTP::Configuration.new(:proxy => :none)
43
+ config[:proxy_host].should == nil
44
+ config[:proxy_port].should == nil
45
+ config[:proxy_user].should == nil
46
+ config[:proxy_password].should == nil
47
+ end
48
+
49
+ it "should be able to set a global configuration" do
50
+ Net::HTTP::Configuration.set_global(:proxy_host => 'localhost', :proxy_port => 8080, :read_timeout => 5)
51
+ Net::HTTP::Configuration.global[:proxy_host].should == 'localhost'
52
+ Net::HTTP::Configuration.global[:proxy_port].should == 8080
53
+ Net::HTTP::Configuration.global[:read_timeout].should == 5
54
+ Net::HTTP::Configuration.set_global(nil)
55
+ Net::HTTP::Configuration.global.should == nil
56
+ end
57
+
58
+ it "should be able to apply a configuration to a block" do
59
+ Net::HTTP::Configuration.set_global(nil)
60
+ config = Net::HTTP::Configuration.new(:proxy_host => "proxy#{rand(1000)}", :read_timeout => 30)
61
+ retval = config.apply(:read_timeout => 5) do
62
+ Net::HTTP::Configuration.current[:proxy_host].should == config[:proxy_host]
63
+ Net::HTTP::Configuration.current[:read_timeout].should == 5
64
+ :done
65
+ end
66
+ retval.should == :done
67
+ end
68
+
69
+ it "should be able to determine the current configuration" do
70
+ Net::HTTP::Configuration.set_global(:proxy_host => 'global')
71
+ config1 = Net::HTTP::Configuration.new(:proxy_host => 'config1')
72
+ config2 = Net::HTTP::Configuration.new(:proxy_host => 'config2')
73
+
74
+ Net::HTTP::Configuration.current[:proxy_host].should == 'global'
75
+ config1.apply do
76
+ Net::HTTP::Configuration.current[:proxy_host].should == 'config1'
77
+ config2.apply do
78
+ Net::HTTP::Configuration.current[:proxy_host].should == 'config2'
79
+ end
80
+ Net::HTTP::Configuration.current[:proxy_host].should == 'config1'
81
+ end
82
+ Net::HTTP::Configuration.current[:proxy_host].should == 'global'
83
+
84
+ Net::HTTP::Configuration.set_global(nil)
85
+ Net::HTTP::Configuration.current.should == nil
86
+ end
87
+
88
+ it "should be able to determine if a host does not require a proxy" do
89
+ Net::HTTP::Configuration.no_proxy?('host.local', :no_proxy => ['host.local']).should == true
90
+ Net::HTTP::Configuration.no_proxy?('host.local', :no_proxy => ['not_this_one', '.local']).should == true
91
+ Net::HTTP::Configuration.no_proxy?('HOST.LOCAL', :no_proxy => ['.local']).should == true
92
+ Net::HTTP::Configuration.no_proxy?('host.local', :no_proxy => ['.LOCAL']).should == true
93
+ Net::HTTP::Configuration.no_proxy?('host.local', :no_proxy => ['other.host.local']).should == false
94
+ Net::HTTP::Configuration.no_proxy?('external.host', :no_proxy => ['.local']).should == false
95
+ Net::HTTP::Configuration.no_proxy?('external.host', :no_proxy => nil).should == false
96
+ end
97
+
98
+ end
99
+
100
+ describe "Net::HTTP" do
101
+
102
+ it "should work normally if no configuration has been set" do
103
+ Net::HTTP.should_receive(:new_without_configuration).with('localhost', 80, nil, nil, nil, nil)
104
+ Net::HTTP.new('localhost', 80)
105
+ end
106
+
107
+ it "should use proxy settings if they have been set" do
108
+ config = Net::HTTP::Configuration.new(:proxy_host => 'proxy', :proxy_port => 8080, :proxy_user => 'user', :proxy_password => 'password')
109
+ Net::HTTP.should_receive(:new_without_configuration).with('localhost', 80, 'proxy', 8080, 'user', 'password')
110
+ config.apply do
111
+ Net::HTTP.new('localhost', 80)
112
+ end
113
+ end
114
+
115
+ it "should honor no_proxy hosts" do
116
+ config = Net::HTTP::Configuration.new(:proxy_host => 'proxy', :proxy_port => 8080, :no_proxy => ['localhost'])
117
+ Net::HTTP.should_receive(:new_without_configuration).with('localhost', 80, nil, nil, nil, nil)
118
+ config.apply do
119
+ Net::HTTP.new('localhost', 80)
120
+ end
121
+ end
122
+
123
+ it "should honor proxies explicitly passed" do
124
+ config = Net::HTTP::Configuration.new(:proxy_host => 'proxy', :proxy_port => 8080, :proxy_user => 'user', :proxy_password => 'password')
125
+ Net::HTTP.should_receive(:new_without_configuration).with('localhost', 80, 'other_proxy', 9000, nil, nil)
126
+ config.apply do
127
+ Net::HTTP.new('localhost', 80, 'other_proxy', 9000)
128
+ end
129
+ end
130
+
131
+ it "should set read_timeout from the configuration" do
132
+ config = Net::HTTP::Configuration.new(:read_timeout => 7)
133
+ config.apply do
134
+ http = Net::HTTP.new('localhost', 80)
135
+ http.read_timeout.should == 7
136
+ end
137
+ end
138
+
139
+ it "should set open_timeout from the configuration" do
140
+ config = Net::HTTP::Configuration.new(:open_timeout => 6)
141
+ config.apply do
142
+ http = Net::HTTP.new('localhost', 80)
143
+ http.open_timeout.should == 6
144
+ end
145
+ end
146
+
147
+ end
148
+
149
+ describe "Net::BufferedIO" do
150
+
151
+ it "should timeout without an interrupt" do
152
+ io = Net::BufferedIO.new(StringIO.new)
153
+ lambda do
154
+ io.send(:timeout, 1){sleep(2)}
155
+ end.should raise_error(Net::NetworkTimeoutError)
156
+ end
157
+
158
+ end
159
+
160
+ describe "Net::Protocol" do
161
+
162
+ it "should timeout without an interrupt" do
163
+ http = Net::HTTP.new('localhost')
164
+ lambda do
165
+ http.send(:timeout, 1){sleep(2)}
166
+ end.should raise_error(Net::NetworkTimeoutError)
167
+ end
168
+
169
+ end
metadata ADDED
@@ -0,0 +1,50 @@
1
+ --- !ruby/object:Gem::Specification
2
+ rubygems_version: 0.9.0
3
+ specification_version: 1
4
+ name: http_configuration
5
+ version: !ruby/object:Gem::Version
6
+ version: 1.0.0
7
+ date: 2007-10-24 00:00:00 -05:00
8
+ summary: Provide configuration options for Net::HTTP
9
+ require_paths:
10
+ - lib
11
+ email:
12
+ homepage:
13
+ rubyforge_project:
14
+ description:
15
+ autorequire: init.rb
16
+ default_executable:
17
+ bindir: bin
18
+ has_rdoc: true
19
+ required_ruby_version: !ruby/object:Gem::Version::Requirement
20
+ requirements:
21
+ - - ">"
22
+ - !ruby/object:Gem::Version
23
+ version: 0.0.0
24
+ version:
25
+ platform: ruby
26
+ signing_key:
27
+ cert_chain:
28
+ post_install_message:
29
+ authors:
30
+ - Brian Durand
31
+ files:
32
+ - lib/http_configuration.rb
33
+ - init.rb
34
+ - MIT-LICENSE
35
+ - Rakefile
36
+ - README
37
+ test_files:
38
+ - test/http_configuration_test.rb
39
+ rdoc_options: []
40
+
41
+ extra_rdoc_files:
42
+ - README
43
+ executables: []
44
+
45
+ extensions: []
46
+
47
+ requirements: []
48
+
49
+ dependencies: []
50
+