bpardee-net-http-persistent 1.0.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
@@ -0,0 +1,7 @@
1
+ # -*- ruby -*-
2
+
3
+ require 'autotest/restart'
4
+
5
+ Autotest.add_hook :initialize do |at|
6
+ at.testlib = 'minitest/unit'
7
+ end
@@ -0,0 +1,6 @@
1
+ === 1.0.0 / 2010-09-08
2
+
3
+ * Major Enhancements
4
+ * Fork of drbrain/net-http-persistent 1.3 prerelease
5
+ * Uses a connection pool instead of connection/thread
6
+ * force_retry option added to retry on POST as well as idempotent requests
@@ -0,0 +1,8 @@
1
+ .autotest
2
+ History.txt
3
+ Manifest.txt
4
+ README.txt
5
+ Rakefile
6
+ lib/net/http/faster.rb
7
+ lib/net/http/persistent.rb
8
+ test/test_net_http_persistent.rb
@@ -0,0 +1,112 @@
1
+ = net_http_persistent
2
+
3
+ * http://seattlerb.rubyforge.org/net-http-persistent
4
+
5
+ == DESCRIPTION:
6
+
7
+ Persistent connections using Net::HTTP plus a speed fix for 1.8. It's
8
+ thread-safe too!
9
+
10
+ == FORK DESCRIPTION
11
+
12
+ This is an experimental branch that implements a connection pool of
13
+ Net::HTTP objects instead of a connection/thread. C/T is fine if
14
+ you're only using your http threads to make connections but if you
15
+ use them in child threads then I suspect you will have a thread memory
16
+ leak. Also, I want to see if I get less connection resets if the
17
+ most recently used connection is always returned.
18
+
19
+ Also added a :force_retry option that if set to true will retry POST
20
+ requests as well as idempotent requests.
21
+
22
+ This branch is currently incompatible with the master branch in the
23
+ following ways:
24
+
25
+ * It doesn't allow you to recreate the Net::HTTP::Persistent object
26
+ on the fly. This is possible in the master version since all the
27
+ data is kept in thread local storage. For this version, you should
28
+ probably create a class instance of the object and use that in your
29
+ instance methods.
30
+
31
+ * It uses a hash in the initialize method. This was easier for me
32
+ as I use a HashWithIndifferentAccess created from a YAML file to
33
+ define my options. This should probably be modified to check the
34
+ arguments to achieve backwards compatibility.
35
+
36
+ * The method shutdown is unimplemented as I wasn't sure how I should
37
+ implement it and I don't need it as I do a graceful shutdown from
38
+ nginx to finish up my connections.
39
+
40
+ For connection issues, I completely recreate a new Net::HTTP instance.
41
+ I was running into an issue which I suspect is a JRuby bug where an
42
+ SSL connection that times out would leave the ssl context in a frozen
43
+ state which would then make that connection unusable so each time that
44
+ thread handled a connection a 500 error with the exception "TypeError:
45
+ can't modify frozen". I think Joseph West's fork resolves this issue
46
+ but I'm paranoid so I recreate the object.
47
+
48
+ Compatibility with the master version could probably be achieved by
49
+ creating a Strategy wrapper class for GenePool and a separate strategy
50
+ class with the connection/thread implementation.
51
+
52
+ == FEATURES/PROBLEMS:
53
+
54
+ * Supports SSL
55
+ * Thread-safe
56
+ * Pure ruby
57
+ * Timeout-less speed boost for 1.8 (by Aaron Patterson)
58
+
59
+ == INSTALL:
60
+
61
+ gem install bpardee-net-http-persistent
62
+
63
+ == EXAMPLE USAGE:
64
+
65
+ class MyHttpClient
66
+ @@http ||= Net::HTTP::Persistent.new(
67
+ :name => 'MyHttpClient',
68
+ :logger => Rails.logger,
69
+ :pool_size => 10,
70
+ :warn_timeout => 0.25,
71
+ :force_retry => true
72
+ )
73
+
74
+ def send_get_message
75
+ uri = URI.parse('https://www.example.com/echo/foo')
76
+ response = @@http.request(uri)
77
+ ... Handle response as you would a normal Net::HTTPResponse ...
78
+ end
79
+
80
+ def send_post_message
81
+ uri = URI.parse('https://www.example.com/echo/foo')
82
+ request = Net::HTTP::Post.new(uri.request_uri)
83
+ ... Modify request as needed ...
84
+ response = @@http.request(uri, request)
85
+ ... Handle response as you would a normal Net::HTTPResponse ...
86
+ end
87
+ end
88
+
89
+ == LICENSE:
90
+
91
+ (The MIT License)
92
+
93
+ Copyright (c) 2010 Eric Hodel, Aaron Patterson
94
+
95
+ Permission is hereby granted, free of charge, to any person obtaining
96
+ a copy of this software and associated documentation files (the
97
+ 'Software'), to deal in the Software without restriction, including
98
+ without limitation the rights to use, copy, modify, merge, publish,
99
+ distribute, sublicense, and/or sell copies of the Software, and to
100
+ permit persons to whom the Software is furnished to do so, subject to
101
+ the following conditions:
102
+
103
+ The above copyright notice and this permission notice shall be
104
+ included in all copies or substantial portions of the Software.
105
+
106
+ THE SOFTWARE IS PROVIDED 'AS IS', WITHOUT WARRANTY OF ANY KIND,
107
+ EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
108
+ MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.
109
+ IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY
110
+ CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT,
111
+ TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE
112
+ SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
@@ -0,0 +1,14 @@
1
+ # -*- ruby -*-
2
+
3
+ require 'rubygems'
4
+ require 'hoe'
5
+
6
+ Hoe.plugin :git
7
+ Hoe.plugin :minitest
8
+
9
+ Hoe.spec 'bpardee-net-http-persistent' do |p|
10
+ developer 'Eric Hodel', 'drbrain@segment7.net'
11
+ extra_deps << ['gene_pool', '>= 1.0']
12
+ end
13
+
14
+ # vim: syntax=Ruby
@@ -0,0 +1,27 @@
1
+ require 'net/protocol'
2
+
3
+ ##
4
+ # Aaron Patterson's monkeypatch (accepted into 1.9.1) to fix Net::HTTP's speed
5
+ # problems.
6
+ #
7
+ # http://gist.github.com/251244
8
+
9
+ class Net::BufferedIO #:nodoc:
10
+ alias :old_rbuf_fill :rbuf_fill
11
+
12
+ def rbuf_fill
13
+ if @io.respond_to? :read_nonblock then
14
+ begin
15
+ @rbuf << @io.read_nonblock(65536)
16
+ rescue Errno::EWOULDBLOCK => e
17
+ retry if IO.select [@io], nil, nil, @read_timeout
18
+ raise Timeout::Error, e.message
19
+ end
20
+ else # SSL sockets do not have read_nonblock
21
+ timeout @read_timeout do
22
+ @rbuf << @io.sysread(65536)
23
+ end
24
+ end
25
+ end
26
+ end if RUBY_VERSION < '1.9'
27
+
@@ -0,0 +1,444 @@
1
+ require 'net/http'
2
+ require 'net/http/faster'
3
+ require 'uri'
4
+ require 'gene_pool'
5
+
6
+ ##
7
+ # Persistent connections for Net::HTTP
8
+ #
9
+ # Net::HTTP::Persistent maintains persistent connections across all the
10
+ # servers you wish to talk to. For each host:port you communicate with a
11
+ # single persistent connection is created.
12
+ #
13
+ # Multiple Net::HTTP::Persistent objects will share the same set of
14
+ # connections which will be checked out of a pool.
15
+ #
16
+ # You can shut down the HTTP connections when done by calling #shutdown. You
17
+ # should name your Net::HTTP::Persistent object if you intend to call this
18
+ # method.
19
+ #
20
+ # Example:
21
+ #
22
+ # uri = URI.parse 'http://example.com/awesome/web/service'
23
+ # http = Net::HTTP::Persistent.new
24
+ # stuff = http.request uri # performs a GET
25
+ #
26
+ # # perform a POST
27
+ # post_uri = uri + 'create'
28
+ # post = Net::HTTP::Post.new post_uri.path
29
+ # post.set_form_data 'some' => 'cool data'
30
+ # http.request post_uri, post # URI is always required
31
+
32
+ class Net::HTTP::Persistent
33
+
34
+ ##
35
+ # The version of Net::HTTP::Persistent use are using
36
+
37
+ VERSION = '1.0.0'
38
+
39
+ ##
40
+ # Error class for errors raised by Net::HTTP::Persistent. Various
41
+ # SystemCallErrors are re-raised with a human-readable message under this
42
+ # class.
43
+
44
+ class Error < StandardError; end
45
+
46
+ ##
47
+ # An SSL certificate authority. Setting this will set verify_mode to
48
+ # VERIFY_PEER.
49
+
50
+ attr_accessor :ca_file
51
+
52
+ ##
53
+ # This client's OpenSSL::X509::Certificate
54
+
55
+ attr_accessor :certificate
56
+
57
+ ##
58
+ # Sends debug_output to this IO via Net::HTTP#set_debug_output.
59
+ #
60
+ # Never use this method in production code, it causes a serious security
61
+ # hole.
62
+
63
+ attr_accessor :debug_output
64
+
65
+ ##
66
+ # Retry even for non-idempotent (POST) requests.
67
+
68
+ attr_accessor :force_retry
69
+
70
+ ##
71
+ # Headers that are added to every request
72
+
73
+ attr_accessor :headers
74
+
75
+ ##
76
+ # Maps host:port to an HTTP version. This allows us to enable version
77
+ # specific features.
78
+
79
+ attr_reader :http_versions
80
+
81
+ ##
82
+ # The value sent in the Keep-Alive header. Defaults to 30. Not needed for
83
+ # HTTP/1.1 servers.
84
+ #
85
+ # This may not work correctly for HTTP/1.0 servers
86
+ #
87
+ # This method may be removed in a future version as RFC 2616 does not
88
+ # require this header.
89
+
90
+ attr_accessor :keep_alive
91
+
92
+ ##
93
+ # Logger for message logging.
94
+
95
+ attr_accessor :logger
96
+
97
+ ##
98
+ # A name for this connection. Allows you to keep your connections apart
99
+ # from everybody else's.
100
+
101
+ attr_reader :name
102
+
103
+ ##
104
+ # Seconds to wait until a connection is opened. See Net::HTTP#open_timeout
105
+
106
+ attr_accessor :open_timeout
107
+
108
+ ##
109
+ # The maximum size of the connection pool
110
+
111
+ attr_reader :pool_size
112
+
113
+ ##
114
+ # This client's SSL private key
115
+
116
+ attr_accessor :private_key
117
+
118
+ ##
119
+ # The URL through which requests will be proxied
120
+
121
+ attr_reader :proxy_uri
122
+
123
+ ##
124
+ # Seconds to wait until reading one block. See Net::HTTP#read_timeout
125
+
126
+ attr_accessor :read_timeout
127
+
128
+ ##
129
+ # SSL verification callback. Used when ca_file is set.
130
+
131
+ attr_accessor :verify_callback
132
+
133
+ ##
134
+ # HTTPS verify mode. Defaults to OpenSSL::SSL::VERIFY_NONE which ignores
135
+ # certificate problems.
136
+ #
137
+ # You can use +verify_mode+ to override any default values.
138
+
139
+ attr_accessor :verify_mode
140
+
141
+ ##
142
+ # The threshold in seconds for checking out a connection at which a warning
143
+ # will be logged via the logger
144
+
145
+ attr_accessor :warn_timeout
146
+
147
+ ##
148
+ # Creates a new Net::HTTP::Persistent.
149
+ #
150
+ # Set +name+ to keep your connections apart from everybody else's. Not
151
+ # required currently, but highly recommended. Your library name should be
152
+ # good enough. This parameter will be required in a future version.
153
+ #
154
+ # +proxy+ may be set to a URI::HTTP or :ENV to pick up proxy options from
155
+ # the environment. See proxy_from_env for details.
156
+ #
157
+ # In order to use a URI for the proxy you'll need to do some extra work
158
+ # beyond URI.parse:
159
+ #
160
+ # proxy = URI.parse 'http://proxy.example'
161
+ # proxy.user = 'AzureDiamond'
162
+ # proxy.password = 'hunter2'
163
+
164
+ def initialize(options={})
165
+ @name = options[:name] || 'Net::HTTP::Persistent'
166
+ proxy = options[:proxy]
167
+
168
+ @proxy_uri = case proxy
169
+ when :ENV then proxy_from_env
170
+ when URI::HTTP then proxy
171
+ when nil then # ignore
172
+ else raise ArgumentError, 'proxy must be :ENV or a URI::HTTP'
173
+ end
174
+
175
+ if @proxy_uri then
176
+ @proxy_args = [
177
+ @proxy_uri.host,
178
+ @proxy_uri.port,
179
+ @proxy_uri.user,
180
+ @proxy_uri.password,
181
+ ]
182
+
183
+ @proxy_connection_id = [nil, *@proxy_args].join ':'
184
+ end
185
+
186
+ @ca_file = options[:ca_file]
187
+ @certificate = options[:certificate]
188
+ @debug_output = options[:debug_output]
189
+ @force_retry = options[:force_retry]
190
+ @headers = options[:header] || {}
191
+ @http_versions = {}
192
+ @keep_alive = options[:keep_alive] || 30
193
+ @logger = options[:logger]
194
+ @open_timeout = options[:open_timeout]
195
+ @pool_size = options[:pool_size] || 1
196
+ @private_key = options[:private_key]
197
+ @read_timeout = options[:read_timeout]
198
+ @verify_callback = options[:verify_callback]
199
+ @verify_mode = options[:verify_mode]
200
+ @warn_timeout = options[:warn_timeout] || 0.5
201
+
202
+ # Hash containing connection pools based on key of host:port
203
+ @pool_hash = {}
204
+
205
+ # Hash containing the request counts based on the connection
206
+ @count_hash = Hash.new(0)
207
+ end
208
+
209
+ ##
210
+ # Makes a request on +uri+. If +req+ is nil a Net::HTTP::Get is performed
211
+ # against +uri+.
212
+ #
213
+ # If a block is passed #request behaves like Net::HTTP#request (the body of
214
+ # the response will not have been read).
215
+ #
216
+ # +req+ must be a Net::HTTPRequest subclass (see Net::HTTP for a list).
217
+ #
218
+ # If there is an error and the request is idempontent according to RFC 2616
219
+ # it will be retried automatically.
220
+
221
+ def request uri, req = nil, &block
222
+ retried = false
223
+ bad_response = false
224
+
225
+ req = Net::HTTP::Get.new uri.request_uri unless req
226
+
227
+ headers.each do |pair|
228
+ req.add_field(*pair)
229
+ end
230
+
231
+ req.add_field 'Connection', 'keep-alive'
232
+ req.add_field 'Keep-Alive', @keep_alive
233
+
234
+ pool = pool_for uri
235
+ pool.with_connection do |connection|
236
+ begin
237
+ count = @count_hash[connection.object_id] += 1
238
+ response = connection.request req, &block
239
+ @http_versions["#{uri.host}:#{uri.port}"] ||= response.http_version
240
+ return response
241
+
242
+ rescue Timeout::Error => e
243
+ due_to = "(due to #{e.message} - #{e.class})"
244
+ message = error_message connection
245
+ @logger.info "#{name}: Removing connection #{due_to} #{message}" if @logger
246
+ remove pool, connection
247
+ raise
248
+
249
+ rescue Net::HTTPBadResponse => e
250
+ message = error_message connection
251
+ if bad_response or not (idempotent? req or @force_retry)
252
+ @logger.info "#{name}: Removing connection because of too many bad responses #{message}" if @logger
253
+ remove pool, connection
254
+ raise Error, "too many bad responses #{message}"
255
+ else
256
+ bad_response = true
257
+ @logger.info "#{name}: Renewing connection because of bad response #{message}" if @logger
258
+ connection = renew pool, connection
259
+ retry
260
+ end
261
+
262
+ rescue IOError, EOFError, Errno::ECONNABORTED, Errno::ECONNRESET, Errno::EPIPE => e
263
+ due_to = "(due to #{e.message} - #{e.class})"
264
+ message = error_message connection
265
+ if retried or not (idempotent? req or @force_retry)
266
+ @logger.info "#{name}: Removing connection #{due_to} #{message}" if @logger
267
+ remove pool, connection
268
+ raise Error, "too many connection resets #{due_to} #{message}"
269
+ else
270
+ retried = true
271
+ @logger.info "#{name}: Renewing connection #{due_to} #{message}" if @logger
272
+ connection = renew pool, connection
273
+ retry
274
+ end
275
+ end
276
+ end
277
+ end
278
+
279
+ ##
280
+ # Returns the HTTP protocol version for +uri+
281
+
282
+ def http_version uri
283
+ @http_versions["#{uri.host}:#{uri.port}"]
284
+ end
285
+
286
+ ##
287
+ # Shuts down all connections.
288
+
289
+ def shutdown
290
+ raise 'Shutdown not implemented'
291
+ # TBD - need to think about this one
292
+ @count_hash = nil
293
+ end
294
+
295
+ #######
296
+ private
297
+ #######
298
+
299
+ ##
300
+ # Returns an error message containing the number of requests performed on
301
+ # this connection
302
+
303
+ def error_message connection
304
+ requests = @count_hash[connection.object_id] || 0
305
+ "after #{requests} requests on #{connection.object_id}"
306
+ end
307
+
308
+ ##
309
+ # URI::escape wrapper
310
+
311
+ def escape str
312
+ URI.escape str if str
313
+ end
314
+
315
+ ##
316
+ # Finishes the Net::HTTP +connection+
317
+
318
+ def finish connection
319
+ @count_hash.delete(connection.object_id)
320
+ connection.finish
321
+ rescue IOError
322
+ end
323
+
324
+ ##
325
+ # Is +req+ idempotent according to RFC 2616?
326
+
327
+ def idempotent? req
328
+ case req
329
+ when Net::HTTP::Delete, Net::HTTP::Get, Net::HTTP::Head,
330
+ Net::HTTP::Options, Net::HTTP::Put, Net::HTTP::Trace then
331
+ true
332
+ end
333
+ end
334
+
335
+ ##
336
+ # Adds "http://" to the String +uri+ if it is missing.
337
+
338
+ def normalize_uri uri
339
+ (uri =~ /^https?:/) ? uri : "http://#{uri}"
340
+ end
341
+
342
+ ##
343
+ # Get the connection pool associated with this +uri+
344
+ def pool_for uri
345
+ net_http_args = [uri.host, uri.port]
346
+ connection_id = net_http_args.join ':'
347
+
348
+ if @proxy_uri then
349
+ connection_id << @proxy_connection_id
350
+ net_http_args.concat @proxy_args
351
+ end
352
+ @pool_hash[connection_id] ||= GenePool.new(:name => name + '-' + connection_id,
353
+ :pool_size => @pool_size,
354
+ :warn_timeout => @warn_timeout,
355
+ :logger => @logger) do
356
+ begin
357
+ connection = Net::HTTP.new(*net_http_args)
358
+ connection.set_debug_output @debug_output if @debug_output
359
+ connection.open_timeout = @open_timeout if @open_timeout
360
+ connection.read_timeout = @read_timeout if @read_timeout
361
+
362
+ ssl connection if uri.scheme == 'https'
363
+
364
+ connection.start
365
+ connection
366
+ rescue Errno::ECONNREFUSED
367
+ raise Error, "connection refused: #{connection.address}:#{connection.port}"
368
+ rescue Errno::EHOSTDOWN
369
+ raise Error, "host down: #{connection.address}:#{connection.port}"
370
+ end
371
+ end
372
+ end
373
+
374
+ ##
375
+ # Creates a URI for an HTTP proxy server from ENV variables.
376
+ #
377
+ # If +HTTP_PROXY+ is set a proxy will be returned.
378
+ #
379
+ # If +HTTP_PROXY_USER+ or +HTTP_PROXY_PASS+ are set the URI is given the
380
+ # indicated user and password unless HTTP_PROXY contains either of these in
381
+ # the URI.
382
+ #
383
+ # For Windows users lowercase ENV variables are preferred over uppercase ENV
384
+ # variables.
385
+
386
+ def proxy_from_env
387
+ env_proxy = ENV['http_proxy'] || ENV['HTTP_PROXY']
388
+
389
+ return nil if env_proxy.nil? or env_proxy.empty?
390
+
391
+ uri = URI.parse(normalize_uri(env_proxy))
392
+
393
+ unless uri.user or uri.password then
394
+ uri.user = escape ENV['http_proxy_user'] || ENV['HTTP_PROXY_USER']
395
+ uri.password = escape ENV['http_proxy_pass'] || ENV['HTTP_PROXY_PASS']
396
+ end
397
+
398
+ uri
399
+ end
400
+
401
+ ##
402
+ # Finishes then removes the Net::HTTP +connection+
403
+
404
+ def remove pool, connection
405
+ finish connection
406
+ pool.remove(connection)
407
+ end
408
+
409
+ ##
410
+ # Finishes then renews the Net::HTTP +connection+. It may be unnecessary
411
+ # to completely recreate the connection but connections that get timed out
412
+ # in JRuby leave the ssl context in a frozen object state.
413
+
414
+ def renew pool, connection
415
+ finish connection
416
+ connection = pool.renew(connection)
417
+ end
418
+
419
+ ##
420
+ # Enables SSL on +connection+
421
+
422
+ def ssl connection
423
+ require 'net/https'
424
+ connection.use_ssl = true
425
+
426
+ # suppress warning but allow override
427
+ connection.verify_mode = OpenSSL::SSL::VERIFY_NONE unless @verify_mode
428
+
429
+ if @ca_file then
430
+ connection.ca_file = @ca_file
431
+ connection.verify_mode = OpenSSL::SSL::VERIFY_PEER
432
+ connection.verify_callback = @verify_callback if @verify_callback
433
+ end
434
+
435
+ if @certificate and @private_key then
436
+ connection.cert = @certificate
437
+ connection.key = @private_key
438
+ end
439
+
440
+ connection.verify_mode = @verify_mode if @verify_mode
441
+ end
442
+
443
+ end
444
+