bterlson-httpclient 2.1.4

Sign up to get free protection for your applications and to get access to all the features.
data/README.txt ADDED
@@ -0,0 +1,368 @@
1
+ httpclient - HTTP accessing library.
2
+ Copyright (C) 2000-2009 NAKAMURA, Hiroshi <nahi@ruby-lang.org>.
3
+
4
+ 'httpclient' gives something like the functionality of libwww-perl (LWP) in
5
+ Ruby. 'httpclient' formerly known as 'http-access2'.
6
+
7
+ See HTTPClient for documentation.
8
+
9
+
10
+ == Features
11
+
12
+ * methods like GET/HEAD/POST/* via HTTP/1.1.
13
+ * HTTPS(SSL), Cookies, proxy, authentication(Digest, NTLM, Basic), etc.
14
+ * asynchronous HTTP request, streaming HTTP request.
15
+
16
+ * by contrast with net/http in standard distribution;
17
+ * Cookies support
18
+ * MT-safe
19
+ * streaming POST (POST with File/IO)
20
+ * Digest auth
21
+ * Negotiate/NTLM auth for WWW-Authenticate (requires net/htlm module)
22
+ * NTLM auth for Proxy-Authenticate (requires win32/sspi module)
23
+ * extensible with filter interface
24
+ * you don't have to care HTTP/1.1 persistent connection
25
+ (httpclient cares instead of you)
26
+
27
+ * Not supported now
28
+ * Cache
29
+ * Rather advanced HTTP/1.1 usage such as Range, deflate, etc.
30
+ (of course you can set it in header by yourself)
31
+
32
+
33
+ == Author
34
+
35
+ Name:: NAKAMURA, Hiroshi
36
+ E-mail:: nahi@ruby-lang.org
37
+ Project web site:: http://dev.ctor.org/httpclient/
38
+
39
+
40
+ == License
41
+
42
+ This program is copyrighted free software by NAKAMURA, Hiroshi. You can
43
+ redistribute it and/or modify it under the same terms of Ruby's license;
44
+ either the dual license version in 2003, or any later version.
45
+
46
+ httpclient/session.rb is based on http-access.rb in http-access/0.0.4.
47
+ Some part of code in http-access.rb was recycled in http-access2.rb.
48
+ Those part is copyrighted by Maehashi-san who made and distributed
49
+ http-access/0.0.4. Many thanks to Maehashi-san.
50
+
51
+
52
+ == Install
53
+
54
+ $ ruby install.rb
55
+
56
+ It will install lib/* to your site_ruby directory such as
57
+ /usr/local/lib/ruby/site_ruby/1.8/.
58
+
59
+
60
+ == Uninstall
61
+
62
+ Delete installed files from your site_ruby directory.
63
+
64
+
65
+ == Usage
66
+
67
+ See HTTPClient for documentation.
68
+ You can also check sample/howto.rb how to use APIs.
69
+
70
+
71
+ == Download
72
+
73
+ * Stable: http://dev.ctor.org/download/httpclient-2.1.4.tar.gz (tar + gzip)
74
+ * Stable: http://dev.ctor.org/download/httpclient-2.1.4.zip (ZIP)
75
+
76
+ * Older versions: http://dev.ctor.org/download/archive/
77
+
78
+ * Gem repository for stable version
79
+ * (at default remove source at rubyforge.org)
80
+ * Gem repository for development version
81
+ * http://dev.ctor.org/download/
82
+
83
+ * svn: http://dev.ctor.org/svn/http-access2/trunk/
84
+
85
+ === Gem
86
+
87
+ You can install httpclient with rubygems.
88
+
89
+ % gem install httpclient --source http://dev.ctor.org/download/
90
+
91
+
92
+ == Bug report or Feature request
93
+
94
+ Please file a ticket at the project web site.
95
+
96
+ 1. find a similar ticket from http://dev.ctor.org/http-access2/search?ticket=on
97
+ 2. login as an anonymous user from http://dev.ctor.org/http-access2/login
98
+ 3. create a new ticket from http://dev.ctor.org/http-access2/newticket
99
+
100
+ Thanks in advance.
101
+
102
+
103
+ == Changes
104
+
105
+ Feb 13, 2009 - version 2.1.4
106
+
107
+ * Bug fixes
108
+ * When we hit some site through http-proxy we get a response without
109
+ Content-Length header. httpclient/2.1.3 drops response body for such
110
+ case. fixed. (#199)
111
+ * Avoid duplicated 'Date' header in request. Fixed. (#194)
112
+ * Avoid to add port number to 'Host' header. Some servers like GFE/1.3
113
+ dislike it. Thanks to anonymous user for investigating the behavior.
114
+ (#195)
115
+ * httpclient/2.1.3 does not work when you fork a process after requiring
116
+ httpclient module (Passenger). Thanks to Akira Yamada for tracing down
117
+ this bug. (#197)
118
+ * httpclient/2.1.3 cannot handle Cookie header with 'expires=' and
119
+ 'expires=""'. Empty String for Time.parse returns Time.now unlike
120
+ ParseDate.parsedate. Thanks to Mark for the patch. (#200)
121
+
122
+ Jan 8, 2009 - version 2.1.3.1
123
+
124
+ * Security fix introduced at 2.1.3.
125
+ * get_content/post_content of httpclient/2.1.3 may send secure cookies
126
+ for a https site to non-secure (non-https) site when the https site
127
+ redirects the request to a non-https site. httpclient/2.1.3 caches
128
+ request object and reuses it for redirection. It should not be cached
129
+ and recreated for each time as httpclient <= 2.1.2 and http-access2.
130
+ * I realized this bug when I was reading open-uri story on
131
+ [ruby-core:21205]. Ruby users should use open-uri rather than using
132
+ net/http directly wherever possible.
133
+
134
+ Dec 29, 2008 - version 2.1.3
135
+
136
+ * Features
137
+ * Proxy Authentication for SSL.
138
+ * Performance improvements.
139
+ * Full RDoc. Please tell me any English problem. Thanks in advance.
140
+ * Do multipart file upload when a given body includes a File. You don't
141
+ need to set 'Content-Type' and boundary String any more.
142
+ * Added propfind and proppatch methods.
143
+
144
+ * Changes
145
+ * Avoid unnecessary memory consuming for get_content/post_content with
146
+ block. get_content returns nil when you call it with a block.
147
+ * post_content with IO did not work when redirect/auth cycle is required.
148
+ (CAUTION: post_content now correctly follows redirection and posts the
149
+ given content)
150
+ * Exception handling cleanups.
151
+ * Raises HTTPClient::ConfigurationError? for environment problem.
152
+ (trying to do SSL without openssl installed for example)
153
+ * Raises HTTPClient::BadResponse? for HTTP response problem. You can
154
+ get the response HTTPMessage returned via $!.res.
155
+ * Raises SocketError? for connection problem (as same as before).
156
+
157
+ * Bug fixes
158
+ * Avoid unnecessary negotiation cycle for Negotiate(NTLM) authentication.
159
+ Thanks Rishav for great support for debugging Negotiate authentication.
160
+ * get_content/post_content with block yielded unexpected message body
161
+ during redirect/auth cycle.
162
+ * Relative URI redirection should be allowed from 2.1.2 but it did not
163
+ work... fixed.
164
+ * Avoid unnecessary timeout waiting when no message body returned such as
165
+ '204 No Content' for DAV.
166
+ * Avoid blocking on socket closing when the socket is already closed by
167
+ foreign host and the client runs under MT-condition.
168
+
169
+ Sep 22, 2007 - version 2.1.2
170
+
171
+ * HTTP
172
+ * implemented Negotiate authentication with a support from exterior
173
+ modules. 'rubyntlm' module is required for Negotiate auth with IIS.
174
+ 'win32/sspi' module is required for Negotiate auth with ISA.
175
+ * a workaround for Ubuntu + SonicWALL timeout problem. try to send HTTP
176
+ request in one chunk.
177
+
178
+ * SSL
179
+ * create new self-signing dist-cert which has serial number 0x01 and
180
+ embed it in httpclient.rb.
181
+ * update cacert.p7s. certificates are imported from cacerts in JRE 6
182
+ Update 2. 1 expired CA certificate
183
+ 'C=US, O=GTE Corporation, CN=GTE CyberTrust Root' is removed.
184
+
185
+ * Bug fix
186
+ * [BUG] SSL + debug_dev didn't work under version 2.1.1.
187
+ * [BUG] Reason-Phrase of HTTP response status line can be empty according
188
+ * to RFC2616.
189
+
190
+ Aug 28, 2007 - version 2.1.1
191
+
192
+ * bug fix
193
+ * domain_match should be case insensitive. thanks to Brian for the patch.
194
+ * before calling SSLSocket#post_connection_check, check if
195
+ RUBY_VERSION > "1.8.4" for CN based wildcard certificate. when
196
+ RUBY_VERSION <= "1.8.4", it fallbacks to the post_connection_check
197
+ method in HTTPClient so httpclient should run on 1.8.4 fine as before.
198
+
199
+ * misc
200
+ * added HTTPClient#test_loopback_http_response which accepts test
201
+ loopback response which contains HTTP header.
202
+
203
+ Jul 14, 2007 - version 2.1.0
204
+
205
+ * program/project renamed from 'http-access2' to 'httpclient'.
206
+ there's compatibility layer included so existing programs for
207
+ http-access2 which uses HTTPAccess2::Client should work with
208
+ httpclient/2.1.0 correctly.
209
+
210
+ * misc
211
+ * install.rb did not install cacerts.p7s. Thanks to knu.
212
+ * now HTTPClient loads http_proxy/HTTP_PROXY and no_proxy/NO_PROXY
213
+ environment variable at initialization time. bear in mind that it
214
+ doesn't load http_proxy/HTTP_PROXY when a library is considered to be
215
+ running under CGI environment (checked by ENVREQUEST_METHOD existence.
216
+ cgi_http_proxy/CGI_HTTP_PROXY is loaded instead.
217
+
218
+ Jul 4, 2007 - version 2.0.9
219
+
220
+ * bug fix
221
+ * fix the BasicAuth regression problem in 2.0.8. A server may return
222
+ "BASIC" as an authenticate scheme label instead of "Basic". It must be
223
+ treated as a case-insensitive token according to RFC2617 section 1.2.
224
+ Thanks to mwedeme for contributing the patch. (#159)
225
+
226
+ Jun 30, 2007 - version 2.0.8
227
+
228
+ * HTTP
229
+ * added request/response filter interface and implemented DigestAuth
230
+ based on the filter interface. DigestAuth calc engine is based on
231
+ http://tools.assembla.com/breakout/wiki/DigestForSoap
232
+ Thanks to sromano. (#155)
233
+ * re-implemented BasicAuth based on the filter interface. send BasicAuth
234
+ header only if it's needed. (#31)
235
+ * handle a response which has 2XX status code as a successfull response
236
+ while retry check. applied the patch from Micah Wedemeyer.
237
+ Thanks! (#158)
238
+
239
+ * Connection
240
+ * show more friendly error message for unconnectable URL. (#156)
241
+
242
+ * bug fixes
243
+ * to avoid MIME format incompatibility, add empty epilogue chunk
244
+ explicitly. Thanks to the anonymous user who reported #154 (#154)
245
+ * rescue EPIPE for keep-alive reconnecting. Thanks to anonymous user
246
+ who posted a patch at #124. (#124)
247
+
248
+ May 13, 2007 - version 2.0.7
249
+
250
+ * HTTP
251
+ * added proxyauth support. (#6)
252
+ * let developer allow to rescue a redirect with relative URI. (#28)
253
+ * changed last-chunk condition statement to allow "0000\r\n" marker from
254
+ WebLogic Server 7.0 SP5 instead of "0\r\n". (#30)
255
+ * fixed multipart form submit. (#29, #116)
256
+ * use http_date format as a date in a request header. (#35)
257
+ * avoid duplicated Date header when running under mod_ruby. (#127)
258
+ * reason phrase in Message#reason contains \r. (#122)
259
+ * trim "\n"s in base64 encoded BasicAuth value for interoperability.
260
+ (#149)
261
+ * let retry_connect return a Message not a content. (#119)
262
+ * rescue SocketError and dump a message when a wrong address given. (#152)
263
+
264
+ * HTTP-Cookies
265
+ * changed "domain" parameter matching condition statement to allow
266
+ followings; (#24, #32, #118, #147)
267
+ * [host, domain] = [rubyforge.com, .rubyforge.com]
268
+ * [host, domain] = [reddit.com, reddit.com]
269
+
270
+ * SSL
271
+ * bundles CA certificates as trust anchors.
272
+ * allow user to get peer_cert. (#117, #123)
273
+ * added wildcard certificate support. (#151)
274
+ * SSL + HTTP keep-alive + long wait causes uncaught exception. fixed.
275
+ (#120)
276
+
277
+ * Connection
278
+ * fixed a loop condition bug that caused intermittent empty response.
279
+ (#150, #26, #125)
280
+
281
+ September 16, 2005 - version 2.0.6
282
+
283
+ * HTTP
284
+ * allows redirects from a "POST" request. imported a patch from sveit.
285
+ Thanks! (#7)
286
+ * add 'content-type: application/application/x-www-form-urlencoded' when
287
+ a request contains message-body. (#11)
288
+ * HTTP/0.9 support. (#15)
289
+ * allows submitting multipart forms. imported a patch from sveit.
290
+ Thanks! (#7)
291
+
292
+ * HTTP-Cookies
293
+ * avoid NameError when a cookie value is nil. (#10)
294
+ * added netscape_rule property to CookieManager (false by default). You
295
+ can turn on the domain attribute test of Netscape rule with the
296
+ property. cf. http://wp.netscape.com/newsref/std/cookie_spec.html
297
+ * added HTTPClient#cookie_manager property for accessing its properties.
298
+ (#13)
299
+ * added save_all_cookies method to save unused and discarded cookies as
300
+ well. The patch is from Christian Lademann. Thanks! (#21)
301
+ * allow to set cookie_manager. raise an error when set_cookie_store
302
+ called and cookie_store has already been set. (#20)
303
+
304
+ * SSL
305
+ * allows SSL connection debugging when debug_dev != nil. (#14)
306
+ * skip post_connection_check when
307
+ verify_mode == OpenSSL::SSL::VERIFY_NONE. Thanks to kdraper. (#12)
308
+ * post_connection_check: support a certificate with a wildcard in the
309
+ hostname. (#18)
310
+ * avoid NameError when no peer_cert and VERIFY_FAIL_IF_NO_PEER_CERT
311
+ given. Thanks to Christian Lademann.
312
+
313
+ * Connection
314
+ * insert a connecting host and port to an exception message when
315
+ connecting failed. (#5)
316
+ * added socket_sync property to HTTPClient(HTTPAccess2::Client) that
317
+ controls socket's sync property. the default value is true. CAUTION:
318
+ if your ruby is older than 2005-09-06 and you want to use SSL
319
+ connection, do not set socket_sync = false to avoid a blocking bug of
320
+ openssl/buffering.rb.
321
+
322
+ December 24, 2004 - version 2.0.5
323
+ This is a minor bug fix release.
324
+ - Connect/Send/Receive timeout cannot be configured. fixed.
325
+ - IPSocket#addr caused SocketError? on Mac OS X 10.3.6 + ruby-1.8.1 GA.
326
+ fixed.
327
+ - There is a server which does not like 'foo.bar.com:80' style Host header.
328
+ The server for http://rubyforge.org/export/rss_sfnews.php seems to
329
+ dislike HTTP/1.1 Host header "Host: rubyforge.net:80". It returns
330
+ HTTP 302: Found and redirects to the page again, causes
331
+ HTTPAccess2::Client to raise "retry count exceeded". Keat found that the
332
+ server likes "Host: rubyforge.net" (not with port number).
333
+
334
+ February 11, 2004 - version 2.0.4
335
+ - add Client#redirect_uri_callback interface.
336
+ - refactorings and bug fixes found during negative test.
337
+ - add SSL test.
338
+
339
+ December 16, 2003 - version 2.0.3
340
+ - no_proxy was broken in 2.0.2.
341
+ - do not dump 'Host' header under protocol_version == 'HTTP/1.0'
342
+
343
+ December ?, 2003 - version 2.0.2
344
+ - do not trust HTTP_PROXY environment variable. set proxy server manually.
345
+ http://ftp.ics.uci.edu/pub/websoft/libwww-perl/archive/2001h1/0072.html
346
+ http://ftp.ics.uci.edu/pub/websoft/libwww-perl/archive/2001h1/0241.html
347
+ http://curl.haxx.se/mail/archive-2001-12/0034.html
348
+ - follow ossl2 change.
349
+
350
+ October 4, 2003 - version 2.0.1
351
+ Query was not escaped when query was given as an Array or a Hash. Fixed.
352
+ Do not use http_proxy defined by ENV['http_proxy'] or ENV['HTTP_PROXY'] if
353
+ the destination host is 'localhost'.
354
+ Hosts which matches ENV['no_proxy'] or ENV['NO_PROXY'] won't be proxyed.
355
+ [,:] separated. ("ruby-lang.org:rubyist.net")
356
+ No regexp. (give "ruby-lang.org", not "*.ruby-lang.org")
357
+ If you want specify hot by IP address, give full address.
358
+ ("192.168.1.1, 192.168.1.2")
359
+
360
+ September 10, 2003 - version 2.0
361
+ CamelCase to non_camel_case.
362
+ SSL support (requires Ruby/OpenSSL).
363
+ Cookies support. lib/http-access2/cookie.rb is redistributed file which is
364
+ originally included in Webagent by TAKAHASHI `Maki' Masayoshi. You can
365
+ download the entire package from http://www.rubycolor.org/arc/.
366
+
367
+ January 11, 2003 - version J
368
+ ruby/1.8 support.
data/VERSION.yml ADDED
@@ -0,0 +1,4 @@
1
+ ---
2
+ :major: 2
3
+ :minor: 1
4
+ :patch: 4
@@ -0,0 +1,53 @@
1
+ # HTTPAccess2 - HTTP accessing library.
2
+ # Copyright (C) 2000-2007 NAKAMURA, Hiroshi <nakahiro@sarion.co.jp>.
3
+
4
+ # This program is copyrighted free software by NAKAMURA, Hiroshi. You can
5
+ # redistribute it and/or modify it under the same terms of Ruby's license;
6
+ # either the dual license version in 2003, or any later version.
7
+
8
+ # http-access2.rb is based on http-access.rb in http-access/0.0.4. Some part
9
+ # of code in http-access.rb was recycled in http-access2.rb. Those part is
10
+ # copyrighted by Maehashi-san.
11
+
12
+
13
+ require 'httpclient'
14
+
15
+
16
+ module HTTPAccess2
17
+ VERSION = ::HTTPClient::VERSION
18
+ RUBY_VERSION_STRING = ::HTTPClient::RUBY_VERSION_STRING
19
+ SSLEnabled = ::HTTPClient::SSLEnabled
20
+ SSPIEnabled = ::HTTPClient::SSPIEnabled
21
+ DEBUG_SSL = true
22
+
23
+ Util = ::HTTPClient::Util
24
+
25
+ class Client < ::HTTPClient
26
+ class RetryableResponse < StandardError
27
+ end
28
+ end
29
+
30
+ SSLConfig = ::HTTPClient::SSLConfig
31
+ BasicAuth = ::HTTPClient::BasicAuth
32
+ DigestAuth = ::HTTPClient::DigestAuth
33
+ NegotiateAuth = ::HTTPClient::NegotiateAuth
34
+ AuthFilterBase = ::HTTPClient::AuthFilterBase
35
+ WWWAuth = ::HTTPClient::WWWAuth
36
+ ProxyAuth = ::HTTPClient::ProxyAuth
37
+ Site = ::HTTPClient::Site
38
+ Connection = ::HTTPClient::Connection
39
+ SessionManager = ::HTTPClient::SessionManager
40
+ SSLSocketWrap = ::HTTPClient::SSLSocketWrap
41
+ DebugSocket = ::HTTPClient::DebugSocket
42
+
43
+ class Session < ::HTTPClient::Session
44
+ class Error < StandardError
45
+ end
46
+ class InvalidState < Error
47
+ end
48
+ class BadResponse < Error
49
+ end
50
+ class KeepAliveDisconnected < Error
51
+ end
52
+ end
53
+ end
@@ -0,0 +1 @@
1
+ require 'httpclient/cookie'
@@ -0,0 +1 @@
1
+ require 'httpclient/http'
data/lib/httpclient.rb ADDED
@@ -0,0 +1,1016 @@
1
+ # HTTPClient - HTTP client library.
2
+ # Copyright (C) 2000-2009 NAKAMURA, Hiroshi <nahi@ruby-lang.org>.
3
+ #
4
+ # This program is copyrighted free software by NAKAMURA, Hiroshi. You can
5
+ # redistribute it and/or modify it under the same terms of Ruby's license;
6
+ # either the dual license version in 2003, or any later version.
7
+
8
+ $:.unshift(File.dirname(__FILE__))
9
+
10
+ require 'uri'
11
+ require 'stringio'
12
+ require 'digest/sha1'
13
+
14
+ # Extra library
15
+ require 'httpclient/util'
16
+ require 'httpclient/ssl_config'
17
+ require 'httpclient/connection'
18
+ require 'httpclient/session'
19
+ require 'httpclient/http'
20
+ require 'httpclient/auth'
21
+ require 'httpclient/cookie'
22
+
23
+
24
+ # The HTTPClient class provides several methods for accessing Web resources
25
+ # via HTTP.
26
+ #
27
+ # HTTPClient instance is designed to be MT-safe. You can call a HTTPClient
28
+ # instance from several threads without synchronization after setting up an
29
+ # instance.
30
+ #
31
+ # clnt = HTTPClient.new
32
+ # clnt.set_cookie_store('/home/nahi/cookie.dat')
33
+ # urls.each do |url|
34
+ # Thread.new(url) do |u|
35
+ # p clnt.head(u).status
36
+ # end
37
+ # end
38
+ #
39
+ # == How to use
40
+ #
41
+ # At first, how to create your client. See initialize for more detail.
42
+ #
43
+ # 1. Create simple client.
44
+ #
45
+ # clnt = HTTPClient.new
46
+ #
47
+ # 2. Accessing resources through HTTP proxy. You can use environment
48
+ # variable 'http_proxy' or 'HTTP_PROXY' instead.
49
+ #
50
+ # clnt = HTTPClient.new('http://myproxy:8080')
51
+ #
52
+ # === How to retrieve web resources
53
+ #
54
+ # See get_content.
55
+ #
56
+ # 1. Get content of specified URL. It returns a String of whole result.
57
+ #
58
+ # puts clnt.get_content('http://dev.ctor.org/')
59
+ #
60
+ # 2. Get content as chunks of String. It yields chunks of String.
61
+ #
62
+ # clnt.get_content('http://dev.ctor.org/') do |chunk|
63
+ # puts chunk
64
+ # end
65
+ #
66
+ # === Invoking other HTTP methods
67
+ #
68
+ # See head, get, post, put, delete, options, propfind, proppatch and trace.
69
+ # It returns a HTTP::Message instance as a response.
70
+ #
71
+ # 1. Do HEAD request.
72
+ #
73
+ # res = clnt.head(uri)
74
+ # p res.header['Last-Modified'][0]
75
+ #
76
+ # 2. Do GET request with query.
77
+ #
78
+ # query = { 'keyword' => 'ruby', 'lang' => 'en' }
79
+ # res = clnt.get(uri, query)
80
+ # p res.status
81
+ # p res.contenttype
82
+ # p res.header['X-Custom']
83
+ # puts res.content
84
+ #
85
+ # === How to POST
86
+ #
87
+ # See post.
88
+ #
89
+ # 1. Do POST a form data.
90
+ #
91
+ # body = { 'keyword' => 'ruby', 'lang' => 'en' }
92
+ # res = clnt.post(uri, body)
93
+ #
94
+ # 2. Do multipart file upload with POST. No need to set extra header by
95
+ # yourself from httpclient/2.1.4.
96
+ #
97
+ # File.open('/tmp/post_data') do |file|
98
+ # body = { 'upload' => file, 'user' => 'nahi' }
99
+ # res = clnt.post(uri, body)
100
+ # end
101
+ #
102
+ # === Accessing via SSL
103
+ #
104
+ # Ruby needs to be compiled with OpenSSL.
105
+ #
106
+ # 1. Get content of specified URL via SSL.
107
+ # Just pass an URL which starts with 'https://'.
108
+ #
109
+ # https_url = 'https://www.rsa.com'
110
+ # clnt.get_content(https_url)
111
+ #
112
+ # 2. Getting peer certificate from response.
113
+ #
114
+ # res = clnt.get(https_url)
115
+ # p res.peer_cert #=> returns OpenSSL::X509::Certificate
116
+ #
117
+ # 3. Configuring OpenSSL options. See HTTPClient::SSLConfig for more details.
118
+ #
119
+ # user_cert_file = 'cert.pem'
120
+ # user_key_file = 'privkey.pem'
121
+ # clnt.ssl_config.set_client_cert_file(user_cert_file, user_key_file)
122
+ # clnt.get_content(https_url)
123
+ #
124
+ # === Handling Cookies
125
+ #
126
+ # 1. Using volatile Cookies. Nothing to do. HTTPClient handles Cookies.
127
+ #
128
+ # clnt = HTTPClient.new
129
+ # clnt.get_content(url1) # receives Cookies.
130
+ # clnt.get_content(url2) # sends Cookies if needed.
131
+ #
132
+ # 2. Saving non volatile Cookies to a specified file. Need to set a file at
133
+ # first and invoke save method at last.
134
+ #
135
+ # clnt = HTTPClient.new
136
+ # clnt.set_cookie_store('/home/nahi/cookie.dat')
137
+ # clnt.get_content(url)
138
+ # ...
139
+ # clnt.save_cookie_store
140
+ #
141
+ # 3. Disabling Cookies.
142
+ #
143
+ # clnt = HTTPClient.new
144
+ # clnt.cookie_manager = nil
145
+ #
146
+ # === Configuring authentication credentials
147
+ #
148
+ # 1. Authentication with Web server. Supports BasicAuth, DigestAuth, and
149
+ # Negotiate/NTLM (requires ruby/ntlm module).
150
+ #
151
+ # clnt = HTTPClient.new
152
+ # domain = 'http://dev.ctor.org/http-access2/'
153
+ # user = 'user'
154
+ # password = 'user'
155
+ # clnt.set_auth(domain, user, password)
156
+ # p clnt.get_content('http://dev.ctor.org/http-access2/login').status
157
+ #
158
+ # 2. Authentication with Proxy server. Supports BasicAuth and NTLM
159
+ # (requires win32/sspi)
160
+ #
161
+ # clnt = HTTPClient.new(proxy)
162
+ # user = 'proxy'
163
+ # password = 'proxy'
164
+ # clnt.set_proxy_auth(user, password)
165
+ # p clnt.get_content(url)
166
+ #
167
+ # === Invoking HTTP methods with custom header
168
+ #
169
+ # Pass a Hash or an Array for extheader argument.
170
+ #
171
+ # extheader = { 'Accept' => '*/*' }
172
+ # clnt.get_content(uri, query, extheader)
173
+ #
174
+ # extheader = [['Accept', 'image/jpeg'], ['Accept', 'image/png']]
175
+ # clnt.get_content(uri, query, extheader)
176
+ #
177
+ # === Invoking HTTP methods asynchronously
178
+ #
179
+ # See head_async, get_async, post_async, put_async, delete_async,
180
+ # options_async, propfind_async, proppatch_async, and trace_async.
181
+ # It immediately returns a HTTPClient::Connection instance as a returning value.
182
+ #
183
+ # connection = clnt.post_async(url, body)
184
+ # print 'posting.'
185
+ # while true
186
+ # break if connection.finished?
187
+ # print '.'
188
+ # sleep 1
189
+ # end
190
+ # puts '.'
191
+ # res = connection.pop
192
+ # p res.status
193
+ # p res.content.read # res.content is an IO for the res of async method.
194
+ #
195
+ # === Shortcut methods
196
+ #
197
+ # You can invoke get_content, get, etc. without creating HTTPClient instance.
198
+ #
199
+ # ruby -rhttpclient -e 'puts HTTPClient.get_content(ARGV.shift)' http://dev.ctor.org/
200
+ # ruby -rhttpclient -e 'p HTTPClient.head(ARGV.shift).header["last-modified"]' http://dev.ctor.org/
201
+ #
202
+ class HTTPClient
203
+ VERSION = '2.1.4.1'
204
+ RUBY_VERSION_STRING = "ruby #{RUBY_VERSION} (#{RUBY_RELEASE_DATE}) [#{RUBY_PLATFORM}]"
205
+ /: (\S+) (\S+)/ =~ %q$Id$
206
+ LIB_NAME = "(#{$1}/#{$2}, #{RUBY_VERSION_STRING})"
207
+
208
+ include Util
209
+
210
+ # Raised for indicating running environment configuration error for example
211
+ # accessing via SSL under the ruby which is not compiled with OpenSSL.
212
+ class ConfigurationError < StandardError
213
+ end
214
+
215
+ # Raised for indicating HTTP response error.
216
+ class BadResponseError < RuntimeError
217
+ # HTTP::Message:: a response
218
+ attr_reader :res
219
+
220
+ def initialize(msg, res = nil) # :nodoc:
221
+ super(msg)
222
+ @res = res
223
+ end
224
+ end
225
+
226
+ # Raised for indicating a timeout error.
227
+ class TimeoutError < RuntimeError
228
+ end
229
+
230
+ # Raised for indicating a connection timeout error.
231
+ # You can configure connection timeout via HTTPClient#connect_timeout=.
232
+ class ConnectTimeoutError < TimeoutError
233
+ end
234
+
235
+ # Raised for indicating a request sending timeout error.
236
+ # You can configure request sending timeout via HTTPClient#send_timeout=.
237
+ class SendTimeoutError < TimeoutError
238
+ end
239
+
240
+ # Raised for indicating a response receiving timeout error.
241
+ # You can configure response receiving timeout via
242
+ # HTTPClient#receive_timeout=.
243
+ class ReceiveTimeoutError < TimeoutError
244
+ end
245
+
246
+ # Deprecated. just for backward compatibility
247
+ class Session
248
+ BadResponse = ::HTTPClient::BadResponseError
249
+ end
250
+
251
+ class << self
252
+ %w(get_content post_content head get post put delete options propfind proppatch trace).each do |name|
253
+ eval <<-EOD
254
+ def #{name}(*arg)
255
+ new.#{name}(*arg)
256
+ end
257
+ EOD
258
+ end
259
+
260
+ private
261
+
262
+ def attr_proxy(symbol, assignable = false)
263
+ name = symbol.to_s
264
+ define_method(name) {
265
+ @session_manager.__send__(name)
266
+ }
267
+ if assignable
268
+ aname = name + '='
269
+ define_method(aname) { |rhs|
270
+ reset_all
271
+ @session_manager.__send__(aname, rhs)
272
+ }
273
+ end
274
+ end
275
+ end
276
+
277
+ # HTTPClient::SSLConfig:: SSL configurator.
278
+ attr_reader :ssl_config
279
+ # WebAgent::CookieManager:: Cookies configurator.
280
+ attr_accessor :cookie_manager
281
+ # An array of response HTTP message body String which is used for loop-back
282
+ # test. See test/* to see how to use it. If you want to do loop-back test
283
+ # of HTTP header, use test_loopback_http_response instead.
284
+ attr_reader :test_loopback_response
285
+ # An array of request filter which can trap HTTP request/response.
286
+ # See HTTPClient::WWWAuth to see how to use it.
287
+ attr_reader :request_filter
288
+ # HTTPClient::ProxyAuth:: Proxy authentication handler.
289
+ attr_reader :proxy_auth
290
+ # HTTPClient::WWWAuth:: WWW authentication handler.
291
+ attr_reader :www_auth
292
+ # How many times get_content and post_content follows HTTP redirect.
293
+ # 10 by default.
294
+ attr_accessor :follow_redirect_count
295
+
296
+ # Set HTTP version as a String:: 'HTTP/1.0' or 'HTTP/1.1'
297
+ attr_proxy(:protocol_version, true)
298
+ # Connect timeout in sec.
299
+ attr_proxy(:connect_timeout, true)
300
+ # Request sending timeout in sec.
301
+ attr_proxy(:send_timeout, true)
302
+ # Response receiving timeout in sec.
303
+ attr_proxy(:receive_timeout, true)
304
+ # Negotiation retry count for authentication. 5 by default.
305
+ attr_proxy(:protocol_retry_count, true)
306
+ # if your ruby is older than 2005-09-06, do not set socket_sync = false to
307
+ # avoid an SSL socket blocking bug in openssl/buffering.rb.
308
+ attr_proxy(:socket_sync, true)
309
+ # User-Agent header in HTTP request.
310
+ attr_proxy(:agent_name, true)
311
+ # From header in HTTP request.
312
+ attr_proxy(:from, true)
313
+ # An array of response HTTP String (not a HTTP message body) which is used
314
+ # for loopback test. See test/* to see how to use it.
315
+ attr_proxy(:test_loopback_http_response)
316
+
317
+ # Default extheader for PROPFIND request.
318
+ PROPFIND_DEFAULT_EXTHEADER = { 'Depth' => '0' }
319
+
320
+ # Creates a HTTPClient instance which manages sessions, cookies, etc.
321
+ #
322
+ # HTTPClient.new takes 3 optional arguments for proxy url string,
323
+ # User-Agent String and From header String. User-Agent and From are embedded
324
+ # in HTTP request Header if given. No User-Agent and From header added
325
+ # without setting it explicitly.
326
+ #
327
+ # proxy = 'http://myproxy:8080'
328
+ # agent_name = 'MyAgent/0.1'
329
+ # from = 'from@example.com'
330
+ # HTTPClient.new(proxy, agent_name, from)
331
+ #
332
+ # You can use a keyword argument style Hash. Keys are :proxy, :agent_name
333
+ # and :from.
334
+ #
335
+ # HTTPClient.new(:agent_name = 'MyAgent/0.1')
336
+ def initialize(*args)
337
+ proxy, agent_name, from = keyword_argument(args, :proxy, :agent_name, :from)
338
+ @proxy = nil # assigned later.
339
+ @no_proxy = nil
340
+ @www_auth = WWWAuth.new
341
+ @proxy_auth = ProxyAuth.new
342
+ @request_filter = [@proxy_auth, @www_auth]
343
+ @debug_dev = nil
344
+ @redirect_uri_callback = method(:default_redirect_uri_callback)
345
+ @test_loopback_response = []
346
+ @session_manager = SessionManager.new(self)
347
+ @session_manager.agent_name = agent_name
348
+ @session_manager.from = from
349
+ @session_manager.ssl_config = @ssl_config = SSLConfig.new(self)
350
+ @cookie_manager = WebAgent::CookieManager.new
351
+ @follow_redirect_count = 10
352
+ load_environment
353
+ self.proxy = proxy if proxy
354
+ end
355
+
356
+ # Returns debug device if exists. See debug_dev=.
357
+ def debug_dev
358
+ @debug_dev
359
+ end
360
+
361
+ # Sets debug device. Once debug device is set, all HTTP requests and
362
+ # responses are dumped to given device. dev must respond to << for dump.
363
+ #
364
+ # Calling this method resets all existing sessions.
365
+ def debug_dev=(dev)
366
+ @debug_dev = dev
367
+ reset_all
368
+ @session_manager.debug_dev = dev
369
+ end
370
+
371
+ # Returns URI object of HTTP proxy if exists.
372
+ def proxy
373
+ @proxy
374
+ end
375
+
376
+ # Sets HTTP proxy used for HTTP connection. Given proxy can be an URI,
377
+ # a String or nil. You can set user/password for proxy authentication like
378
+ # HTTPClient#proxy = 'http://user:passwd@myproxy:8080'
379
+ #
380
+ # You can use environment variable 'http_proxy' or 'HTTP_PROXY' for it.
381
+ # You need to use 'cgi_http_proxy' or 'CGI_HTTP_PROXY' instead if you run
382
+ # HTTPClient from CGI environment from security reason. (HTTPClient checks
383
+ # 'REQUEST_METHOD' environment variable whether it's CGI or not)
384
+ #
385
+ # Calling this method resets all existing sessions.
386
+ def proxy=(proxy)
387
+ if proxy.nil?
388
+ @proxy = nil
389
+ @proxy_auth.reset_challenge
390
+ else
391
+ @proxy = urify(proxy)
392
+ if @proxy.scheme == nil or @proxy.scheme.downcase != 'http' or
393
+ @proxy.host == nil or @proxy.port == nil
394
+ raise ArgumentError.new("unsupported proxy #{proxy}")
395
+ end
396
+ @proxy_auth.reset_challenge
397
+ if @proxy.user || @proxy.password
398
+ @proxy_auth.set_auth(@proxy.user, @proxy.password)
399
+ end
400
+ end
401
+ reset_all
402
+ @session_manager.proxy = @proxy
403
+ @proxy
404
+ end
405
+
406
+ # Returns NO_PROXY setting String if given.
407
+ def no_proxy
408
+ @no_proxy
409
+ end
410
+
411
+ # Sets NO_PROXY setting String. no_proxy must be a comma separated String.
412
+ # Each entry must be 'host' or 'host:port' such as;
413
+ # HTTPClient#no_proxy = 'example.com,example.co.jp:443'
414
+ #
415
+ # 'localhost' is treated as a no_proxy site regardless of explicitly listed.
416
+ # HTTPClient checks given URI objects before accessing it.
417
+ # 'host' is tail string match. No IP-addr conversion.
418
+ #
419
+ # You can use environment variable 'no_proxy' or 'NO_PROXY' for it.
420
+ #
421
+ # Calling this method resets all existing sessions.
422
+ def no_proxy=(no_proxy)
423
+ @no_proxy = no_proxy
424
+ reset_all
425
+ end
426
+
427
+ # Sets credential for Web server authentication.
428
+ # domain:: a String or an URI to specify where HTTPClient should use this
429
+ # credential. If you set uri to nil, HTTPClient uses this credential
430
+ # wherever a server requires it.
431
+ # user:: username String.
432
+ # passwd:: password String.
433
+ #
434
+ # You can set multiple credentials for each uri.
435
+ #
436
+ # clnt.set_auth('http://www.example.com/foo/', 'foo_user', 'passwd')
437
+ # clnt.set_auth('http://www.example.com/bar/', 'bar_user', 'passwd')
438
+ #
439
+ # Calling this method resets all existing sessions.
440
+ def set_auth(domain, user, passwd)
441
+ uri = urify(domain)
442
+ @www_auth.set_auth(uri, user, passwd)
443
+ reset_all
444
+ end
445
+
446
+ # Deprecated. Use set_auth instead.
447
+ def set_basic_auth(domain, user, passwd)
448
+ uri = urify(domain)
449
+ @www_auth.basic_auth.set(uri, user, passwd)
450
+ reset_all
451
+ end
452
+
453
+ # Sets credential for Proxy authentication.
454
+ # user:: username String.
455
+ # passwd:: password String.
456
+ #
457
+ # Calling this method resets all existing sessions.
458
+ def set_proxy_auth(user, passwd)
459
+ @proxy_auth.set_auth(user, passwd)
460
+ reset_all
461
+ end
462
+
463
+ # Sets the filename where non-volatile Cookies be saved by calling
464
+ # save_cookie_store.
465
+ # This method tries to load and managing Cookies from the specified file.
466
+ #
467
+ # Calling this method resets all existing sessions.
468
+ def set_cookie_store(filename)
469
+ @cookie_manager.cookies_file = filename
470
+ @cookie_manager.load_cookies if filename
471
+ reset_all
472
+ end
473
+
474
+ # Try to save Cookies to the file specified in set_cookie_store. Unexpected
475
+ # error will be raised if you don't call set_cookie_store first.
476
+ # (interface mismatch between WebAgent::CookieManager implementation)
477
+ def save_cookie_store
478
+ @cookie_manager.save_cookies
479
+ end
480
+
481
+ # Sets callback proc when HTTP redirect status is returned for get_content
482
+ # and post_content. default_redirect_uri_callback is used by default.
483
+ #
484
+ # If you need strict implementation which does not allow relative URI
485
+ # redirection, set strict_redirect_uri_callback instead.
486
+ #
487
+ # clnt.redirect_uri_callback = clnt.method(:strict_redirect_uri_callback)
488
+ #
489
+ def redirect_uri_callback=(redirect_uri_callback)
490
+ @redirect_uri_callback = redirect_uri_callback
491
+ end
492
+
493
+ # Retrieves a web resource.
494
+ #
495
+ # uri:: a String or an URI object which represents an URL of web resource.
496
+ # query:: a Hash or an Array of query part of URL.
497
+ # e.g. { "a" => "b" } => 'http://host/part?a=b'.
498
+ # Give an array to pass multiple value like
499
+ # [["a", "b"], ["a", "c"]] => 'http://host/part?a=b&a=c'.
500
+ # extheader:: a Hash or an Array of extra headers. e.g.
501
+ # { 'Accept' => '*/*' } or
502
+ # [['Accept', 'image/jpeg'], ['Accept', 'image/png']].
503
+ # &block:: Give a block to get chunked message-body of response like
504
+ # get_content(uri) { |chunked_body| ... }.
505
+ # Size of each chunk may not be the same.
506
+ #
507
+ # get_content follows HTTP redirect status (see HTTP::Status.redirect?)
508
+ # internally and try to retrieve content from redirected URL. See
509
+ # redirect_uri_callback= how HTTP redirection is handled.
510
+ #
511
+ # If you need to get full HTTP response including HTTP status and headers,
512
+ # use get method. get returns HTTP::Message as a response and you need to
513
+ # follow HTTP redirect by yourself if you need.
514
+ def get_content(uri, query = nil, extheader = {}, &block)
515
+ follow_redirect(:get, uri, query, nil, extheader, &block).content
516
+ end
517
+
518
+ # Posts a content.
519
+ #
520
+ # uri:: a String or an URI object which represents an URL of web resource.
521
+ # body:: a Hash or an Array of body part.
522
+ # e.g. { "a" => "b" } => 'a=b'.
523
+ # Give an array to pass multiple value like
524
+ # [["a", "b"], ["a", "c"]] => 'a=b&a=c'.
525
+ # When you pass a File as a value, it will be posted as a
526
+ # multipart/form-data. e.g. { 'upload' => file }
527
+ # extheader:: a Hash or an Array of extra headers. e.g.
528
+ # { 'Accept' => '*/*' } or
529
+ # [['Accept', 'image/jpeg'], ['Accept', 'image/png']].
530
+ # &block:: Give a block to get chunked message-body of response like
531
+ # post_content(uri) { |chunked_body| ... }.
532
+ # Size of each chunk may not be the same.
533
+ #
534
+ # post_content follows HTTP redirect status (see HTTP::Status.redirect?)
535
+ # internally and try to post the content to redirected URL. See
536
+ # redirect_uri_callback= how HTTP redirection is handled.
537
+ #
538
+ # If you need to get full HTTP response including HTTP status and headers,
539
+ # use post method.
540
+ def post_content(uri, body = nil, extheader = {}, &block)
541
+ follow_redirect(:post, uri, nil, body, extheader, &block).content
542
+ end
543
+
544
+ # A method for redirect uri callback. How to use:
545
+ # clnt.redirect_uri_callback = clnt.method(:strict_redirect_uri_callback)
546
+ # This callback does not allow relative redirect such as
547
+ # Location: ../foo/
548
+ # in HTTP header. (raises BadResponseError instead)
549
+ def strict_redirect_uri_callback(uri, res)
550
+ newuri = URI.parse(res.header['location'][0])
551
+ if https?(uri) && !https?(newuri)
552
+ raise BadResponseError.new("redirecting to non-https resource")
553
+ end
554
+ unless newuri.is_a?(URI::HTTP)
555
+ raise BadResponseError.new("unexpected location: #{newuri}", res)
556
+ end
557
+ puts "redirect to: #{newuri}" if $DEBUG
558
+ newuri
559
+ end
560
+
561
+ # A default method for redirect uri callback. This method is used by
562
+ # HTTPClient instance by default.
563
+ # This callback allows relative redirect such as
564
+ # Location: ../foo/
565
+ # in HTTP header.
566
+ def default_redirect_uri_callback(uri, res)
567
+ newuri = URI.parse(res.header['location'][0])
568
+ if https?(uri) && !https?(newuri)
569
+ raise BadResponseError.new("redirecting to non-https resource")
570
+ end
571
+ unless newuri.is_a?(URI::HTTP)
572
+ newuri = uri + newuri
573
+ STDERR.puts("could be a relative URI in location header which is not recommended")
574
+ STDERR.puts("'The field value consists of a single absolute URI' in HTTP spec")
575
+ end
576
+ puts "redirect to: #{newuri}" if $DEBUG
577
+ newuri
578
+ end
579
+
580
+ # Sends HEAD request to the specified URL. See request for arguments.
581
+ def head(uri, query = nil, extheader = {})
582
+ request(:head, uri, query, nil, extheader)
583
+ end
584
+
585
+ # Sends GET request to the specified URL. See request for arguments.
586
+ def get(uri, query = nil, extheader = {}, &block)
587
+ request(:get, uri, query, nil, extheader, &block)
588
+ end
589
+
590
+ # Sends POST request to the specified URL. See request for arguments.
591
+ def post(uri, body = nil, extheader = {}, &block)
592
+ request(:post, uri, nil, body, extheader, &block)
593
+ end
594
+
595
+ # Sends PUT request to the specified URL. See request for arguments.
596
+ def put(uri, body = nil, extheader = {}, &block)
597
+ request(:put, uri, nil, body, extheader, &block)
598
+ end
599
+
600
+ # Sends DELETE request to the specified URL. See request for arguments.
601
+ def delete(uri, extheader = {}, &block)
602
+ request(:delete, uri, nil, nil, extheader, &block)
603
+ end
604
+
605
+ # Sends OPTIONS request to the specified URL. See request for arguments.
606
+ def options(uri, extheader = {}, &block)
607
+ request(:options, uri, nil, nil, extheader, &block)
608
+ end
609
+
610
+ # Sends PROPFIND request to the specified URL. See request for arguments.
611
+ def propfind(uri, extheader = PROPFIND_DEFAULT_EXTHEADER, &block)
612
+ request(:propfind, uri, nil, nil, extheader, &block)
613
+ end
614
+
615
+ # Sends PROPPATCH request to the specified URL. See request for arguments.
616
+ def proppatch(uri, body = nil, extheader = {}, &block)
617
+ request(:proppatch, uri, nil, body, extheader, &block)
618
+ end
619
+
620
+ # Sends TRACE request to the specified URL. See request for arguments.
621
+ def trace(uri, query = nil, body = nil, extheader = {}, &block)
622
+ request('TRACE', uri, query, body, extheader, &block)
623
+ end
624
+
625
+ # Sends a request to the specified URL.
626
+ #
627
+ # method:: HTTP method to be sent. method.to_s.upcase is used.
628
+ # uri:: a String or an URI object which represents an URL of web resource.
629
+ # query:: a Hash or an Array of query part of URL.
630
+ # e.g. { "a" => "b" } => 'http://host/part?a=b'
631
+ # Give an array to pass multiple value like
632
+ # [["a", "b"], ["a", "c"]] => 'http://host/part?a=b&a=c'
633
+ # body:: a Hash or an Array of body part.
634
+ # e.g. { "a" => "b" } => 'a=b'.
635
+ # Give an array to pass multiple value like
636
+ # [["a", "b"], ["a", "c"]] => 'a=b&a=c'.
637
+ # When the given method is 'POST' and the given body contains a file
638
+ # as a value, it will be posted as a multipart/form-data.
639
+ # e.g. { 'upload' => file }
640
+ # See HTTP::Message.file? for actual condition of 'a file'.
641
+ # extheader:: a Hash or an Array of extra headers. e.g.
642
+ # { 'Accept' => '*/*' } or
643
+ # [['Accept', 'image/jpeg'], ['Accept', 'image/png']].
644
+ # &block:: Give a block to get chunked message-body of response like
645
+ # get(uri) { |chunked_body| ... }.
646
+ # Size of each chunk may not be the same.
647
+ #
648
+ # You can also pass a String as a body. HTTPClient just sends a String as
649
+ # a HTTP request message body.
650
+ #
651
+ # When you pass an IO as a body, HTTPClient sends it as a HTTP request with
652
+ # chunked encoding (Transfer-Encoding: chunked in HTTP header). Bear in mind
653
+ # that some server application does not support chunked request. At least
654
+ # cgi.rb does not support it.
655
+ def request(method, uri, query = nil, body = nil, extheader = {}, &block)
656
+ uri = urify(uri)
657
+ proxy = no_proxy?(uri) ? nil : @proxy
658
+ req = create_request(method, uri, query, body, extheader)
659
+ if block
660
+ filtered_block = proc { |res, str|
661
+ block.call(str)
662
+ }
663
+ end
664
+ do_request(req, proxy, &filtered_block)
665
+ end
666
+
667
+ # Sends HEAD request in async style. See request_async for arguments.
668
+ # It immediately returns a HTTPClient::Connection instance as a result.
669
+ def head_async(uri, query = nil, extheader = {})
670
+ request_async(:head, uri, query, nil, extheader)
671
+ end
672
+
673
+ # Sends GET request in async style. See request_async for arguments.
674
+ # It immediately returns a HTTPClient::Connection instance as a result.
675
+ def get_async(uri, query = nil, extheader = {})
676
+ request_async(:get, uri, query, nil, extheader)
677
+ end
678
+
679
+ # Sends POST request in async style. See request_async for arguments.
680
+ # It immediately returns a HTTPClient::Connection instance as a result.
681
+ def post_async(uri, body = nil, extheader = {})
682
+ request_async(:post, uri, nil, body, extheader)
683
+ end
684
+
685
+ # Sends PUT request in async style. See request_async for arguments.
686
+ # It immediately returns a HTTPClient::Connection instance as a result.
687
+ def put_async(uri, body = nil, extheader = {})
688
+ request_async(:put, uri, nil, body, extheader)
689
+ end
690
+
691
+ # Sends DELETE request in async style. See request_async for arguments.
692
+ # It immediately returns a HTTPClient::Connection instance as a result.
693
+ def delete_async(uri, extheader = {})
694
+ request_async(:delete, uri, nil, nil, extheader)
695
+ end
696
+
697
+ # Sends OPTIONS request in async style. See request_async for arguments.
698
+ # It immediately returns a HTTPClient::Connection instance as a result.
699
+ def options_async(uri, extheader = {})
700
+ request_async(:options, uri, nil, nil, extheader)
701
+ end
702
+
703
+ # Sends PROPFIND request in async style. See request_async for arguments.
704
+ # It immediately returns a HTTPClient::Connection instance as a result.
705
+ def propfind_async(uri, extheader = PROPFIND_DEFAULT_EXTHEADER)
706
+ request_async(:propfind, uri, nil, nil, extheader)
707
+ end
708
+
709
+ # Sends PROPPATCH request in async style. See request_async for arguments.
710
+ # It immediately returns a HTTPClient::Connection instance as a result.
711
+ def proppatch_async(uri, body = nil, extheader = {})
712
+ request_async(:proppatch, uri, nil, body, extheader)
713
+ end
714
+
715
+ # Sends TRACE request in async style. See request_async for arguments.
716
+ # It immediately returns a HTTPClient::Connection instance as a result.
717
+ def trace_async(uri, query = nil, body = nil, extheader = {})
718
+ request_async(:trace, uri, query, body, extheader)
719
+ end
720
+
721
+ # Sends a request in async style. request method creates new Thread for
722
+ # HTTP connection and returns a HTTPClient::Connection instance immediately.
723
+ #
724
+ # Arguments definition is the same as request.
725
+ def request_async(method, uri, query = nil, body = nil, extheader = {})
726
+ uri = urify(uri)
727
+ proxy = no_proxy?(uri) ? nil : @proxy
728
+ req = create_request(method, uri, query, body, extheader)
729
+ do_request_async(req, proxy)
730
+ end
731
+
732
+ # Resets internal session for the given URL. Keep-alive connection for the
733
+ # site (host-port pair) is disconnected if exists.
734
+ def reset(uri)
735
+ uri = urify(uri)
736
+ @session_manager.reset(uri)
737
+ end
738
+
739
+ # Resets all of internal sessions. Keep-alive connections are disconnected.
740
+ def reset_all
741
+ @session_manager.reset_all
742
+ end
743
+
744
+ private
745
+
746
+ class RetryableResponse < StandardError # :nodoc:
747
+ end
748
+
749
+ class KeepAliveDisconnected < StandardError # :nodoc:
750
+ end
751
+
752
+ def do_request(req, proxy, &block)
753
+ conn = Connection.new
754
+ res = nil
755
+ retry_count = @session_manager.protocol_retry_count
756
+ while retry_count > 0
757
+ begin
758
+ protect_keep_alive_disconnected do
759
+ do_get_block(req, proxy, conn, &block)
760
+ end
761
+ res = conn.pop
762
+ break
763
+ rescue RetryableResponse
764
+ res = conn.pop
765
+ retry_count -= 1
766
+ end
767
+ end
768
+ res
769
+ end
770
+
771
+ def do_request_async(req, proxy)
772
+ conn = Connection.new
773
+ t = Thread.new(conn) { |tconn|
774
+ retry_count = @session_manager.protocol_retry_count
775
+ while retry_count > 0
776
+ begin
777
+ protect_keep_alive_disconnected do
778
+ do_get_stream(req, proxy, tconn)
779
+ end
780
+ break
781
+ rescue RetryableResponse
782
+ retry_count -= 1
783
+ end
784
+ end
785
+ }
786
+ conn.async_thread = t
787
+ conn
788
+ end
789
+
790
+ def load_environment
791
+ # http_proxy
792
+ if getenv('REQUEST_METHOD')
793
+ # HTTP_PROXY conflicts with the environment variable usage in CGI where
794
+ # HTTP_* is used for HTTP header information. Unlike open-uri, we
795
+ # simply ignore http_proxy in CGI env and use cgi_http_proxy instead.
796
+ self.proxy = getenv('cgi_http_proxy')
797
+ else
798
+ self.proxy = getenv('http_proxy')
799
+ end
800
+ # no_proxy
801
+ self.no_proxy = getenv('no_proxy')
802
+ end
803
+
804
+ def getenv(name)
805
+ ENV[name.downcase] || ENV[name.upcase]
806
+ end
807
+
808
+ def follow_redirect(method, uri, query, body, extheader, &block)
809
+ uri = urify(uri)
810
+ if block
811
+ filtered_block = proc { |r, str|
812
+ block.call(str) if HTTP::Status.successful?(r.status)
813
+ }
814
+ end
815
+ if HTTP::Message.file?(body)
816
+ pos = body.pos rescue nil
817
+ end
818
+ retry_number = 0
819
+ while retry_number < @follow_redirect_count
820
+ body.pos = pos if pos
821
+ req = create_request(method, uri, query, body, extheader)
822
+ proxy = no_proxy?(req.header.request_uri) ? nil : @proxy
823
+ res = do_request(req, proxy, &filtered_block)
824
+ if HTTP::Status.successful?(res.status)
825
+ return res
826
+ elsif HTTP::Status.redirect?(res.status)
827
+ uri = urify(@redirect_uri_callback.call(req.header.request_uri, res))
828
+ retry_number += 1
829
+ else
830
+ raise BadResponseError.new("unexpected response: #{res.header.inspect}", res)
831
+ end
832
+ end
833
+ raise BadResponseError.new("retry count exceeded", res)
834
+ end
835
+
836
+ def protect_keep_alive_disconnected
837
+ begin
838
+ yield
839
+ rescue KeepAliveDisconnected
840
+ yield
841
+ end
842
+ end
843
+
844
+ def create_request(method, uri, query, body, extheader)
845
+ method = method.to_s.upcase
846
+ if extheader.is_a?(Hash)
847
+ extheader = extheader.to_a
848
+ else
849
+ extheader = extheader.dup
850
+ end
851
+ boundary = nil
852
+ if body
853
+ dummy, content_type = extheader.find { |key, value|
854
+ key.downcase == 'content-type'
855
+ }
856
+ if content_type
857
+ if /\Amultipart/ =~ content_type
858
+ if content_type =~ /boundary=(.+)\z/
859
+ boundary = $1
860
+ else
861
+ boundary = create_boundary
862
+ content_type = "#{content_type}; boundary=#{boundary}"
863
+ extheader = override_header(extheader, 'Content-Type', content_type)
864
+ end
865
+ end
866
+ elsif method == 'POST'
867
+ if file_in_form_data?(body)
868
+ boundary = create_boundary
869
+ content_type = "multipart/form-data; boundary=#{boundary}"
870
+ else
871
+ content_type = 'application/x-www-form-urlencoded'
872
+ end
873
+ extheader << ['Content-Type', content_type]
874
+ end
875
+ end
876
+ req = HTTP::Message.new_request(method, uri, query, body, boundary)
877
+ extheader.each do |key, value|
878
+ req.header.add(key, value)
879
+ end
880
+ if @cookie_manager && cookie = @cookie_manager.find(uri)
881
+ req.header.add('Cookie', cookie)
882
+ end
883
+ req
884
+ end
885
+
886
+ def create_boundary
887
+ Digest::SHA1.hexdigest(Time.now.to_s)
888
+ end
889
+
890
+ def file_in_form_data?(body)
891
+ HTTP::Message.multiparam_query?(body) &&
892
+ body.any? { |k, v| HTTP::Message.file?(v) }
893
+ end
894
+
895
+ def override_header(extheader, key, value)
896
+ result = []
897
+ extheader.each do |k, v|
898
+ if k.downcase == key.downcase
899
+ result << [key, value]
900
+ else
901
+ result << [k, v]
902
+ end
903
+ end
904
+ result
905
+ end
906
+
907
+ NO_PROXY_HOSTS = ['localhost']
908
+
909
+ def no_proxy?(uri)
910
+ if !@proxy or NO_PROXY_HOSTS.include?(uri.host)
911
+ return true
912
+ end
913
+ unless @no_proxy
914
+ return false
915
+ end
916
+ @no_proxy.scan(/([^:,]+)(?::(\d+))?/) do |host, port|
917
+ if /(\A|\.)#{Regexp.quote(host)}\z/i =~ uri.host &&
918
+ (!port || uri.port == port.to_i)
919
+ return true
920
+ end
921
+ end
922
+ false
923
+ end
924
+
925
+ def https?(uri)
926
+ uri.scheme.downcase == 'https'
927
+ end
928
+
929
+ # !! CAUTION !!
930
+ # Method 'do_get*' runs under MT conditon. Be careful to change.
931
+ def do_get_block(req, proxy, conn, &block)
932
+ @request_filter.each do |filter|
933
+ filter.filter_request(req)
934
+ end
935
+ if str = @test_loopback_response.shift
936
+ dump_dummy_request_response(req.body.dump, str) if @debug_dev
937
+ conn.push(HTTP::Message.new_response(str))
938
+ return
939
+ end
940
+ content = block ? nil : ''
941
+ res = HTTP::Message.new_response(content)
942
+ @debug_dev << "= Request\n\n" if @debug_dev
943
+ sess = @session_manager.query(req, proxy)
944
+ res.peer_cert = sess.ssl_peer_cert
945
+ @debug_dev << "\n\n= Response\n\n" if @debug_dev
946
+ do_get_header(req, res, sess)
947
+ conn.push(res)
948
+ sess.get_body do |part|
949
+ if block
950
+ block.call(res, part)
951
+ else
952
+ content << part
953
+ end
954
+ end
955
+ @session_manager.keep(sess) unless sess.closed?
956
+ commands = @request_filter.collect { |filter|
957
+ filter.filter_response(req, res)
958
+ }
959
+ if commands.find { |command| command == :retry }
960
+ raise RetryableResponse.new
961
+ end
962
+ end
963
+
964
+ def do_get_stream(req, proxy, conn)
965
+ @request_filter.each do |filter|
966
+ filter.filter_request(req)
967
+ end
968
+ if str = @test_loopback_response.shift
969
+ dump_dummy_request_response(req.body.dump, str) if @debug_dev
970
+ conn.push(HTTP::Message.new_response(StringIO.new(str)))
971
+ return
972
+ end
973
+ piper, pipew = IO.pipe
974
+ res = HTTP::Message.new_response(piper)
975
+ @debug_dev << "= Request\n\n" if @debug_dev
976
+ sess = @session_manager.query(req, proxy)
977
+ res.peer_cert = sess.ssl_peer_cert
978
+ @debug_dev << "\n\n= Response\n\n" if @debug_dev
979
+ do_get_header(req, res, sess)
980
+ conn.push(res)
981
+ sess.get_body do |part|
982
+ pipew.syswrite(part)
983
+ end
984
+ pipew.close
985
+ @session_manager.keep(sess) unless sess.closed?
986
+ commands = @request_filter.collect { |filter|
987
+ filter.filter_response(req, res)
988
+ }
989
+ # ignore commands (not retryable in async mode)
990
+ end
991
+
992
+ def do_get_header(req, res, sess)
993
+ res.version, res.status, res.reason, headers = sess.get_header
994
+ headers.each do |key, value|
995
+ res.header.add(key, value)
996
+ end
997
+ if @cookie_manager
998
+ res.header['set-cookie'].each do |cookie|
999
+ @cookie_manager.parse(cookie, req.header.request_uri)
1000
+ end
1001
+
1002
+ # Set the cookie header in the current request so cookies are present in the event we have to retry the
1003
+ # request, such as when responding to an authorization challenge.
1004
+ if (cookie = @cookie_manager.find(req.header.request_uri)) && req.header['Cookie'].empty?
1005
+ req.header.add('Cookie', cookie)
1006
+ end
1007
+ end
1008
+ end
1009
+
1010
+ def dump_dummy_request_response(req, res)
1011
+ @debug_dev << "= Dummy Request\n\n"
1012
+ @debug_dev << req
1013
+ @debug_dev << "\n\n= Dummy Response\n\n"
1014
+ @debug_dev << res
1015
+ end
1016
+ end