httpclient 2.3.0.1 → 2.8.3

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (41) hide show
  1. checksums.yaml +7 -0
  2. data/README.md +85 -0
  3. data/bin/httpclient +18 -6
  4. data/bin/jsonclient +85 -0
  5. data/lib/http-access2.rb +1 -1
  6. data/lib/httpclient.rb +262 -88
  7. data/lib/httpclient/auth.rb +269 -244
  8. data/lib/httpclient/cacert.pem +3952 -0
  9. data/lib/httpclient/cacert1024.pem +3866 -0
  10. data/lib/httpclient/connection.rb +1 -1
  11. data/lib/httpclient/cookie.rb +161 -514
  12. data/lib/httpclient/http.rb +57 -21
  13. data/lib/httpclient/include_client.rb +2 -0
  14. data/lib/httpclient/jruby_ssl_socket.rb +588 -0
  15. data/lib/httpclient/session.rb +259 -317
  16. data/lib/httpclient/ssl_config.rb +141 -188
  17. data/lib/httpclient/ssl_socket.rb +150 -0
  18. data/lib/httpclient/timeout.rb +1 -1
  19. data/lib/httpclient/util.rb +62 -1
  20. data/lib/httpclient/version.rb +1 -1
  21. data/lib/httpclient/webagent-cookie.rb +459 -0
  22. data/lib/jsonclient.rb +63 -0
  23. data/lib/oauthclient.rb +2 -1
  24. data/sample/jsonclient.rb +67 -0
  25. data/sample/oauth_twitter.rb +4 -4
  26. data/test/{ca-chain.cert → ca-chain.pem} +0 -0
  27. data/test/client-pass.key +18 -0
  28. data/test/helper.rb +10 -8
  29. data/test/jruby_ssl_socket/test_pemutils.rb +32 -0
  30. data/test/test_auth.rb +175 -4
  31. data/test/test_cookie.rb +147 -243
  32. data/test/test_http-access2.rb +17 -16
  33. data/test/test_httpclient.rb +458 -77
  34. data/test/test_jsonclient.rb +80 -0
  35. data/test/test_ssl.rb +341 -17
  36. data/test/test_webagent-cookie.rb +465 -0
  37. metadata +57 -55
  38. data/README.txt +0 -721
  39. data/lib/httpclient/cacert.p7s +0 -1858
  40. data/lib/httpclient/cacert_sha1.p7s +0 -1858
  41. data/sample/oauth_salesforce_10.rb +0 -63
@@ -0,0 +1,7 @@
1
+ ---
2
+ SHA1:
3
+ metadata.gz: 45217dcc777d36d71246dd468e40b79caad351d6
4
+ data.tar.gz: afcf1a175414e0a1dde95eb5823b0fa339655d9c
5
+ SHA512:
6
+ metadata.gz: f5ae105eb3b269d67521a35446e3518b359e7a359c0c8a14500ef9e9ff8c4681c20b103bb4d5a2f96cdd9caa337fc149f39df3754bb9f3185b6914f284adfdc0
7
+ data.tar.gz: 005d1769b6906e0c107ba63e7a178c4d73aae59198ed3efe866e8722fdadcf4c4142c3724c167b6db5f8a45cb03f517083164c18fdd1aa65074dc4ab362bc9de
@@ -0,0 +1,85 @@
1
+ httpclient - HTTP accessing library. [![Gem Version](https://badge.fury.io/rb/httpclient.svg)](http://badge.fury.io/rb/httpclient)
2
+
3
+ Copyright (C) 2000-2015 NAKAMURA, Hiroshi <nahi@ruby-lang.org>.
4
+
5
+ 'httpclient' gives something like the functionality of libwww-perl (LWP) in
6
+ Ruby. 'httpclient' formerly known as 'http-access2'.
7
+
8
+ See [HTTPClient](http://www.rubydoc.info/gems/httpclient/frames) for documentation.
9
+
10
+
11
+ ## Features
12
+
13
+ * methods like GET/HEAD/POST/* via HTTP/1.1.
14
+ * HTTPS(SSL), Cookies, proxy, authentication(Digest, NTLM, Basic), etc.
15
+ * asynchronous HTTP request, streaming HTTP request.
16
+ * debug mode CLI.
17
+ * by contrast with net/http in standard distribution;
18
+ * Cookies support
19
+ * MT-safe
20
+ * streaming POST (POST with File/IO)
21
+ * Digest auth
22
+ * Negotiate/NTLM auth for WWW-Authenticate (requires net/ntlm module; rubyntlm gem)
23
+ * NTLM auth for Proxy-Authenticate (requires 'win32/sspi' module; rubysspi gem)
24
+ * extensible with filter interface
25
+ * you don't have to care HTTP/1.1 persistent connection
26
+ (httpclient cares instead of you)
27
+ * Not supported now
28
+ * Cache
29
+ * Rather advanced HTTP/1.1 usage such as Range, deflate, etc.
30
+ (of course you can set it in header by yourself)
31
+
32
+ ## httpclient command
33
+
34
+ Usage: 1) `httpclient get https://www.google.co.jp/?q=ruby`
35
+ Usage: 2) `httpclient`
36
+
37
+ For 1) it issues a GET request to the given URI and shows the wiredump and
38
+ the parsed result. For 2) it invokes irb shell with the binding that has a
39
+ HTTPClient as 'self'. You can call HTTPClient instance methods like;
40
+
41
+ ```ruby
42
+ get "https://www.google.co.jp/", :q => :ruby
43
+ ```
44
+
45
+ ## Author
46
+
47
+ * Name:: Hiroshi Nakamura
48
+ * E-mail:: nahi@ruby-lang.org
49
+ * Project web site:: http://github.com/nahi/httpclient
50
+
51
+
52
+ ## License
53
+
54
+ This program is copyrighted free software by NAKAMURA, Hiroshi. You can
55
+ redistribute it and/or modify it under the same terms of Ruby's license;
56
+ either the dual license version in 2003, or any later version.
57
+
58
+ httpclient/session.rb is based on http-access.rb in http-access/0.0.4. Some
59
+ part of it is copyrighted by Maebashi-san who made and published
60
+ http-access/0.0.4. http-access/0.0.4 did not include license notice but when
61
+ I asked Maebashi-san he agreed that I can redistribute it under the same terms
62
+ of Ruby. Many thanks to Maebashi-san.
63
+
64
+
65
+ ## Install
66
+
67
+ You can install httpclient via rubygems: `gem install httpclient`
68
+
69
+
70
+ ## Usage
71
+
72
+ See [HTTPClient](http://www.rubydoc.info/gems/httpclient/frames) for documentation.
73
+ You can also check sample/howto.rb how to use APIs.
74
+
75
+ ## Bug report or Feature request
76
+
77
+ Please file a ticket at the project web site.
78
+
79
+ 1. find a similar ticket from https://github.com/nahi/httpclient/issues
80
+ 2. create a new ticket by clicking 'Create Issue' button.
81
+ 3. you can use github features such as pull-request if you like.
82
+
83
+ ## Changes
84
+
85
+ See [ChangeLog](https://github.com/nahi/httpclient/blob/master/CHANGELOG.md)
@@ -11,13 +11,24 @@
11
11
  # > get "https://www.google.co.jp/", :q => :ruby
12
12
  require 'httpclient'
13
13
 
14
- METHODS = ['head', 'get', 'post', 'put', 'delete', 'options', 'propfind', 'proppatch', 'trace']
15
- if ARGV.size >= 2 && METHODS.include?(ARGV[0])
14
+ method = ARGV.shift
15
+ if method == 'version'
16
+ puts HTTPClient::VERSION
17
+ exit
18
+ end
19
+
20
+ url = ARGV.shift
21
+ if method && url
16
22
  client = HTTPClient.new
17
- client.debug_dev = STDERR
18
- $DEBUG = true
19
- require 'pp'
20
- pp client.send(*ARGV)
23
+ client.strict_response_size_check = true
24
+ if method == 'download'
25
+ print client.get_content(url)
26
+ else
27
+ client.debug_dev = STDERR
28
+ $DEBUG = true
29
+ require 'pp'
30
+ pp client.send(method, url, *ARGV)
31
+ end
21
32
  exit
22
33
  end
23
34
 
@@ -27,6 +38,7 @@ require 'irb/completion'
27
38
  class Runner
28
39
  def initialize
29
40
  @httpclient = HTTPClient.new
41
+ @httpclient.strict_response_size_check = true
30
42
  end
31
43
 
32
44
  def method_missing(msg, *a, &b)
@@ -0,0 +1,85 @@
1
+ #!/usr/bin/env ruby
2
+
3
+ # jsonclient shell command.
4
+ #
5
+ # Usage: 1) % jsonclient post https://www.example.com/ content.json
6
+ # Usage: 2) % jsonclient
7
+ #
8
+ # For 1) it issues a GET request to the given URI and shows the wiredump and
9
+ # the parsed result. For 2) it invokes irb shell with the binding that has a
10
+ # JSONClient as 'self'. You can call JSONClient instance methods like;
11
+ # > post "https://www.example.com/resource", {'hello' => 'world'}
12
+ require 'jsonclient'
13
+
14
+ method = ARGV.shift
15
+ url = ARGV.shift
16
+ body = []
17
+ if ['post', 'put'].include?(method)
18
+ if ARGV.size == 1 && File.exist?(ARGV[0])
19
+ body << File.read(ARGV[0])
20
+ else
21
+ body << ARGF.read
22
+ end
23
+ end
24
+ if method && url
25
+ require 'pp'
26
+ client = JSONClient.new
27
+ client.debug_dev = STDERR if $DEBUG
28
+ res = client.send(method, url, *body)
29
+ STDERR.puts('RESPONSE HEADER: ')
30
+ PP.pp(res.headers, STDERR)
31
+ if res.ok?
32
+ begin
33
+ puts JSON.pretty_generate(res.content)
34
+ rescue JSON::GeneratorError
35
+ puts res.content
36
+ end
37
+ exit 0
38
+ else
39
+ STDERR.puts res.content
40
+ exit 1
41
+ end
42
+ end
43
+
44
+ require 'irb'
45
+ require 'irb/completion'
46
+
47
+ class Runner
48
+ def initialize
49
+ @httpclient = JSONClient.new
50
+ end
51
+
52
+ def method_missing(msg, *a, &b)
53
+ debug, $DEBUG = $DEBUG, true
54
+ begin
55
+ @httpclient.send(msg, *a, &b)
56
+ ensure
57
+ $DEBUG = debug
58
+ end
59
+ end
60
+
61
+ def run
62
+ IRB.setup(nil)
63
+ ws = IRB::WorkSpace.new(binding)
64
+ irb = IRB::Irb.new(ws)
65
+ IRB.conf[:MAIN_CONTEXT] = irb.context
66
+
67
+ trap("SIGINT") do
68
+ irb.signal_handle
69
+ end
70
+
71
+ begin
72
+ catch(:IRB_EXIT) do
73
+ irb.eval_input
74
+ end
75
+ ensure
76
+ IRB.irb_at_exit
77
+ end
78
+ end
79
+
80
+ def to_s
81
+ 'JSONClient'
82
+ end
83
+ end
84
+
85
+ Runner.new.run
@@ -39,7 +39,7 @@ module HTTPAccess2
39
39
  Site = ::HTTPClient::Site
40
40
  Connection = ::HTTPClient::Connection
41
41
  SessionManager = ::HTTPClient::SessionManager
42
- SSLSocketWrap = ::HTTPClient::SSLSocketWrap
42
+ SSLSocketWrap = ::HTTPClient::SSLSocket
43
43
  DebugSocket = ::HTTPClient::DebugSocket
44
44
 
45
45
  class Session < ::HTTPClient::Session
@@ -1,5 +1,5 @@
1
1
  # HTTPClient - HTTP client library.
2
- # Copyright (C) 2000-2009 NAKAMURA, Hiroshi <nahi@ruby-lang.org>.
2
+ # Copyright (C) 2000-2015 NAKAMURA, Hiroshi <nahi@ruby-lang.org>.
3
3
  #
4
4
  # This program is copyrighted free software by NAKAMURA, Hiroshi. You can
5
5
  # redistribute it and/or modify it under the same terms of Ruby's license;
@@ -116,7 +116,7 @@ require 'httpclient/cookie'
116
116
  # res = clnt.post(uri, body)
117
117
  # end
118
118
  #
119
- # 3. Do multipart wth custom body.
119
+ # 3. Do multipart with custom body.
120
120
  #
121
121
  # File.open('/tmp/post_data') do |file|
122
122
  # body = [{ 'Content-Type' => 'application/atom+xml; charset=UTF-8',
@@ -149,6 +149,16 @@ require 'httpclient/cookie'
149
149
  # clnt.ssl_config.set_client_cert_file(user_cert_file, user_key_file)
150
150
  # clnt.get(https_url)
151
151
  #
152
+ # 4. Revocation check. On JRuby you can set following options to let
153
+ # HTTPClient to perform revocation check with CRL and OCSP:
154
+ #
155
+ # -J-Dcom.sun.security.enableCRLDP=true -J-Dcom.sun.net.ssl.checkRevocation=true
156
+ # ex. jruby -J-Dcom.sun.security.enableCRLDP=true -J-Dcom.sun.net.ssl.checkRevocation=true app.rb
157
+ # Revoked cert example: https://test-sspev.verisign.com:2443/test-SSPEV-revoked-verisign.html
158
+ #
159
+ # On other platform you can download CRL by yourself and set it via
160
+ # SSLConfig#add_crl.
161
+ #
152
162
  # === Handling Cookies
153
163
  #
154
164
  # 1. Using volatile Cookies. Nothing to do. HTTPClient handles Cookies.
@@ -229,7 +239,7 @@ require 'httpclient/cookie'
229
239
  # ruby -rhttpclient -e 'p HTTPClient.head(ARGV.shift).header["last-modified"]' http://dev.ctor.org/
230
240
  #
231
241
  class HTTPClient
232
- RUBY_VERSION_STRING = "ruby #{RUBY_VERSION} (#{RUBY_RELEASE_DATE}) [#{RUBY_PLATFORM}]"
242
+ RUBY_VERSION_STRING = "ruby #{RUBY_VERSION} (#{RUBY_RELEASE_DATE})"
233
243
  LIB_NAME = "(#{VERSION}, #{RUBY_VERSION_STRING})"
234
244
 
235
245
  include Util
@@ -299,7 +309,6 @@ class HTTPClient
299
309
  if assignable
300
310
  aname = name + '='
301
311
  define_method(aname) { |rhs|
302
- reset_all
303
312
  @session_manager.__send__(aname, rhs)
304
313
  }
305
314
  end
@@ -308,7 +317,7 @@ class HTTPClient
308
317
 
309
318
  # HTTPClient::SSLConfig:: SSL configurator.
310
319
  attr_reader :ssl_config
311
- # WebAgent::CookieManager:: Cookies configurator.
320
+ # HTTPClient::CookieManager:: Cookies configurator.
312
321
  attr_accessor :cookie_manager
313
322
  # An array of response HTTP message body String which is used for loop-back
314
323
  # test. See test/* to see how to use it. If you want to do loop-back test
@@ -324,6 +333,10 @@ class HTTPClient
324
333
  # How many times get_content and post_content follows HTTP redirect.
325
334
  # 10 by default.
326
335
  attr_accessor :follow_redirect_count
336
+ # Base url of resources.
337
+ attr_accessor :base_url
338
+ # Default request header.
339
+ attr_accessor :default_header
327
340
 
328
341
  # Set HTTP version as a String:: 'HTTP/1.0' or 'HTTP/1.1'
329
342
  attr_proxy(:protocol_version, true)
@@ -342,6 +355,8 @@ class HTTPClient
342
355
  # if your ruby is older than 2005-09-06, do not set socket_sync = false to
343
356
  # avoid an SSL socket blocking bug in openssl/buffering.rb.
344
357
  attr_proxy(:socket_sync, true)
358
+ # Enables TCP keepalive; no timing settings exist at present
359
+ attr_proxy(:tcp_keepalive, true)
345
360
  # User-Agent header in HTTP request.
346
361
  attr_proxy(:agent_name, true)
347
362
  # From header in HTTP request.
@@ -351,48 +366,90 @@ class HTTPClient
351
366
  attr_proxy(:test_loopback_http_response)
352
367
  # Decompress a compressed (with gzip or deflate) content body transparently. false by default.
353
368
  attr_proxy(:transparent_gzip_decompression, true)
369
+ # Raise BadResponseError if response size does not match with Content-Length header in response. false by default.
370
+ # TODO: enable by default
371
+ attr_proxy(:strict_response_size_check, true)
354
372
  # Local socket address. Set HTTPClient#socket_local.host and HTTPClient#socket_local.port to specify local binding hostname and port of TCP socket.
355
373
  attr_proxy(:socket_local, true)
356
374
 
357
375
  # Default header for PROPFIND request.
358
376
  PROPFIND_DEFAULT_EXTHEADER = { 'Depth' => '0' }
359
377
 
378
+ # Default User-Agent header
379
+ DEFAULT_AGENT_NAME = 'HTTPClient/1.0'
380
+
360
381
  # Creates a HTTPClient instance which manages sessions, cookies, etc.
361
382
  #
362
- # HTTPClient.new takes 3 optional arguments for proxy url string,
363
- # User-Agent String and From header String. User-Agent and From are embedded
364
- # in HTTP request Header if given. No User-Agent and From header added
365
- # without setting it explicitly.
383
+ # HTTPClient.new takes optional arguments as a Hash.
384
+ # * :proxy - proxy url string
385
+ # * :agent_name - User-Agent String
386
+ # * :from - from header String
387
+ # * :base_url - base URL of resources
388
+ # * :default_header - header Hash all HTTP requests should have
389
+ # * :force_basic_auth - flag for sending Authorization header w/o gettin 401 first
390
+ # User-Agent and From are embedded in HTTP request Header if given.
391
+ # From header is not set without setting it explicitly.
366
392
  #
367
393
  # proxy = 'http://myproxy:8080'
368
394
  # agent_name = 'MyAgent/0.1'
369
395
  # from = 'from@example.com'
370
396
  # HTTPClient.new(proxy, agent_name, from)
371
397
  #
372
- # You can use a keyword argument style Hash. Keys are :proxy, :agent_name
373
- # and :from.
398
+ # After you set :base_url, all resources you pass to get, post and other
399
+ # methods are recognized to be prefixed with base_url. Say base_url is
400
+ # 'https://api.example.com/v1/, get('users') is the same as
401
+ # get('https://api.example.com/v1/users') internally. You can also pass
402
+ # full URL from 'http://' even after setting base_url.
403
+ #
404
+ # The expected base_url and path behavior is the following. Please take
405
+ # care of '/' in base_url and path.
374
406
  #
375
- # HTTPClient.new(:agent_name => 'MyAgent/0.1')
376
- def initialize(*args)
377
- proxy, agent_name, from = keyword_argument(args, :proxy, :agent_name, :from)
407
+ # The last '/' is important for base_url:
408
+ # 1. http://foo/bar/baz/ + path -> http://foo/bar/baz/path
409
+ # 2. http://foo/bar/baz + path -> http://foo/bar/path
410
+ # Relative path handling:
411
+ # 3. http://foo/bar/baz/ + ../path -> http://foo/bar/path
412
+ # 4. http://foo/bar/baz + ../path -> http://foo/path
413
+ # 5. http://foo/bar/baz/ + ./path -> http://foo/bar/baz/path
414
+ # 6. http://foo/bar/baz + ./path -> http://foo/bar/path
415
+ # The leading '/' of path means absolute path:
416
+ # 7. http://foo/bar/baz/ + /path -> http://foo/path
417
+ # 8. http://foo/bar/baz + /path -> http://foo/path
418
+ #
419
+ # :default_header is for providing default headers Hash that all HTTP
420
+ # requests should have, such as custom 'Authorization' header in API.
421
+ # You can override :default_header with :header Hash parameter in HTTP
422
+ # request methods.
423
+ #
424
+ # :force_basic_auth turns on/off the BasicAuth force flag. Generally
425
+ # HTTP client must send Authorization header after it gets 401 error
426
+ # from server from security reason. But in some situation (e.g. API
427
+ # client) you might want to send Authorization from the beginning.
428
+ def initialize(*args, &block)
429
+ proxy, agent_name, from, base_url, default_header, force_basic_auth =
430
+ keyword_argument(args, :proxy, :agent_name, :from, :base_url, :default_header, :force_basic_auth)
378
431
  @proxy = nil # assigned later.
379
432
  @no_proxy = nil
380
433
  @no_proxy_regexps = []
434
+ @base_url = base_url
435
+ @default_header = default_header || {}
381
436
  @www_auth = WWWAuth.new
382
437
  @proxy_auth = ProxyAuth.new
438
+ @www_auth.basic_auth.force_auth = @proxy_auth.basic_auth.force_auth = force_basic_auth
383
439
  @request_filter = [@proxy_auth, @www_auth]
384
440
  @debug_dev = nil
385
441
  @redirect_uri_callback = method(:default_redirect_uri_callback)
386
442
  @test_loopback_response = []
387
443
  @session_manager = SessionManager.new(self)
388
- @session_manager.agent_name = agent_name
444
+ @session_manager.agent_name = agent_name || DEFAULT_AGENT_NAME
389
445
  @session_manager.from = from
390
446
  @session_manager.ssl_config = @ssl_config = SSLConfig.new(self)
391
- @cookie_manager = WebAgent::CookieManager.new
447
+ @cookie_manager = CookieManager.new
392
448
  @follow_redirect_count = 10
393
449
  load_environment
394
450
  self.proxy = proxy if proxy
395
451
  keep_webmock_compat
452
+ instance_eval(&block) if block
396
453
  end
397
454
 
398
455
  # webmock 1.6.2 depends on HTTP::Message#body.content to work.
@@ -480,7 +537,7 @@ class HTTPClient
480
537
  @no_proxy = no_proxy
481
538
  @no_proxy_regexps.clear
482
539
  if @no_proxy
483
- @no_proxy.scan(/([^:,]+)(?::(\d+))?/) do |host, port|
540
+ @no_proxy.scan(/([^\s:,]+)(?::(\d+))?/) do |host, port|
484
541
  if host[0] == ?.
485
542
  regexp = /#{Regexp.quote(host)}\z/i
486
543
  else
@@ -506,14 +563,14 @@ class HTTPClient
506
563
  #
507
564
  # Calling this method resets all existing sessions.
508
565
  def set_auth(domain, user, passwd)
509
- uri = urify(domain)
566
+ uri = to_resource_url(domain)
510
567
  @www_auth.set_auth(uri, user, passwd)
511
568
  reset_all
512
569
  end
513
570
 
514
571
  # Deprecated. Use set_auth instead.
515
572
  def set_basic_auth(domain, user, passwd)
516
- uri = urify(domain)
573
+ uri = to_resource_url(domain)
517
574
  @www_auth.basic_auth.set(uri, user, passwd)
518
575
  reset_all
519
576
  end
@@ -528,6 +585,14 @@ class HTTPClient
528
585
  reset_all
529
586
  end
530
587
 
588
+ # Turn on/off the BasicAuth force flag. Generally HTTP client must
589
+ # send Authorization header after it gets 401 error from server from
590
+ # security reason. But in some situation (e.g. API client) you might
591
+ # want to send Authorization from the beginning.
592
+ def force_basic_auth=(force_basic_auth)
593
+ @www_auth.basic_auth.force_auth = @proxy_auth.basic_auth.force_auth = force_basic_auth
594
+ end
595
+
531
596
  # Sets the filename where non-volatile Cookies be saved by calling
532
597
  # save_cookie_store.
533
598
  # This method tries to load and managing Cookies from the specified file.
@@ -541,7 +606,6 @@ class HTTPClient
541
606
 
542
607
  # Try to save Cookies to the file specified in set_cookie_store. Unexpected
543
608
  # error will be raised if you don't call set_cookie_store first.
544
- # (interface mismatch between WebAgent::CookieManager implementation)
545
609
  def save_cookie_store
546
610
  @cookie_manager.save_cookies
547
611
  end
@@ -624,8 +688,13 @@ class HTTPClient
624
688
  # If you need to get full HTTP response including HTTP status and headers,
625
689
  # use post method.
626
690
  def post_content(uri, *args, &block)
627
- body, header = keyword_argument(args, :body, :header)
628
- success_content(follow_redirect(:post, uri, nil, body, header || {}, &block))
691
+ if hashy_argument_has_keys(args, :query, :body)
692
+ query, body, header = keyword_argument(args, :query, :body, :header)
693
+ else
694
+ query = nil
695
+ body, header = keyword_argument(args, :body, :header)
696
+ end
697
+ success_content(follow_redirect(:post, uri, query, body, header || {}, &block))
629
698
  end
630
699
 
631
700
  # A method for redirect uri callback. How to use:
@@ -653,9 +722,9 @@ class HTTPClient
653
722
  def default_redirect_uri_callback(uri, res)
654
723
  newuri = urify(res.header['location'][0])
655
724
  if !http?(newuri) && !https?(newuri)
656
- newuri = uri + newuri
657
- warn("could be a relative URI in location header which is not recommended")
725
+ warn("#{newuri}: a relative URI in location header which is not recommended")
658
726
  warn("'The field value consists of a single absolute URI' in HTTP spec")
727
+ newuri = uri + newuri
659
728
  end
660
729
  if https?(uri) && !https?(newuri)
661
730
  raise BadResponseError.new("redirecting to non-https resource")
@@ -674,26 +743,47 @@ class HTTPClient
674
743
  request(:get, uri, argument_to_hash(args, :query, :header, :follow_redirect), &block)
675
744
  end
676
745
 
746
+ # Sends PATCH request to the specified URL. See request for arguments.
747
+ def patch(uri, *args, &block)
748
+ if hashy_argument_has_keys(args, :query, :body)
749
+ new_args = args[0]
750
+ else
751
+ new_args = argument_to_hash(args, :body, :header)
752
+ end
753
+ request(:patch, uri, new_args, &block)
754
+ end
755
+
677
756
  # Sends POST request to the specified URL. See request for arguments.
678
757
  # You should not depend on :follow_redirect => true for POST method. It
679
758
  # sends the same POST method to the new location which is prohibited in HTTP spec.
680
759
  def post(uri, *args, &block)
681
- request(:post, uri, argument_to_hash(args, :body, :header, :follow_redirect), &block)
760
+ if hashy_argument_has_keys(args, :query, :body)
761
+ new_args = args[0]
762
+ else
763
+ new_args = argument_to_hash(args, :body, :header, :follow_redirect)
764
+ end
765
+ request(:post, uri, new_args, &block)
682
766
  end
683
767
 
684
768
  # Sends PUT request to the specified URL. See request for arguments.
685
769
  def put(uri, *args, &block)
686
- request(:put, uri, argument_to_hash(args, :body, :header), &block)
770
+ if hashy_argument_has_keys(args, :query, :body)
771
+ new_args = args[0]
772
+ else
773
+ new_args = argument_to_hash(args, :body, :header)
774
+ end
775
+ request(:put, uri, new_args, &block)
687
776
  end
688
777
 
689
778
  # Sends DELETE request to the specified URL. See request for arguments.
690
779
  def delete(uri, *args, &block)
691
- request(:delete, uri, argument_to_hash(args, :body, :header), &block)
780
+ request(:delete, uri, argument_to_hash(args, :body, :header, :query), &block)
692
781
  end
693
782
 
694
783
  # Sends OPTIONS request to the specified URL. See request for arguments.
695
784
  def options(uri, *args, &block)
696
- request(:options, uri, argument_to_hash(args, :header), &block)
785
+ new_args = argument_to_hash(args, :header, :query, :body)
786
+ request(:options, uri, new_args, &block)
697
787
  end
698
788
 
699
789
  # Sends PROPFIND request to the specified URL. See request for arguments.
@@ -747,23 +837,18 @@ class HTTPClient
747
837
  #
748
838
  # When you pass an IO as a body, HTTPClient sends it as a HTTP request with
749
839
  # chunked encoding (Transfer-Encoding: chunked in HTTP header) if IO does not
750
- # respond to :read. Bear in mind that some server application does not support
840
+ # respond to :size. Bear in mind that some server application does not support
751
841
  # chunked request. At least cgi.rb does not support it.
752
842
  def request(method, uri, *args, &block)
753
843
  query, body, header, follow_redirect = keyword_argument(args, :query, :body, :header, :follow_redirect)
754
- if [:post, :put].include?(method)
755
- body ||= ''
756
- end
757
844
  if method == :propfind
758
845
  header ||= PROPFIND_DEFAULT_EXTHEADER
759
846
  else
760
847
  header ||= {}
761
848
  end
762
- uri = urify(uri)
849
+ uri = to_resource_url(uri)
763
850
  if block
764
- filtered_block = proc { |res, str|
765
- block.call(str)
766
- }
851
+ filtered_block = adapt_block(&block)
767
852
  end
768
853
  if follow_redirect
769
854
  follow_redirect(method, uri, query, body, header, &block)
@@ -775,64 +860,76 @@ class HTTPClient
775
860
  # Sends HEAD request in async style. See request_async for arguments.
776
861
  # It immediately returns a HTTPClient::Connection instance as a result.
777
862
  def head_async(uri, *args)
778
- query, header = keyword_argument(args, :query, :header)
779
- request_async(:head, uri, query, nil, header || {})
863
+ request_async2(:head, uri, argument_to_hash(args, :query, :header))
780
864
  end
781
865
 
782
866
  # Sends GET request in async style. See request_async for arguments.
783
867
  # It immediately returns a HTTPClient::Connection instance as a result.
784
868
  def get_async(uri, *args)
785
- query, header = keyword_argument(args, :query, :header)
786
- request_async(:get, uri, query, nil, header || {})
869
+ request_async2(:get, uri, argument_to_hash(args, :query, :header))
870
+ end
871
+
872
+ # Sends PATCH request in async style. See request_async2 for arguments.
873
+ # It immediately returns a HTTPClient::Connection instance as a result.
874
+ def patch_async(uri, *args)
875
+ if hashy_argument_has_keys(args, :query, :body)
876
+ new_args = args[0]
877
+ else
878
+ new_args = argument_to_hash(args, :body, :header)
879
+ end
880
+ request_async2(:patch, uri, new_args)
787
881
  end
788
882
 
789
883
  # Sends POST request in async style. See request_async for arguments.
790
884
  # It immediately returns a HTTPClient::Connection instance as a result.
791
885
  def post_async(uri, *args)
792
- body, header = keyword_argument(args, :body, :header)
793
- request_async(:post, uri, nil, body || '', header || {})
886
+ if hashy_argument_has_keys(args, :query, :body)
887
+ new_args = args[0]
888
+ else
889
+ new_args = argument_to_hash(args, :body, :header)
890
+ end
891
+ request_async2(:post, uri, new_args)
794
892
  end
795
893
 
796
- # Sends PUT request in async style. See request_async for arguments.
894
+ # Sends PUT request in async style. See request_async2 for arguments.
797
895
  # It immediately returns a HTTPClient::Connection instance as a result.
798
896
  def put_async(uri, *args)
799
- body, header = keyword_argument(args, :body, :header)
800
- request_async(:put, uri, nil, body || '', header || {})
897
+ if hashy_argument_has_keys(args, :query, :body)
898
+ new_args = args[0]
899
+ else
900
+ new_args = argument_to_hash(args, :body, :header)
901
+ end
902
+ request_async2(:put, uri, new_args)
801
903
  end
802
904
 
803
- # Sends DELETE request in async style. See request_async for arguments.
905
+ # Sends DELETE request in async style. See request_async2 for arguments.
804
906
  # It immediately returns a HTTPClient::Connection instance as a result.
805
907
  def delete_async(uri, *args)
806
- header = keyword_argument(args, :header)
807
- request_async(:delete, uri, nil, nil, header || {})
908
+ request_async2(:delete, uri, argument_to_hash(args, :body, :header, :query))
808
909
  end
809
910
 
810
- # Sends OPTIONS request in async style. See request_async for arguments.
911
+ # Sends OPTIONS request in async style. See request_async2 for arguments.
811
912
  # It immediately returns a HTTPClient::Connection instance as a result.
812
913
  def options_async(uri, *args)
813
- header = keyword_argument(args, :header)
814
- request_async(:options, uri, nil, nil, header || {})
914
+ request_async2(:options, uri, argument_to_hash(args, :header, :query, :body))
815
915
  end
816
916
 
817
- # Sends PROPFIND request in async style. See request_async for arguments.
917
+ # Sends PROPFIND request in async style. See request_async2 for arguments.
818
918
  # It immediately returns a HTTPClient::Connection instance as a result.
819
919
  def propfind_async(uri, *args)
820
- header = keyword_argument(args, :header)
821
- request_async(:propfind, uri, nil, nil, header || PROPFIND_DEFAULT_EXTHEADER)
920
+ request_async2(:propfind, uri, argument_to_hash(args, :body, :header))
822
921
  end
823
922
 
824
- # Sends PROPPATCH request in async style. See request_async for arguments.
923
+ # Sends PROPPATCH request in async style. See request_async2 for arguments.
825
924
  # It immediately returns a HTTPClient::Connection instance as a result.
826
925
  def proppatch_async(uri, *args)
827
- body, header = keyword_argument(args, :body, :header)
828
- request_async(:proppatch, uri, nil, body, header || {})
926
+ request_async2(:proppatch, uri, argument_to_hash(args, :body, :header))
829
927
  end
830
928
 
831
- # Sends TRACE request in async style. See request_async for arguments.
929
+ # Sends TRACE request in async style. See request_async2 for arguments.
832
930
  # It immediately returns a HTTPClient::Connection instance as a result.
833
931
  def trace_async(uri, *args)
834
- query, body, header = keyword_argument(args, :query, :body, :header)
835
- request_async(:trace, uri, query, body, header || {})
932
+ request_async2(:trace, uri, argument_to_hash(args, :query, :header))
836
933
  end
837
934
 
838
935
  # Sends a request in async style. request method creates new Thread for
@@ -840,14 +937,29 @@ class HTTPClient
840
937
  #
841
938
  # Arguments definition is the same as request.
842
939
  def request_async(method, uri, query = nil, body = nil, header = {})
843
- uri = urify(uri)
940
+ uri = to_resource_url(uri)
941
+ do_request_async(method, uri, query, body, header)
942
+ end
943
+
944
+ # new method that has same signature as 'request'
945
+ def request_async2(method, uri, *args)
946
+ query, body, header = keyword_argument(args, :query, :body, :header)
947
+ if [:post, :put].include?(method)
948
+ body ||= ''
949
+ end
950
+ if method == :propfind
951
+ header ||= PROPFIND_DEFAULT_EXTHEADER
952
+ else
953
+ header ||= {}
954
+ end
955
+ uri = to_resource_url(uri)
844
956
  do_request_async(method, uri, query, body, header)
845
957
  end
846
958
 
847
959
  # Resets internal session for the given URL. Keep-alive connection for the
848
960
  # site (host-port pair) is disconnected if exists.
849
961
  def reset(uri)
850
- uri = urify(uri)
962
+ uri = to_resource_url(uri)
851
963
  @session_manager.reset(uri)
852
964
  end
853
965
 
@@ -859,34 +971,62 @@ class HTTPClient
859
971
  private
860
972
 
861
973
  class RetryableResponse < StandardError # :nodoc:
974
+ attr_reader :res
975
+
976
+ def initialize(res = nil)
977
+ @res = res
978
+ end
862
979
  end
863
980
 
864
981
  class KeepAliveDisconnected < StandardError # :nodoc:
865
982
  attr_reader :sess
866
- def initialize(sess = nil)
983
+ attr_reader :cause
984
+ def initialize(sess = nil, cause = nil)
985
+ super("#{self.class.name}: #{cause ? cause.message : nil}")
867
986
  @sess = sess
987
+ @cause = cause
868
988
  end
869
989
  end
870
990
 
991
+ def hashy_argument_has_keys(args, *key)
992
+ # if the given arg is a single Hash and...
993
+ args.size == 1 and Hash === args[0] and
994
+ # it has any one of the key
995
+ key.all? { |e| args[0].key?(e) }
996
+ end
997
+
871
998
  def do_request(method, uri, query, body, header, &block)
872
- conn = Connection.new
873
999
  res = nil
874
1000
  if HTTP::Message.file?(body)
875
1001
  pos = body.pos rescue nil
876
1002
  end
877
1003
  retry_count = @session_manager.protocol_retry_count
878
1004
  proxy = no_proxy?(uri) ? nil : @proxy
1005
+ previous_request = previous_response = nil
879
1006
  while retry_count > 0
880
1007
  body.pos = pos if pos
881
1008
  req = create_request(method, uri, query, body, header)
1009
+ if previous_request
1010
+ # to remember IO positions to read
1011
+ req.http_body.positions = previous_request.http_body.positions
1012
+ end
882
1013
  begin
883
1014
  protect_keep_alive_disconnected do
884
- do_get_block(req, proxy, conn, &block)
1015
+ # TODO: remove Connection.new
1016
+ # We want to delete Connection usage in do_get_block but Newrelic gem depends on it.
1017
+ # https://github.com/newrelic/rpm/blob/master/lib/new_relic/agent/instrumentation/httpclient.rb#L34-L36
1018
+ conn = Connection.new
1019
+ res = do_get_block(req, proxy, conn, &block)
1020
+ # Webmock's do_get_block returns ConditionVariable
1021
+ if !res.respond_to?(:previous)
1022
+ res = conn.pop
1023
+ end
885
1024
  end
886
- res = conn.pop
1025
+ res.previous = previous_response
887
1026
  break
888
- rescue RetryableResponse
889
- res = conn.pop
1027
+ rescue RetryableResponse => e
1028
+ previous_request = req
1029
+ previous_response = res = e.res
890
1030
  retry_count -= 1
891
1031
  end
892
1032
  end
@@ -914,8 +1054,8 @@ private
914
1054
  retry_count -= 1
915
1055
  end
916
1056
  end
917
- rescue Exception
918
- conn.push $!
1057
+ rescue Exception => e
1058
+ conn.push e
919
1059
  end
920
1060
  }
921
1061
  conn.async_thread = t
@@ -940,23 +1080,38 @@ private
940
1080
  ENV[name.downcase] || ENV[name.upcase]
941
1081
  end
942
1082
 
1083
+ def adapt_block(&block)
1084
+ return block if block.arity == 2
1085
+ proc { |r, str| block.call(str) }
1086
+ end
1087
+
943
1088
  def follow_redirect(method, uri, query, body, header, &block)
944
- uri = urify(uri)
1089
+ uri = to_resource_url(uri)
945
1090
  if block
1091
+ b = adapt_block(&block)
946
1092
  filtered_block = proc { |r, str|
947
- block.call(str) if r.ok?
1093
+ b.call(r, str) if r.ok?
948
1094
  }
949
1095
  end
950
1096
  if HTTP::Message.file?(body)
951
1097
  pos = body.pos rescue nil
952
1098
  end
953
1099
  retry_number = 0
1100
+ previous = nil
1101
+ request_query = query
954
1102
  while retry_number < @follow_redirect_count
955
1103
  body.pos = pos if pos
956
- res = do_request(method, uri, query, body, header, &filtered_block)
1104
+ res = do_request(method, uri, request_query, body, header, &filtered_block)
1105
+ res.previous = previous
957
1106
  if res.redirect?
1107
+ if res.header['location'].empty?
1108
+ raise BadResponseError.new("Missing Location header for redirect", res)
1109
+ end
958
1110
  method = :get if res.see_other? # See RFC2616 10.3.4
959
1111
  uri = urify(@redirect_uri_callback.call(uri, res))
1112
+ # To avoid duped query parameter. 'location' must include query part.
1113
+ request_query = nil
1114
+ previous = res
960
1115
  retry_number += 1
961
1116
  else
962
1117
  return res
@@ -976,25 +1131,28 @@ private
976
1131
  def protect_keep_alive_disconnected
977
1132
  begin
978
1133
  yield
979
- rescue KeepAliveDisconnected => e
980
- if e.sess
981
- @session_manager.invalidate(e.sess.dest)
1134
+ rescue KeepAliveDisconnected
1135
+ # Force to create new connection
1136
+ Thread.current[:HTTPClient_AcquireNewConnection] = true
1137
+ begin
1138
+ yield
1139
+ ensure
1140
+ Thread.current[:HTTPClient_AcquireNewConnection] = false
982
1141
  end
983
- yield
984
1142
  end
985
1143
  end
986
1144
 
987
1145
  def create_request(method, uri, query, body, header)
988
1146
  method = method.to_s.upcase
989
1147
  if header.is_a?(Hash)
990
- header = header.to_a
1148
+ header = @default_header.merge(header).to_a
991
1149
  else
992
- header = header.dup
1150
+ header = @default_header.to_a + header.dup
993
1151
  end
994
1152
  boundary = nil
995
1153
  if body
996
1154
  _, content_type = header.find { |key, value|
997
- key.downcase == 'content-type'
1155
+ key.to_s.downcase == 'content-type'
998
1156
  }
999
1157
  if content_type
1000
1158
  if /\Amultipart/ =~ content_type
@@ -1003,7 +1161,7 @@ private
1003
1161
  else
1004
1162
  boundary = create_boundary
1005
1163
  content_type = "#{content_type}; boundary=#{boundary}"
1006
- header = override_header(header, 'Content-Type', content_type)
1164
+ header = override_header(header, 'content-type', content_type)
1007
1165
  end
1008
1166
  end
1009
1167
  else
@@ -1020,8 +1178,11 @@ private
1020
1178
  header.each do |key, value|
1021
1179
  req.header.add(key.to_s, value)
1022
1180
  end
1023
- if @cookie_manager && cookie = @cookie_manager.find(uri)
1024
- req.header.add('Cookie', cookie)
1181
+ if @cookie_manager
1182
+ cookie_value = @cookie_manager.cookie_value(uri)
1183
+ if cookie_value
1184
+ req.header.add('Cookie', cookie_value)
1185
+ end
1025
1186
  end
1026
1187
  req
1027
1188
  end
@@ -1038,7 +1199,7 @@ private
1038
1199
  def override_header(header, key, value)
1039
1200
  result = []
1040
1201
  header.each do |k, v|
1041
- if k.downcase == key.downcase
1202
+ if k.to_s.downcase == key
1042
1203
  result << [key, value]
1043
1204
  else
1044
1205
  result << [k, v]
@@ -1071,8 +1232,9 @@ private
1071
1232
  end
1072
1233
  if str = @test_loopback_response.shift
1073
1234
  dump_dummy_request_response(req.http_body.dump, str) if @debug_dev
1074
- conn.push(HTTP::Message.new_response(str, req.header))
1075
- return
1235
+ res = HTTP::Message.new_response(str, req.header)
1236
+ conn.push(res)
1237
+ return res
1076
1238
  end
1077
1239
  content = block ? nil : ''
1078
1240
  res = HTTP::Message.new_response(content, req.header)
@@ -1085,7 +1247,7 @@ private
1085
1247
  sess.get_body do |part|
1086
1248
  set_encoding(part, res.body_encoding)
1087
1249
  if block
1088
- block.call(res, part)
1250
+ block.call(res, part.dup)
1089
1251
  else
1090
1252
  content << part
1091
1253
  end
@@ -1097,8 +1259,9 @@ private
1097
1259
  filter.filter_response(req, res)
1098
1260
  }
1099
1261
  if commands.find { |command| command == :retry }
1100
- raise RetryableResponse.new
1262
+ raise RetryableResponse.new(res)
1101
1263
  end
1264
+ res
1102
1265
  end
1103
1266
 
1104
1267
  def do_get_stream(req, proxy, conn)
@@ -1111,6 +1274,7 @@ private
1111
1274
  return
1112
1275
  end
1113
1276
  piper, pipew = IO.pipe
1277
+ pipew.binmode
1114
1278
  res = HTTP::Message.new_response(piper, req.header)
1115
1279
  @debug_dev << "= Request\n\n" if @debug_dev
1116
1280
  sess = @session_manager.query(req, proxy)
@@ -1128,6 +1292,7 @@ private
1128
1292
  filter.filter_response(req, res)
1129
1293
  }
1130
1294
  # ignore commands (not retryable in async mode)
1295
+ res
1131
1296
  end
1132
1297
 
1133
1298
  def do_get_header(req, res, sess)
@@ -1150,4 +1315,13 @@ private
1150
1315
  def set_encoding(str, encoding)
1151
1316
  str.force_encoding(encoding) if encoding
1152
1317
  end
1318
+
1319
+ def to_resource_url(uri)
1320
+ u = urify(uri)
1321
+ if @base_url && u.scheme.nil? && u.host.nil?
1322
+ urify(@base_url) + uri
1323
+ else
1324
+ u
1325
+ end
1326
+ end
1153
1327
  end