typhoeus 0.6.3 → 0.6.4

Sign up to get free protection for your applications and to get access to all the features.
Files changed (61) hide show
  1. checksums.yaml +15 -0
  2. data/.gitignore +8 -0
  3. data/.rspec +4 -0
  4. data/.travis.yml +10 -0
  5. data/CHANGELOG.md +7 -1
  6. data/CONTRIBUTING.md +16 -0
  7. data/Guardfile +9 -0
  8. data/README.md +303 -1
  9. data/UPGRADE.md +55 -0
  10. data/lib/rack/typhoeus/middleware/params_decoder.rb +1 -1
  11. data/lib/rack/typhoeus/middleware/params_decoder/helper.rb +6 -2
  12. data/lib/typhoeus/adapters/faraday.rb +4 -1
  13. data/lib/typhoeus/easy_factory.rb +36 -15
  14. data/lib/typhoeus/hydra/before.rb +2 -1
  15. data/lib/typhoeus/hydra/cacheable.rb +1 -0
  16. data/lib/typhoeus/hydra/memoizable.rb +1 -0
  17. data/lib/typhoeus/hydra/queueable.rb +15 -3
  18. data/lib/typhoeus/request.rb +6 -2
  19. data/lib/typhoeus/request/marshal.rb +3 -2
  20. data/lib/typhoeus/response/informations.rb +4 -0
  21. data/lib/typhoeus/version.rb +1 -1
  22. data/perf/profile.rb +14 -0
  23. data/perf/vs_nethttp.rb +64 -0
  24. data/spec/rack/typhoeus/middleware/params_decoder/helper_spec.rb +132 -0
  25. data/spec/rack/typhoeus/middleware/params_decoder_spec.rb +31 -0
  26. data/spec/spec_helper.rb +35 -0
  27. data/spec/support/localhost_server.rb +94 -0
  28. data/spec/support/server.rb +108 -0
  29. data/spec/typhoeus/adapters/faraday_spec.rb +245 -0
  30. data/spec/typhoeus/config_spec.rb +15 -0
  31. data/spec/typhoeus/easy_factory_spec.rb +96 -0
  32. data/spec/typhoeus/errors/no_stub_spec.rb +13 -0
  33. data/spec/typhoeus/expectation_spec.rb +273 -0
  34. data/spec/typhoeus/hydra/addable_spec.rb +22 -0
  35. data/spec/typhoeus/hydra/before_spec.rb +97 -0
  36. data/spec/typhoeus/hydra/block_connection_spec.rb +18 -0
  37. data/spec/typhoeus/hydra/cacheable_spec.rb +68 -0
  38. data/spec/typhoeus/hydra/memoizable_spec.rb +53 -0
  39. data/spec/typhoeus/hydra/queueable_spec.rb +34 -0
  40. data/spec/typhoeus/hydra/runnable_spec.rb +155 -0
  41. data/spec/typhoeus/hydra/stubbable_spec.rb +28 -0
  42. data/spec/typhoeus/hydra_spec.rb +26 -0
  43. data/spec/typhoeus/pool_spec.rb +79 -0
  44. data/spec/typhoeus/request/actions_spec.rb +19 -0
  45. data/spec/typhoeus/request/before_spec.rb +92 -0
  46. data/spec/typhoeus/request/block_connection_spec.rb +75 -0
  47. data/spec/typhoeus/request/cacheable_spec.rb +80 -0
  48. data/spec/typhoeus/request/callbacks_spec.rb +91 -0
  49. data/spec/typhoeus/request/marshal_spec.rb +62 -0
  50. data/spec/typhoeus/request/memoizable_spec.rb +34 -0
  51. data/spec/typhoeus/request/operations_spec.rb +70 -0
  52. data/spec/typhoeus/request/responseable_spec.rb +13 -0
  53. data/spec/typhoeus/request/stubbable_spec.rb +27 -0
  54. data/spec/typhoeus/request_spec.rb +131 -0
  55. data/spec/typhoeus/response/header_spec.rb +97 -0
  56. data/spec/typhoeus/response/informations_spec.rb +218 -0
  57. data/spec/typhoeus/response/status_spec.rb +218 -0
  58. data/spec/typhoeus/response_spec.rb +81 -0
  59. data/spec/typhoeus_spec.rb +105 -0
  60. data/typhoeus.gemspec +25 -0
  61. metadata +101 -27
checksums.yaml ADDED
@@ -0,0 +1,15 @@
1
+ ---
2
+ !binary "U0hBMQ==":
3
+ metadata.gz: !binary |-
4
+ MmQ5ZDMxYzRhNGY1MDRlYWFkZjc2YTJiNDVhY2RlZWQzYjE4ZGEzNw==
5
+ data.tar.gz: !binary |-
6
+ YmRjNWEyZWY0ZDM4YWEyZjE4ZjM3ZGEyNThhMzUzMjUyMGU2N2I4ZQ==
7
+ !binary "U0hBNTEy":
8
+ metadata.gz: !binary |-
9
+ NGY0ZWM1M2YwMGY2NTExNTljOTNkZTNiYjc1YTAwZjA5MzExZjM1MzY2MDUy
10
+ YjM0ZDk5Y2U4OGU1MGVhMmRhOTQzNTI0ZGY2OGI3ZjNmMWY5Yjc1MGVkMmZj
11
+ YTQxMjdkNWE4NmI0MTU2ZGVmOGU5NzY5MDZhNDRlZmExMzI1ZDU=
12
+ data.tar.gz: !binary |-
13
+ NTdmNmMzOTY2YTk1NGU2NWE3MTZlMzg3MmVlMjRlMDYzMmRiNzE5ZTE5MmNi
14
+ NTJiMjcxZGUwNzg5MGExODFiOTQ3ODhkMDJkYTM1MmUxMTYwODgyNzBhNzE4
15
+ NmYyZWRlYjVlODVmOTdmZjI3OTc4NmFiNDVlOTE3YjBjNWUxYzA=
data/.gitignore ADDED
@@ -0,0 +1,8 @@
1
+ *.gem
2
+ Gemfile.lock
3
+ doc/
4
+ .yardoc
5
+ .rvmrc
6
+ coverage
7
+ check.sh
8
+ tags
data/.rspec ADDED
@@ -0,0 +1,4 @@
1
+ --tty
2
+ --color
3
+ --format documentation
4
+ --backtrace
data/.travis.yml ADDED
@@ -0,0 +1,10 @@
1
+ script: "bundle exec rake"
2
+ rvm:
3
+ - 1.8.7
4
+ - 1.9.2
5
+ - 1.9.3
6
+ - jruby-head
7
+ - jruby-18mode
8
+ - jruby-19mode
9
+ - rbx-18mode
10
+ - rbx-19mode
data/CHANGELOG.md CHANGED
@@ -2,7 +2,13 @@
2
2
 
3
3
  ## Master
4
4
 
5
- [Full Changelog](http://github.com/typhoeus/typhoeus/compare/v0.6.3...master)
5
+ [Full Changelog](http://github.com/typhoeus/typhoeus/compare/v0.6.4...master)
6
+
7
+ ## Master
8
+
9
+ [Full Changelog](http://github.com/typhoeus/typhoeus/compare/v0.6.3...v0.6.4)
10
+
11
+ The changelog entries are coming soon!
6
12
 
7
13
  ## 0.6.3
8
14
 
data/CONTRIBUTING.md ADDED
@@ -0,0 +1,16 @@
1
+ We love pull requests. Here's a quick guide:
2
+
3
+ 1. Fork the repo.
4
+
5
+ 2. Run the tests. We only take pull requests with passing tests, and it's great
6
+ to know that you have a clean slate: `bundle && bundle exec rake`
7
+
8
+ 3. Add a test for your change. Only refactoring and documentation changes
9
+ require no new tests. If you are adding functionality or fixing a bug, we need
10
+ a test!
11
+
12
+ 4. Make the test pass.
13
+
14
+ 5. Push to your fork and submit a pull request.
15
+
16
+ And in case we didn't emphasize it enough: we love tests!
data/Guardfile ADDED
@@ -0,0 +1,9 @@
1
+ # vim:set filetype=ruby:
2
+ guard(
3
+ "rspec",
4
+ all_after_pass: false,
5
+ cli: "--fail-fast --tty --format documentation --colour") do
6
+
7
+ watch(%r{^spec/.+_spec\.rb$})
8
+ watch(%r{^lib/(.+)\.rb$}) { |match| "spec/#{match[1]}_spec.rb" }
9
+ end
data/README.md CHANGED
@@ -23,6 +23,7 @@ hydra.run
23
23
  ```
24
24
  gem install typhoeus
25
25
  ```
26
+
26
27
  ```
27
28
  gem "typhoeus"
28
29
  ```
@@ -30,9 +31,310 @@ gem "typhoeus"
30
31
  ## Project Tracking
31
32
 
32
33
  * [Documentation](http://rubydoc.info/github/typhoeus/typhoeus/frames/Typhoeus) (GitHub master)
33
- * [Website](http://typhoeus.github.com/) (v0.4.2)
34
34
  * [Mailing list](http://groups.google.com/group/typhoeus)
35
35
 
36
+ ## Usage
37
+
38
+ ### Introduction
39
+
40
+ The primary interface for Typhoeus is comprised of three classes: Request, Response, and Hydra. Request represents an HTTP request object, response represents an HTTP response, and Hydra manages making parallel HTTP connections.
41
+
42
+ ```ruby
43
+ request = Typhoeus::Request.new(
44
+ "www.example.com",
45
+ method: :post,
46
+ body: "this is a request body",
47
+ params: { field1: "a field" },
48
+ headers: { Accept: "text/html" }
49
+ )
50
+ ```
51
+
52
+ We can see from this that the first argument is the url. The second is a set of options.
53
+ The options are all optional. The default for `:method` is `:get`.
54
+
55
+ When you want to send URL parameters, you can use `:params` hash to do so. Please note that in case of you should send a request via `x-www-form-urlencoded` parameters, you need to use `:body` hash instead. `params` are for URL parameters and `:body` is for the request body.
56
+
57
+
58
+ You can run the query either on its own or through the hydra:
59
+
60
+ ``` ruby
61
+ request.run
62
+ #=> <Typhoeus::Response ... >
63
+ ```
64
+
65
+ ```ruby
66
+ hydra = Typhoeus::Hydra.hydra
67
+ hydra.queue(request)
68
+ hydra.run
69
+ ```
70
+
71
+ The response object will be set after the request is run.
72
+
73
+ ```ruby
74
+ response = request.response
75
+ response.code
76
+ response.total_time
77
+ response.headers_has
78
+ response.body
79
+ ```
80
+
81
+ ### Making Quick Requests
82
+
83
+ Typhoeus has some convenience methods for performing single HTTP requests. The arguments are the same as those you pass into the request constructor.
84
+
85
+ ```ruby
86
+ Typhoeus.get("www.example.com")
87
+ Typhoeus.head("www.example.com")
88
+ Typhoeus.put("www.example.com/posts/1", body: "whoo, a body")
89
+ Typhoeus.post("www.example.com/posts", body: { title: "test post", content: "this is my test"})
90
+ Typhoeus.delete("www.example.com/posts/1")
91
+ ```
92
+
93
+ ### Handling HTTP errors
94
+
95
+ You can query the response object to figure out if you had a successful
96
+ request or not. Here’s some example code that you might use to handle errors.
97
+ The callbacks are executed right after the request is finished, make sure to define
98
+ them before running the request.
99
+
100
+ ```ruby
101
+ request = Typhoeus::Request.new("www.example.com", followlocation: true)
102
+
103
+ request.on_complete do |response|
104
+ if response.success?
105
+ # hell yeah
106
+ elsif response.timed_out?
107
+ # aw hell no
108
+ log("got a time out")
109
+ elsif response.code == 0
110
+ # Could not get an http response, something's wrong.
111
+ log(response.return_message)
112
+ else
113
+ # Received a non-successful http response.
114
+ log("HTTP request failed: " + response.code.to_s)
115
+ end
116
+ end
117
+
118
+ request.run
119
+ ```
120
+
121
+ This also works with serial (blocking) requests in the same fashion. Both
122
+ serial and parallel requests return a Response object.
123
+
124
+ ### Handling file uploads
125
+
126
+ A File object can be passed as a param for a POST request to handle uploading
127
+ files to the server. Typhoeus will upload the file as the original file name
128
+ and use Mime::Types to set the content type.
129
+
130
+ ```ruby
131
+ Typhoeus.post(
132
+ "http://localhost:3000/posts",
133
+ body: {
134
+ title: "test post",
135
+ content: "this is my test",
136
+ file: File.open("thesis.txt","r")
137
+ }
138
+ )
139
+ ```
140
+
141
+ ### Making Parallel Requests
142
+
143
+ Generally, you should be running requests through hydra. Here is how that looks
144
+
145
+ ```ruby
146
+ hydra = Typhoeus::Hydra.hydra
147
+
148
+ first_request = Typhoeus::Request.new("www.example.com/posts/1.json")
149
+ first_request.on_complete do |response|
150
+ third_request = Typhoeus::Request.new(www.example.com/posts/3.json)
151
+ hydra.queue third_request
152
+ end
153
+ second_request = Typhoeus::Request.new("www.example.com/posts/2.json")
154
+
155
+ hydra.queue first_request
156
+ hydra.queue second_request
157
+ # this is a blocking call that returns once all requests are complete
158
+ hydra.run
159
+ ```
160
+
161
+ The execution of that code goes something like this. The first and second requests are built and queued. When hydra is run the first and second requests run in parallel. When the first request completes, the third request is then built and queued up. The moment it is queued Hydra starts executing it. Meanwhile the second request would continue to run (or it could have completed before the first). Once the third request is done, `hydra.run` returns.
162
+
163
+ ### Specifying Max Concurrency
164
+
165
+ Hydra will also handle how many requests you can make in parallel. Things will get flakey if you try to make too many requests at the same time. The built in limit is 200. When more requests than that are queued up, hydra will save them for later and start the requests as others are finished. You can raise or lower the concurrency limit through the Hydra constructor.
166
+
167
+ ```ruby
168
+ Typhoeus::Hydra.new(max_concurrency: 20)
169
+ ```
170
+
171
+ ### Memoization
172
+
173
+ Hydra memoizes requests within a single run call. You have to enable memoization.
174
+ This will result in a single request being issued. However, the on_complete handlers of both will be called.
175
+
176
+ ```ruby
177
+ Typhoeus::Config.memoize = true
178
+
179
+ hydra = Typhoeus::Hydra.new(max_concurrency: 1)
180
+ 2.times do
181
+ hydra.queue Typhoeus::Request.new("www.example.com")
182
+ end
183
+ hydra.run
184
+ ```
185
+
186
+ This will result in a two requests.
187
+
188
+ ```ruby
189
+ Typhoeus::Config.memoize = false
190
+
191
+ hydra = Typhoeus::Hydra.new(max_concurrency: 1)
192
+ 2.times do
193
+ hydra.queue Typhoeus::Request.new("www.example.com")
194
+ end
195
+ hydra.run
196
+ ```
197
+
198
+ ### Caching
199
+
200
+ Typhoeus includes built in support for caching. In the following example, if there is a cache hit, the cached object is passed to the on_complete handler of the request object.
201
+
202
+ ```ruby
203
+ class Cache
204
+ attr_accessor :memory
205
+
206
+ def initialize
207
+ @memory = {}
208
+ end
209
+
210
+ def get(request)
211
+ memory[request]
212
+ end
213
+
214
+ def set(request, reseponse)
215
+ memory[request] = reseponse
216
+ end
217
+ end
218
+
219
+ Typhoeus::Config.cache = Cache.new
220
+
221
+ Typhoeus.get("www.example.com") == Typhoeus.get("www.example.com")
222
+ #=> true
223
+ ```
224
+
225
+ ### Direct Stubbing
226
+
227
+ Hydra allows you to stub out specific urls and patterns to avoid hitting
228
+ remote servers while testing.
229
+
230
+ ```ruby
231
+ response = Typhoeus::Response.new(code: 200, body: "{'name' : 'paul'}")
232
+ Typhoeus.stub('www.example.com').and_return(response)
233
+
234
+ Typhoeus.get("www.example.com") == response
235
+ #=> true
236
+ ```
237
+
238
+ The queued request will hit the stub. You can also specify a regex to match urls.
239
+
240
+ ```ruby
241
+ response = Typhoeus::Response.new(code: 200, body: "{'name' : 'paul'}")
242
+ Typhoeus.stub(/example/).and_return(response)
243
+
244
+ Typhoeus.get("www.example.com") == response
245
+ #=> true
246
+ ```
247
+
248
+ ### Timeouts
249
+
250
+ No exceptions are raised on HTTP timeouts. You can check whether a request timed out with the following methods:
251
+
252
+ ```ruby
253
+ Typhoeus.get("www.example.com").timed_out?
254
+ ```
255
+
256
+ ### Following Redirections
257
+
258
+ Use `followlocation: true`, eg:
259
+
260
+ ```ruby
261
+ Typhoeus.get("www.example.com", followlocation: true)
262
+ ```
263
+
264
+ ### Basic Authentication
265
+
266
+ ```ruby
267
+ Typhoeus::Request.get("www.example.com", userpwd: "user:password")
268
+ ```
269
+
270
+ ### Cookies
271
+
272
+ ```ruby
273
+ Typhoeus::Request.get("www.example.com", cookiefile: "/path/to/file", cookiejar: "/path/to/file")
274
+ ```
275
+
276
+ Here, `cookiefile` is a file to read cookies to send from, and `cookiejar` is a file to write received cookies to.
277
+ So, if you just want cookies enabled, you need to pass in a same filename in both options.
278
+
279
+ ### Other CURL options
280
+
281
+ Are available and documented [here](http://rubydoc.info/github/typhoeus/ethon/Ethon/Easy/Options)
282
+
283
+ ### SSL
284
+
285
+ SSL comes built in to libcurl so it’s in Typhoeus as well. If you pass in a
286
+ url with "https" it should just work assuming that you have your [cert
287
+ bundle](http://curl.haxx.se/docs/caextract.html) in order and the server is
288
+ verifiable. You must also have libcurl built with SSL support enabled. You can
289
+ check that by doing this:
290
+
291
+ ```
292
+ curl --version
293
+ ```
294
+
295
+ Now, even if you have libcurl built with OpenSSL you may still have a messed
296
+ up cert bundle or if you’re hitting a non-verifiable SSL server then you’ll
297
+ have to disable peer verification to make SSL work. Like this:
298
+
299
+ ```ruby
300
+ Typhoeus.get("https://www.example.com", ssl_verifypeer: false)
301
+ ```
302
+
303
+ If you are getting "SSL: certificate subject name does not match target host
304
+ name" from curl (ex:- you are trying to access to b.c.host.com when the
305
+ certificate subject is \*.host.com). You can disable host verification. Like
306
+ this:
307
+
308
+ ```ruby
309
+ Typhoeus.get("https://www.example.com", ssl_verifyhost: 2)
310
+ ```
311
+
312
+ ### Verbose debug output
313
+
314
+ Sometime it’s useful to see verbose output from curl. You may now enable it on a per request basis:
315
+
316
+ ```ruby
317
+ Typhoeus.get("http://example.com", verbose: true)
318
+ ```
319
+
320
+ or global:
321
+
322
+ ```ruby
323
+ Typhoeus::Config.verbose = true
324
+ ```
325
+
326
+ Just remember that libcurl prints it’s debug output to the console (to
327
+ STDERR), so you’ll need to run your scripts from the console to see it.
328
+
329
+ ### Running the specs
330
+
331
+ Running the specs should be as easy as:
332
+
333
+ ```
334
+ bundle install
335
+ bundle exec rake
336
+ ```
337
+
36
338
  ## LICENSE
37
339
 
38
340
  (The MIT License)
data/UPGRADE.md ADDED
@@ -0,0 +1,55 @@
1
+ # Upgrade guide
2
+
3
+ ## 0.5
4
+
5
+ ### Options
6
+
7
+ Fix the option names, because some were renamed. The errors should point you in the right direction:
8
+
9
+ ```ruby
10
+ Typhoeus.get("www.example.com", follow_location: true)
11
+ # Ethon::Errors::InvalidOption: The option: follow_location is invalid.
12
+ # Please try followlocation instead of follow_location.
13
+ # ... [Backtrace]
14
+
15
+ Typhoeus.get("www.example.com", followlocation: true).code
16
+ #=> 200
17
+ ```
18
+
19
+ ### Headers
20
+
21
+ `Response#headers` returns a hash now and replaces `Response#headers_hash`, use `Response#response_headers` for the raw string:
22
+
23
+ ```ruby
24
+ Typhoeus.get("www.example.com", followlocation: true).headers
25
+ #=> {
26
+ # "date"=>"Tue, 06 Nov 2012 09:07:27 GMT",
27
+ # "server"=>"Apache/2.2.3 (CentOS)",
28
+ # "last-modified"=>"Wed, 09 Feb 2011 17:13:15 GMT",
29
+ # "vary"=>"Accept-Encoding",
30
+ # "connection"=>"close",
31
+ # "content-type"=>"text/html; charset=UTF-8"
32
+ # }
33
+
34
+ Typhoeus.get("www.example.com", followlocation: true).response_headers
35
+ #=> "HTTP/1.0 302 Found\r\nLocation: http://www.iana.org/domains/example/ [...]"
36
+ ```
37
+
38
+ ### Params vs body
39
+
40
+ Make sure every request sends proper params and body (especially POST/PUT). `:params` becomes url parameter and `:body` request body. Before params for POST was smashed into the body.
41
+
42
+ ### Configuration
43
+
44
+ Create a global configuration in case you want to turn on verbose, memoize or block_connection:
45
+
46
+ ```ruby
47
+ Typhoeus.configure do |config|
48
+ config.verbose = true
49
+ config.memoize = true
50
+ end
51
+ ```
52
+
53
+ ### Docs
54
+
55
+ When in doubt, read the [docs](http://rubydoc.info/github/typhoeus/typhoeus/frames/Typhoeus) or the [code](https://www.github.com/typhoeus).