typhoeus 1.7.1 → 1.8.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGELOG.md +69 -2
- data/README.md +77 -18
- data/UPGRADE.md +17 -0
- data/lib/typhoeus/adapters/faraday.rb +53 -2
- data/lib/typhoeus/config.rb +19 -0
- data/lib/typhoeus/hydra.rb +66 -4
- data/lib/typhoeus/multipart.rb +32 -0
- data/lib/typhoeus/pool.rb +86 -1
- data/lib/typhoeus/request.rb +9 -0
- data/lib/typhoeus/response/informations.rb +15 -0
- data/lib/typhoeus/version.rb +1 -1
- data/lib/typhoeus.rb +1 -0
- data/typhoeus.gemspec +1 -1
- metadata +8 -7
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: 8ab758d7e029f91567c9c57f9d81a2286e0bcd05ff8b0221f9e59251867ef774
|
|
4
|
+
data.tar.gz: f956be592758e62cfcd58ef6b4c12f988323e089ec374263a9da00858b15129f
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: 2b0326b3e22303e82ab66ffb6da46b54cd43c488913b5cc725bafbdb12351c23a4e56d33b4df6a5795255e0a84af479916ef6ceda74914eb9390aeb20c6d61d6
|
|
7
|
+
data.tar.gz: 1922d2d332ed0717394c3535f2d1683b054532dfb04bf3d3812356ca92ad1dfcefce52fac793634ffaf0189638b9efbc6ed56eb310dc6370f11aaa2101c302e0
|
data/CHANGELOG.md
CHANGED
|
@@ -1,8 +1,75 @@
|
|
|
1
1
|
# Changelog
|
|
2
2
|
|
|
3
|
-
##
|
|
3
|
+
## 1.8.0
|
|
4
4
|
|
|
5
|
-
[Full Changelog](http://github.com/typhoeus/typhoeus/compare/v1.7.1...
|
|
5
|
+
[Full Changelog](http://github.com/typhoeus/typhoeus/compare/v1.7.1...v1.8.0)
|
|
6
|
+
|
|
7
|
+
This release adds response streaming to the Faraday adapter, multipart parts
|
|
8
|
+
with their own content type, reusable multi handles, `Response#certinfo` and
|
|
9
|
+
fork-safe hydra memoization. It requires Ethon 0.19.2 or newer.
|
|
10
|
+
|
|
11
|
+
### Breaking changes
|
|
12
|
+
|
|
13
|
+
* Require Ethon >= 0.19.2. `Ethon::Multipart::Part` and the `certinfo` mirror
|
|
14
|
+
need it, and so does the fork opt-in the README describes. Ethon is still
|
|
15
|
+
constrained to `< 0.20`.
|
|
16
|
+
|
|
17
|
+
### Behavior changes
|
|
18
|
+
|
|
19
|
+
* Stream response bodies through Faraday's `on_data` callback. On Faraday
|
|
20
|
+
2.5+, the callback receives `env` as a third argument with the status and
|
|
21
|
+
headers already set, like with Faraday's `net_http` adapter. A request with
|
|
22
|
+
`on_data` set now gets an empty `response.body`; it used to get the full
|
|
23
|
+
body, the callback being ignored.
|
|
24
|
+
([#834](https://github.com/typhoeus/typhoeus/pull/834))
|
|
25
|
+
|
|
26
|
+
### Enhancements
|
|
27
|
+
|
|
28
|
+
* Add `Typhoeus::Multipart::Part` for setting the content type and, optionally,
|
|
29
|
+
the filename of a single `multipart/form-data` part. This makes it possible
|
|
30
|
+
to send a JSON field with `Content-Type: application/json` or to override the
|
|
31
|
+
detected content type and filename of an uploaded file.
|
|
32
|
+
([#494](https://github.com/typhoeus/typhoeus/issues/494))
|
|
33
|
+
|
|
34
|
+
* Add `Hydra#release`, which returns the native multi handle to a pool so the
|
|
35
|
+
next hydra reuses its connection cache instead of paying for a new TLS
|
|
36
|
+
handshake. `Typhoeus::Config.max_idle_multis` controls the pool (default 5,
|
|
37
|
+
`0` disables it). In a forked child, a hydra drops a handle it built before
|
|
38
|
+
the fork instead of pooling it, since Ethon rejects the parent's handle there.
|
|
39
|
+
([#832](https://github.com/typhoeus/typhoeus/issues/832))
|
|
40
|
+
|
|
41
|
+
* Add `Response#certinfo`, the server's certificate chain when the request is
|
|
42
|
+
made with `certinfo: true`.
|
|
43
|
+
([#427](https://github.com/typhoeus/typhoeus/issues/427))
|
|
44
|
+
|
|
45
|
+
* Return a new hydra from `Typhoeus::Hydra.hydra` in a forked child instead of
|
|
46
|
+
the one it memoized in the parent. The child no longer inherits requests
|
|
47
|
+
queued on that hydra, so the parent and child no longer both send them, and it
|
|
48
|
+
no longer uses the parent's multi handle, which Ethon 0.19 rejects in the
|
|
49
|
+
child. A server that forks workers after using libcurl can opt in with
|
|
50
|
+
`Ethon::Curl.allow_handles_after_fork!` in each worker before it creates a
|
|
51
|
+
handle (not supported on macOS).
|
|
52
|
+
([ethon#462](https://github.com/typhoeus/ethon/issues/462))
|
|
53
|
+
|
|
54
|
+
### Fixes and hardening
|
|
55
|
+
|
|
56
|
+
* Keep a discarded easy in the pool when its cleanup fails. A cleanup error
|
|
57
|
+
raised from an `on_complete` callback aborted the hydra run and left the
|
|
58
|
+
remaining requests without a response, which made CocoaPods' CDN source wait
|
|
59
|
+
forever.
|
|
60
|
+
([#825](https://github.com/typhoeus/typhoeus/issues/825))
|
|
61
|
+
|
|
62
|
+
### Documentation and maintenance
|
|
63
|
+
|
|
64
|
+
* Update the macOS fork advice for Ethon 0.19: initializing libcurl in the
|
|
65
|
+
parent now makes the child raise, so create handles after the fork or use
|
|
66
|
+
`fork` followed by `exec`.
|
|
67
|
+
([#836](https://github.com/typhoeus/typhoeus/pull/836))
|
|
68
|
+
* Run the suite against Faraday 1.x as well.
|
|
69
|
+
([#834](https://github.com/typhoeus/typhoeus/pull/834))
|
|
70
|
+
* Skip the memory-usage and leader-change specs on JRuby, where they cannot
|
|
71
|
+
assert the same thing.
|
|
72
|
+
([#831](https://github.com/typhoeus/typhoeus/pull/831))
|
|
6
73
|
|
|
7
74
|
## 1.7.1
|
|
8
75
|
|
data/README.md
CHANGED
|
@@ -41,7 +41,7 @@ Or install it directly:
|
|
|
41
41
|
gem install typhoeus
|
|
42
42
|
```
|
|
43
43
|
|
|
44
|
-
Typhoeus requires Ruby 2.6 or later and Ethon 0.19.
|
|
44
|
+
Typhoeus requires Ruby 2.6 or later and Ethon 0.19.2 or later.
|
|
45
45
|
|
|
46
46
|
## Documentation
|
|
47
47
|
|
|
@@ -208,6 +208,42 @@ Typhoeus.post(
|
|
|
208
208
|
)
|
|
209
209
|
```
|
|
210
210
|
|
|
211
|
+
Set the content type of a part, or override the filename and content type of a
|
|
212
|
+
file, with `Typhoeus::Multipart::Part`:
|
|
213
|
+
|
|
214
|
+
```ruby
|
|
215
|
+
Typhoeus.post(
|
|
216
|
+
"http://localhost:3000/posts",
|
|
217
|
+
body: {
|
|
218
|
+
metadata: Typhoeus::Multipart::Part.new(
|
|
219
|
+
JSON.generate(title: "test post"),
|
|
220
|
+
content_type: "application/json"
|
|
221
|
+
),
|
|
222
|
+
file: Typhoeus::Multipart::Part.new(
|
|
223
|
+
File.open("thesis.txt", "r"),
|
|
224
|
+
filename: "paper.txt",
|
|
225
|
+
content_type: "text/plain"
|
|
226
|
+
)
|
|
227
|
+
}
|
|
228
|
+
)
|
|
229
|
+
```
|
|
230
|
+
|
|
231
|
+
A string part with a filename is sent as an in-memory upload, so the data does
|
|
232
|
+
not need a file on disk:
|
|
233
|
+
|
|
234
|
+
```ruby
|
|
235
|
+
Typhoeus.post(
|
|
236
|
+
"http://localhost:3000/posts",
|
|
237
|
+
body: {
|
|
238
|
+
file: Typhoeus::Multipart::Part.new(
|
|
239
|
+
"file contents",
|
|
240
|
+
filename: "notes.txt",
|
|
241
|
+
content_type: "text/plain"
|
|
242
|
+
)
|
|
243
|
+
}
|
|
244
|
+
)
|
|
245
|
+
```
|
|
246
|
+
|
|
211
247
|
### Streaming the response body
|
|
212
248
|
|
|
213
249
|
Set `on_body` to receive response chunks as they arrive. This is useful for
|
|
@@ -309,6 +345,28 @@ end
|
|
|
309
345
|
hydra.run
|
|
310
346
|
```
|
|
311
347
|
|
|
348
|
+
#### Reusing connections between hydras
|
|
349
|
+
|
|
350
|
+
A hydra keeps libcurl's connection cache for as long as its native multi
|
|
351
|
+
handle lives, so a new `Hydra` starts with no open connections and pays for a
|
|
352
|
+
fresh TLS handshake. Returning the handle to the pool lets the next hydra
|
|
353
|
+
reuse it:
|
|
354
|
+
|
|
355
|
+
```ruby
|
|
356
|
+
hydra = Typhoeus::Hydra.new
|
|
357
|
+
hydra.queue(Typhoeus::Request.new("https://example.com"))
|
|
358
|
+
hydra.run
|
|
359
|
+
hydra.release # keeps the connection cache for the next hydra
|
|
360
|
+
|
|
361
|
+
# Takes the released handle, so the TLS connection is still there.
|
|
362
|
+
hydra = Typhoeus::Hydra.new
|
|
363
|
+
```
|
|
364
|
+
|
|
365
|
+
`release` does nothing while requests are still queued. A hydra created with
|
|
366
|
+
multi options, such as `maxconnects:`, is not pooled, because a later hydra
|
|
367
|
+
with different options cannot reconfigure the handle. `Typhoeus::Config.max_idle_multis`
|
|
368
|
+
(default 5) limits the pool; set it to `0` to disable multi pooling.
|
|
369
|
+
|
|
312
370
|
### Making Parallel Requests with Faraday + Typhoeus
|
|
313
371
|
|
|
314
372
|
Typhoeus ships a Faraday adapter that supports Faraday 1.x and 2.x.
|
|
@@ -336,6 +394,9 @@ response1.body
|
|
|
336
394
|
response2.body
|
|
337
395
|
```
|
|
338
396
|
|
|
397
|
+
A request with an `on_data` callback streams the body to it in chunks, and its
|
|
398
|
+
`response.body` is then empty.
|
|
399
|
+
|
|
339
400
|
### Specifying Max Concurrency
|
|
340
401
|
|
|
341
402
|
Hydra runs at most 200 requests concurrently by default. Additional queued
|
|
@@ -527,26 +588,24 @@ defaults to 300 seconds; passing 0 also uses 300 seconds.
|
|
|
527
588
|
|
|
528
589
|
Typhoeus delegates handle management to Ethon. Ethon raises
|
|
529
590
|
`Ethon::Errors::ForkSafety` for `Ethon::Easy` and `Ethon::Multi` handles
|
|
530
|
-
inherited across `fork`. Typhoeus clears its easy pool in the child process
|
|
531
|
-
|
|
532
|
-
|
|
591
|
+
inherited across `fork`. Typhoeus clears its easy pool in the child process,
|
|
592
|
+
and `Typhoeus::Hydra.hydra` replaces the hydra it memoized in the parent.
|
|
593
|
+
A hydra creates its native multi handle lazily, so creating or memoizing a
|
|
594
|
+
hydra during boot does not initialize libcurl. Typhoeus cannot reset libcurl's
|
|
533
595
|
process-global state. If libcurl is initialized before forking, for example by
|
|
534
596
|
making a request during boot, use `fork` followed by `exec`, or fork before the
|
|
535
|
-
first request.
|
|
597
|
+
first request. A server that forks workers from a process that has already
|
|
598
|
+
made requests, such as Pitchfork with reforking or Puma with `fork_worker`, can
|
|
599
|
+
instead call `Ethon::Curl.allow_handles_after_fork!` in each worker before its
|
|
600
|
+
first request, except on macOS, as Ethon's README describes.
|
|
536
601
|
|
|
537
602
|
On macOS, libcurl's global initialization can touch Objective-C and crash a
|
|
538
|
-
forked child with `objc_initializeAfterForkError`.
|
|
539
|
-
parent
|
|
540
|
-
|
|
541
|
-
|
|
542
|
-
|
|
543
|
-
|
|
544
|
-
Ethon::Curl.init
|
|
545
|
-
```
|
|
546
|
-
|
|
547
|
-
If you cannot initialize libcurl before forking, set
|
|
548
|
-
`OBJC_DISABLE_INITIALIZE_FORK_SAFETY=YES` in the environment. It skips the
|
|
549
|
-
safety check in the child, so prefer initializing libcurl before the fork.
|
|
603
|
+
forked child with `objc_initializeAfterForkError`. Initializing libcurl in the
|
|
604
|
+
parent doesn't avoid that: the child then raises `Ethon::Errors::ForkSafety`,
|
|
605
|
+
and `Ethon::Curl.allow_handles_after_fork!` raises there too. Create handles
|
|
606
|
+
only after the fork, and if the child crashes, set
|
|
607
|
+
`OBJC_DISABLE_INITIALIZE_FORK_SAFETY=YES` in the environment, which skips the
|
|
608
|
+
Objective-C check, or use `fork` followed by `exec`.
|
|
550
609
|
|
|
551
610
|
Typhoeus keeps at most `Typhoeus::Config.max_idle_easies` idle easy handles in
|
|
552
611
|
the pool (20 by default). Handles beyond that limit are cleaned up instead of
|
|
@@ -557,7 +616,7 @@ Ethon. Typhoeus releases the underlying easy handle before re-raising.
|
|
|
557
616
|
|
|
558
617
|
### Ethon 0.19
|
|
559
618
|
|
|
560
|
-
Typhoeus requires Ethon 0.19. Ethon 0.19 makes several previously
|
|
619
|
+
Typhoeus requires Ethon 0.19.2. Ethon 0.19 makes several previously
|
|
561
620
|
silent errors visible:
|
|
562
621
|
|
|
563
622
|
- Unknown enum and bitmask option values raise `Ethon::Errors::InvalidValue`,
|
data/UPGRADE.md
CHANGED
|
@@ -1,5 +1,22 @@
|
|
|
1
1
|
# Upgrade guide
|
|
2
2
|
|
|
3
|
+
## 1.8
|
|
4
|
+
|
|
5
|
+
### Ethon 0.19.2
|
|
6
|
+
|
|
7
|
+
Typhoeus 1.8 requires Ethon `>= 0.19.2`, `< 0.20.0`. Ethon 0.19.2 mirrors
|
|
8
|
+
`certinfo`, adds `Ethon::Multipart::Part`, and lets a forked child opt in to
|
|
9
|
+
creating new handles with `Ethon::Curl.allow_handles_after_fork!`. The macOS
|
|
10
|
+
restriction on that opt-in is covered in the README.
|
|
11
|
+
|
|
12
|
+
### Faraday `on_data`
|
|
13
|
+
|
|
14
|
+
A Faraday request that sets `on_data` now streams the response body to the
|
|
15
|
+
callback, like Faraday's `net_http` adapter, and its `response.body` is empty.
|
|
16
|
+
Before, Typhoeus buffered the body and ignored the callback. On Faraday 2.5+ the
|
|
17
|
+
callback receives the response `env`, with the status and headers already set,
|
|
18
|
+
as a third argument.
|
|
19
|
+
|
|
3
20
|
## 1.7
|
|
4
21
|
|
|
5
22
|
### Ethon 0.19
|
|
@@ -82,13 +82,62 @@ module Faraday # :nodoc:
|
|
|
82
82
|
configure_timeout req, env
|
|
83
83
|
configure_socket req, env
|
|
84
84
|
|
|
85
|
+
on_data = env[:request][:on_data]
|
|
86
|
+
on_data = nil unless on_data.respond_to?(:call)
|
|
87
|
+
# Faraday 2.5+ passes env, with status and headers set, as a third argument.
|
|
88
|
+
on_data_args = env.respond_to?(:stream_response) ? [env] : []
|
|
89
|
+
streamed_size = 0
|
|
90
|
+
streaming_error = nil
|
|
91
|
+
|
|
92
|
+
# Ethon aborts the transfer when a callback returns :abort, and turns an
|
|
93
|
+
# error raised in it into a write failure, which would be reported as
|
|
94
|
+
# Faraday::ConnectionFailed: keep the error to raise it as is. Ethon
|
|
95
|
+
# re-raises it only after on_complete, so this also rescues Interrupt
|
|
96
|
+
# and SystemExit, which would otherwise come out as a connection failure.
|
|
97
|
+
stream_chunk = lambda do |chunk, size|
|
|
98
|
+
on_data.call(chunk, size, *on_data_args)
|
|
99
|
+
nil
|
|
100
|
+
rescue Exception => exception
|
|
101
|
+
streaming_error ||= exception
|
|
102
|
+
:abort
|
|
103
|
+
end
|
|
104
|
+
|
|
105
|
+
if on_data
|
|
106
|
+
unless on_data_args.empty?
|
|
107
|
+
req.on_headers do |resp|
|
|
108
|
+
env.status = resp.code
|
|
109
|
+
env.reason_phrase = resp.status_message
|
|
110
|
+
env.response_headers = Faraday::Utils::Headers.new.tap do |headers|
|
|
111
|
+
headers.parse(resp.response_headers)
|
|
112
|
+
end
|
|
113
|
+
end
|
|
114
|
+
end
|
|
115
|
+
|
|
116
|
+
req.on_body do |chunk|
|
|
117
|
+
# A stubbed response without a body passes nil.
|
|
118
|
+
chunk = chunk.to_s
|
|
119
|
+
next if chunk.empty? && streamed_size.zero?
|
|
120
|
+
|
|
121
|
+
streamed_size += chunk.bytesize
|
|
122
|
+
stream_chunk.call(chunk, streamed_size)
|
|
123
|
+
end
|
|
124
|
+
end
|
|
125
|
+
|
|
85
126
|
req.on_complete do |resp|
|
|
127
|
+
failed = resp.timed_out? || (resp.response_code == 0) || ((resp.return_code != :ok) && !resp.mock?)
|
|
128
|
+
stream_chunk.call(+'', 0) if on_data && streamed_size.zero? && !failed
|
|
129
|
+
|
|
130
|
+
# In parallel mode, raising would leave the other requests unfinished:
|
|
131
|
+
# the write failure it caused is reported below like any other failure.
|
|
132
|
+
reported_as_failure = parallel?(env) && failed && streaming_error.is_a?(StandardError)
|
|
133
|
+
raise streaming_error if streaming_error && !reported_as_failure
|
|
134
|
+
|
|
86
135
|
if resp.timed_out?
|
|
87
136
|
env[:typhoeus_timed_out] = true
|
|
88
137
|
unless parallel?(env)
|
|
89
138
|
raise Faraday::TimeoutError, "request timed out"
|
|
90
139
|
end
|
|
91
|
-
elsif
|
|
140
|
+
elsif failed
|
|
92
141
|
env[:typhoeus_connection_failed] = true
|
|
93
142
|
env[:typhoeus_return_message] = resp.return_message
|
|
94
143
|
unless parallel?(env)
|
|
@@ -96,7 +145,9 @@ module Faraday # :nodoc:
|
|
|
96
145
|
end
|
|
97
146
|
end
|
|
98
147
|
|
|
99
|
-
|
|
148
|
+
# A stubbed response hands its whole body to on_body and keeps it.
|
|
149
|
+
body = on_data ? +'' : resp.body
|
|
150
|
+
save_response(env, resp.code, body, nil, resp.status_message) do |response_headers|
|
|
100
151
|
response_headers.parse resp.response_headers
|
|
101
152
|
end
|
|
102
153
|
# in async mode, :response is initialized at this point
|
data/lib/typhoeus/config.rb
CHANGED
|
@@ -125,5 +125,24 @@ module Typhoeus
|
|
|
125
125
|
end
|
|
126
126
|
|
|
127
127
|
@max_idle_easies = 20
|
|
128
|
+
|
|
129
|
+
# Defines the maximum number of idle multi handles kept in the
|
|
130
|
+
# process-wide pool. Defaults to 5. Set it to nil to keep every handle,
|
|
131
|
+
# or to 0 to disable multi pooling.
|
|
132
|
+
#
|
|
133
|
+
# @return [ Integer, nil ]
|
|
134
|
+
#
|
|
135
|
+
# @see Typhoeus::Pool
|
|
136
|
+
attr_reader :max_idle_multis
|
|
137
|
+
|
|
138
|
+
def max_idle_multis=(value)
|
|
139
|
+
if !value.nil? && (!value.is_a?(Integer) || value < 0)
|
|
140
|
+
raise ArgumentError, "max_idle_multis must be nil or a non-negative Integer"
|
|
141
|
+
end
|
|
142
|
+
|
|
143
|
+
@max_idle_multis = value
|
|
144
|
+
end
|
|
145
|
+
|
|
146
|
+
@max_idle_multis = 5
|
|
128
147
|
end
|
|
129
148
|
end
|
data/lib/typhoeus/hydra.rb
CHANGED
|
@@ -44,14 +44,29 @@ module Typhoeus
|
|
|
44
44
|
|
|
45
45
|
class << self
|
|
46
46
|
|
|
47
|
-
# Returns a memoized
|
|
47
|
+
# Returns a hydra memoized in Thread.current[:typhoeus_hydra], which is
|
|
48
|
+
# local to the current fiber. A forked child gets a new one, since
|
|
49
|
+
# Ethon 0.19 rejects the parent's multi handle.
|
|
48
50
|
#
|
|
49
51
|
# @example Get a hydra.
|
|
50
52
|
# Typhoeus::Hydra.hydra
|
|
51
53
|
#
|
|
52
|
-
# @return [Typhoeus::Hydra]
|
|
54
|
+
# @return [Typhoeus::Hydra] The memoized hydra.
|
|
53
55
|
def hydra
|
|
54
|
-
Thread.current[:typhoeus_hydra]
|
|
56
|
+
hydra = Thread.current[:typhoeus_hydra]
|
|
57
|
+
# Only replace a hydra memoized here, not one that other code stored.
|
|
58
|
+
# Ruby < 3.0 warns under -w when reading an ivar that was never set.
|
|
59
|
+
if hydra && hydra.instance_variable_defined?(:@memoized_pid)
|
|
60
|
+
pid = hydra.instance_variable_get(:@memoized_pid)
|
|
61
|
+
end
|
|
62
|
+
if !hydra || (pid && pid != Process.pid)
|
|
63
|
+
# Keep max_concurrency, which callers can change. This method sets
|
|
64
|
+
# no other option.
|
|
65
|
+
options = hydra ? { max_concurrency: hydra.max_concurrency } : {}
|
|
66
|
+
hydra = Thread.current[:typhoeus_hydra] = new(options)
|
|
67
|
+
hydra.instance_variable_set(:@memoized_pid, Process.pid)
|
|
68
|
+
end
|
|
69
|
+
hydra
|
|
55
70
|
end
|
|
56
71
|
end
|
|
57
72
|
|
|
@@ -91,6 +106,38 @@ module Typhoeus
|
|
|
91
106
|
@multi ||= build_multi
|
|
92
107
|
end
|
|
93
108
|
|
|
109
|
+
# Return the native multi handle and its connection cache to the pool, so
|
|
110
|
+
# a later hydra can reuse both. The hydra creates a new handle on next
|
|
111
|
+
# use. Does nothing while requests are still queued. In a forked child, a
|
|
112
|
+
# handle built before the fork is dropped instead.
|
|
113
|
+
#
|
|
114
|
+
# @example Release the handle after a run.
|
|
115
|
+
# hydra.run
|
|
116
|
+
# hydra.release
|
|
117
|
+
#
|
|
118
|
+
# @return [ nil ]
|
|
119
|
+
def release
|
|
120
|
+
return nil if queued_requests.any?
|
|
121
|
+
return nil unless defined?(@multi) && @multi
|
|
122
|
+
|
|
123
|
+
multi_handle = @multi
|
|
124
|
+
@multi = nil
|
|
125
|
+
# The parent's handle would raise for whichever hydra took it from the
|
|
126
|
+
# pool, and freeing it here raises too.
|
|
127
|
+
return nil unless defined?(@multi_pid) && @multi_pid == Process.pid
|
|
128
|
+
|
|
129
|
+
if multi_options.empty?
|
|
130
|
+
Pool.release_multi(multi_handle)
|
|
131
|
+
else
|
|
132
|
+
begin
|
|
133
|
+
multi_handle.cleanup
|
|
134
|
+
rescue StandardError
|
|
135
|
+
# A handle that cannot be freed must not raise from a cleanup path.
|
|
136
|
+
end
|
|
137
|
+
end
|
|
138
|
+
nil
|
|
139
|
+
end
|
|
140
|
+
|
|
94
141
|
# Add a request to the hydra.
|
|
95
142
|
#
|
|
96
143
|
# This is the hot path for every request. The checks are unrolled instead
|
|
@@ -414,10 +461,25 @@ module Typhoeus
|
|
|
414
461
|
# @api private
|
|
415
462
|
def build_multi
|
|
416
463
|
@multi_creation_failed = false
|
|
417
|
-
|
|
464
|
+
@multi_pid = Process.pid
|
|
465
|
+
if multi_options.empty?
|
|
466
|
+
# A hydra without multi options can take a pooled handle and reuse its
|
|
467
|
+
# connection cache. Handles created with options are not pooled,
|
|
468
|
+
# because a later hydra with different options cannot change them.
|
|
469
|
+
Pool.get_multi(multi_options)
|
|
470
|
+
else
|
|
471
|
+
Ethon::Multi.new(multi_options)
|
|
472
|
+
end
|
|
418
473
|
rescue StandardError
|
|
419
474
|
@multi_creation_failed = true
|
|
420
475
|
raise
|
|
421
476
|
end
|
|
477
|
+
|
|
478
|
+
# Options that belong to the Ethon multi rather than to the hydra.
|
|
479
|
+
#
|
|
480
|
+
# @return [ Hash ] The multi options.
|
|
481
|
+
def multi_options
|
|
482
|
+
@options.reject { |key, _| key == :max_concurrency }
|
|
483
|
+
end
|
|
422
484
|
end
|
|
423
485
|
end
|
|
@@ -0,0 +1,32 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
module Typhoeus
|
|
4
|
+
# Multipart form data support.
|
|
5
|
+
module Multipart
|
|
6
|
+
# A single part of a multipart form body.
|
|
7
|
+
#
|
|
8
|
+
# Wrap a body value to control the content type, and optionally the
|
|
9
|
+
# filename, that Typhoeus sends for that field.
|
|
10
|
+
#
|
|
11
|
+
# @example A JSON field.
|
|
12
|
+
# Typhoeus::Multipart::Part.new(
|
|
13
|
+
# JSON.generate(:a => 1),
|
|
14
|
+
# content_type: "application/json"
|
|
15
|
+
# )
|
|
16
|
+
#
|
|
17
|
+
# @example A file with an explicit filename and content type.
|
|
18
|
+
# Typhoeus::Multipart::Part.new(
|
|
19
|
+
# File.open("photo.jpg"),
|
|
20
|
+
# filename: "avatar.png",
|
|
21
|
+
# content_type: "image/png"
|
|
22
|
+
# )
|
|
23
|
+
#
|
|
24
|
+
# @example An in-memory upload.
|
|
25
|
+
# Typhoeus::Multipart::Part.new(
|
|
26
|
+
# "file contents",
|
|
27
|
+
# filename: "notes.txt",
|
|
28
|
+
# content_type: "text/plain"
|
|
29
|
+
# )
|
|
30
|
+
Part = Ethon::Multipart::Part
|
|
31
|
+
end
|
|
32
|
+
end
|
data/lib/typhoeus/pool.rb
CHANGED
|
@@ -40,7 +40,7 @@ module Typhoeus
|
|
|
40
40
|
easies << easy
|
|
41
41
|
end
|
|
42
42
|
end
|
|
43
|
-
discarded
|
|
43
|
+
cleanup_discarded(discarded) if discarded
|
|
44
44
|
rescue StandardError => e
|
|
45
45
|
# Ethon 0.19+ refuses to reset handles created before a fork.
|
|
46
46
|
raise unless defined?(Ethon::Errors::ForkSafety) && e.is_a?(Ethon::Errors::ForkSafety)
|
|
@@ -80,9 +80,66 @@ module Typhoeus
|
|
|
80
80
|
end
|
|
81
81
|
end
|
|
82
82
|
|
|
83
|
+
cleared_multis = @mutex.synchronize do
|
|
84
|
+
current = multis.dup
|
|
85
|
+
multis.clear
|
|
86
|
+
current
|
|
87
|
+
end
|
|
88
|
+
|
|
89
|
+
cleared_multis.each { |multi| cleanup_multi(multi) }
|
|
90
|
+
|
|
83
91
|
easies
|
|
84
92
|
end
|
|
85
93
|
|
|
94
|
+
# Pop a multi handle from the pool, or create one when the pool is empty.
|
|
95
|
+
#
|
|
96
|
+
# A pooled handle keeps the connection cache of the hydra that released
|
|
97
|
+
# it, so the next hydra reuses TLS connections instead of opening new
|
|
98
|
+
# ones.
|
|
99
|
+
#
|
|
100
|
+
# @example Get a multi.
|
|
101
|
+
# Typhoeus::Pool.get_multi
|
|
102
|
+
#
|
|
103
|
+
# @param [ Hash ] options Ethon::Multi options used when a new handle is
|
|
104
|
+
# created. Ignored for a pooled handle, which keeps its own options.
|
|
105
|
+
#
|
|
106
|
+
# @return [ Ethon::Multi ] The multi handle.
|
|
107
|
+
def self.get_multi(options = {})
|
|
108
|
+
reset_after_fork
|
|
109
|
+
@mutex.synchronize { multis.pop } || Ethon::Multi.new(options)
|
|
110
|
+
end
|
|
111
|
+
|
|
112
|
+
# Release a multi handle into the pool, or free it when the pool is full
|
|
113
|
+
# or disabled.
|
|
114
|
+
#
|
|
115
|
+
# @example Release a multi.
|
|
116
|
+
# Typhoeus::Pool.release_multi(multi)
|
|
117
|
+
#
|
|
118
|
+
# @param [ Ethon::Multi ] multi The multi handle.
|
|
119
|
+
#
|
|
120
|
+
# @return [ Ethon::Multi ] The released handle.
|
|
121
|
+
def self.release_multi(multi)
|
|
122
|
+
reset_after_fork
|
|
123
|
+
|
|
124
|
+
limit = max_multi_size
|
|
125
|
+
if limit == 0
|
|
126
|
+
cleanup_multi(multi)
|
|
127
|
+
return multi
|
|
128
|
+
end
|
|
129
|
+
|
|
130
|
+
discarded = nil
|
|
131
|
+
@mutex.synchronize do
|
|
132
|
+
if limit && multis.size >= limit
|
|
133
|
+
discarded = multi
|
|
134
|
+
else
|
|
135
|
+
multis << multi
|
|
136
|
+
end
|
|
137
|
+
end
|
|
138
|
+
|
|
139
|
+
cleanup_multi(discarded) if discarded
|
|
140
|
+
multi
|
|
141
|
+
end
|
|
142
|
+
|
|
86
143
|
# Use yielded easy, will be released automatically afterwards.
|
|
87
144
|
#
|
|
88
145
|
# @example Use easy.
|
|
@@ -108,6 +165,7 @@ module Typhoeus
|
|
|
108
165
|
# the new pid cannot pick up the inherited lock.
|
|
109
166
|
@mutex = Mutex.new
|
|
110
167
|
@easies = []
|
|
168
|
+
@multis = []
|
|
111
169
|
@pid = Process.pid
|
|
112
170
|
end
|
|
113
171
|
|
|
@@ -115,10 +173,37 @@ module Typhoeus
|
|
|
115
173
|
Typhoeus::Config.max_idle_easies if defined?(Typhoeus::Config)
|
|
116
174
|
end
|
|
117
175
|
|
|
176
|
+
# Free an easy that does not fit into the pool. Some Ethon versions reject
|
|
177
|
+
# cleanup from an on_complete callback, which is where Typhoeus releases
|
|
178
|
+
# the easy. Raising there aborts the hydra run and leaves every other
|
|
179
|
+
# request without a response, so keep the reset handle pooled instead. The
|
|
180
|
+
# pool may temporarily exceed max_idle_easies; a later release or
|
|
181
|
+
# Pool.clear frees the handle.
|
|
182
|
+
def self.cleanup_discarded(easy)
|
|
183
|
+
easy.cleanup
|
|
184
|
+
rescue Ethon::Errors::EthonError
|
|
185
|
+
@mutex.synchronize { easies << easy }
|
|
186
|
+
end
|
|
187
|
+
|
|
118
188
|
def self.easies
|
|
119
189
|
@easies ||= []
|
|
120
190
|
end
|
|
121
191
|
|
|
192
|
+
def self.multis
|
|
193
|
+
@multis ||= []
|
|
194
|
+
end
|
|
195
|
+
|
|
196
|
+
def self.max_multi_size
|
|
197
|
+
Typhoeus::Config.max_idle_multis if defined?(Typhoeus::Config)
|
|
198
|
+
end
|
|
199
|
+
|
|
200
|
+
def self.cleanup_multi(multi)
|
|
201
|
+
multi.cleanup
|
|
202
|
+
rescue StandardError
|
|
203
|
+
# Freeing a handle is best effort. A handle inherited across a fork is
|
|
204
|
+
# dropped instead of freed.
|
|
205
|
+
end
|
|
206
|
+
|
|
122
207
|
# Flush cookies to the configured jar and clear them from the handle.
|
|
123
208
|
# libcurl builds without cookie support reject CURLOPT_COOKIELIST; that
|
|
124
209
|
# must not prevent the handle from being recycled.
|
data/lib/typhoeus/request.rb
CHANGED
|
@@ -465,6 +465,15 @@ module Typhoeus
|
|
|
465
465
|
when Array
|
|
466
466
|
values = obj.map { |value| hashable_string_for(value) }
|
|
467
467
|
"array:#{obj.size}:#{values.join}"
|
|
468
|
+
when Ethon::Multipart::Part
|
|
469
|
+
# A part's metadata changes what is sent, so it belongs in the key
|
|
470
|
+
# even though the value alone would be enough for a regular field.
|
|
471
|
+
# Files are keyed by path because File#to_s contains the object
|
|
472
|
+
# address, which would give every upload its own cache entry.
|
|
473
|
+
value = obj.file? ? obj.value.path : obj.value
|
|
474
|
+
"part:#{hashable_string_for(obj.content_type)}:" \
|
|
475
|
+
"#{hashable_string_for(obj.filename)}:" \
|
|
476
|
+
"#{hashable_string_for(value)}"
|
|
468
477
|
else
|
|
469
478
|
value = obj.to_s
|
|
470
479
|
"#{obj.class}:#{value.bytesize}:#{value}"
|
|
@@ -241,6 +241,21 @@ module Typhoeus
|
|
|
241
241
|
options[:redirect_url]
|
|
242
242
|
end
|
|
243
243
|
|
|
244
|
+
# Return the server's certificate chain, when the request was made with
|
|
245
|
+
# `certinfo: true` over TLS. Each certificate is an Array of
|
|
246
|
+
# "Name:value" Strings, as libcurl reports them. libcurl only collects
|
|
247
|
+
# the chain during a TLS handshake, so a request that reuses a
|
|
248
|
+
# connection gets an empty Array unless it also sets `fresh_connect`.
|
|
249
|
+
#
|
|
250
|
+
# @example Get certinfo.
|
|
251
|
+
# response.certinfo
|
|
252
|
+
#
|
|
253
|
+
# @return [ Array<Array<String>>, nil ] The certificate chain, empty
|
|
254
|
+
# when none was collected, nil for a stubbed response.
|
|
255
|
+
def certinfo
|
|
256
|
+
options[:certinfo]
|
|
257
|
+
end
|
|
258
|
+
|
|
244
259
|
def request_size
|
|
245
260
|
options[:request_size]
|
|
246
261
|
end
|
data/lib/typhoeus/version.rb
CHANGED
data/lib/typhoeus.rb
CHANGED
data/typhoeus.gemspec
CHANGED
|
@@ -25,7 +25,7 @@ Gem::Specification.new do |s|
|
|
|
25
25
|
'source_code_uri' => "https://github.com/typhoeus/typhoeus/tree/v#{s.version}"
|
|
26
26
|
}
|
|
27
27
|
|
|
28
|
-
s.add_dependency('ethon', [">= 0.19.
|
|
28
|
+
s.add_dependency('ethon', [">= 0.19.2", "< 0.20.0"])
|
|
29
29
|
|
|
30
30
|
s.files = Dir.chdir(__dir__) do
|
|
31
31
|
files = begin
|
metadata
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
|
2
2
|
name: typhoeus
|
|
3
3
|
version: !ruby/object:Gem::Version
|
|
4
|
-
version: 1.
|
|
4
|
+
version: 1.8.0
|
|
5
5
|
platform: ruby
|
|
6
6
|
authors:
|
|
7
7
|
- David Balatero
|
|
@@ -10,7 +10,7 @@ authors:
|
|
|
10
10
|
autorequire:
|
|
11
11
|
bindir: bin
|
|
12
12
|
cert_chain: []
|
|
13
|
-
date: 2026-
|
|
13
|
+
date: 2026-11-20 00:00:00.000000000 Z
|
|
14
14
|
dependencies:
|
|
15
15
|
- !ruby/object:Gem::Dependency
|
|
16
16
|
name: ethon
|
|
@@ -18,7 +18,7 @@ dependencies:
|
|
|
18
18
|
requirements:
|
|
19
19
|
- - ">="
|
|
20
20
|
- !ruby/object:Gem::Version
|
|
21
|
-
version: 0.19.
|
|
21
|
+
version: 0.19.2
|
|
22
22
|
- - "<"
|
|
23
23
|
- !ruby/object:Gem::Version
|
|
24
24
|
version: 0.20.0
|
|
@@ -28,7 +28,7 @@ dependencies:
|
|
|
28
28
|
requirements:
|
|
29
29
|
- - ">="
|
|
30
30
|
- !ruby/object:Gem::Version
|
|
31
|
-
version: 0.19.
|
|
31
|
+
version: 0.19.2
|
|
32
32
|
- - "<"
|
|
33
33
|
- !ruby/object:Gem::Version
|
|
34
34
|
version: 0.20.0
|
|
@@ -63,6 +63,7 @@ files:
|
|
|
63
63
|
- lib/typhoeus/expectation.rb
|
|
64
64
|
- lib/typhoeus/hydra.rb
|
|
65
65
|
- lib/typhoeus/hydra/queueable.rb
|
|
66
|
+
- lib/typhoeus/multipart.rb
|
|
66
67
|
- lib/typhoeus/pool.rb
|
|
67
68
|
- lib/typhoeus/railtie.rb
|
|
68
69
|
- lib/typhoeus/request.rb
|
|
@@ -82,10 +83,10 @@ licenses:
|
|
|
82
83
|
- MIT
|
|
83
84
|
metadata:
|
|
84
85
|
bug_tracker_uri: https://github.com/typhoeus/typhoeus/issues
|
|
85
|
-
changelog_uri: https://github.com/typhoeus/typhoeus/blob/v1.
|
|
86
|
-
documentation_uri: https://www.rubydoc.info/gems/typhoeus/1.
|
|
86
|
+
changelog_uri: https://github.com/typhoeus/typhoeus/blob/v1.8.0/CHANGELOG.md
|
|
87
|
+
documentation_uri: https://www.rubydoc.info/gems/typhoeus/1.8.0
|
|
87
88
|
rubygems_mfa_required: 'true'
|
|
88
|
-
source_code_uri: https://github.com/typhoeus/typhoeus/tree/v1.
|
|
89
|
+
source_code_uri: https://github.com/typhoeus/typhoeus/tree/v1.8.0
|
|
89
90
|
post_install_message:
|
|
90
91
|
rdoc_options: []
|
|
91
92
|
require_paths:
|