restify 1.12.0 → 1.15.1

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: 19ee2486506eef195ed64b0a37ff921b97fefa46356e69d2fc3b28ca404ebf98
4
- data.tar.gz: 3b81f57ac5b84452dc9e9b6e1bb39705b601138ee5e5f9e2d1bb33d1107e286f
3
+ metadata.gz: 95d9cf12fd58e2f17bd1d3c046dfda23eaa38ec2fb99fe0c9b7e1034e014cd1f
4
+ data.tar.gz: b995cf5d935ba1c83b403fedf18d4c35154ea0ad60d4be0437fa98c89281d687
5
5
  SHA512:
6
- metadata.gz: e2e87e8c7e31b32d7da5f70321887a7ca28831a78e088fe3c934a08dc94481413d4a24c64eb8f765d60c7195a2d9830a11d11df6fa7c9c96ee5033ae26659f80
7
- data.tar.gz: 58697191071fa894417566779fcca6f74bb817ae6173bf9d7752b91af0b252d5ae4911fd29507b04fe6adf327e8b73cbf5109509fd9771078936259cb1911e46
6
+ metadata.gz: 96b5f22cc74ca967876f2ecc86bf30c70fa21ceee6f8726117862ad0e94a6319a9384673cd83a4222bd40b49f4b6eeb48cdec7bd5d1e94ac70fcdab787a8ed5b
7
+ data.tar.gz: 98199df1ecf1b9467b609ae57f58f2ae459f42f7121c1947763a9d49acb7224d6b8dc50088e74cd7f563719fb81f507d88fca2252a823856e57bbf995db5cbbe
data/CHANGELOG.md CHANGED
@@ -18,6 +18,42 @@ and this project adheres to [Semantic Versioning](http://semver.org/spec/v2.0.0.
18
18
  ### Breaks
19
19
 
20
20
 
21
+ ## 1.15.1 - (2021-07-15)
22
+ ---
23
+
24
+ ### Fixes
25
+ * Typhoeus internal exception when request timed out
26
+
27
+
28
+ ## 1.15.0 - (2021-07-09)
29
+ ---
30
+
31
+ ### New
32
+ * Improve memory usage when running lots of requests with typhoeus adapter
33
+ * Use hydra for synchronous requests
34
+ * Increased thread stability of typhoeus adapter (new internal queuing mechanism)
35
+
36
+ ### Changes
37
+ * Use Ruby 2.5 as baseline for testing and linting
38
+ * Add Ruby 3.0 to automated testing
39
+ * Changed timing behavior for multiple requests due to new internal queuing mechanism for the typhoeus adapter
40
+
41
+
42
+ ## 1.14.0 - (2020-12-15)
43
+ ---
44
+
45
+ ### New
46
+ * Allow making requests with non-JSON bodies and custom content types (#42)
47
+
48
+
49
+ ## 1.13.0 - (2020-06-12)
50
+ ---
51
+
52
+ ### New
53
+ * typhoeus: Support setting per-request libcurl options on adapter
54
+ * typhoeus: Enable short TCP keepalive probes by default (5s/5s)
55
+
56
+
21
57
  ## 1.12.0 - (2020-04-01)
22
58
  ---
23
59
 
data/README.md CHANGED
@@ -15,20 +15,17 @@ Restify is build upon the following libraries:
15
15
  * [addressable](https://github.com/sporkmonger/addressable)
16
16
  * [typhoeus](https://github.com/typhoeus/typhoeus)
17
17
 
18
- It has optional HTTP adapters using:
19
-
20
- * [em-http-request](https://github.com/igrigorik/em-http-request)
21
-
22
18
  The HTTP adapters are mostly run in a background thread and may not survive mid-application forks.
23
19
 
24
- Included processors can handle:
20
+ Restify includes processors to parse responses and to extract links between resources. The following formats are can be parsed:
25
21
 
26
- * Plain JSON with GitHub-style relations
27
- * MessagePack with GitHub-style relations *(currently experimental)*
22
+ * JSON
23
+ * MessagePack
28
24
 
29
- (Beside HTTP Link header that's always supported)
25
+ Links are extracted from
30
26
 
31
- Restify requires Ruby 2.0+.
27
+ * HTTP Link header
28
+ * Github-style relations in payloads
32
29
 
33
30
  ### Planned features
34
31
 
@@ -106,9 +103,9 @@ commit = repo.rel(:commits).get.value.first
106
103
  And print it:
107
104
 
108
105
  ```ruby
109
- puts "Last commit: #{commit[:sha]}"
110
- puts "By #{commit[:commit][:author][:name]} <#{commit[:commit][:author][:email]}>"
111
- puts "#{commit[:commit][:message]}"
106
+ puts "Last commit: #{commit['sha']}"
107
+ puts "By #{commit['commit']['author']['name']} <#{commit['commit']['author']['email']}>"
108
+ puts "#{commit['commit']['message']}"
112
109
  ```
113
110
 
114
111
  See commented example in main spec [`spec/restify_spec.rb`](https://github.com/jgraichen/restify/blob/master/spec/restify_spec.rb#L100) or in the `examples` directory.
@@ -63,8 +63,8 @@ module Restify
63
63
  query: request.uri.normalized_query,
64
64
  body: request.body,
65
65
  head: request.headers
66
- rescue Exception => err # rubocop:disable RescueException
67
- writer.reject err
66
+ rescue Exception => e # rubocop:disable Lint/RescueException
67
+ writer.reject e
68
68
  requests.shift unless pipeline?
69
69
  return
70
70
  end
@@ -128,12 +128,10 @@ module Restify
128
128
  return if EventMachine.reactor_running?
129
129
 
130
130
  Thread.new do
131
- begin
132
- EventMachine.run {}
133
- rescue StandardError => e
134
- puts "#{self.class} -> #{e}\n#{e.backtrace.join("\n")}"
135
- raise e
136
- end
131
+ EventMachine.run {}
132
+ rescue StandardError => e
133
+ puts "#{self.class} -> #{e}\n#{e.backtrace.join("\n")}"
134
+ raise e
137
135
  end
138
136
  end
139
137
  end
@@ -63,7 +63,7 @@ module Restify
63
63
  #
64
64
  def release(conn)
65
65
  @available.unshift(conn) if @available.size < @size
66
- @used -= 1 if @used > 0
66
+ @used -= 1 if @used.positive?
67
67
 
68
68
  logger.debug do
69
69
  "[#{conn.uri}] Released to pool (#{@available.size}/#{@used}/#{size})"
@@ -97,7 +97,7 @@ module Restify
97
97
  private
98
98
 
99
99
  def close(conn)
100
- @used -= 1 if @used > 0
100
+ @used -= 1 if @used.positive?
101
101
  @host[conn.uri.to_s] -= 1
102
102
 
103
103
  conn.close
@@ -200,9 +200,9 @@ module Restify
200
200
  @pool = Pool.new(**kwargs)
201
201
  end
202
202
 
203
- # rubocop:disable MethodLength
204
- # rubocop:disable AbcSize
205
- # rubocop:disable BlockLength
203
+ # rubocop:disable Metrics/MethodLength
204
+ # rubocop:disable Metrics/AbcSize
205
+ # rubocop:disable Metrics/BlockLength
206
206
  def call_native(request, writer)
207
207
  next_tick do
208
208
  defer = @pool.get(request)
@@ -212,39 +212,37 @@ module Restify
212
212
  end
213
213
 
214
214
  defer.callback do |conn|
215
- begin
216
- req = conn.send request.method.downcase,
217
- keepalive: true,
218
- redirects: 3,
219
- path: request.uri.normalized_path,
220
- query: request.uri.normalized_query,
221
- body: request.body,
222
- head: request.headers
223
-
224
- req.callback do
225
- writer.fulfill Response.new(
226
- request,
227
- req.last_effective_url,
228
- req.response_header.status,
229
- req.response_header,
230
- req.response
231
- )
232
-
233
- if req.response_header['CONNECTION'] == 'close'
234
- @pool.remove(conn)
235
- else
236
- @pool << conn
237
- end
238
- end
239
-
240
- req.errback do
215
+ req = conn.send request.method.downcase,
216
+ keepalive: true,
217
+ redirects: 3,
218
+ path: request.uri.normalized_path,
219
+ query: request.uri.normalized_query,
220
+ body: request.body,
221
+ head: request.headers
222
+
223
+ req.callback do
224
+ writer.fulfill Response.new(
225
+ request,
226
+ req.last_effective_url,
227
+ req.response_header.status,
228
+ req.response_header,
229
+ req.response
230
+ )
231
+
232
+ if req.response_header['CONNECTION'] == 'close'
241
233
  @pool.remove(conn)
242
- writer.reject(req.error)
234
+ else
235
+ @pool << conn
243
236
  end
244
- rescue Exception => ex # rubocop:disable RescueException
237
+ end
238
+
239
+ req.errback do
245
240
  @pool.remove(conn)
246
- writer.reject(ex)
241
+ writer.reject(req.error)
247
242
  end
243
+ rescue Exception => e # rubocop:disable Lint/RescueException
244
+ @pool.remove(conn)
245
+ writer.reject(e)
248
246
  end
249
247
  end
250
248
  end
@@ -261,12 +259,10 @@ module Restify
261
259
  return if EventMachine.reactor_running?
262
260
 
263
261
  Thread.new do
264
- begin
265
- EventMachine.run {}
266
- rescue StandardError => e
267
- logger.error(e)
268
- raise e
269
- end
262
+ EventMachine.run {}
263
+ rescue StandardError => e
264
+ logger.error(e)
265
+ raise e
270
266
  end
271
267
  end
272
268
  end
@@ -16,76 +16,85 @@ module Restify
16
16
  'Transfer-Encoding' => ''
17
17
  }.freeze
18
18
 
19
- def initialize(sync: false, **options)
20
- @sync = sync
21
- @hydra = ::Typhoeus::Hydra.new(**options)
22
- @mutex = Mutex.new
23
- @signal = ConditionVariable.new
24
- @thread = nil
19
+ DEFAULT_OPTIONS = {
20
+ followlocation: true,
21
+ tcp_keepalive: true,
22
+ tcp_keepidle: 5,
23
+ tcp_keepintvl: 5
24
+ }.freeze
25
+
26
+ def initialize(sync: false, options: {}, **kwargs)
27
+ @hydra = ::Typhoeus::Hydra.new(**kwargs)
28
+ @mutex = Mutex.new
29
+ @options = DEFAULT_OPTIONS.merge(options)
30
+ @queue = Queue.new
31
+ @sync = sync
32
+ @thread = nil
33
+
34
+ super()
25
35
  end
26
36
 
27
37
  def sync?
28
38
  @sync
29
39
  end
30
40
 
31
- # rubocop:disable AbcSize
32
- # rubocop:disable MethodLength
41
+ # rubocop:disable Metrics/MethodLength
33
42
  def call_native(request, writer)
34
43
  req = convert(request, writer)
35
44
 
36
45
  if sync?
37
- req.run
46
+ @hydra.queue(req)
47
+ @hydra.run
38
48
  else
39
- @mutex.synchronize do
40
- debug 'request:add',
41
- tag: request.object_id,
42
- method: request.method.upcase,
43
- url: request.uri
49
+ debug 'request:add',
50
+ tag: request.object_id,
51
+ method: request.method.upcase,
52
+ url: request.uri,
53
+ timeout: request.timeout
44
54
 
45
- @hydra.queue(req)
46
- @hydra.dequeue_many
55
+ @queue << convert(request, writer)
47
56
 
48
- thread.run unless thread.status
49
- end
50
-
51
- debug 'request:signal', tag: request.object_id
52
-
53
- @signal.signal
57
+ thread.run unless thread.status
54
58
  end
55
59
  end
56
- # rubocop:enable all
60
+ # rubocop:enable Metrics/MethodLength
57
61
 
58
62
  private
59
63
 
60
- # rubocop:disable AbcSize
61
- # rubocop:disable MethodLength
64
+ # rubocop:disable Metrics/AbcSize
65
+ # rubocop:disable Metrics/MethodLength
62
66
  def convert(request, writer)
63
67
  ::Typhoeus::Request.new(
64
68
  request.uri,
69
+ **@options,
65
70
  method: request.method,
66
71
  headers: DEFAULT_HEADERS.merge(request.headers),
67
72
  body: request.body,
68
- followlocation: true,
69
73
  timeout: request.timeout,
70
74
  connecttimeout: request.timeout
71
75
  ).tap do |req|
72
76
  req.on_complete do |response|
73
77
  debug 'request:complete',
74
78
  tag: request.object_id,
75
- status: response.code
79
+ status: response.code,
80
+ message: response.return_message,
81
+ timeout: response.timed_out?
76
82
 
77
- if response.timed_out?
78
- writer.reject Restify::Timeout.new request
79
- elsif response.code.zero?
83
+ if response.timed_out? || response.code.zero?
80
84
  writer.reject \
81
85
  Restify::NetworkError.new(request, response.return_message)
82
86
  else
83
87
  writer.fulfill convert_back(response, request)
84
88
  end
89
+
90
+ # Add all newly queued requests to active hydra, e.g. requests
91
+ # queued in a completion callback.
92
+ dequeue_all
85
93
  end
86
94
  end
87
95
  end
88
- # rubocop:enable all
96
+ # rubocop:enable Metrics/MethodLength
97
+ # rubocop:enable Metrics/AbcSize
89
98
 
90
99
  def convert_back(response, request)
91
100
  uri = request.uri
@@ -109,38 +118,51 @@ module Restify
109
118
  # Recreate thread if nil or dead
110
119
  debug 'hydra:spawn'
111
120
 
112
- @thread = Thread.new { _loop }
121
+ @thread = Thread.new do
122
+ Thread.current.name = 'Restify/Typhoeus Background'
123
+ run
124
+ end
113
125
  end
114
126
 
115
127
  @thread
116
128
  end
117
129
 
118
- def _loop
119
- Thread.current.name = 'Restify/Typhoeus Background'
120
- loop { _run }
121
- end
130
+ # rubocop:disable Metrics/MethodLength
131
+ def run
132
+ runs = 0
122
133
 
123
- def _ongoing?
124
- @hydra.queued_requests.any? || @hydra.multi.easy_handles.any?
125
- end
134
+ loop do
135
+ if @queue.empty? && runs > 100
136
+ debug 'hydra:gc'
137
+ GC.start(full_mark: false, immediate_sweep: false)
138
+ runs = 0
139
+ end
126
140
 
127
- # rubocop:disable MethodLength
128
- def _run
129
- debug 'hydra:run'
130
- @hydra.run while _ongoing?
131
- debug 'hydra:completed'
141
+ debug 'hydra:pop'
132
142
 
133
- @mutex.synchronize do
134
- return if _ongoing?
143
+ # Wait for next item and add all available requests to hydra
144
+ @hydra.queue @queue.pop
145
+ dequeue_all
146
+
147
+ debug 'hydra:run'
148
+ @hydra.run
149
+ runs += 1
150
+ debug 'hydra:completed'
151
+ rescue StandardError => e
152
+ logger.error(e)
153
+ end
154
+ ensure
155
+ debug 'hydra:exit'
156
+ end
157
+ # rubocop:enable Metrics/MethodLength
135
158
 
136
- debug 'hydra:pause'
137
- @signal.wait(@mutex, 60)
138
- debug 'hydra:resumed'
159
+ def dequeue_all
160
+ loop do
161
+ @hydra.queue @queue.pop(true)
162
+ rescue ThreadError
163
+ break
139
164
  end
140
- rescue StandardError => e
141
- logger.error(e)
142
165
  end
143
- # rubocop:enable all
144
166
 
145
167
  def _log_prefix
146
168
  "[#{object_id}/#{Thread.current.object_id}]"
@@ -36,7 +36,7 @@ module Restify
36
36
 
37
37
  def inherit(uri, **kwargs)
38
38
  uri ||= self.uri
39
- Context.new uri, kwargs.merge(options)
39
+ Context.new(uri, **kwargs, **options)
40
40
  end
41
41
 
42
42
  def process(response)
@@ -59,10 +59,10 @@ module Restify
59
59
 
60
60
  ret = cache.call(request) {|req| adapter.call(req) }
61
61
  ret.then do |response|
62
- if !response.errored?
63
- process response
64
- else
62
+ if response.errored?
65
63
  raise ResponseError.from_code(response)
64
+ else
65
+ process response
66
66
  end
67
67
  end
68
68
  end