px-service-client 1.2.3 → 2.0.0
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +4 -4
- data/README.md +35 -32
- data/lib/px/service/client/base.rb +56 -1
- data/lib/px/service/client/caching.rb +39 -36
- data/lib/px/service/client/circuit_breaker.rb +11 -4
- data/lib/px/service/client/hmac_signing.rb +22 -30
- data/lib/px/service/client/null_statsd_client.rb +30 -0
- data/lib/px/service/client/version.rb +1 -1
- data/lib/px/service/client.rb +1 -0
- data/spec/px/service/client/base_spec.rb +27 -3
- data/spec/px/service/client/caching/caching_spec.rb +8 -6
- data/spec/px/service/client/hmac_signing_spec.rb +10 -8
- metadata +3 -2
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
2
|
SHA1:
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
3
|
+
metadata.gz: 9a10d1168c3cd74e1fb75eb062481a9aa8dfa25d
|
4
|
+
data.tar.gz: 81dd905f9e51020552ed8c655c11678e8fe5fdc5
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: 53fce4eecc310c99c01ccf635de6616d46dc61edcfb40362657caf28390a99a3b6b51dc502812dbb5c4e006cd90d735295d812bfee4903452ceb2f4fe1069e5e
|
7
|
+
data.tar.gz: 43d57237b4ab1e3b8f2882baa419ad14fb784cce1f270690d11fcd72268ff046bb1b012c4b96236b08546666a5525e8817671e4b5a6f837bd2a2549eaf122833
|
data/README.md
CHANGED
@@ -38,14 +38,20 @@ This gem includes several common features used in 500px service client libraries
|
|
38
38
|
The features are:
|
39
39
|
|
40
40
|
#### Px::Service::Client::Base
|
41
|
-
This class provides a basic `make_request(method, url, ...)` method that produces an asynchronous request. The method immediately returns a `Future`. It works together with `Multiplexer`(discussed below) and uses [Typhoeus](https://github.com/typhoeus/typhoeus) as the underlying HTTP client to support asynchronicity.
|
41
|
+
This class provides a basic `make_request(method, url, ...)` method that produces an asynchronous request. The method immediately returns a `Future`. It works together with `Multiplexer`(discussed below) and uses [Typhoeus](https://github.com/typhoeus/typhoeus) as the underlying HTTP client to support asynchronicity.
|
42
42
|
|
43
|
-
**
|
43
|
+
**Clients should subclass this class and include other features/mixins, if needed.**
|
44
|
+
|
45
|
+
# Optional
|
46
|
+
config do |config|
|
47
|
+
config.statsd_client = Statsd.new(host, port)
|
48
|
+
end
|
44
49
|
|
45
|
-
|
50
|
+
|
51
|
+
See the following section for an example of how to use `make_request` and `Multiplexer`.
|
46
52
|
|
47
53
|
#### Px::Service::Client::Multiplexer
|
48
|
-
This class works together with `Px::Service::Client::Base` sub-classes to support request parallel execution.
|
54
|
+
This class works together with `Px::Service::Client::Base` sub-classes to support request parallel execution.
|
49
55
|
|
50
56
|
Example:
|
51
57
|
|
@@ -62,9 +68,9 @@ end
|
|
62
68
|
multi.run # a blocking call, like hydra.run
|
63
69
|
|
64
70
|
```
|
65
|
-
`multi.context` encapsulates the block into a [`Fiber`](http://ruby-doc.org/core-2.2.0/Fiber.html) object and immediately runs (or `resume`, in Fiber's term) that fiber until the block explicitly gives up control. The method returns `multi` itself.
|
71
|
+
`multi.context` encapsulates the block into a [`Fiber`](http://ruby-doc.org/core-2.2.0/Fiber.html) object and immediately runs (or `resume`, in Fiber's term) that fiber until the block explicitly gives up control. The method returns `multi` itself.
|
66
72
|
|
67
|
-
`multi.do(request_or_future,retries)` queues the request into `hydra`. It always returns a `Future`. A [`Typhoeus::Request`](https://github.com/typhoeus/typhoeus) will be converted into a `Future ` in this call.
|
73
|
+
`multi.do(request_or_future,retries)` queues the request into `hydra`. It always returns a `Future`. A [`Typhoeus::Request`](https://github.com/typhoeus/typhoeus) will be converted into a `Future ` in this call.
|
68
74
|
|
69
75
|
Finally, `multi.run` starts `hydra` to execute the requests in parallel. The request is made as soon as the multiplexer is started. You get the results of the request by evaluating the value of the `Future`.
|
70
76
|
|
@@ -76,21 +82,18 @@ Provides client-side response caching of service requests.
|
|
76
82
|
include Px::Service::Client::Caching
|
77
83
|
|
78
84
|
# Optional
|
79
|
-
|
80
|
-
config.cache_strategy = :none
|
85
|
+
config do |config|
|
81
86
|
config.cache_expiry = 30.seconds
|
82
|
-
config.
|
83
|
-
config.cache_options = {}
|
84
|
-
config.cache_options[:policy_group] = 'general'
|
87
|
+
config.cache_default_policy_group = 'general'
|
85
88
|
config.cache_client = Dalli::Client.new(...)
|
86
|
-
config.cache_logger = Logger.new(STDOUT) # or Rails.logger, for example
|
89
|
+
config.cache_logger = Logger.new(STDOUT) # or Rails.logger, for example. Can be nil.
|
87
90
|
end
|
88
91
|
|
89
92
|
# An example of a cached request
|
90
93
|
result = cache_request(url, :last_resort, refresh_probability: 1) do
|
91
94
|
req = make_request(method, url)
|
92
95
|
response = @multi.do(req)
|
93
|
-
|
96
|
+
|
94
97
|
# cache_request() expects a future that returns the result to be cached
|
95
98
|
Px::Service::Client::Future.new do
|
96
99
|
JSON.parse(response.body)
|
@@ -98,7 +101,7 @@ result = cache_request(url, :last_resort, refresh_probability: 1) do
|
|
98
101
|
end
|
99
102
|
```
|
100
103
|
|
101
|
-
`cache_request` expects a block that returns a `Future` object. The return value (usually the response body) of that future will be cached. `cache_request` always returns a future. By evaluating the future, i.e., via the `Future.value!` call, you get the result (whether cached or not).
|
104
|
+
`cache_request` expects a block that returns a `Future` object. The return value (usually the response body) of that future will be cached. `cache_request` always returns a future. By evaluating the future, i.e., via the `Future.value!` call, you get the result (whether cached or not).
|
102
105
|
|
103
106
|
|
104
107
|
**Note**: DO NOT cache the `Typhoeus::Response` directly (See the below code snippet), because the response object cannot be serializable to be stored in memcached. That's the reason why we see warning message: `You are trying to cache a Ruby object which cannot be serialized to memcached.`
|
@@ -107,24 +110,24 @@ end
|
|
107
110
|
# An incorrect example of using cache_request()
|
108
111
|
cache_request(url, :last_resort) do
|
109
112
|
req = make_request(method, url)
|
110
|
-
response = @multi.do(req) # DO NOT do this
|
113
|
+
response = @multi.do(req) # DO NOT do this
|
111
114
|
end
|
112
115
|
|
113
|
-
```
|
116
|
+
```
|
114
117
|
Responses are cached in either a *last-resort* or *first-resort* manner.
|
115
118
|
|
116
119
|
*last-resort* means that the cached value is only used when the service client request fails (with a
|
117
120
|
`ServiceError`). If the service client request succeeds, there is a chance that the cache value may get refreshed. The `refresh_probability` is provided to let the cached value
|
118
|
-
be refreshed probabilistically (rather than on every request).
|
121
|
+
be refreshed probabilistically (rather than on every request).
|
119
122
|
|
120
|
-
If the service client request fails and there is a `ServiceError`, `cache_logger` will record the exception message, and attempt to read the existing cache value.
|
123
|
+
If the service client request fails and there is a `ServiceError`, `cache_logger` will record the exception message, and attempt to read the existing cache value.
|
121
124
|
|
122
|
-
*first-resort* means that the cached value is always used, if present. If the cached value is present but expired, the it sends the service client request and, if the request succeeds, it refreshes the cached value expiry. If the request fails, it uses the expired cached value, but the value remain expired. A retry may be needed.
|
125
|
+
*first-resort* means that the cached value is always used, if present. If the cached value is present but expired, the it sends the service client request and, if the request succeeds, it refreshes the cached value expiry. If the request fails, it uses the expired cached value, but the value remain expired. A retry may be needed.
|
123
126
|
|
124
127
|
|
125
128
|
|
126
129
|
#### Px::Service::Client::CircuitBreaker
|
127
|
-
This mixin overrides `Px::Service::Client::Base#make_request` method and implements the circuit breaker pattern.
|
130
|
+
This mixin overrides `Px::Service::Client::Base#make_request` method and implements the circuit breaker pattern.
|
128
131
|
|
129
132
|
```ruby
|
130
133
|
include Px::Service::Client::CircuitBreaker
|
@@ -146,7 +149,7 @@ Adds a circuit breaker to the client. `make_request` always returns `Future`
|
|
146
149
|
|
147
150
|
The circuit will open on any exception from the wrapped method, or if the request runs for longer than the `invocation_timeout`.
|
148
151
|
|
149
|
-
If the circuit is open, any future request will be get an error message wrapped in `Px::Service::ServiceError`.
|
152
|
+
If the circuit is open, any future request will be get an error message wrapped in `Px::Service::ServiceError`.
|
150
153
|
|
151
154
|
By default, `Px::Service::ServiceRequestError` is excluded by the handler. That is, when the request fails with a `ServiceRequestError` exceptions, the same `ServiceRequestError` will be raised. But it does NOT increase the failure count or trip the breaker, as these exceptions indicate an error on the caller's part (e.g. an HTTP 4xx error).
|
152
155
|
|
@@ -156,37 +159,37 @@ Every instance of the class that includes the `CircuitBreaker` concern will shar
|
|
156
159
|
This module is based on (and uses) the [Circuit Breaker](https://github.com/wsargent/circuit_breaker) gem by Will Sargent.
|
157
160
|
|
158
161
|
#### Px::Service::Client::HmacSigning
|
159
|
-
Similar to `Px::Service::Client::CircuitBreaker`, this mixin overrides `Px::Service::Client::Base#make_request` method and appends a HMAC signature in the request header.
|
162
|
+
Similar to `Px::Service::Client::CircuitBreaker`, this mixin overrides `Px::Service::Client::Base#make_request` method and appends a HMAC signature in the request header.
|
160
163
|
|
161
|
-
To use this mixin:
|
164
|
+
To use this mixin:
|
162
165
|
|
163
166
|
```ruby
|
164
167
|
class MyClient < Px::Service::Client::Base
|
165
168
|
include Px::Service::Client::HmacSigning
|
166
169
|
|
167
170
|
#optional
|
168
|
-
|
169
|
-
config.
|
170
|
-
config.
|
171
|
+
config do |config|
|
172
|
+
config.hmac_secret = 'mykey'
|
173
|
+
config.hmac_keyspan = 300
|
171
174
|
end
|
172
175
|
end
|
173
176
|
```
|
174
177
|
|
175
|
-
Note: `key` and `keyspan` are class variables and shared among instances of the same class.
|
178
|
+
Note: `key` and `keyspan` are class variables and shared among instances of the same class.
|
176
179
|
|
177
|
-
The signature is produced from the secret key, a nonce, HTTP method, url, query, body. The nonce is generated from the timestamp.
|
180
|
+
The signature is produced from the secret key, a nonce, HTTP method, url, query, body. The nonce is generated from the timestamp.
|
178
181
|
|
179
|
-
To retrieve and verify the signature:
|
182
|
+
To retrieve and verify the signature:
|
180
183
|
|
181
184
|
```ruby
|
182
185
|
# Make a request with signed headers
|
183
|
-
resp = make_request(method, url, query, headers, body)
|
186
|
+
resp = make_request(method, url, query, headers, body)
|
184
187
|
|
185
188
|
signature = resp.request.options[:headers]["X-Service-Auth"]
|
186
189
|
timestamp = resp.request.options[:headers]["Timestamp"]
|
187
190
|
|
188
191
|
# Call the class method to regenerate the signature
|
189
|
-
expected_signature = MyClient.generate_signature(method, url, query, body, timestamp)
|
192
|
+
expected_signature = MyClient.generate_signature(method, url, query, body, timestamp)
|
190
193
|
|
191
194
|
# assert signature == expected_signature
|
192
195
|
```
|
@@ -197,7 +200,7 @@ expected_signature = MyClient.generate_signature(method, url, query, body, times
|
|
197
200
|
def get_something(page, page_size)
|
198
201
|
response = JSON.parse(http_get("http://some/url?p=#{page}&l=#{page_size}"))
|
199
202
|
return Px::Service::Client::ListResponse(page_size, response, "items")
|
200
|
-
end
|
203
|
+
end
|
201
204
|
```
|
202
205
|
|
203
206
|
Wraps a deserialized response. A `ListResponse` implements the Ruby `Enumerable` module, as well
|
@@ -1,7 +1,31 @@
|
|
1
1
|
module Px::Service::Client
|
2
2
|
class Base
|
3
3
|
cattr_accessor :logger
|
4
|
-
|
4
|
+
|
5
|
+
class DefaultConfig < OpenStruct
|
6
|
+
def initialize
|
7
|
+
super
|
8
|
+
self.statsd_client = NullStatsdClient.new
|
9
|
+
end
|
10
|
+
end
|
11
|
+
|
12
|
+
##
|
13
|
+
# Configure the client
|
14
|
+
def self.config
|
15
|
+
@config ||= DefaultConfig.new
|
16
|
+
yield(@config) if block_given?
|
17
|
+
@config
|
18
|
+
end
|
19
|
+
|
20
|
+
# Make class config available to instances
|
21
|
+
def config
|
22
|
+
if block_given?
|
23
|
+
self.class.config { |c| yield(c) }
|
24
|
+
else
|
25
|
+
self.class.config
|
26
|
+
end
|
27
|
+
end
|
28
|
+
|
5
29
|
private
|
6
30
|
|
7
31
|
def parsed_body(response)
|
@@ -19,6 +43,22 @@ module Px::Service::Client
|
|
19
43
|
##
|
20
44
|
# Make the request
|
21
45
|
def make_request(method, uri, query: nil, headers: nil, body: nil, timeout: 0)
|
46
|
+
stats_tags = [
|
47
|
+
"method:#{method.downcase}",
|
48
|
+
]
|
49
|
+
if uri.respond_to?(:path)
|
50
|
+
stats_tags << "host:#{uri.host}"
|
51
|
+
stats_tags << "path:#{uri.path}"
|
52
|
+
else
|
53
|
+
actual_uri = URI(uri)
|
54
|
+
stats_tags << "host:#{actual_uri.host}"
|
55
|
+
stats_tags << "path:#{actual_uri.path}"
|
56
|
+
end
|
57
|
+
|
58
|
+
_make_request(method, uri, query: query, headers: headers, body: body, timeout: timeout, stats_tags: stats_tags)
|
59
|
+
end
|
60
|
+
|
61
|
+
def _make_request(method, uri, query: nil, headers: nil, body: nil, timeout: nil, stats_tags: [])
|
22
62
|
req = Typhoeus::Request.new(
|
23
63
|
uri,
|
24
64
|
method: method,
|
@@ -32,11 +72,26 @@ module Px::Service::Client
|
|
32
72
|
|
33
73
|
req.on_complete do |response|
|
34
74
|
elapsed = (Time.now - start_time) * 1000
|
75
|
+
config.statsd_client.histogram("request.duration", elapsed.to_i, tags: stats_tags)
|
76
|
+
config.statsd_client.increment("response.count", tags: stats_tags + ["httpstatus:#{response.response_code}"])
|
77
|
+
case
|
78
|
+
when response.response_code > 100 && response.response_code < 199
|
79
|
+
config.statsd_client.increment("response.status_1xx.count", tags: stats_tags)
|
80
|
+
when response.response_code > 200 && response.response_code < 299
|
81
|
+
config.statsd_client.increment("response.status_2xx.count", tags: stats_tags)
|
82
|
+
when response.response_code > 300 && response.response_code < 399
|
83
|
+
config.statsd_client.increment("response.status_3xx.count", tags: stats_tags)
|
84
|
+
when response.response_code > 400 && response.response_code < 499
|
85
|
+
config.statsd_client.increment("response.status_4xx.count", tags: stats_tags)
|
86
|
+
when response.response_code > 500
|
87
|
+
config.statsd_client.increment("response.status_5xx.count", tags: stats_tags)
|
88
|
+
end
|
35
89
|
logger.debug "Completed request #{method.to_s.upcase} #{uri}, took #{elapsed.to_i}ms, got status #{response.response_code}" if logger
|
36
90
|
end
|
37
91
|
|
38
92
|
RetriableResponseFuture.new(req)
|
39
93
|
end
|
40
94
|
|
95
|
+
|
41
96
|
end
|
42
97
|
end
|
@@ -18,42 +18,26 @@ module Px::Service::Client
|
|
18
18
|
|
19
19
|
included do
|
20
20
|
cattr_accessor :cache_client, :cache_logger
|
21
|
-
end
|
22
21
|
|
23
|
-
|
24
|
-
|
25
|
-
|
26
|
-
|
27
|
-
|
28
|
-
self.max_page = nil
|
29
|
-
self.cache_options = {}
|
30
|
-
self.cache_options[:policy_group] = 'general'
|
31
|
-
self.cache_logger = nil
|
32
|
-
self.cache_client = nil
|
33
|
-
end
|
22
|
+
config do |config|
|
23
|
+
config.cache_expiry = 30.seconds
|
24
|
+
config.cache_default_policy_group = 'general'
|
25
|
+
config.cache_logger = nil
|
26
|
+
config.cache_client = nil
|
34
27
|
end
|
35
28
|
|
36
|
-
|
37
|
-
|
38
|
-
def caching(&block)
|
39
|
-
@cache_config ||= DefaultConfig.new
|
40
|
-
yield(@cache_config) if block_given?
|
41
|
-
@cache_config
|
42
|
-
end
|
43
|
-
end
|
44
|
-
|
45
|
-
def config
|
46
|
-
@cache_config || self.class.caching
|
29
|
+
# DEPRECATED: Use .config (base class method) instead
|
30
|
+
alias_method :caching, :config
|
47
31
|
end
|
48
32
|
|
49
|
-
def cache_request(url, strategy: nil, policy_group: config.
|
33
|
+
def cache_request(url, strategy: nil, policy_group: config.cache_default_policy_group, expires_in: config.cache_expiry, refresh_probability: 1)
|
50
34
|
case strategy
|
51
35
|
when :last_resort
|
52
|
-
cache_last_resort(url, policy_group: policy_group, expires_in: expires_in, refresh_probability: refresh_probability
|
36
|
+
cache_last_resort(url, policy_group: policy_group, expires_in: expires_in, refresh_probability: refresh_probability) { yield }
|
53
37
|
when :first_resort
|
54
|
-
cache_first_resort(url, policy_group: policy_group, expires_in: expires_in
|
38
|
+
cache_first_resort(url, policy_group: policy_group, expires_in: expires_in) { yield }
|
55
39
|
else
|
56
|
-
no_cache
|
40
|
+
no_cache { yield }
|
57
41
|
end
|
58
42
|
end
|
59
43
|
|
@@ -63,10 +47,15 @@ module Px::Service::Client
|
|
63
47
|
# Use the cache as a last resort. This path will make the request each time, caching the result
|
64
48
|
# on success. If an exception occurs, the cache is checked for a result. If the cache has a result, it's
|
65
49
|
# returned and the cache entry is touched to prevent expiry. Otherwise, the original exception is re-raised.
|
66
|
-
def cache_last_resort(url, policy_group: 'general', expires_in: nil, refresh_probability: 1
|
50
|
+
def cache_last_resort(url, policy_group: 'general', expires_in: nil, refresh_probability: 1)
|
51
|
+
tags = [
|
52
|
+
"type:last_resort",
|
53
|
+
"policy_group:#{policy_group}",
|
54
|
+
]
|
55
|
+
|
67
56
|
# Note we use a smaller refresh window here (technically, could even use 0)
|
68
57
|
# since we don't really need the "expired but not really expired" behaviour when caching as a last resort.
|
69
|
-
retry_response =
|
58
|
+
retry_response = yield
|
70
59
|
|
71
60
|
Future.new do
|
72
61
|
begin
|
@@ -76,16 +65,21 @@ module Px::Service::Client
|
|
76
65
|
|
77
66
|
# Only store a new result if we roll a 0
|
78
67
|
r = rand(refresh_probability)
|
79
|
-
|
68
|
+
if r == 0
|
69
|
+
entry.store(expires_in, refresh_window: 1.minute)
|
70
|
+
config.statsd_client.increment("caching.write.count", tags: tags)
|
71
|
+
end
|
80
72
|
resp
|
81
73
|
rescue Px::Service::ServiceError => ex
|
82
|
-
cache_logger.error "Service responded with exception: #{ex.class.name}: #{ex.message}\n#{ex.backtrace.join('\n')}" if cache_logger
|
74
|
+
config.cache_logger.error "Service responded with exception: #{ex.class.name}: #{ex.message}\n#{ex.backtrace.join('\n')}" if config.cache_logger
|
83
75
|
entry = CacheEntry.fetch(config.cache_client, url, policy_group)
|
84
76
|
if entry.nil?
|
85
77
|
# Re-raise the error, no cached response
|
78
|
+
config.statsd_client.increment("caching.fetch.count", tags: tags + ["result:miss"])
|
86
79
|
raise ex
|
87
80
|
end
|
88
81
|
|
82
|
+
config.statsd_client.increment("caching.fetch.count", tags: tags + ["result:hit"])
|
89
83
|
entry.touch(expires_in, refresh_window: 1.minute)
|
90
84
|
entry.data
|
91
85
|
end
|
@@ -97,7 +91,11 @@ module Px::Service::Client
|
|
97
91
|
# or if the cache entry has expired. It follows logic similar to ActiveSupport::Cache. If the cache entry
|
98
92
|
# has expired (but is still present) and the request fails, the cached value is still returned, as if this was
|
99
93
|
# cache_last_resort.
|
100
|
-
def cache_first_resort(url, policy_group: 'general', expires_in: nil
|
94
|
+
def cache_first_resort(url, policy_group: 'general', expires_in: nil)
|
95
|
+
tags = [
|
96
|
+
"type:last_resort",
|
97
|
+
"policy_group:#{policy_group}",
|
98
|
+
]
|
101
99
|
entry = CacheEntry.fetch(config.cache_client, url, policy_group)
|
102
100
|
|
103
101
|
if entry
|
@@ -106,13 +104,15 @@ module Px::Service::Client
|
|
106
104
|
# calling the block, but to prevent lots of others from also trying to refresh, first it updates
|
107
105
|
# the expiry date on the entry so that other callers that come in while we're requesting the update
|
108
106
|
# don't also try to update the cache.
|
107
|
+
config.statsd_client.increment("caching.fetch.count", tags: tags + ["result:expired"])
|
109
108
|
entry.touch(expires_in)
|
110
109
|
else
|
110
|
+
config.statsd_client.increment("caching.fetch.count", tags: tags + ["result:hit"])
|
111
111
|
return Future.new { entry.data }
|
112
112
|
end
|
113
113
|
end
|
114
114
|
|
115
|
-
retry_response =
|
115
|
+
retry_response = yield
|
116
116
|
|
117
117
|
Future.new do
|
118
118
|
begin
|
@@ -120,15 +120,18 @@ module Px::Service::Client
|
|
120
120
|
resp = retry_response.value!
|
121
121
|
entry = CacheEntry.new(config.cache_client, url, policy_group, resp)
|
122
122
|
entry.store(expires_in)
|
123
|
+
config.statsd_client.increment("caching.write.count", tags: tags)
|
123
124
|
resp
|
124
125
|
rescue Px::Service::ServiceError => ex
|
125
|
-
cache_logger.error "Service responded with exception: #{ex.class.name}: #{ex.message}\n#{ex.backtrace.join('\n')}" if cache_logger
|
126
|
+
config.cache_logger.error "Service responded with exception: #{ex.class.name}: #{ex.message}\n#{ex.backtrace.join('\n')}" if config.cache_logger
|
126
127
|
|
127
128
|
entry = CacheEntry.fetch(config.cache_client, url, policy_group)
|
128
129
|
if entry.nil?
|
129
130
|
# Re-raise the error, no cached response
|
131
|
+
# config.statsd_client.increment("caching.fetch.count", tags: tags + ["result:miss"])
|
130
132
|
raise ex
|
131
133
|
end
|
134
|
+
config.statsd_client.increment("caching.fetch.count", tags: tags + ["result:hit"])
|
132
135
|
|
133
136
|
# Set the entry to be expired again (but reset the refresh window). This allows the next call to try again
|
134
137
|
# (assuming the circuit breaker is reset) but keeps the value in the cache in the meantime
|
@@ -141,8 +144,8 @@ module Px::Service::Client
|
|
141
144
|
Future.new { ex }
|
142
145
|
end
|
143
146
|
|
144
|
-
def no_cache
|
145
|
-
retry_response =
|
147
|
+
def no_cache
|
148
|
+
retry_response = yield
|
146
149
|
|
147
150
|
Future.new do
|
148
151
|
raise ArgumentError.new('Block did not return a Future.') unless retry_response.is_a?(Future)
|
@@ -18,12 +18,12 @@ module Px::Service::Client
|
|
18
18
|
::CircuitBreaker::CircuitState.new
|
19
19
|
end
|
20
20
|
|
21
|
-
alias_method_chain :
|
21
|
+
alias_method_chain :_make_request, :breaker
|
22
22
|
end
|
23
23
|
|
24
24
|
##
|
25
25
|
# Make the request, respecting the circuit breaker, if configured
|
26
|
-
def
|
26
|
+
def _make_request_with_breaker(method, uri, query: nil, headers: nil, body: nil, timeout: nil, stats_tags: [])
|
27
27
|
state = self.class.circuit_state
|
28
28
|
handler = self.class.circuit_handler
|
29
29
|
|
@@ -38,20 +38,27 @@ module Px::Service::Client
|
|
38
38
|
end
|
39
39
|
end
|
40
40
|
|
41
|
-
|
41
|
+
config.statsd_client.increment("breakers.ready.count", tags: stats_tags) if circuit_state.half_open?
|
42
|
+
|
43
|
+
retry_request = _make_request_without_breaker(
|
42
44
|
method,
|
43
45
|
uri,
|
44
46
|
query: query,
|
45
47
|
headers: headers,
|
46
48
|
body: body,
|
47
|
-
timeout: handler.invocation_timeout
|
49
|
+
timeout: handler.invocation_timeout,
|
50
|
+
stats_tags: stats_tags)
|
48
51
|
|
49
52
|
retry_request.request.on_complete do |response|
|
50
53
|
# Wait for request to exhaust retries
|
51
54
|
if retry_request.completed?
|
52
55
|
if response.response_code >= 500 || response.response_code == 0
|
56
|
+
config.statsd_client.increment("breakers.fail.count", tags: stats_tags)
|
57
|
+
config.statsd_client.increment("breakers.tripped.count", tags: stats_tags) if circuit_state.closed?
|
58
|
+
|
53
59
|
handler.on_failure(state)
|
54
60
|
else
|
61
|
+
config.statsd_client.increment("breakers.reset.count", tags: stats_tags) unless circuit_state.closed?
|
55
62
|
handler.on_success(state)
|
56
63
|
end
|
57
64
|
end
|
@@ -2,43 +2,33 @@ module Px::Service::Client
|
|
2
2
|
module HmacSigning
|
3
3
|
extend ActiveSupport::Concern
|
4
4
|
included do
|
5
|
-
alias_method_chain :
|
6
|
-
|
5
|
+
alias_method_chain :_make_request, :signing
|
6
|
+
|
7
7
|
cattr_accessor :secret do
|
8
8
|
DEFAULT_SECRET
|
9
9
|
end
|
10
|
-
|
10
|
+
|
11
11
|
cattr_accessor :keyspan do
|
12
12
|
DEFAULT_KEYSPAN
|
13
13
|
end
|
14
|
-
|
15
|
-
end
|
16
14
|
|
17
|
-
|
18
|
-
|
19
|
-
|
20
|
-
|
21
|
-
self.keyspan = DEFAULT_KEYSPAN
|
22
|
-
end
|
15
|
+
# Default config for signing
|
16
|
+
config do |config|
|
17
|
+
config.hmac_secret = DEFAULT_SECRET
|
18
|
+
config.hmac_keyspan = DEFAULT_KEYSPAN
|
23
19
|
end
|
24
20
|
|
25
|
-
|
26
|
-
|
27
|
-
|
28
|
-
|
29
|
-
|
30
|
-
|
31
|
-
yield(@signing_config)
|
32
|
-
self.secret = @signing_config.secret
|
33
|
-
self.keyspan = @signing_config.keyspan
|
34
|
-
end
|
35
|
-
end
|
36
|
-
|
21
|
+
##
|
22
|
+
# DEPRECATED: Use .config (base class method) instead
|
23
|
+
alias_method :hmac_signing, :config
|
24
|
+
end
|
25
|
+
|
26
|
+
module ClassMethods
|
37
27
|
##
|
38
28
|
# Generate a nonce that's used to expire message after keyspan seconds
|
39
29
|
def generate_signature(method, uri, query, body, timestamp)
|
40
|
-
secret = self.
|
41
|
-
keyspan = self.
|
30
|
+
secret = self.config.hmac_secret
|
31
|
+
keyspan = self.config.hmac_keyspan
|
42
32
|
nonce = (timestamp - (timestamp % keyspan)) + keyspan
|
43
33
|
data = "#{method.capitalize},#{uri},#{query},#{body},#{nonce.to_s}"
|
44
34
|
digest = OpenSSL::Digest.new('sha256')
|
@@ -46,21 +36,23 @@ module Px::Service::Client
|
|
46
36
|
return Base64.urlsafe_encode64(digest).strip()
|
47
37
|
end
|
48
38
|
end
|
49
|
-
|
50
|
-
def
|
39
|
+
|
40
|
+
def _make_request_with_signing(method, uri, query: nil, headers: nil, body: nil, timeout: nil, stats_tags: [])
|
51
41
|
timestamp = Time.now.to_i
|
52
42
|
signature = self.class.generate_signature(method, uri, query, body, timestamp)
|
53
|
-
|
43
|
+
|
54
44
|
headers = {} if headers.nil?
|
55
45
|
headers.merge!("X-Service-Auth" => signature)
|
56
46
|
headers.merge!("Timestamp" => timestamp)
|
57
47
|
|
58
|
-
|
48
|
+
_make_request_without_signing(
|
59
49
|
method,
|
60
50
|
uri,
|
61
51
|
query: query,
|
62
52
|
headers: headers,
|
63
|
-
body: body
|
53
|
+
body: body,
|
54
|
+
timeout: timeout,
|
55
|
+
stats_tags: stats_tags)
|
64
56
|
end
|
65
57
|
|
66
58
|
end
|
@@ -0,0 +1,30 @@
|
|
1
|
+
module Px::Service::Client
|
2
|
+
# Does nothing, gracefully
|
3
|
+
class NullStatsdClient
|
4
|
+
def increment(*args)
|
5
|
+
end
|
6
|
+
|
7
|
+
def gauge(*args)
|
8
|
+
end
|
9
|
+
|
10
|
+
def histogram(*args)
|
11
|
+
end
|
12
|
+
|
13
|
+
def time(*args)
|
14
|
+
yield if block_given?
|
15
|
+
end
|
16
|
+
|
17
|
+
def timing(*args)
|
18
|
+
end
|
19
|
+
|
20
|
+
def set(*args)
|
21
|
+
end
|
22
|
+
|
23
|
+
def count(*args)
|
24
|
+
end
|
25
|
+
|
26
|
+
def batch(*args)
|
27
|
+
yield(self) if block_given?
|
28
|
+
end
|
29
|
+
end
|
30
|
+
end
|
data/lib/px/service/client.rb
CHANGED
@@ -14,6 +14,7 @@ end
|
|
14
14
|
|
15
15
|
require "px/service/client/version"
|
16
16
|
require "px/service/client/future"
|
17
|
+
require "px/service/client/null_statsd_client"
|
17
18
|
require "px/service/client/caching"
|
18
19
|
require "px/service/client/circuit_breaker"
|
19
20
|
require "px/service/client/hmac_signing"
|
@@ -6,8 +6,9 @@ describe Px::Service::Client::Base do
|
|
6
6
|
let(:dalli) { Dalli::Client.new(dalli_host, dalli_options) }
|
7
7
|
|
8
8
|
subject {
|
9
|
-
|
10
|
-
c.
|
9
|
+
Class.new(Px::Service::Client::Base).tap do |c|
|
10
|
+
c.include(Px::Service::Client::Caching)
|
11
|
+
c.config do |config|
|
11
12
|
config.cache_client = dalli
|
12
13
|
end
|
13
14
|
end.new
|
@@ -20,6 +21,30 @@ describe Px::Service::Client::Base do
|
|
20
21
|
headers: { "Content-Type" => "application/json"} )
|
21
22
|
end
|
22
23
|
|
24
|
+
describe '#config' do
|
25
|
+
let(:other_subclass) {
|
26
|
+
Class.new(Px::Service::Client::Base).tap do |c|
|
27
|
+
c.include(Px::Service::Client::Caching)
|
28
|
+
end.new
|
29
|
+
}
|
30
|
+
|
31
|
+
context "when there are separate subclasses" do
|
32
|
+
before :each do
|
33
|
+
subject.config do |c|
|
34
|
+
c.subject_field = "value"
|
35
|
+
end
|
36
|
+
end
|
37
|
+
|
38
|
+
it "sets the config value on the subject" do
|
39
|
+
expect(subject.config.subject_field).to eq("value")
|
40
|
+
end
|
41
|
+
|
42
|
+
it "does not set the config value on other subclasses" do
|
43
|
+
expect(other_subclass.config.subject_field).not_to eq("value")
|
44
|
+
end
|
45
|
+
end
|
46
|
+
end
|
47
|
+
|
23
48
|
describe '#make_request' do
|
24
49
|
let(:url) { 'http://localhost:3000/path' }
|
25
50
|
|
@@ -308,4 +333,3 @@ describe Px::Service::Client::Base do
|
|
308
333
|
end
|
309
334
|
end
|
310
335
|
end
|
311
|
-
|
@@ -7,11 +7,13 @@ describe Px::Service::Client::Caching do
|
|
7
7
|
let(:dalli) { Dalli::Client.new(dalli_host, dalli_options) }
|
8
8
|
|
9
9
|
subject {
|
10
|
-
Class.new
|
10
|
+
Class.new(Px::Service::Client::Base).tap do |c|
|
11
|
+
c.include(Px::Service::Client::Caching)
|
12
|
+
|
11
13
|
# Anonymous classes don't have a name. Stub out :name so that things work
|
12
14
|
allow(c).to receive(:name).and_return("Caching")
|
13
15
|
|
14
|
-
c.
|
16
|
+
c.config do |config|
|
15
17
|
config.cache_client = dalli
|
16
18
|
end
|
17
19
|
end.new
|
@@ -71,7 +73,7 @@ describe Px::Service::Client::Caching do
|
|
71
73
|
|
72
74
|
shared_examples_for "a request with no cached response" do
|
73
75
|
it "raises the exception" do
|
74
|
-
expect {
|
76
|
+
expect {
|
75
77
|
subject.cache_request(url, strategy: strategy) do
|
76
78
|
raise Px::Service::ServiceError.new("Error", 500)
|
77
79
|
end.value!
|
@@ -85,7 +87,7 @@ describe Px::Service::Client::Caching do
|
|
85
87
|
|
86
88
|
context 'when cache client is not set' do
|
87
89
|
before :each do
|
88
|
-
subject.
|
90
|
+
subject.config do |config|
|
89
91
|
config.cache_client = nil
|
90
92
|
end
|
91
93
|
end
|
@@ -109,7 +111,7 @@ describe Px::Service::Client::Caching do
|
|
109
111
|
context "when there is a cached response" do
|
110
112
|
context 'when cache client is not set' do
|
111
113
|
before :each do
|
112
|
-
subject.
|
114
|
+
subject.config do |config|
|
113
115
|
config.cache_client = nil
|
114
116
|
end
|
115
117
|
end
|
@@ -164,7 +166,7 @@ describe Px::Service::Client::Caching do
|
|
164
166
|
context "when there is a cached response" do
|
165
167
|
context 'when cache client is not set' do
|
166
168
|
before :each do
|
167
|
-
subject.
|
169
|
+
subject.config do |config|
|
168
170
|
config.cache_client = nil
|
169
171
|
end
|
170
172
|
end
|
@@ -8,21 +8,23 @@ describe Px::Service::Client::HmacSigning do
|
|
8
8
|
c.include(Px::Service::Client::HmacSigning)
|
9
9
|
end
|
10
10
|
}
|
11
|
-
|
11
|
+
|
12
12
|
let(:another_class) {
|
13
|
-
Class.new(Px::Service::Client::Base).
|
14
|
-
c.
|
15
|
-
|
13
|
+
Class.new(Px::Service::Client::Base).tap do |c|
|
14
|
+
c.include(Px::Service::Client::HmacSigning)
|
15
|
+
|
16
|
+
c.config do |config|
|
17
|
+
config.hmac_secret = "different secret"
|
16
18
|
end
|
17
19
|
end
|
18
20
|
}
|
19
|
-
|
21
|
+
|
20
22
|
subject { subject_class.new }
|
21
23
|
let(:another_object) { another_class.new }
|
22
|
-
|
24
|
+
|
23
25
|
describe '#make_request' do
|
24
26
|
context "when the underlying request method succeeds" do
|
25
|
-
let(:url) { 'http://localhost:3000/path' }
|
27
|
+
let(:url) { 'http://localhost:3000/path' }
|
26
28
|
let(:resp) { subject.send(:make_request, 'get', url) }
|
27
29
|
let(:headers) { resp.request.options[:headers] }
|
28
30
|
|
@@ -34,7 +36,7 @@ describe Px::Service::Client::HmacSigning do
|
|
34
36
|
expect(headers).to have_key("X-Service-Auth")
|
35
37
|
expect(headers).to have_key("Timestamp")
|
36
38
|
end
|
37
|
-
|
39
|
+
|
38
40
|
let(:resp2) { another_object.send(:make_request, 'get', url) }
|
39
41
|
let(:headers2) { resp2.request.options[:headers] }
|
40
42
|
it "is different from the object of another class with a different key" do
|
metadata
CHANGED
@@ -1,14 +1,14 @@
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
2
2
|
name: px-service-client
|
3
3
|
version: !ruby/object:Gem::Version
|
4
|
-
version:
|
4
|
+
version: 2.0.0
|
5
5
|
platform: ruby
|
6
6
|
authors:
|
7
7
|
- Chris Micacchi
|
8
8
|
autorequire:
|
9
9
|
bindir: bin
|
10
10
|
cert_chain: []
|
11
|
-
date: 2016-
|
11
|
+
date: 2016-06-20 00:00:00.000000000 Z
|
12
12
|
dependencies:
|
13
13
|
- !ruby/object:Gem::Dependency
|
14
14
|
name: will_paginate
|
@@ -221,6 +221,7 @@ files:
|
|
221
221
|
- lib/px/service/client/hmac_signing.rb
|
222
222
|
- lib/px/service/client/list_response.rb
|
223
223
|
- lib/px/service/client/multiplexer.rb
|
224
|
+
- lib/px/service/client/null_statsd_client.rb
|
224
225
|
- lib/px/service/client/retriable_response_future.rb
|
225
226
|
- lib/px/service/client/version.rb
|
226
227
|
- lib/px/service/errors.rb
|