circuitbox 0.5.2 → 0.7.0
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +5 -13
- data/Gemfile +2 -1
- data/README.md +12 -12
- data/Rakefile +2 -2
- data/circuitbox.gemspec +1 -0
- data/lib/circuitbox/circuit_breaker.rb +75 -20
- data/lib/circuitbox/notifier.rb +28 -6
- data/lib/circuitbox/version.rb +1 -1
- data/test/circuit_breaker_test.rb +260 -7
- data/test/notifier_test.rb +15 -4
- metadata +25 -11
checksums.yaml
CHANGED
@@ -1,15 +1,7 @@
|
|
1
1
|
---
|
2
|
-
|
3
|
-
metadata.gz:
|
4
|
-
|
5
|
-
data.tar.gz: !binary |-
|
6
|
-
MTJiMjk1YWFkZThhY2FjZDdmYzA1MjUwYWY3OGNiNzE0N2JjMzM5OQ==
|
2
|
+
SHA1:
|
3
|
+
metadata.gz: 00e479410c5dfdf8769c828dfbfdcae074ee524c
|
4
|
+
data.tar.gz: c2f06d9ceaa7ca95c6837a7abc01e79a93ecb81c
|
7
5
|
SHA512:
|
8
|
-
metadata.gz:
|
9
|
-
|
10
|
-
Y2EyYjIwNGZmYTEzMjY4MjVhYTIyODAwNDE5ZTE1Nzk2MWRiNWMyZDI3Yzgx
|
11
|
-
NzM4OWRhMWY1MWU0OTQzMWZiNmU4M2U3NzdiODgwOTE3MDllMmY=
|
12
|
-
data.tar.gz: !binary |-
|
13
|
-
ODA5NDcxMjk0NDI1Mzc1ODU1OWQyMDQ5ZTA5ZWE4NWRhNDczNzFmOWFkMmY4
|
14
|
-
ZDBlNGYzYWEzNDgzMGY0YzM4YzZiZTNjMzI4ZjE2MGFmYzYyMTJiYmViNGEx
|
15
|
-
NDE5YWE1N2VmNjFkZWViNDI5MTdhZWM4ODFjMzRjNDNmNDM3Y2U=
|
6
|
+
metadata.gz: d43fff58af3f25b93be58ecaac16d7b8a815ac275d4337ff6cbf65833c9e72c3c4b1ddd930165c1393df426fa0f9250b29585e987288a4af798b951073073dcf
|
7
|
+
data.tar.gz: a1913c31ddc136a114ea659fe602f8358f6fecb7076cb2af6d954d04f014c70f9685c1fe871963b6cdd180e62701b9fd968aa4b5f70f5cc20ccd5e6ee8d79982
|
data/Gemfile
CHANGED
data/README.md
CHANGED
@@ -1,7 +1,7 @@
|
|
1
|
-
# Circuitbox
|
2
1
|
|
3
|
-
|
2
|
+
# Circuitbox
|
4
3
|
|
4
|
+
Circuitbox is a Ruby circuit breaker gem. It protects your application from failures of it's service dependencies. It wraps calls to external services and monitors for failures in one minute intervals. Once more than 10 requests have been made with a 50% failure rate, Circuitbox stops sending requests to that failing service for one minute. This helps your application gracefully degrade.
|
5
5
|
Resources about the circuit breaker pattern:
|
6
6
|
* [http://martinfowler.com/bliki/CircuitBreaker.html](http://martinfowler.com/bliki/CircuitBreaker.html)
|
7
7
|
* [https://github.com/Netflix/Hystrix/wiki/How-it-Works#CircuitBreaker](https://github.com/Netflix/Hystrix/wiki/How-it-Works#CircuitBreaker)
|
@@ -15,14 +15,14 @@ end
|
|
15
15
|
```
|
16
16
|
|
17
17
|
Circuitbox will return nil for failed requests and open circuits.
|
18
|
-
If your HTTP client has it's own conditions for failure, you can pass an `exceptions` option.
|
18
|
+
If your HTTP client has it's own conditions for failure, you can pass an `exceptions` option.
|
19
19
|
|
20
20
|
```ruby
|
21
21
|
class ExampleServiceClient
|
22
22
|
def circuit
|
23
23
|
Circuitbox.circuit(:yammer, exceptions: [Zephyr::FailedRequest])
|
24
24
|
end
|
25
|
-
|
25
|
+
|
26
26
|
def http_get
|
27
27
|
circuit.run do
|
28
28
|
Zephyr.new("http://example.com").get(200, 1000, "/api/messages")
|
@@ -40,16 +40,16 @@ class ExampleServiceClient
|
|
40
40
|
exceptions: [YourCustomException],
|
41
41
|
|
42
42
|
# seconds the circuit stays open once it has passed the error threshold
|
43
|
-
sleep_window: 300,
|
43
|
+
sleep_window: 300,
|
44
44
|
|
45
45
|
# number of requests within 1 minute before it calculates error rates
|
46
|
-
volume_threshold: 10,
|
46
|
+
volume_threshold: 10,
|
47
47
|
|
48
|
-
# exceeding this rate will open the circuit
|
48
|
+
# exceeding this rate will open the circuit
|
49
49
|
error_threshold: 50,
|
50
50
|
|
51
|
-
# seconds before the circuit times out
|
52
|
-
timeout_seconds: 1
|
51
|
+
# seconds before the circuit times out
|
52
|
+
timeout_seconds: 1
|
53
53
|
})
|
54
54
|
end
|
55
55
|
end
|
@@ -58,20 +58,20 @@ end
|
|
58
58
|
You can also pass a Proc as an option value which will evaluate each time the circuit breaker is used. This lets you configure the circuit breaker without having to restart the processes.
|
59
59
|
|
60
60
|
```ruby
|
61
|
-
Circuitbox.circuit(:yammer, {
|
61
|
+
Circuitbox.circuit(:yammer, {
|
62
62
|
sleep_window: Proc.new { Configuration.get(:sleep_window) }
|
63
63
|
})
|
64
64
|
```
|
65
65
|
|
66
66
|
## Monitoring & Statistics
|
67
67
|
|
68
|
-
You can also run `rake circuits:stats SERVICE={service_name}` to see successes, failures and opened circuits.
|
68
|
+
You can also run `rake circuits:stats SERVICE={service_name}` to see successes, failures and opened circuits.
|
69
69
|
Add `PARTITION={partition_key}` to see the circuit for a particular partition.
|
70
70
|
The stats are aggregated into 1 minute intervals.
|
71
71
|
|
72
72
|
## Faraday (Caveat: Open circuits return a nil response object)
|
73
73
|
|
74
|
-
Circuitbox ships with [Faraday HTTP client](https://github.com/lostisland/faraday) middleware.
|
74
|
+
Circuitbox ships with [Faraday HTTP client](https://github.com/lostisland/faraday) middleware.
|
75
75
|
|
76
76
|
```ruby
|
77
77
|
require 'faraday'
|
data/Rakefile
CHANGED
@@ -1,5 +1,5 @@
|
|
1
1
|
require 'rake/testtask'
|
2
|
-
require
|
2
|
+
require "bundler/gem_version_tasks"
|
3
3
|
|
4
4
|
Rake::TestTask.new do |t|
|
5
5
|
t.libs << 'test'
|
@@ -7,4 +7,4 @@ Rake::TestTask.new do |t|
|
|
7
7
|
end
|
8
8
|
|
9
9
|
desc "Run tests"
|
10
|
-
task :default => :test
|
10
|
+
task :default => :test
|
data/circuitbox.gemspec
CHANGED
@@ -7,7 +7,8 @@ class Circuitbox
|
|
7
7
|
sleep_window: 300,
|
8
8
|
volume_threshold: 5,
|
9
9
|
error_threshold: 50,
|
10
|
-
timeout_seconds: 1
|
10
|
+
timeout_seconds: 1,
|
11
|
+
time_window: 60,
|
11
12
|
}
|
12
13
|
|
13
14
|
#
|
@@ -18,18 +19,21 @@ class Circuitbox
|
|
18
19
|
# `error_threshold` - percentage of failed requests needed to trip circuit
|
19
20
|
# `timeout_seconds` - seconds until it will timeout the request
|
20
21
|
# `exceptions` - exceptions other than Timeout::Error that count as failures
|
22
|
+
# `time_window` - interval of time used to calculate error_rate (in seconds) - default is 60s
|
21
23
|
#
|
22
24
|
def initialize(service, options = {})
|
23
25
|
@service = service
|
24
26
|
@circuit_options = options
|
25
27
|
@circuit_store = options.fetch(:cache) { Circuitbox.circuit_store }
|
26
|
-
@notifier =
|
28
|
+
@notifier = options.fetch(:notifier_class) { Notifier }
|
27
29
|
|
28
30
|
@exceptions = options.fetch(:exceptions) { [] }
|
29
31
|
@exceptions = [Timeout::Error] if @exceptions.blank?
|
30
32
|
|
31
33
|
@logger = defined?(Rails) ? Rails.logger : Logger.new(STDOUT)
|
32
34
|
@stat_store = options.fetch(:stat_store) { Circuitbox.stat_store }
|
35
|
+
@time_class = options.fetch(:time_class) { Time }
|
36
|
+
sanitize_options
|
33
37
|
end
|
34
38
|
|
35
39
|
def option_value(name)
|
@@ -46,6 +50,7 @@ class Circuitbox
|
|
46
50
|
response = nil
|
47
51
|
open! unless open_flag?
|
48
52
|
else
|
53
|
+
close! if was_open?
|
49
54
|
logger.debug "[CIRCUIT] closed: querying #{service}"
|
50
55
|
|
51
56
|
begin
|
@@ -100,14 +105,48 @@ class Circuitbox
|
|
100
105
|
stats
|
101
106
|
end
|
102
107
|
|
108
|
+
def error_rate(failures = failure_count, success = success_count)
|
109
|
+
all_count = failures + success
|
110
|
+
return 0.0 unless all_count > 0
|
111
|
+
failure_count.to_f / all_count.to_f * 100
|
112
|
+
end
|
113
|
+
|
114
|
+
def failure_count
|
115
|
+
circuit_store.read(stat_storage_key(:failure)).to_i
|
116
|
+
end
|
117
|
+
|
118
|
+
def success_count
|
119
|
+
circuit_store.read(stat_storage_key(:success)).to_i
|
120
|
+
end
|
121
|
+
|
122
|
+
def try_close_next_time
|
123
|
+
circuit_store.delete(storage_key(:asleep))
|
124
|
+
end
|
125
|
+
|
103
126
|
private
|
104
127
|
def open!
|
105
128
|
log_event :open
|
106
129
|
logger.debug "[CIRCUIT] opening #{service} circuit"
|
107
130
|
circuit_store.write(storage_key(:asleep), true, expires_in: option_value(:sleep_window).seconds)
|
108
131
|
half_open!
|
132
|
+
was_open!
|
109
133
|
end
|
110
134
|
|
135
|
+
### BEGIN - all this is just here to produce a close notification
|
136
|
+
def close!
|
137
|
+
log_event :close
|
138
|
+
circuit_store.delete(storage_key(:was_open))
|
139
|
+
end
|
140
|
+
|
141
|
+
def was_open!
|
142
|
+
circuit_store.write(storage_key(:was_open), true)
|
143
|
+
end
|
144
|
+
|
145
|
+
def was_open?
|
146
|
+
circuit_store.read(storage_key(:was_open)).present?
|
147
|
+
end
|
148
|
+
### END
|
149
|
+
|
111
150
|
def half_open!
|
112
151
|
circuit_store.write(storage_key(:half_open), true)
|
113
152
|
end
|
@@ -125,27 +164,20 @@ class Circuitbox
|
|
125
164
|
end
|
126
165
|
|
127
166
|
def passed_rate_threshold?
|
128
|
-
|
129
|
-
end
|
130
|
-
|
131
|
-
def failure_count
|
132
|
-
circuit_store.read(stat_storage_key(:failure)).to_i
|
133
|
-
end
|
134
|
-
|
135
|
-
def success_count
|
136
|
-
circuit_store.read(stat_storage_key(:success)).to_i
|
167
|
+
read_and_log_error_rate >= option_value(:error_threshold)
|
137
168
|
end
|
138
169
|
|
139
|
-
def
|
140
|
-
|
141
|
-
|
142
|
-
|
170
|
+
def read_and_log_error_rate
|
171
|
+
failures = failure_count
|
172
|
+
success = success_count
|
173
|
+
rate = error_rate(failures, success)
|
174
|
+
log_metrics(rate, failures, success)
|
175
|
+
rate
|
143
176
|
end
|
144
177
|
|
145
178
|
def success!
|
146
179
|
log_event :success
|
147
180
|
circuit_store.delete(storage_key(:half_open))
|
148
|
-
clear_failures!
|
149
181
|
end
|
150
182
|
|
151
183
|
def failure!
|
@@ -154,7 +186,7 @@ class Circuitbox
|
|
154
186
|
|
155
187
|
# Store success/failure/open/close data in memcache
|
156
188
|
def log_event(event)
|
157
|
-
notifier.
|
189
|
+
notifier.new(service,partition).notify(event)
|
158
190
|
log_event_to_process(event)
|
159
191
|
|
160
192
|
if stat_store.present?
|
@@ -163,6 +195,22 @@ class Circuitbox
|
|
163
195
|
end
|
164
196
|
end
|
165
197
|
|
198
|
+
def log_metrics(error_rate, failures, successes)
|
199
|
+
n = notifier.new(service,partition)
|
200
|
+
n.metric_gauge(:error_rate, error_rate)
|
201
|
+
n.metric_gauge(:failure_count, failures)
|
202
|
+
n.metric_gauge(:success_count, successes)
|
203
|
+
end
|
204
|
+
|
205
|
+
def sanitize_options
|
206
|
+
sleep_window = option_value(:sleep_window)
|
207
|
+
time_window = option_value(:time_window)
|
208
|
+
if sleep_window < time_window
|
209
|
+
notifier.new(service,partition).notify_warning("sleep_window:#{sleep_window} is shorter than time_window:#{time_window}, the error_rate could not be reset properly after a sleep. sleep_window as been set to equal time_window.")
|
210
|
+
@circuit_options[:sleep_window] = option_value(:time_window)
|
211
|
+
end
|
212
|
+
end
|
213
|
+
|
166
214
|
# When there is a successful response within a stat interval, clear the failures.
|
167
215
|
def clear_failures!
|
168
216
|
circuit_store.write(stat_storage_key(:failure), 0, raw: true)
|
@@ -201,7 +249,14 @@ class Circuitbox
|
|
201
249
|
end
|
202
250
|
|
203
251
|
def stat_storage_key(event, options = {})
|
204
|
-
storage_key(:stats,
|
252
|
+
storage_key(:stats, align_time_on_minute, event, options)
|
253
|
+
end
|
254
|
+
|
255
|
+
# return time representation in seconds
|
256
|
+
def align_time_on_minute(time=nil)
|
257
|
+
time ||= @time_class.now.to_i
|
258
|
+
time_window = option_value(:time_window)
|
259
|
+
time - ( time % time_window ) # remove rest of integer division
|
205
260
|
end
|
206
261
|
|
207
262
|
def storage_key(*args)
|
@@ -225,11 +280,11 @@ class Circuitbox
|
|
225
280
|
end
|
226
281
|
|
227
282
|
def stats_for_time(time, options = {})
|
228
|
-
stats = { time: time }
|
283
|
+
stats = { time: align_time_on_minute(time) }
|
229
284
|
[:success, :failure, :open].each do |event|
|
230
285
|
stats[event] = stat_store.read(storage_key(:stats, time, event, options), raw: true) || 0
|
231
286
|
end
|
232
287
|
stats
|
233
288
|
end
|
234
289
|
end
|
235
|
-
end
|
290
|
+
end
|
data/lib/circuitbox/notifier.rb
CHANGED
@@ -1,12 +1,34 @@
|
|
1
1
|
class Circuitbox
|
2
2
|
class Notifier
|
3
|
-
def
|
4
|
-
|
5
|
-
|
6
|
-
|
7
|
-
circuit_name += ":#{partition}" if partition
|
3
|
+
def initialize(service, partition=nil)
|
4
|
+
@service = service
|
5
|
+
@partition = partition
|
6
|
+
end
|
8
7
|
|
8
|
+
def notify(event)
|
9
|
+
return unless notification_available?
|
9
10
|
ActiveSupport::Notifications.instrument("circuit_#{event}", circuit: circuit_name)
|
10
11
|
end
|
12
|
+
|
13
|
+
def notify_warning(message)
|
14
|
+
return unless notification_available?
|
15
|
+
ActiveSupport::Notifications.instrument("circuit_warning", { circuit: circuit_name, message: message})
|
16
|
+
end
|
17
|
+
|
18
|
+
def metric_gauge(gauge, value)
|
19
|
+
return unless notification_available?
|
20
|
+
ActiveSupport::Notifications.instrument("circuit_gauge", { circuit: circuit_name, gauge: gauge.to_s, value: value })
|
21
|
+
end
|
22
|
+
|
23
|
+
private
|
24
|
+
def circuit_name
|
25
|
+
circuit_name = @service.to_s
|
26
|
+
circuit_name += ":#{@partition}" if @partition
|
27
|
+
circuit_name
|
28
|
+
end
|
29
|
+
|
30
|
+
def notification_available?
|
31
|
+
defined? ActiveSupport::Notifications
|
32
|
+
end
|
11
33
|
end
|
12
|
-
end
|
34
|
+
end
|
data/lib/circuitbox/version.rb
CHANGED
@@ -1,5 +1,6 @@
|
|
1
1
|
require 'test_helper'
|
2
2
|
require 'circuitbox'
|
3
|
+
require 'ostruct'
|
3
4
|
|
4
5
|
class CircuitBreakerTest < Minitest::Test
|
5
6
|
SUCCESSFUL_RESPONSE_STRING = "Success!"
|
@@ -11,12 +12,183 @@ class CircuitBreakerTest < Minitest::Test
|
|
11
12
|
Circuitbox::CircuitBreaker.reset
|
12
13
|
end
|
13
14
|
|
15
|
+
describe 'initialize' do
|
16
|
+
it 'force sleep_window to equal time_window if it is too short' do
|
17
|
+
circuit = Circuitbox::CircuitBreaker.new(:yammer,
|
18
|
+
:sleep_window => 1,
|
19
|
+
:time_window => 10
|
20
|
+
)
|
21
|
+
assert_equal circuit.option_value(:sleep_window),
|
22
|
+
circuit.option_value(:time_window),
|
23
|
+
'sleep_window has not been corrected properly'
|
24
|
+
end
|
25
|
+
end
|
26
|
+
|
14
27
|
def test_goes_into_half_open_state_on_sleep
|
15
28
|
circuit = Circuitbox::CircuitBreaker.new(:yammer)
|
16
29
|
circuit.send(:open!)
|
17
30
|
assert circuit.send(:half_open?)
|
18
31
|
end
|
19
32
|
|
33
|
+
|
34
|
+
describe 'ratio' do
|
35
|
+
def cb_options
|
36
|
+
{
|
37
|
+
sleep_window: 300,
|
38
|
+
volume_threshold: 5,
|
39
|
+
error_threshold: 33,
|
40
|
+
timeout_seconds: 1
|
41
|
+
}
|
42
|
+
end
|
43
|
+
|
44
|
+
def setup
|
45
|
+
Circuitbox::CircuitBreaker.reset
|
46
|
+
@circuit = Circuitbox::CircuitBreaker.new(:yammer, cb_options)
|
47
|
+
end
|
48
|
+
|
49
|
+
|
50
|
+
it 'open the circuit on 100% failure' do
|
51
|
+
run_counter = 0
|
52
|
+
10.times do
|
53
|
+
@circuit.run do
|
54
|
+
run_counter += 1
|
55
|
+
raise RequestFailureError
|
56
|
+
end
|
57
|
+
end
|
58
|
+
assert_equal 6, run_counter, 'the circuit did not open after 6 failures (5 failures + 10%)'
|
59
|
+
end
|
60
|
+
|
61
|
+
it 'keep circuit closed on 0% failure' do
|
62
|
+
run_counter = 0
|
63
|
+
10.times do
|
64
|
+
@circuit.run do
|
65
|
+
run_counter += 1
|
66
|
+
'sucess'
|
67
|
+
end
|
68
|
+
end
|
69
|
+
assert_equal 10, run_counter, 'run block was not executed 10 times'
|
70
|
+
end
|
71
|
+
|
72
|
+
it 'open the circuit even after 1 success' do
|
73
|
+
run_counter = 0
|
74
|
+
5.times do
|
75
|
+
@circuit.run do
|
76
|
+
run_counter += 1
|
77
|
+
raise RequestFailureError
|
78
|
+
end
|
79
|
+
end
|
80
|
+
|
81
|
+
# one success
|
82
|
+
@circuit.run { 'success'}
|
83
|
+
assert_equal 5, @circuit.failure_count, 'the total count of failures is not 5'
|
84
|
+
|
85
|
+
5.times do
|
86
|
+
@circuit.run do
|
87
|
+
run_counter += 1
|
88
|
+
raise RequestFailureError
|
89
|
+
end
|
90
|
+
end
|
91
|
+
assert_equal 5, run_counter, 'the circuit did not open after 5 failures (5 failures + 10%)'
|
92
|
+
end
|
93
|
+
|
94
|
+
it 'keep circuit closed when failure ratio do not exceed limit' do
|
95
|
+
run_counter = 0
|
96
|
+
7.times do
|
97
|
+
@circuit.run do
|
98
|
+
run_counter += 1
|
99
|
+
'sucess'
|
100
|
+
end
|
101
|
+
end
|
102
|
+
assert_equal 0, @circuit.failure_count, 'some errors were counted'
|
103
|
+
|
104
|
+
3.times do
|
105
|
+
@circuit.run do
|
106
|
+
run_counter += 1
|
107
|
+
raise RequestFailureError
|
108
|
+
end
|
109
|
+
end
|
110
|
+
assert_equal 10, run_counter, 'block was not executed 10 times'
|
111
|
+
assert @circuit.error_rate < 33, 'error_rate pass over 33%'
|
112
|
+
end
|
113
|
+
|
114
|
+
it 'circuit open when failure ratio exceed limit' do
|
115
|
+
run_counter = 0
|
116
|
+
10.times do
|
117
|
+
@circuit.run do
|
118
|
+
run_counter += 1
|
119
|
+
'sucess'
|
120
|
+
end
|
121
|
+
end
|
122
|
+
assert_equal 0, @circuit.failure_count, 'some errors were counted'
|
123
|
+
|
124
|
+
10.times do
|
125
|
+
@circuit.run do
|
126
|
+
run_counter += 1
|
127
|
+
raise RequestFailureError
|
128
|
+
end
|
129
|
+
end
|
130
|
+
# 5 failure on 15 run is 33%
|
131
|
+
assert_equal 15, run_counter, 'block was not executed 10 times'
|
132
|
+
assert @circuit.error_rate >= 33, 'error_rate pass over 33%'
|
133
|
+
end
|
134
|
+
|
135
|
+
end
|
136
|
+
|
137
|
+
describe 'closing the circuit after sleep' do
|
138
|
+
class GodTime < SimpleDelegator
|
139
|
+
def now
|
140
|
+
self
|
141
|
+
end
|
142
|
+
|
143
|
+
def initialize(now=nil)
|
144
|
+
@now = now || Time.now
|
145
|
+
super(@now)
|
146
|
+
end
|
147
|
+
|
148
|
+
def __getobj__
|
149
|
+
@now
|
150
|
+
end
|
151
|
+
|
152
|
+
def __setobj__(obj)
|
153
|
+
@now = obj
|
154
|
+
end
|
155
|
+
|
156
|
+
def jump(interval)
|
157
|
+
__setobj__ @now + interval
|
158
|
+
end
|
159
|
+
end
|
160
|
+
|
161
|
+
def cb_options
|
162
|
+
{
|
163
|
+
sleep_window: 70,
|
164
|
+
time_window: 60,
|
165
|
+
volume_threshold: 5,
|
166
|
+
error_threshold: 33,
|
167
|
+
timeout_seconds: 1,
|
168
|
+
time_class: @timer
|
169
|
+
}
|
170
|
+
end
|
171
|
+
|
172
|
+
def setup
|
173
|
+
@timer = GodTime.new
|
174
|
+
@circuit = Circuitbox::CircuitBreaker.new(:yammer, cb_options)
|
175
|
+
end
|
176
|
+
|
177
|
+
|
178
|
+
it 'close the circuit after sleeping time' do
|
179
|
+
# lets open the circuit
|
180
|
+
10.times { @circuit.run { raise RequestFailureError } }
|
181
|
+
run_count = 0
|
182
|
+
@circuit.run { run_count += 1 }
|
183
|
+
assert_equal 0, run_count, 'circuit is not open'
|
184
|
+
|
185
|
+
@timer.jump(cb_options[:sleep_window] + 1)
|
186
|
+
@circuit.try_close_next_time
|
187
|
+
@circuit.run { run_count += 1 }
|
188
|
+
assert_equal 1, run_count, 'circuit is not closed'
|
189
|
+
end
|
190
|
+
end
|
191
|
+
|
20
192
|
describe "when in half open state" do
|
21
193
|
before do
|
22
194
|
Circuitbox::CircuitBreaker.reset
|
@@ -129,7 +301,7 @@ class CircuitBreakerTest < Minitest::Test
|
|
129
301
|
|
130
302
|
def test_open_checks_error_rate_threshold
|
131
303
|
circuit = Circuitbox::CircuitBreaker.new(:yammer)
|
132
|
-
circuit.stubs(:open_flag? => false,
|
304
|
+
circuit.stubs(:open_flag? => false,
|
133
305
|
:passed_volume_threshold? => true)
|
134
306
|
|
135
307
|
circuit.expects(:passed_rate_threshold?).once
|
@@ -138,7 +310,7 @@ class CircuitBreakerTest < Minitest::Test
|
|
138
310
|
|
139
311
|
def test_open_is_false_if_awake_and_under_rate_threshold
|
140
312
|
circuit = Circuitbox::CircuitBreaker.new(:yammer)
|
141
|
-
circuit.stubs(:open_flag? => false,
|
313
|
+
circuit.stubs(:open_flag? => false,
|
142
314
|
:passed_volume_threshold? => false,
|
143
315
|
:passed_rate_threshold => false)
|
144
316
|
|
@@ -185,12 +357,93 @@ class CircuitBreakerTest < Minitest::Test
|
|
185
357
|
assert_equal 0, circuit.send(:success_count)
|
186
358
|
end
|
187
359
|
|
188
|
-
|
189
|
-
|
190
|
-
|
191
|
-
|
360
|
+
describe 'notifications' do
|
361
|
+
|
362
|
+
def setup
|
363
|
+
Circuitbox::CircuitBreaker.reset
|
364
|
+
end
|
365
|
+
|
366
|
+
def circuit
|
367
|
+
Circuitbox::CircuitBreaker.new(:yammer, :notifier_class => @notifier)
|
368
|
+
end
|
369
|
+
|
370
|
+
|
371
|
+
it 'notifies on open circuit' do
|
372
|
+
@notifier = gimme_notifier
|
373
|
+
c = circuit
|
374
|
+
10.times { c.run { raise RequestFailureError }}
|
375
|
+
assert @notifier.notified?, 'no notification sent'
|
376
|
+
end
|
377
|
+
|
378
|
+
it 'notifies on close circuit' do
|
379
|
+
@notifier = gimme_notifier
|
380
|
+
c = circuit
|
381
|
+
5.times { c.run { raise RequestFailureError }}
|
382
|
+
clear_notified!
|
383
|
+
10.times { c.run { 'success' }}
|
384
|
+
assert @notifier.notified?, 'no notification sent'
|
385
|
+
end
|
386
|
+
|
387
|
+
it 'notifies warning if sleep_window is shorter than time_window' do
|
388
|
+
@notifier = gimme_notifier
|
389
|
+
Circuitbox::CircuitBreaker.new(:yammer,
|
390
|
+
:notifier_class => @notifier,
|
391
|
+
:sleep_window => 1,
|
392
|
+
:time_window => 10
|
393
|
+
)
|
394
|
+
assert @notifier.notified?, 'no notification sent'
|
395
|
+
end
|
396
|
+
|
397
|
+
it 'DO NOT notifies warning if sleep_window is longer than time_window' do
|
398
|
+
@notifier = gimme_notifier
|
399
|
+
Circuitbox::CircuitBreaker.new(:yammer,
|
400
|
+
:notifier_class => @notifier,
|
401
|
+
:sleep_window => 11,
|
402
|
+
:time_window => 10
|
403
|
+
)
|
404
|
+
assert_equal false, @notifier.notified?, 'no notification sent'
|
405
|
+
end
|
406
|
+
|
407
|
+
|
408
|
+
it 'notifies error_rate on error_rate calculation' do
|
409
|
+
@notifier = gimme_notifier(metric: :error_rate, metric_value: 0.0)
|
410
|
+
10.times { circuit.run {'success' }}
|
411
|
+
assert @notifier.notified?, 'no notification sent'
|
412
|
+
end
|
413
|
+
|
414
|
+
it 'notifies failure_count on error_rate calculation' do
|
415
|
+
@notifier = gimme_notifier(metric: :failure_count, metric_value: 1)
|
416
|
+
10.times { circuit.run { raise RequestFailureError }}
|
417
|
+
assert @notifier.notified?, 'no notification sent'
|
418
|
+
end
|
419
|
+
|
420
|
+
it 'notifies success_count on error_rate calculation' do
|
421
|
+
@notifier = gimme_notifier(metric: :success_count, metric_value: 6)
|
422
|
+
10.times { circuit.run { 'success' }}
|
423
|
+
assert @notifier.notified?, 'no notification sent'
|
424
|
+
end
|
425
|
+
|
426
|
+
def clear_notified!
|
427
|
+
@notified = false
|
428
|
+
end
|
429
|
+
|
430
|
+
def gimme_notifier(opts={})
|
431
|
+
clear_notified!
|
432
|
+
metric = opts.fetch(:metric,:error_rate)
|
433
|
+
metric_value = opts.fetch(:metric_value, 0.0)
|
434
|
+
warning_msg = opts.fetch(:warning_msg, '')
|
435
|
+
fake_notifier = gimme
|
436
|
+
give(fake_notifier).notify(:open) { @notified=true }
|
437
|
+
give(fake_notifier).notify(:close) { @notified=true }
|
438
|
+
give(fake_notifier).notify_warning(Gimme::Matchers::Anything.new) { @notified = true }
|
439
|
+
give(fake_notifier).metric_gauge(metric, metric_value) { @notified=true }
|
440
|
+
fake_notifier_class = gimme
|
441
|
+
give(fake_notifier_class).new(:yammer,nil) { fake_notifier }
|
442
|
+
give(fake_notifier_class).notified? { @notified }
|
443
|
+
fake_notifier_class
|
444
|
+
end
|
192
445
|
end
|
193
|
-
|
446
|
+
|
194
447
|
def emulate_circuit_run(circuit, response_type, response_value)
|
195
448
|
circuit.run do
|
196
449
|
case response_type
|
data/test/notifier_test.rb
CHANGED
@@ -3,8 +3,19 @@ require 'circuitbox/notifier'
|
|
3
3
|
require 'active_support/notifications'
|
4
4
|
|
5
5
|
describe Circuitbox::Notifier do
|
6
|
-
it "sends an ActiveSupport::Notification" do
|
7
|
-
ActiveSupport::Notifications.expects(:instrument).with("circuit_open", circuit: :
|
8
|
-
Circuitbox::Notifier.
|
6
|
+
it "[notify] sends an ActiveSupport::Notification" do
|
7
|
+
ActiveSupport::Notifications.expects(:instrument).with("circuit_open", circuit: 'yammer:12')
|
8
|
+
Circuitbox::Notifier.new(:yammer, 12).notify(:open)
|
9
9
|
end
|
10
|
-
|
10
|
+
|
11
|
+
it "[notify_warning] sends an ActiveSupport::Notification" do
|
12
|
+
ActiveSupport::Notifications.expects(:instrument).with("circuit_warning", { circuit: 'yammer:12', message: 'hello'})
|
13
|
+
Circuitbox::Notifier.new(:yammer, 12).notify_warning('hello')
|
14
|
+
end
|
15
|
+
|
16
|
+
it '[gauge] sends an ActiveSupport::Notifier' do
|
17
|
+
ActiveSupport::Notifications.expects(:instrument).with("circuit_gauge", { circuit: 'yammer:12', gauge: 'ratio', value: 12})
|
18
|
+
Circuitbox::Notifier.new(:yammer, 12).metric_gauge(:ratio, 12)
|
19
|
+
|
20
|
+
end
|
21
|
+
end
|
metadata
CHANGED
@@ -1,55 +1,69 @@
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
2
2
|
name: circuitbox
|
3
3
|
version: !ruby/object:Gem::Version
|
4
|
-
version: 0.
|
4
|
+
version: 0.7.0
|
5
5
|
platform: ruby
|
6
6
|
authors:
|
7
7
|
- Fahim Ferdous
|
8
8
|
autorequire:
|
9
9
|
bindir: bin
|
10
10
|
cert_chain: []
|
11
|
-
date: 2014-
|
11
|
+
date: 2014-10-10 00:00:00.000000000 Z
|
12
12
|
dependencies:
|
13
13
|
- !ruby/object:Gem::Dependency
|
14
14
|
name: bundler
|
15
15
|
requirement: !ruby/object:Gem::Requirement
|
16
16
|
requirements:
|
17
|
-
- - ~>
|
17
|
+
- - "~>"
|
18
18
|
- !ruby/object:Gem::Version
|
19
19
|
version: '1.4'
|
20
20
|
type: :development
|
21
21
|
prerelease: false
|
22
22
|
version_requirements: !ruby/object:Gem::Requirement
|
23
23
|
requirements:
|
24
|
-
- - ~>
|
24
|
+
- - "~>"
|
25
25
|
- !ruby/object:Gem::Version
|
26
26
|
version: '1.4'
|
27
27
|
- !ruby/object:Gem::Dependency
|
28
28
|
name: rake
|
29
29
|
requirement: !ruby/object:Gem::Requirement
|
30
30
|
requirements:
|
31
|
-
- -
|
31
|
+
- - ">="
|
32
32
|
- !ruby/object:Gem::Version
|
33
33
|
version: '0'
|
34
34
|
type: :development
|
35
35
|
prerelease: false
|
36
36
|
version_requirements: !ruby/object:Gem::Requirement
|
37
37
|
requirements:
|
38
|
-
- -
|
38
|
+
- - ">="
|
39
|
+
- !ruby/object:Gem::Version
|
40
|
+
version: '0'
|
41
|
+
- !ruby/object:Gem::Dependency
|
42
|
+
name: bundler-gem_version_tasks
|
43
|
+
requirement: !ruby/object:Gem::Requirement
|
44
|
+
requirements:
|
45
|
+
- - ">="
|
46
|
+
- !ruby/object:Gem::Version
|
47
|
+
version: '0'
|
48
|
+
type: :development
|
49
|
+
prerelease: false
|
50
|
+
version_requirements: !ruby/object:Gem::Requirement
|
51
|
+
requirements:
|
52
|
+
- - ">="
|
39
53
|
- !ruby/object:Gem::Version
|
40
54
|
version: '0'
|
41
55
|
- !ruby/object:Gem::Dependency
|
42
56
|
name: activesupport
|
43
57
|
requirement: !ruby/object:Gem::Requirement
|
44
58
|
requirements:
|
45
|
-
- -
|
59
|
+
- - ">="
|
46
60
|
- !ruby/object:Gem::Version
|
47
61
|
version: '0'
|
48
62
|
type: :runtime
|
49
63
|
prerelease: false
|
50
64
|
version_requirements: !ruby/object:Gem::Requirement
|
51
65
|
requirements:
|
52
|
-
- -
|
66
|
+
- - ">="
|
53
67
|
- !ruby/object:Gem::Version
|
54
68
|
version: '0'
|
55
69
|
description: A robust circuit breaker that manages failing external services.
|
@@ -59,7 +73,7 @@ executables: []
|
|
59
73
|
extensions: []
|
60
74
|
extra_rdoc_files: []
|
61
75
|
files:
|
62
|
-
- .gitignore
|
76
|
+
- ".gitignore"
|
63
77
|
- Gemfile
|
64
78
|
- Guardfile
|
65
79
|
- LICENSE
|
@@ -89,12 +103,12 @@ require_paths:
|
|
89
103
|
- lib
|
90
104
|
required_ruby_version: !ruby/object:Gem::Requirement
|
91
105
|
requirements:
|
92
|
-
- -
|
106
|
+
- - ">="
|
93
107
|
- !ruby/object:Gem::Version
|
94
108
|
version: '0'
|
95
109
|
required_rubygems_version: !ruby/object:Gem::Requirement
|
96
110
|
requirements:
|
97
|
-
- -
|
111
|
+
- - ">="
|
98
112
|
- !ruby/object:Gem::Version
|
99
113
|
version: '0'
|
100
114
|
requirements: []
|