supervision 0.1.0 → 0.2.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGELOG.md +12 -0
- data/README.md +20 -10
- data/lib/supervision.rb +30 -2
- data/lib/supervision/circuit_breaker.rb +85 -21
- data/lib/supervision/circuit_control.rb +61 -19
- data/lib/supervision/circuit_monitor.rb +47 -2
- data/lib/supervision/circuit_system.rb +31 -2
- data/lib/supervision/configuration.rb +15 -2
- data/lib/supervision/counter.rb +49 -0
- data/lib/supervision/registry.rb +37 -3
- data/lib/supervision/version.rb +1 -1
- data/spec/spec_helper.rb +25 -0
- data/spec/unit/circuit_breaker_spec.rb +37 -12
- data/spec/unit/circuit_control_spec.rb +49 -18
- data/spec/unit/circuit_monitor_spec.rb +21 -0
- data/spec/unit/circuit_system_spec.rb +42 -0
- data/spec/unit/configuration_spec.rb +29 -3
- data/spec/unit/counter_spec.rb +34 -0
- data/spec/unit/initialize_spec.rb +68 -11
- data/spec/unit/registry_spec.rb +38 -7
- data/supervision.gemspec +1 -1
- data/tasks/console.rake +1 -1
- metadata +10 -4
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
2
|
SHA1:
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
3
|
+
metadata.gz: 25069ea70f148108ba3d577443b7a0d1a7deda62
|
4
|
+
data.tar.gz: 66d41d2d5178262807f6bab0a33f8b38189d981d
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: b4541f7b303608839038831f34f57968eabcaf924dcd7b57b638831e2370dbe9d8eb555f5b4e288a5186dbd6a30eed343b0910e3716ada40e8f69056f8f4e90c
|
7
|
+
data.tar.gz: 67fdf192edd72807c1406535be44daeecf982e5576dca42189b570d7b8b9b8499e94d7c984f7d3291ae2f1b371e3c1667acb26b1c88804f7623a783b3257e876
|
data/CHANGELOG.md
ADDED
@@ -0,0 +1,12 @@
|
|
1
|
+
0.2.0 (May 12, 2014)
|
2
|
+
|
3
|
+
* Add InvalidParameterError, DuplicateEntryError types
|
4
|
+
* Add on_success & on_failure callbacks
|
5
|
+
* Change configuration to have more expressive setters
|
6
|
+
* Add shutdown to circuit system
|
7
|
+
* Add ability for dynamic calls on Supervision module
|
8
|
+
* Add ability to call supervised methods directly on object
|
9
|
+
when Supervision is included as a module
|
10
|
+
* Add ability to force reset circuit to closed state
|
11
|
+
* Add tests to ensure reset scheduler works properly
|
12
|
+
* Add ability to query configuration options on supervision instance
|
data/README.md
CHANGED
@@ -2,10 +2,12 @@
|
|
2
2
|
[][gem]
|
3
3
|
[][travis]
|
4
4
|
[][codeclimate]
|
5
|
+
[][coverage]
|
5
6
|
|
6
7
|
[gem]: http://badge.fury.io/rb/supervision
|
7
8
|
[travis]: http://travis-ci.org/peter-murach/supervision
|
8
9
|
[codeclimate]: https://codeclimate.com/github/peter-murach/supervision
|
10
|
+
[coverage]: https://coveralls.io/r/peter-murach/supervision
|
9
11
|
|
10
12
|
Write distributed systems that are resilient and self-heal. Remote calls can fail or hang indefinietly without a response.
|
11
13
|
**Supervision** will help to isolate failure and keep individual components from bringing down the whole system.
|
@@ -52,21 +54,29 @@ Once the call is wrapped you can execute it by sending `call` messsage with argu
|
|
52
54
|
@supervision.call({user: 'Piotr'})
|
53
55
|
```
|
54
56
|
|
55
|
-
|
57
|
+
## 2 System
|
58
|
+
|
59
|
+
You can register more than one **Supervision** by using internal register system. Simply register name under which you want the circuit to be available by calling `supervise_as` helper:
|
56
60
|
|
57
61
|
```ruby
|
58
62
|
Supervision.supervise_as(:danger) { remote_api_call }
|
59
63
|
```
|
60
64
|
|
61
|
-
|
65
|
+
In order to retrieve registered circuit you can use hash syntax:
|
66
|
+
|
67
|
+
```ruby
|
68
|
+
Supervision[:danger] # => returns registered circuit
|
69
|
+
```
|
70
|
+
|
71
|
+
The name under which method is registerd will be available as a method call. Consequently, to execute registered circuit do:
|
62
72
|
|
63
73
|
```ruby
|
64
|
-
Supervision.danger
|
74
|
+
Supervision.danger(:foo, :bar) # => will call underlying method and pass :foo, :barr
|
65
75
|
```
|
66
76
|
|
67
|
-
##
|
77
|
+
## 3 Mixin
|
68
78
|
|
69
|
-
**Supervision** can also act as a mixin and expose `supervise` and `supervise_as` accordingly.
|
79
|
+
**Supervision** can also act as a mixin and expose `supervise` and `supervise_as` accordingly. Use `supervise_as` if you want to be able to register supervised calls inside **Supervision** system. Otherwise, use `supervise` helper to create anonymous supervised call.
|
70
80
|
|
71
81
|
```ruby
|
72
82
|
class Api
|
@@ -75,10 +85,10 @@ class Api
|
|
75
85
|
def remote_call
|
76
86
|
...
|
77
87
|
end
|
78
|
-
|
88
|
+
supervise_as :danger { remote_call } # => register supervision as :danger
|
79
89
|
|
80
90
|
def fetch(repository)
|
81
|
-
danger
|
91
|
+
danger(repository)
|
82
92
|
rescue Supervision::CircuitBreakerOpenError
|
83
93
|
nil
|
84
94
|
end
|
@@ -88,7 +98,7 @@ end
|
|
88
98
|
@api.fetch('github_api')
|
89
99
|
```
|
90
100
|
|
91
|
-
##
|
101
|
+
## 4 Callbacks
|
92
102
|
|
93
103
|
You can listen for `failure` and `success` by attaching `on_failure`, `on_success` listeners respectively:
|
94
104
|
|
@@ -100,7 +110,7 @@ def notify_me
|
|
100
110
|
end
|
101
111
|
```
|
102
112
|
|
103
|
-
##
|
113
|
+
## 5 Configuration
|
104
114
|
|
105
115
|
If you want to configure **Supervision**, you can either pass options directly
|
106
116
|
|
@@ -120,7 +130,7 @@ or use `configure` helper
|
|
120
130
|
end
|
121
131
|
```
|
122
132
|
|
123
|
-
##
|
133
|
+
## 6 Time
|
124
134
|
|
125
135
|
All the numeric types are extended with time related helpers to allow for more fluid parameters when creating **Supervision**
|
126
136
|
|
data/lib/supervision.rb
CHANGED
@@ -2,6 +2,7 @@
|
|
2
2
|
|
3
3
|
require "thread"
|
4
4
|
require "timeout"
|
5
|
+
require "forwardable"
|
5
6
|
require "finite_machine"
|
6
7
|
|
7
8
|
require "supervision/version"
|
@@ -13,6 +14,7 @@ require "supervision/circuit_control"
|
|
13
14
|
require "supervision/circuit_breaker"
|
14
15
|
require "supervision/circuit_system"
|
15
16
|
require "supervision/circuit_monitor"
|
17
|
+
require "supervision/counter"
|
16
18
|
|
17
19
|
module Supervision
|
18
20
|
# Generic error
|
@@ -21,8 +23,15 @@ module Supervision
|
|
21
23
|
# Raised when circuit opens
|
22
24
|
CircuitBreakerOpenError = Class.new(SupervisionError)
|
23
25
|
|
26
|
+
# Raised when checking circuit type
|
24
27
|
TypeError = Class.new(SupervisionError)
|
25
28
|
|
29
|
+
# Raised when invalid configuration parameter is specified
|
30
|
+
InvalidParameterError = Class.new(SupervisionError)
|
31
|
+
|
32
|
+
# Raised when registering duplicate circuit breaker name
|
33
|
+
DuplicateEntryError = Class.new(SupervisionError)
|
34
|
+
|
26
35
|
class << self
|
27
36
|
def included(base)
|
28
37
|
base.send :extend, ClassMethods
|
@@ -32,6 +41,9 @@ module Supervision
|
|
32
41
|
@configuration ||= Configuration.new
|
33
42
|
end
|
34
43
|
|
44
|
+
# Initialize a circuit system
|
45
|
+
#
|
46
|
+
# @api private
|
35
47
|
def init
|
36
48
|
@circuit_system = CircuitSystem.new
|
37
49
|
end
|
@@ -46,6 +58,22 @@ module Supervision
|
|
46
58
|
def new(name = nil, options = {}, &block)
|
47
59
|
name ? supervise_as(name, options, &block) : supervise(options, &block)
|
48
60
|
end
|
61
|
+
|
62
|
+
# Retrieve circuit by name
|
63
|
+
#
|
64
|
+
# @return [Supervision::CircuitBreaker]
|
65
|
+
#
|
66
|
+
# @api public
|
67
|
+
def [](name)
|
68
|
+
circuit_system[name]
|
69
|
+
end
|
70
|
+
|
71
|
+
private
|
72
|
+
|
73
|
+
def method_missing(method_name, *args, &block)
|
74
|
+
super unless circuit_system.registered?(method_name)
|
75
|
+
self[method_name].call(*args)
|
76
|
+
end
|
49
77
|
end
|
50
78
|
|
51
79
|
module ClassMethods
|
@@ -54,8 +82,8 @@ module Supervision
|
|
54
82
|
end
|
55
83
|
|
56
84
|
def supervise_as(name, options = {}, &block)
|
57
|
-
circuit = supervise(options, &block)
|
58
|
-
Supervision.circuit_system
|
85
|
+
circuit = supervise(options.merge!(name: name), &block)
|
86
|
+
Supervision.circuit_system.register(name, circuit)
|
59
87
|
send(:define_method, name) { |*args| circuit.call(args) }
|
60
88
|
circuit
|
61
89
|
end
|
@@ -4,19 +4,29 @@ module Supervision
|
|
4
4
|
# A class responsible for protecting remote calls
|
5
5
|
class CircuitBreaker
|
6
6
|
include Timeout
|
7
|
+
extend Forwardable
|
7
8
|
|
8
9
|
attr_reader :control
|
9
10
|
|
11
|
+
attr_reader :name
|
12
|
+
|
13
|
+
def_delegators :@control, :current, :max_failures, :call_timeout,
|
14
|
+
:reset_timeout
|
15
|
+
|
16
|
+
# Create a CircuitBreaker
|
17
|
+
#
|
18
|
+
# @example
|
19
|
+
# circuit = CircuitBreaker { ... }
|
20
|
+
#
|
21
|
+
# @api public
|
10
22
|
def initialize(options = {}, &block)
|
11
23
|
if block.nil?
|
12
|
-
raise
|
24
|
+
raise InvalidParameterError, 'CircuitBreaker.new requires a block'
|
13
25
|
end
|
26
|
+
@name = options.delete(:name)
|
14
27
|
@control = CircuitControl.new(options)
|
15
28
|
@circuit = Atomic.new(block)
|
16
29
|
@mutex = Mutex.new
|
17
|
-
@before_hook = -> {}
|
18
|
-
@success_hook = -> {}
|
19
|
-
@failure_hook = -> {}
|
20
30
|
end
|
21
31
|
|
22
32
|
# Configure circuit instance parameters
|
@@ -25,11 +35,7 @@ module Supervision
|
|
25
35
|
#
|
26
36
|
# @api public
|
27
37
|
def configure(&block)
|
28
|
-
|
29
|
-
control.config.instance_eval(&block)
|
30
|
-
else
|
31
|
-
yield control.config
|
32
|
-
end
|
38
|
+
control.config.configure(&block)
|
33
39
|
end
|
34
40
|
|
35
41
|
# Executes the dangerous call
|
@@ -39,42 +45,100 @@ module Supervision
|
|
39
45
|
#
|
40
46
|
# @api public
|
41
47
|
def call(*args)
|
42
|
-
|
48
|
+
handle_before
|
43
49
|
begin
|
44
50
|
result = dispatch(*args)
|
45
|
-
|
51
|
+
handle_success
|
46
52
|
result
|
47
53
|
rescue Exception => error
|
48
|
-
|
49
|
-
@failure_hook.call
|
54
|
+
handle_failure(error)
|
50
55
|
end
|
51
56
|
end
|
52
57
|
|
53
|
-
|
54
|
-
|
55
|
-
|
56
|
-
|
58
|
+
# Reset this circuit to closed state
|
59
|
+
#
|
60
|
+
# @example
|
61
|
+
# supervision.reset!
|
62
|
+
#
|
63
|
+
# @return [nil]
|
64
|
+
#
|
65
|
+
# @api public
|
66
|
+
def reset!
|
67
|
+
control.reset!
|
57
68
|
end
|
58
69
|
|
70
|
+
# Define before handler
|
71
|
+
#
|
72
|
+
# @api public
|
59
73
|
def before(&block)
|
60
|
-
@
|
74
|
+
@before = block
|
75
|
+
self
|
61
76
|
end
|
62
77
|
|
63
|
-
#
|
78
|
+
# Define success handler
|
79
|
+
#
|
80
|
+
# @return [Supervision::CircuitBreaker]
|
64
81
|
#
|
65
82
|
# @api public
|
66
83
|
def on_success(&block)
|
67
|
-
@
|
84
|
+
@on_success = block
|
85
|
+
self
|
68
86
|
end
|
69
87
|
alias_method :on_closed, :on_success
|
70
88
|
|
89
|
+
# Define failure handler
|
90
|
+
#
|
91
|
+
# @return [Supervision::CircuitBreaker]
|
92
|
+
#
|
93
|
+
# @api public
|
71
94
|
def on_failure(&block)
|
72
|
-
@
|
95
|
+
@on_failure = block
|
96
|
+
self
|
73
97
|
end
|
74
98
|
alias_method :on_open, :on_failure
|
75
99
|
|
100
|
+
# Detailed string representation of this circuit
|
101
|
+
#
|
102
|
+
# @return [String]
|
103
|
+
#
|
104
|
+
# @api public
|
105
|
+
def inspect
|
106
|
+
"#<#{self.class.name}:#{object_id} @name=#{name}>"
|
107
|
+
end
|
108
|
+
|
109
|
+
# Detailed string representation of this circuit
|
110
|
+
#
|
111
|
+
# @return [String]
|
112
|
+
#
|
113
|
+
# @api public
|
114
|
+
def to_s
|
115
|
+
"#<#{self.class.name}:#{object_id} @name=#{name}>"
|
116
|
+
end
|
117
|
+
|
76
118
|
private
|
77
119
|
|
120
|
+
# Invoke before handler
|
121
|
+
#
|
122
|
+
# @api private
|
123
|
+
def handle_before
|
124
|
+
@before.call if @before
|
125
|
+
end
|
126
|
+
|
127
|
+
# Invoke success handler
|
128
|
+
#
|
129
|
+
# @api private
|
130
|
+
def handle_success
|
131
|
+
@on_success.call if @on_success
|
132
|
+
end
|
133
|
+
|
134
|
+
# Invoke failure handler and instrument circuit controller
|
135
|
+
#
|
136
|
+
# @api private
|
137
|
+
def handle_failure(error)
|
138
|
+
control.handle_failure(error)
|
139
|
+
@on_failure.call(error) if @on_failure
|
140
|
+
end
|
141
|
+
|
78
142
|
# Dispatch message to the current circuit
|
79
143
|
#
|
80
144
|
# @api private
|
@@ -5,10 +5,24 @@ module Supervision
|
|
5
5
|
class CircuitControl
|
6
6
|
extend Forwardable
|
7
7
|
|
8
|
-
def_delegators :@config, :max_failures, :call_timeout, :
|
8
|
+
def_delegators :@config, :max_failures, :call_timeout, :reset_timeout,
|
9
|
+
:failure_count
|
9
10
|
|
11
|
+
# The circuit configuration
|
12
|
+
#
|
13
|
+
# @api private
|
10
14
|
attr_reader :config
|
11
15
|
|
16
|
+
# The reset timeout scheduler
|
17
|
+
#
|
18
|
+
# @api private
|
19
|
+
attr_reader :scheduler
|
20
|
+
|
21
|
+
# The circuit performance monitor
|
22
|
+
#
|
23
|
+
# @api private
|
24
|
+
attr_reader :monitor
|
25
|
+
|
12
26
|
MAX_THREAD_LIFETIME = 5
|
13
27
|
|
14
28
|
# Create a circuit control
|
@@ -21,6 +35,7 @@ module Supervision
|
|
21
35
|
@failure_count = Atomic.new(0)
|
22
36
|
@last_failure_time = Atomic.new
|
23
37
|
@lock = Mutex.new
|
38
|
+
@monitor = CircuitMonitor.new
|
24
39
|
fsm
|
25
40
|
end
|
26
41
|
|
@@ -38,13 +53,13 @@ module Supervision
|
|
38
53
|
|
39
54
|
target context
|
40
55
|
|
41
|
-
events
|
56
|
+
events do
|
42
57
|
event :trip, [:closed, :half_open] => :open
|
43
|
-
event :attempt_reset, :open
|
44
|
-
event :reset, :half_open
|
45
|
-
|
58
|
+
event :attempt_reset, :open => :half_open
|
59
|
+
event :reset, :half_open => :closed
|
60
|
+
end
|
46
61
|
|
47
|
-
callbacks
|
62
|
+
callbacks do
|
48
63
|
on_enter :closed do |event|
|
49
64
|
reset_failure
|
50
65
|
end
|
@@ -55,12 +70,14 @@ module Supervision
|
|
55
70
|
end
|
56
71
|
|
57
72
|
on_enter :half_open do |event|
|
73
|
+
monitor.measure(:half_open_circuit)
|
58
74
|
end
|
59
|
-
|
75
|
+
end
|
60
76
|
end
|
61
77
|
end
|
62
78
|
|
63
|
-
def_delegators :@fsm, :trip, :
|
79
|
+
def_delegators :@fsm, :trip, :trip!, :attempt_reset, :attempt_reset!,
|
80
|
+
:reset, :current
|
64
81
|
|
65
82
|
# Total failure count for current circuit
|
66
83
|
#
|
@@ -80,13 +97,24 @@ module Supervision
|
|
80
97
|
@last_failure_time.value
|
81
98
|
end
|
82
99
|
|
100
|
+
# Force closed state and reset failure statistics
|
101
|
+
#
|
102
|
+
# @return [nil]
|
103
|
+
#
|
104
|
+
# @api public
|
105
|
+
def reset!
|
106
|
+
fsm.reset!
|
107
|
+
reset_failure
|
108
|
+
throw(:terminate) if @scheduler && @scheduler.alive?
|
109
|
+
end
|
110
|
+
|
83
111
|
# Fail fast on any call
|
84
112
|
#
|
85
113
|
# @raise [CircuitBreakerOpenError]
|
86
114
|
#
|
87
115
|
# @api private
|
88
116
|
def fail_fast!
|
89
|
-
|
117
|
+
monitor.measure(:open_circuit)
|
90
118
|
raise CircuitBreakerOpenError
|
91
119
|
end
|
92
120
|
|
@@ -120,18 +148,23 @@ module Supervision
|
|
120
148
|
# Handler exception
|
121
149
|
#
|
122
150
|
# @api public
|
123
|
-
def
|
151
|
+
def handle_failure(error = nil)
|
124
152
|
fail_fast! if fsm.open?
|
125
153
|
record_failure
|
154
|
+
monitor.record_failure
|
126
155
|
trip if failure_count_exceeded? || fsm.half_open?
|
127
156
|
end
|
128
157
|
|
158
|
+
# Record successful call
|
159
|
+
#
|
160
|
+
# @api public
|
129
161
|
def record_success
|
130
162
|
reset if fsm.half_open?
|
131
163
|
reset_failure
|
164
|
+
monitor.record_success
|
132
165
|
end
|
133
166
|
|
134
|
-
#
|
167
|
+
# Record failure count
|
135
168
|
#
|
136
169
|
# @api public
|
137
170
|
def record_failure
|
@@ -153,18 +186,27 @@ module Supervision
|
|
153
186
|
#
|
154
187
|
# @api private
|
155
188
|
def measure_timeout
|
156
|
-
Thread.new do
|
189
|
+
@scheduler = Thread.new do
|
157
190
|
Thread.current.abort_on_exception = true
|
158
191
|
thread = Thread.current
|
159
192
|
thread[:created_at] = Time.now
|
160
193
|
@lock.synchronize do
|
161
|
-
|
162
|
-
|
163
|
-
|
164
|
-
|
165
|
-
|
166
|
-
|
167
|
-
|
194
|
+
run_loop(thread)
|
195
|
+
end
|
196
|
+
end
|
197
|
+
end
|
198
|
+
|
199
|
+
# Run scheduler loop
|
200
|
+
#
|
201
|
+
# @api private
|
202
|
+
def run_loop(thread)
|
203
|
+
catch(:terminate) do
|
204
|
+
loop do
|
205
|
+
if tripped?
|
206
|
+
attempt_reset && break
|
207
|
+
elsif Time.now - thread[:created_at] > max_thread_lifetime
|
208
|
+
thread.kill if thread.alive?
|
209
|
+
break
|
168
210
|
end
|
169
211
|
end
|
170
212
|
end
|