supervision 0.1.0 → 0.2.0
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +4 -4
- data/CHANGELOG.md +12 -0
- data/README.md +20 -10
- data/lib/supervision.rb +30 -2
- data/lib/supervision/circuit_breaker.rb +85 -21
- data/lib/supervision/circuit_control.rb +61 -19
- data/lib/supervision/circuit_monitor.rb +47 -2
- data/lib/supervision/circuit_system.rb +31 -2
- data/lib/supervision/configuration.rb +15 -2
- data/lib/supervision/counter.rb +49 -0
- data/lib/supervision/registry.rb +37 -3
- data/lib/supervision/version.rb +1 -1
- data/spec/spec_helper.rb +25 -0
- data/spec/unit/circuit_breaker_spec.rb +37 -12
- data/spec/unit/circuit_control_spec.rb +49 -18
- data/spec/unit/circuit_monitor_spec.rb +21 -0
- data/spec/unit/circuit_system_spec.rb +42 -0
- data/spec/unit/configuration_spec.rb +29 -3
- data/spec/unit/counter_spec.rb +34 -0
- data/spec/unit/initialize_spec.rb +68 -11
- data/spec/unit/registry_spec.rb +38 -7
- data/supervision.gemspec +1 -1
- data/tasks/console.rake +1 -1
- metadata +10 -4
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
2
|
SHA1:
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
3
|
+
metadata.gz: 25069ea70f148108ba3d577443b7a0d1a7deda62
|
4
|
+
data.tar.gz: 66d41d2d5178262807f6bab0a33f8b38189d981d
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: b4541f7b303608839038831f34f57968eabcaf924dcd7b57b638831e2370dbe9d8eb555f5b4e288a5186dbd6a30eed343b0910e3716ada40e8f69056f8f4e90c
|
7
|
+
data.tar.gz: 67fdf192edd72807c1406535be44daeecf982e5576dca42189b570d7b8b9b8499e94d7c984f7d3291ae2f1b371e3c1667acb26b1c88804f7623a783b3257e876
|
data/CHANGELOG.md
ADDED
@@ -0,0 +1,12 @@
|
|
1
|
+
0.2.0 (May 12, 2014)
|
2
|
+
|
3
|
+
* Add InvalidParameterError, DuplicateEntryError types
|
4
|
+
* Add on_success & on_failure callbacks
|
5
|
+
* Change configuration to have more expressive setters
|
6
|
+
* Add shutdown to circuit system
|
7
|
+
* Add ability for dynamic calls on Supervision module
|
8
|
+
* Add ability to call supervised methods directly on object
|
9
|
+
when Supervision is included as a module
|
10
|
+
* Add ability to force reset circuit to closed state
|
11
|
+
* Add tests to ensure reset scheduler works properly
|
12
|
+
* Add ability to query configuration options on supervision instance
|
data/README.md
CHANGED
@@ -2,10 +2,12 @@
|
|
2
2
|
[![Gem Version](https://badge.fury.io/rb/supervision.png)][gem]
|
3
3
|
[![Build Status](https://secure.travis-ci.org/peter-murach/supervision.png?branch=master)][travis]
|
4
4
|
[![Code Climate](https://codeclimate.com/github/peter-murach/supervision.png)][codeclimate]
|
5
|
+
[![Coverage Status](https://coveralls.io/repos/peter-murach/supervision/badge.png)][coverage]
|
5
6
|
|
6
7
|
[gem]: http://badge.fury.io/rb/supervision
|
7
8
|
[travis]: http://travis-ci.org/peter-murach/supervision
|
8
9
|
[codeclimate]: https://codeclimate.com/github/peter-murach/supervision
|
10
|
+
[coverage]: https://coveralls.io/r/peter-murach/supervision
|
9
11
|
|
10
12
|
Write distributed systems that are resilient and self-heal. Remote calls can fail or hang indefinietly without a response.
|
11
13
|
**Supervision** will help to isolate failure and keep individual components from bringing down the whole system.
|
@@ -52,21 +54,29 @@ Once the call is wrapped you can execute it by sending `call` messsage with argu
|
|
52
54
|
@supervision.call({user: 'Piotr'})
|
53
55
|
```
|
54
56
|
|
55
|
-
|
57
|
+
## 2 System
|
58
|
+
|
59
|
+
You can register more than one **Supervision** by using internal register system. Simply register name under which you want the circuit to be available by calling `supervise_as` helper:
|
56
60
|
|
57
61
|
```ruby
|
58
62
|
Supervision.supervise_as(:danger) { remote_api_call }
|
59
63
|
```
|
60
64
|
|
61
|
-
|
65
|
+
In order to retrieve registered circuit you can use hash syntax:
|
66
|
+
|
67
|
+
```ruby
|
68
|
+
Supervision[:danger] # => returns registered circuit
|
69
|
+
```
|
70
|
+
|
71
|
+
The name under which method is registerd will be available as a method call. Consequently, to execute registered circuit do:
|
62
72
|
|
63
73
|
```ruby
|
64
|
-
Supervision.danger
|
74
|
+
Supervision.danger(:foo, :bar) # => will call underlying method and pass :foo, :barr
|
65
75
|
```
|
66
76
|
|
67
|
-
##
|
77
|
+
## 3 Mixin
|
68
78
|
|
69
|
-
**Supervision** can also act as a mixin and expose `supervise` and `supervise_as` accordingly.
|
79
|
+
**Supervision** can also act as a mixin and expose `supervise` and `supervise_as` accordingly. Use `supervise_as` if you want to be able to register supervised calls inside **Supervision** system. Otherwise, use `supervise` helper to create anonymous supervised call.
|
70
80
|
|
71
81
|
```ruby
|
72
82
|
class Api
|
@@ -75,10 +85,10 @@ class Api
|
|
75
85
|
def remote_call
|
76
86
|
...
|
77
87
|
end
|
78
|
-
|
88
|
+
supervise_as :danger { remote_call } # => register supervision as :danger
|
79
89
|
|
80
90
|
def fetch(repository)
|
81
|
-
danger
|
91
|
+
danger(repository)
|
82
92
|
rescue Supervision::CircuitBreakerOpenError
|
83
93
|
nil
|
84
94
|
end
|
@@ -88,7 +98,7 @@ end
|
|
88
98
|
@api.fetch('github_api')
|
89
99
|
```
|
90
100
|
|
91
|
-
##
|
101
|
+
## 4 Callbacks
|
92
102
|
|
93
103
|
You can listen for `failure` and `success` by attaching `on_failure`, `on_success` listeners respectively:
|
94
104
|
|
@@ -100,7 +110,7 @@ def notify_me
|
|
100
110
|
end
|
101
111
|
```
|
102
112
|
|
103
|
-
##
|
113
|
+
## 5 Configuration
|
104
114
|
|
105
115
|
If you want to configure **Supervision**, you can either pass options directly
|
106
116
|
|
@@ -120,7 +130,7 @@ or use `configure` helper
|
|
120
130
|
end
|
121
131
|
```
|
122
132
|
|
123
|
-
##
|
133
|
+
## 6 Time
|
124
134
|
|
125
135
|
All the numeric types are extended with time related helpers to allow for more fluid parameters when creating **Supervision**
|
126
136
|
|
data/lib/supervision.rb
CHANGED
@@ -2,6 +2,7 @@
|
|
2
2
|
|
3
3
|
require "thread"
|
4
4
|
require "timeout"
|
5
|
+
require "forwardable"
|
5
6
|
require "finite_machine"
|
6
7
|
|
7
8
|
require "supervision/version"
|
@@ -13,6 +14,7 @@ require "supervision/circuit_control"
|
|
13
14
|
require "supervision/circuit_breaker"
|
14
15
|
require "supervision/circuit_system"
|
15
16
|
require "supervision/circuit_monitor"
|
17
|
+
require "supervision/counter"
|
16
18
|
|
17
19
|
module Supervision
|
18
20
|
# Generic error
|
@@ -21,8 +23,15 @@ module Supervision
|
|
21
23
|
# Raised when circuit opens
|
22
24
|
CircuitBreakerOpenError = Class.new(SupervisionError)
|
23
25
|
|
26
|
+
# Raised when checking circuit type
|
24
27
|
TypeError = Class.new(SupervisionError)
|
25
28
|
|
29
|
+
# Raised when invalid configuration parameter is specified
|
30
|
+
InvalidParameterError = Class.new(SupervisionError)
|
31
|
+
|
32
|
+
# Raised when registering duplicate circuit breaker name
|
33
|
+
DuplicateEntryError = Class.new(SupervisionError)
|
34
|
+
|
26
35
|
class << self
|
27
36
|
def included(base)
|
28
37
|
base.send :extend, ClassMethods
|
@@ -32,6 +41,9 @@ module Supervision
|
|
32
41
|
@configuration ||= Configuration.new
|
33
42
|
end
|
34
43
|
|
44
|
+
# Initialize a circuit system
|
45
|
+
#
|
46
|
+
# @api private
|
35
47
|
def init
|
36
48
|
@circuit_system = CircuitSystem.new
|
37
49
|
end
|
@@ -46,6 +58,22 @@ module Supervision
|
|
46
58
|
def new(name = nil, options = {}, &block)
|
47
59
|
name ? supervise_as(name, options, &block) : supervise(options, &block)
|
48
60
|
end
|
61
|
+
|
62
|
+
# Retrieve circuit by name
|
63
|
+
#
|
64
|
+
# @return [Supervision::CircuitBreaker]
|
65
|
+
#
|
66
|
+
# @api public
|
67
|
+
def [](name)
|
68
|
+
circuit_system[name]
|
69
|
+
end
|
70
|
+
|
71
|
+
private
|
72
|
+
|
73
|
+
def method_missing(method_name, *args, &block)
|
74
|
+
super unless circuit_system.registered?(method_name)
|
75
|
+
self[method_name].call(*args)
|
76
|
+
end
|
49
77
|
end
|
50
78
|
|
51
79
|
module ClassMethods
|
@@ -54,8 +82,8 @@ module Supervision
|
|
54
82
|
end
|
55
83
|
|
56
84
|
def supervise_as(name, options = {}, &block)
|
57
|
-
circuit = supervise(options, &block)
|
58
|
-
Supervision.circuit_system
|
85
|
+
circuit = supervise(options.merge!(name: name), &block)
|
86
|
+
Supervision.circuit_system.register(name, circuit)
|
59
87
|
send(:define_method, name) { |*args| circuit.call(args) }
|
60
88
|
circuit
|
61
89
|
end
|
@@ -4,19 +4,29 @@ module Supervision
|
|
4
4
|
# A class responsible for protecting remote calls
|
5
5
|
class CircuitBreaker
|
6
6
|
include Timeout
|
7
|
+
extend Forwardable
|
7
8
|
|
8
9
|
attr_reader :control
|
9
10
|
|
11
|
+
attr_reader :name
|
12
|
+
|
13
|
+
def_delegators :@control, :current, :max_failures, :call_timeout,
|
14
|
+
:reset_timeout
|
15
|
+
|
16
|
+
# Create a CircuitBreaker
|
17
|
+
#
|
18
|
+
# @example
|
19
|
+
# circuit = CircuitBreaker { ... }
|
20
|
+
#
|
21
|
+
# @api public
|
10
22
|
def initialize(options = {}, &block)
|
11
23
|
if block.nil?
|
12
|
-
raise
|
24
|
+
raise InvalidParameterError, 'CircuitBreaker.new requires a block'
|
13
25
|
end
|
26
|
+
@name = options.delete(:name)
|
14
27
|
@control = CircuitControl.new(options)
|
15
28
|
@circuit = Atomic.new(block)
|
16
29
|
@mutex = Mutex.new
|
17
|
-
@before_hook = -> {}
|
18
|
-
@success_hook = -> {}
|
19
|
-
@failure_hook = -> {}
|
20
30
|
end
|
21
31
|
|
22
32
|
# Configure circuit instance parameters
|
@@ -25,11 +35,7 @@ module Supervision
|
|
25
35
|
#
|
26
36
|
# @api public
|
27
37
|
def configure(&block)
|
28
|
-
|
29
|
-
control.config.instance_eval(&block)
|
30
|
-
else
|
31
|
-
yield control.config
|
32
|
-
end
|
38
|
+
control.config.configure(&block)
|
33
39
|
end
|
34
40
|
|
35
41
|
# Executes the dangerous call
|
@@ -39,42 +45,100 @@ module Supervision
|
|
39
45
|
#
|
40
46
|
# @api public
|
41
47
|
def call(*args)
|
42
|
-
|
48
|
+
handle_before
|
43
49
|
begin
|
44
50
|
result = dispatch(*args)
|
45
|
-
|
51
|
+
handle_success
|
46
52
|
result
|
47
53
|
rescue Exception => error
|
48
|
-
|
49
|
-
@failure_hook.call
|
54
|
+
handle_failure(error)
|
50
55
|
end
|
51
56
|
end
|
52
57
|
|
53
|
-
|
54
|
-
|
55
|
-
|
56
|
-
|
58
|
+
# Reset this circuit to closed state
|
59
|
+
#
|
60
|
+
# @example
|
61
|
+
# supervision.reset!
|
62
|
+
#
|
63
|
+
# @return [nil]
|
64
|
+
#
|
65
|
+
# @api public
|
66
|
+
def reset!
|
67
|
+
control.reset!
|
57
68
|
end
|
58
69
|
|
70
|
+
# Define before handler
|
71
|
+
#
|
72
|
+
# @api public
|
59
73
|
def before(&block)
|
60
|
-
@
|
74
|
+
@before = block
|
75
|
+
self
|
61
76
|
end
|
62
77
|
|
63
|
-
#
|
78
|
+
# Define success handler
|
79
|
+
#
|
80
|
+
# @return [Supervision::CircuitBreaker]
|
64
81
|
#
|
65
82
|
# @api public
|
66
83
|
def on_success(&block)
|
67
|
-
@
|
84
|
+
@on_success = block
|
85
|
+
self
|
68
86
|
end
|
69
87
|
alias_method :on_closed, :on_success
|
70
88
|
|
89
|
+
# Define failure handler
|
90
|
+
#
|
91
|
+
# @return [Supervision::CircuitBreaker]
|
92
|
+
#
|
93
|
+
# @api public
|
71
94
|
def on_failure(&block)
|
72
|
-
@
|
95
|
+
@on_failure = block
|
96
|
+
self
|
73
97
|
end
|
74
98
|
alias_method :on_open, :on_failure
|
75
99
|
|
100
|
+
# Detailed string representation of this circuit
|
101
|
+
#
|
102
|
+
# @return [String]
|
103
|
+
#
|
104
|
+
# @api public
|
105
|
+
def inspect
|
106
|
+
"#<#{self.class.name}:#{object_id} @name=#{name}>"
|
107
|
+
end
|
108
|
+
|
109
|
+
# Detailed string representation of this circuit
|
110
|
+
#
|
111
|
+
# @return [String]
|
112
|
+
#
|
113
|
+
# @api public
|
114
|
+
def to_s
|
115
|
+
"#<#{self.class.name}:#{object_id} @name=#{name}>"
|
116
|
+
end
|
117
|
+
|
76
118
|
private
|
77
119
|
|
120
|
+
# Invoke before handler
|
121
|
+
#
|
122
|
+
# @api private
|
123
|
+
def handle_before
|
124
|
+
@before.call if @before
|
125
|
+
end
|
126
|
+
|
127
|
+
# Invoke success handler
|
128
|
+
#
|
129
|
+
# @api private
|
130
|
+
def handle_success
|
131
|
+
@on_success.call if @on_success
|
132
|
+
end
|
133
|
+
|
134
|
+
# Invoke failure handler and instrument circuit controller
|
135
|
+
#
|
136
|
+
# @api private
|
137
|
+
def handle_failure(error)
|
138
|
+
control.handle_failure(error)
|
139
|
+
@on_failure.call(error) if @on_failure
|
140
|
+
end
|
141
|
+
|
78
142
|
# Dispatch message to the current circuit
|
79
143
|
#
|
80
144
|
# @api private
|
@@ -5,10 +5,24 @@ module Supervision
|
|
5
5
|
class CircuitControl
|
6
6
|
extend Forwardable
|
7
7
|
|
8
|
-
def_delegators :@config, :max_failures, :call_timeout, :
|
8
|
+
def_delegators :@config, :max_failures, :call_timeout, :reset_timeout,
|
9
|
+
:failure_count
|
9
10
|
|
11
|
+
# The circuit configuration
|
12
|
+
#
|
13
|
+
# @api private
|
10
14
|
attr_reader :config
|
11
15
|
|
16
|
+
# The reset timeout scheduler
|
17
|
+
#
|
18
|
+
# @api private
|
19
|
+
attr_reader :scheduler
|
20
|
+
|
21
|
+
# The circuit performance monitor
|
22
|
+
#
|
23
|
+
# @api private
|
24
|
+
attr_reader :monitor
|
25
|
+
|
12
26
|
MAX_THREAD_LIFETIME = 5
|
13
27
|
|
14
28
|
# Create a circuit control
|
@@ -21,6 +35,7 @@ module Supervision
|
|
21
35
|
@failure_count = Atomic.new(0)
|
22
36
|
@last_failure_time = Atomic.new
|
23
37
|
@lock = Mutex.new
|
38
|
+
@monitor = CircuitMonitor.new
|
24
39
|
fsm
|
25
40
|
end
|
26
41
|
|
@@ -38,13 +53,13 @@ module Supervision
|
|
38
53
|
|
39
54
|
target context
|
40
55
|
|
41
|
-
events
|
56
|
+
events do
|
42
57
|
event :trip, [:closed, :half_open] => :open
|
43
|
-
event :attempt_reset, :open
|
44
|
-
event :reset, :half_open
|
45
|
-
|
58
|
+
event :attempt_reset, :open => :half_open
|
59
|
+
event :reset, :half_open => :closed
|
60
|
+
end
|
46
61
|
|
47
|
-
callbacks
|
62
|
+
callbacks do
|
48
63
|
on_enter :closed do |event|
|
49
64
|
reset_failure
|
50
65
|
end
|
@@ -55,12 +70,14 @@ module Supervision
|
|
55
70
|
end
|
56
71
|
|
57
72
|
on_enter :half_open do |event|
|
73
|
+
monitor.measure(:half_open_circuit)
|
58
74
|
end
|
59
|
-
|
75
|
+
end
|
60
76
|
end
|
61
77
|
end
|
62
78
|
|
63
|
-
def_delegators :@fsm, :trip, :
|
79
|
+
def_delegators :@fsm, :trip, :trip!, :attempt_reset, :attempt_reset!,
|
80
|
+
:reset, :current
|
64
81
|
|
65
82
|
# Total failure count for current circuit
|
66
83
|
#
|
@@ -80,13 +97,24 @@ module Supervision
|
|
80
97
|
@last_failure_time.value
|
81
98
|
end
|
82
99
|
|
100
|
+
# Force closed state and reset failure statistics
|
101
|
+
#
|
102
|
+
# @return [nil]
|
103
|
+
#
|
104
|
+
# @api public
|
105
|
+
def reset!
|
106
|
+
fsm.reset!
|
107
|
+
reset_failure
|
108
|
+
throw(:terminate) if @scheduler && @scheduler.alive?
|
109
|
+
end
|
110
|
+
|
83
111
|
# Fail fast on any call
|
84
112
|
#
|
85
113
|
# @raise [CircuitBreakerOpenError]
|
86
114
|
#
|
87
115
|
# @api private
|
88
116
|
def fail_fast!
|
89
|
-
|
117
|
+
monitor.measure(:open_circuit)
|
90
118
|
raise CircuitBreakerOpenError
|
91
119
|
end
|
92
120
|
|
@@ -120,18 +148,23 @@ module Supervision
|
|
120
148
|
# Handler exception
|
121
149
|
#
|
122
150
|
# @api public
|
123
|
-
def
|
151
|
+
def handle_failure(error = nil)
|
124
152
|
fail_fast! if fsm.open?
|
125
153
|
record_failure
|
154
|
+
monitor.record_failure
|
126
155
|
trip if failure_count_exceeded? || fsm.half_open?
|
127
156
|
end
|
128
157
|
|
158
|
+
# Record successful call
|
159
|
+
#
|
160
|
+
# @api public
|
129
161
|
def record_success
|
130
162
|
reset if fsm.half_open?
|
131
163
|
reset_failure
|
164
|
+
monitor.record_success
|
132
165
|
end
|
133
166
|
|
134
|
-
#
|
167
|
+
# Record failure count
|
135
168
|
#
|
136
169
|
# @api public
|
137
170
|
def record_failure
|
@@ -153,18 +186,27 @@ module Supervision
|
|
153
186
|
#
|
154
187
|
# @api private
|
155
188
|
def measure_timeout
|
156
|
-
Thread.new do
|
189
|
+
@scheduler = Thread.new do
|
157
190
|
Thread.current.abort_on_exception = true
|
158
191
|
thread = Thread.current
|
159
192
|
thread[:created_at] = Time.now
|
160
193
|
@lock.synchronize do
|
161
|
-
|
162
|
-
|
163
|
-
|
164
|
-
|
165
|
-
|
166
|
-
|
167
|
-
|
194
|
+
run_loop(thread)
|
195
|
+
end
|
196
|
+
end
|
197
|
+
end
|
198
|
+
|
199
|
+
# Run scheduler loop
|
200
|
+
#
|
201
|
+
# @api private
|
202
|
+
def run_loop(thread)
|
203
|
+
catch(:terminate) do
|
204
|
+
loop do
|
205
|
+
if tripped?
|
206
|
+
attempt_reset && break
|
207
|
+
elsif Time.now - thread[:created_at] > max_thread_lifetime
|
208
|
+
thread.kill if thread.alive?
|
209
|
+
break
|
168
210
|
end
|
169
211
|
end
|
170
212
|
end
|