faulty 0.12.0 → 0.13.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGELOG.md +37 -1
- data/README.md +17 -0
- data/lib/faulty/circuit.rb +35 -3
- data/lib/faulty/status.rb +70 -9
- data/lib/faulty/storage/circuit_proxy.rb +1 -0
- data/lib/faulty/storage/fallback_chain.rb +10 -0
- data/lib/faulty/storage/fault_tolerant_proxy.rb +29 -0
- data/lib/faulty/storage/interface.rb +65 -5
- data/lib/faulty/storage/memory.rb +16 -2
- data/lib/faulty/storage/null.rb +6 -0
- data/lib/faulty/storage/redis.rb +42 -11
- data/lib/faulty/version.rb +1 -1
- data/lib/faulty.rb +4 -2
- metadata +3 -3
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: 3642cb65b01572d880aec16ece320bed43b9c12ae13a2dce602c1bf2a5a250c3
|
|
4
|
+
data.tar.gz: 7116acd4f458c31a738d0f90a7ac0bc0f37e392e75105f19e7b9648af9d7b4c7
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: b7078729322061727335102869126e65348f270a71a41b624a2722d62c416672f6b39778338faf570ceac39b7355ebfbc8bf4aff7e866123aada12ba507172ad
|
|
7
|
+
data.tar.gz: 117eae9db002823c62280a7236f5a1969409b6baeffa9c51b0430205f23df5b0238890602cae2217fb19da6e22ea8ebbabf622c750bbb732cd95d0a2caa85554
|
data/CHANGELOG.md
CHANGED
|
@@ -9,6 +9,40 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
|
|
|
9
9
|
[Unreleased]
|
|
10
10
|
-------------------
|
|
11
11
|
|
|
12
|
+
[0.13.0] - 2026-05-13
|
|
13
|
+
---------------------
|
|
14
|
+
|
|
15
|
+
### Added
|
|
16
|
+
|
|
17
|
+
* Reserve half-open test runs across processes so only one process executes
|
|
18
|
+
the test block when a circuit becomes half-open. justinhoward
|
|
19
|
+
|
|
20
|
+
### Changed
|
|
21
|
+
|
|
22
|
+
* `Faulty::Status` now captures `current_time` once when the status is built,
|
|
23
|
+
so all predicates (`open?`, `half_open?`, `reserved?`) reason about the same
|
|
24
|
+
point in time. Previously each predicate called `Faulty.current_time`
|
|
25
|
+
independently. justinhoward
|
|
26
|
+
* `Storage::Redis#close` now also clears the `reserved_at` key. justinhoward
|
|
27
|
+
|
|
28
|
+
### Fixed
|
|
29
|
+
|
|
30
|
+
* `Storage::Redis#reserve` uses safe navigation when serializing
|
|
31
|
+
`previous_reserved_at` for `WATCH`, so the very first CAS against a missing
|
|
32
|
+
key compares correctly. justinhoward
|
|
33
|
+
* `Storage::Redis#reset` now clears the `reserved_at` key. justinhoward
|
|
34
|
+
* `Status#can_run?` now treats `locked_closed?` as an unconditional override,
|
|
35
|
+
so a manually locked-closed circuit runs even when a half-open reservation
|
|
36
|
+
is still in effect from a prior cycle. justinhoward
|
|
37
|
+
|
|
38
|
+
### Breaking Changes
|
|
39
|
+
|
|
40
|
+
* `Storage::Interface` adds a required `#reserve(circuit, reserved_at,
|
|
41
|
+
previous_reserved_at)` method. Custom storage backends must implement it,
|
|
42
|
+
and the `Status` value object must carry the new `reserved_at` attribute.
|
|
43
|
+
See `Storage::Interface#reserve` for the contract and the conformance
|
|
44
|
+
test in `spec/storage/interface_spec.rb` for the structural guarantee.
|
|
45
|
+
|
|
12
46
|
[0.12.0] - 2026-05-13
|
|
13
47
|
---------------------
|
|
14
48
|
|
|
@@ -349,7 +383,9 @@ of AutoWire.
|
|
|
349
383
|
|
|
350
384
|
Initial public release
|
|
351
385
|
|
|
352
|
-
[Unreleased]: https://github.com/ParentSquare/faulty/compare/v0.
|
|
386
|
+
[Unreleased]: https://github.com/ParentSquare/faulty/compare/v0.13.0...HEAD
|
|
387
|
+
[0.13.0]: https://github.com/ParentSquare/faulty/compare/v0.12.0...v0.13.0
|
|
388
|
+
[0.12.0]: https://github.com/ParentSquare/faulty/compare/v0.11.0...v0.12.0
|
|
353
389
|
[0.11.0]: https://github.com/ParentSquare/faulty/compare/v0.10.0...v0.11.0
|
|
354
390
|
[0.10.0]: https://github.com/ParentSquare/faulty/compare/v0.9.0...v0.10.0
|
|
355
391
|
[0.9.0]: https://github.com/ParentSquare/faulty/compare/v0.8.7...v0.9.0
|
data/README.md
CHANGED
|
@@ -1226,6 +1226,23 @@ state, Faulty allows a single execution of the block as a test run. If the test
|
|
|
1226
1226
|
run succeeds, the circuit is fully closed and the circuit state is reset. If the
|
|
1227
1227
|
test run fails, the circuit is opened and the cool-down is reset.
|
|
1228
1228
|
|
|
1229
|
+
When the storage backend supports atomic operations (the default `Memory` and
|
|
1230
|
+
`Redis` backends both do), the half-open test run is reserved exclusively. Other
|
|
1231
|
+
processes or threads that observe the half-open state while a test run is in
|
|
1232
|
+
progress will be skipped with `Faulty::OpenCircuitError`, just as if the circuit
|
|
1233
|
+
were still open. The reservation expires after `cool_down` so that a crashed
|
|
1234
|
+
process can't permanently wedge the circuit.
|
|
1235
|
+
|
|
1236
|
+
This means `cool_down` does double duty: it gates how long the circuit waits
|
|
1237
|
+
before retrying after opening, and it bounds how long a half-open reservation
|
|
1238
|
+
is honored. Test runs that legitimately take longer than `cool_down` (for
|
|
1239
|
+
example, a slow downstream during recovery) will see their reservation expire
|
|
1240
|
+
mid-run, at which point another process can reserve and run the block
|
|
1241
|
+
concurrently. If your protected calls can run longer than `cool_down`, set
|
|
1242
|
+
`cool_down` to comfortably exceed the slowest expected latency for the
|
|
1243
|
+
protected operation, or accept that occasional duplicate half-open test runs
|
|
1244
|
+
are possible during slow recoveries.
|
|
1245
|
+
|
|
1229
1246
|
Each time the circuit changes state or executes the block, events are raised
|
|
1230
1247
|
that are sent to the Faulty event notifier. The notifier should be used to track
|
|
1231
1248
|
circuit failure rates, open circuits, etc.
|
data/lib/faulty/circuit.rb
CHANGED
|
@@ -49,6 +49,10 @@ class Faulty
|
|
|
49
49
|
# @!attribute [r] cool_down
|
|
50
50
|
# @return [Integer] The number of seconds the circuit will
|
|
51
51
|
# stay open after it is tripped. Default 300.
|
|
52
|
+
#
|
|
53
|
+
# Also bounds the half-open reservation TTL — runs longer than
|
|
54
|
+
# `cool_down` lose exclusivity. See the "How it Works" section
|
|
55
|
+
# of the README.
|
|
52
56
|
# @!attribute [r] error_mapper
|
|
53
57
|
# @return [Module, #call] Used by patches to set the namespace module for
|
|
54
58
|
# the faulty errors that will be raised. Should be a module or a callable.
|
|
@@ -308,9 +312,11 @@ class Faulty
|
|
|
308
312
|
return cached_value if !cached_value.nil? && !cache_should_refresh?(cache)
|
|
309
313
|
|
|
310
314
|
current_status = status
|
|
311
|
-
|
|
312
|
-
|
|
313
|
-
|
|
315
|
+
if current_status.can_run? && reserve(current_status)
|
|
316
|
+
run_exec(current_status, cached_value, cache, &block)
|
|
317
|
+
else
|
|
318
|
+
run_skipped(cached_value)
|
|
319
|
+
end
|
|
314
320
|
end
|
|
315
321
|
|
|
316
322
|
# Force the circuit to stay open until unlocked
|
|
@@ -403,6 +409,32 @@ class Faulty
|
|
|
403
409
|
cached_value
|
|
404
410
|
end
|
|
405
411
|
|
|
412
|
+
# Reserves execution for this circuit when it is half-open
|
|
413
|
+
#
|
|
414
|
+
# This prevents concurrent evaluation from allowing multiple simultaneous
|
|
415
|
+
# runs for half-open circuits. For non-half-open states this is a no-op
|
|
416
|
+
# that returns true so closed and locked-closed circuits run unconditionally.
|
|
417
|
+
#
|
|
418
|
+
# `locked_closed?` is checked before `half_open?` to mirror the operator-
|
|
419
|
+
# override hierarchy in {Status#can_run?}: a locked-closed circuit must
|
|
420
|
+
# always proceed regardless of the underlying state, even if another
|
|
421
|
+
# process is currently holding the reservation. Without this, a locked-
|
|
422
|
+
# closed circuit could lose the storage CAS to a concurrent process and
|
|
423
|
+
# be incorrectly skipped.
|
|
424
|
+
#
|
|
425
|
+
# @param status [Status] The current status of the circuit
|
|
426
|
+
# @return [Boolean] True if this call may proceed to execute the block
|
|
427
|
+
def reserve(status)
|
|
428
|
+
return true if status.locked_closed?
|
|
429
|
+
return true unless status.half_open?
|
|
430
|
+
|
|
431
|
+
# Persist a fresh Faulty.current_time, not status.current_time. The
|
|
432
|
+
# snapshot exists for predicate consistency (see Status#current_time);
|
|
433
|
+
# the stored reserved_at should reflect when the reservation was made,
|
|
434
|
+
# not when the snapshot was taken.
|
|
435
|
+
storage.reserve(self, Faulty.current_time, status.reserved_at)
|
|
436
|
+
end
|
|
437
|
+
|
|
406
438
|
# Execute a run
|
|
407
439
|
#
|
|
408
440
|
# @param cached_value The cached value if one is available
|
data/lib/faulty/status.rb
CHANGED
|
@@ -16,9 +16,19 @@ class Faulty
|
|
|
16
16
|
# @return [:open, :closed, nil] If the circuit is locked, the state that
|
|
17
17
|
# it is locked in. Default `nil`.
|
|
18
18
|
# @!attribute [r] opened_at
|
|
19
|
-
# @return [
|
|
20
|
-
# opened. This is not necessarily reset when the circuit
|
|
21
|
-
# Default `nil`.
|
|
19
|
+
# @return [Float, nil] If the circuit is open, the timestamp ({Faulty.current_time})
|
|
20
|
+
# that it was opened. This is not necessarily reset when the circuit
|
|
21
|
+
# is closed. Default `nil`.
|
|
22
|
+
# @!attribute [r] reserved_at
|
|
23
|
+
# @return [Float, nil] If a half-open test run was reserved, the
|
|
24
|
+
# timestamp ({Faulty.current_time}) of that reservation. Cleared when
|
|
25
|
+
# the circuit is closed.
|
|
26
|
+
# Not reset by {Storage::Interface#reopen}; the value naturally expires
|
|
27
|
+
# via `cool_down`. Default `nil`.
|
|
28
|
+
#
|
|
29
|
+
# Only meaningful when {#state} is `:open`. If a backend race or bug
|
|
30
|
+
# produces an inconsistent shape (`state == :closed` with a non-nil
|
|
31
|
+
# `reserved_at`), it is normalized to `nil` at construction.
|
|
22
32
|
# @!attribute [r] failure_rate
|
|
23
33
|
# @return [Float] A number from 0 to 1 representing the percentage of
|
|
24
34
|
# failures for the circuit. For exmaple 0.5 represents a 50% failure rate.
|
|
@@ -34,6 +44,7 @@ class Faulty
|
|
|
34
44
|
:state,
|
|
35
45
|
:lock,
|
|
36
46
|
:opened_at,
|
|
47
|
+
:reserved_at,
|
|
37
48
|
:failure_rate,
|
|
38
49
|
:sample_size,
|
|
39
50
|
:options,
|
|
@@ -43,6 +54,19 @@ class Faulty
|
|
|
43
54
|
class Status
|
|
44
55
|
include ImmutableOptions
|
|
45
56
|
|
|
57
|
+
# @return [Float] The point in time captured when this status was built.
|
|
58
|
+
# All predicates (`open?`, `half_open?`, `reserved?`) reason about
|
|
59
|
+
# this same instant so they are mutually consistent. Held as an
|
|
60
|
+
# instance variable rather than a struct field so it does not leak
|
|
61
|
+
# into `to_h`, `==`, or `members` — those should reflect persisted
|
|
62
|
+
# circuit state, not a transient predicate-consistency snapshot.
|
|
63
|
+
attr_reader :current_time
|
|
64
|
+
|
|
65
|
+
def initialize(hash, &)
|
|
66
|
+
@current_time = hash[:current_time] || Faulty.current_time
|
|
67
|
+
super(hash.except(:current_time), &)
|
|
68
|
+
end
|
|
69
|
+
|
|
46
70
|
# The allowed state values
|
|
47
71
|
STATES = %i[
|
|
48
72
|
open
|
|
@@ -66,7 +90,8 @@ class Faulty
|
|
|
66
90
|
# sample_size
|
|
67
91
|
# @return [Status]
|
|
68
92
|
def self.from_entries(entries, **hash)
|
|
69
|
-
|
|
93
|
+
current_time = Faulty.current_time
|
|
94
|
+
window_start = current_time - hash[:options].evaluation_window
|
|
70
95
|
size = entries.size
|
|
71
96
|
i = 0
|
|
72
97
|
failures = 0
|
|
@@ -84,7 +109,8 @@ class Faulty
|
|
|
84
109
|
|
|
85
110
|
new(hash.merge(
|
|
86
111
|
sample_size: sample_size,
|
|
87
|
-
failure_rate: sample_size.zero? ? 0.0 : failures.to_f / sample_size
|
|
112
|
+
failure_rate: sample_size.zero? ? 0.0 : failures.to_f / sample_size,
|
|
113
|
+
current_time: current_time
|
|
88
114
|
))
|
|
89
115
|
end
|
|
90
116
|
|
|
@@ -94,7 +120,7 @@ class Faulty
|
|
|
94
120
|
#
|
|
95
121
|
# @return [Boolean] True if open
|
|
96
122
|
def open?
|
|
97
|
-
state == :open && opened_at + options.cool_down >
|
|
123
|
+
state == :open && opened_at + options.cool_down > current_time
|
|
98
124
|
end
|
|
99
125
|
|
|
100
126
|
# Whether the circuit is closed
|
|
@@ -112,7 +138,7 @@ class Faulty
|
|
|
112
138
|
#
|
|
113
139
|
# @return [Boolean] True if half-open
|
|
114
140
|
def half_open?
|
|
115
|
-
state == :open && opened_at + options.cool_down <=
|
|
141
|
+
state == :open && opened_at + options.cool_down <= current_time
|
|
116
142
|
end
|
|
117
143
|
|
|
118
144
|
# Whether the circuit is locked open
|
|
@@ -129,15 +155,42 @@ class Faulty
|
|
|
129
155
|
lock == :closed
|
|
130
156
|
end
|
|
131
157
|
|
|
158
|
+
# Whether a half-open test run is currently reserved
|
|
159
|
+
#
|
|
160
|
+
# Process-agnostic: returns true whenever an unexpired reservation exists
|
|
161
|
+
# on this circuit, regardless of who made it. The "did someone else reserve
|
|
162
|
+
# this?" interpretation only applies when this predicate is read on a
|
|
163
|
+
# status snapshot taken *before* the caller attempts {Storage::Interface#reserve};
|
|
164
|
+
# a caller introspecting their own status after a successful reserve will
|
|
165
|
+
# also see `true` here.
|
|
166
|
+
#
|
|
167
|
+
# The reservation expires after `cool_down` to handle the case where the
|
|
168
|
+
# process that made the reservation crashes before resolving the circuit.
|
|
169
|
+
# Side effect: a legitimately-slow test run that exceeds `cool_down`
|
|
170
|
+
# loses exclusivity (another process may reserve and run concurrently).
|
|
171
|
+
# See the "How it Works" section of the README for the full trade-off.
|
|
172
|
+
#
|
|
173
|
+
# @return [Boolean] True if a reservation is in effect
|
|
174
|
+
def reserved?
|
|
175
|
+
return false unless reserved_at
|
|
176
|
+
|
|
177
|
+
state == :open && reserved_at + options.cool_down > current_time
|
|
178
|
+
end
|
|
179
|
+
|
|
132
180
|
# Whether the circuit can be run
|
|
133
181
|
#
|
|
134
|
-
# Takes the circuit state, locks and cooldown into account
|
|
182
|
+
# Takes the circuit state, locks and cooldown into account. Locks are
|
|
183
|
+
# operator overrides and take precedence over both state and reservation,
|
|
184
|
+
# so a `locked_closed?` circuit always runs and a `locked_open?` circuit
|
|
185
|
+
# never runs.
|
|
135
186
|
#
|
|
136
187
|
# @return [Boolean] True if the circuit can be run
|
|
137
188
|
def can_run?
|
|
138
189
|
return false if locked_open?
|
|
190
|
+
return true if locked_closed?
|
|
191
|
+
return false if reserved?
|
|
139
192
|
|
|
140
|
-
closed? ||
|
|
193
|
+
closed? || half_open?
|
|
141
194
|
end
|
|
142
195
|
|
|
143
196
|
# Whether the circuit fails the sample size and rate thresholds
|
|
@@ -155,6 +208,14 @@ class Faulty
|
|
|
155
208
|
raise ArgumentError, "lock must be a symbol in #{self.class}::LOCKS or nil"
|
|
156
209
|
end
|
|
157
210
|
raise ArgumentError, 'opened_at is required if state is open' if state == :open && opened_at.nil?
|
|
211
|
+
|
|
212
|
+
# `reserved_at` is only meaningful while the circuit is open. Backends
|
|
213
|
+
# are expected to clear it on close, but if a brief race or backend bug
|
|
214
|
+
# leaves a stale value paired with `state == :closed`, normalize it
|
|
215
|
+
# here so downstream code can rely on the invariant without checking
|
|
216
|
+
# `state` first. Sanitizing rather than raising avoids turning a
|
|
217
|
+
# transient backend inconsistency into a production crash.
|
|
218
|
+
self.reserved_at = nil if state == :closed && !reserved_at.nil?
|
|
158
219
|
end
|
|
159
220
|
|
|
160
221
|
def required
|
|
@@ -105,6 +105,16 @@ class Faulty
|
|
|
105
105
|
end
|
|
106
106
|
end
|
|
107
107
|
|
|
108
|
+
# Reserve a half-open run in the first available storage backend
|
|
109
|
+
#
|
|
110
|
+
# @param (see Interface#reserve)
|
|
111
|
+
# @return (see Interface#reserve)
|
|
112
|
+
def reserve(circuit, reserved_at, previous_reserved_at)
|
|
113
|
+
send_chain(:reserve, circuit, reserved_at, previous_reserved_at) do |e|
|
|
114
|
+
options.notifier.notify(:storage_failure, circuit: circuit, action: :reserve, error: e)
|
|
115
|
+
end
|
|
116
|
+
end
|
|
117
|
+
|
|
108
118
|
# Lock a circuit in all storage backends
|
|
109
119
|
#
|
|
110
120
|
# @param (see Interface#lock)
|
|
@@ -9,6 +9,19 @@ class Faulty
|
|
|
9
9
|
#
|
|
10
10
|
# If the storage backend raises a `StandardError`, it will be captured and
|
|
11
11
|
# sent to the notifier.
|
|
12
|
+
#
|
|
13
|
+
# The overall design preference is to keep protected code paths running
|
|
14
|
+
# when the storage backend is degraded, even when that means losing
|
|
15
|
+
# circuit-breaker protections that the storage normally provides:
|
|
16
|
+
# `#status` returns a stub closed status (so `Circuit#run` proceeds),
|
|
17
|
+
# `#reserve` returns `true` (so half-open test runs proceed), and the
|
|
18
|
+
# write paths (`#open`, `#reopen`, `#close`, `#entry`) return `false`
|
|
19
|
+
# to safe-deny the *recorded transition* without failing the in-flight
|
|
20
|
+
# call. The trade-off is that a correlated outage of the storage
|
|
21
|
+
# backend and the upstream protected by the circuit will let the fleet
|
|
22
|
+
# converge on the upstream — but that fleet would converge anyway via
|
|
23
|
+
# the stub-closed status path, so individual write methods don't make
|
|
24
|
+
# it worse.
|
|
12
25
|
class FaultTolerantProxy
|
|
13
26
|
extend Forwardable
|
|
14
27
|
|
|
@@ -177,6 +190,22 @@ class Faulty
|
|
|
177
190
|
stub_status(circuit)
|
|
178
191
|
end
|
|
179
192
|
|
|
193
|
+
# Safely reserve execution of a circuit
|
|
194
|
+
#
|
|
195
|
+
# Returns `true` on storage error so half-open test runs proceed when
|
|
196
|
+
# the backend is degraded. See the class-level docs for the gem's
|
|
197
|
+
# fail-open trade-off.
|
|
198
|
+
#
|
|
199
|
+
# @see Interface#reserve
|
|
200
|
+
# @param (see Interface#reserve)
|
|
201
|
+
# @return (see Interface#reserve)
|
|
202
|
+
def reserve(circuit, reserved_at, previous_reserved_at)
|
|
203
|
+
@storage.reserve(circuit, reserved_at, previous_reserved_at)
|
|
204
|
+
rescue StandardError => e
|
|
205
|
+
options.notifier.notify(:storage_failure, circuit: circuit, action: :reserve, error: e)
|
|
206
|
+
true
|
|
207
|
+
end
|
|
208
|
+
|
|
180
209
|
# This cache makes any storage fault tolerant, so this is always `true`
|
|
181
210
|
#
|
|
182
211
|
# @return [true]
|
|
@@ -35,7 +35,8 @@ class Faulty
|
|
|
35
35
|
# long as it can implement the other read methods.
|
|
36
36
|
#
|
|
37
37
|
# @param circuit [Circuit] The circuit that ran
|
|
38
|
-
# @param time [
|
|
38
|
+
# @param time [Float] The unix timestamp for the run, from
|
|
39
|
+
# {Faulty.current_time}
|
|
39
40
|
# @param success [Boolean] True if the run succeeded
|
|
40
41
|
# @param status [Status, nil] The previous status. If given, this method must
|
|
41
42
|
# return an updated status object from the new entry data.
|
|
@@ -58,7 +59,8 @@ class Faulty
|
|
|
58
59
|
# current time.
|
|
59
60
|
#
|
|
60
61
|
# @param circuit [Circuit] The circuit to open
|
|
61
|
-
# @param opened_at [
|
|
62
|
+
# @param opened_at [Float] The timestamp the circuit was opened at,
|
|
63
|
+
# from {Faulty.current_time}
|
|
62
64
|
# @return [Boolean] True if the circuit transitioned from closed to open
|
|
63
65
|
def open(circuit, opened_at)
|
|
64
66
|
raise NotImplementedError
|
|
@@ -75,10 +77,35 @@ class Faulty
|
|
|
75
77
|
# it may always return true, but that could result in duplicate reopen
|
|
76
78
|
# notifications.
|
|
77
79
|
#
|
|
80
|
+
# The backend MUST NOT clear `reserved_at` here.
|
|
81
|
+
#
|
|
82
|
+
# Preserving the prior cycle's `reserved_at` is load-bearing for
|
|
83
|
+
# half-open exclusivity. If a late-arriving caller read status while
|
|
84
|
+
# `reserved_at` was still nil (before the winning process reserved),
|
|
85
|
+
# its subsequent `reserve(circuit, T, nil)` CAS must fail. Clearing
|
|
86
|
+
# `reserved_at` in `reopen` would let that stale CAS incorrectly
|
|
87
|
+
# succeed and produce a duplicate half-open run.
|
|
88
|
+
#
|
|
89
|
+
# Beyond exclusivity, the prior reservation expires naturally at
|
|
90
|
+
# `reserved_at + cool_down`, aligned with the start of the next
|
|
91
|
+
# half-open window (since `reserved_at <= new_opened_at`). An
|
|
92
|
+
# explicit reset would be redundant with the cool-down-aligned expiry
|
|
93
|
+
# that already handles crash recovery.
|
|
94
|
+
#
|
|
95
|
+
# This invariant assumes the reservation TTL equals `cool_down`. If
|
|
96
|
+
# a separate `reservation_ttl` is ever introduced, this method must
|
|
97
|
+
# be revisited and may need to clear `reserved_at` explicitly.
|
|
98
|
+
#
|
|
78
99
|
# @param circuit [Circuit] The circuit to reopen
|
|
79
|
-
# @param opened_at [
|
|
80
|
-
#
|
|
81
|
-
#
|
|
100
|
+
# @param opened_at [Float] The timestamp the circuit was opened at,
|
|
101
|
+
# from {Faulty.current_time}
|
|
102
|
+
# @param previous_opened_at [Float] The last known value of opened_at.
|
|
103
|
+
# Can be used to compare-and-set. Always non-nil — `Circuit#failure!`
|
|
104
|
+
# only enters the reopen branch when `status.half_open?` is true,
|
|
105
|
+
# which requires non-nil `opened_at`. Unlike `previous_reserved_at`
|
|
106
|
+
# on {#reserve}, there is no legitimate "no prior value" call path
|
|
107
|
+
# to `reopen`, so backends may treat this parameter as required and
|
|
108
|
+
# are not expected to handle `nil`.
|
|
82
109
|
# @return [Boolean] True if the opened_at time was updated
|
|
83
110
|
def reopen(circuit, opened_at, previous_opened_at)
|
|
84
111
|
raise NotImplementedError
|
|
@@ -90,6 +117,9 @@ class Faulty
|
|
|
90
117
|
# may be called more than once. If so, this method should return true
|
|
91
118
|
# only once, when the circuit transitions from open to closed.
|
|
92
119
|
#
|
|
120
|
+
# The backend should reset the reserved_at value to empty when closing
|
|
121
|
+
# the circuit.
|
|
122
|
+
#
|
|
93
123
|
# If the backend does not support locking or atomic operations, then
|
|
94
124
|
# it may always return true, but that could result in duplicate close
|
|
95
125
|
# notifications.
|
|
@@ -99,6 +129,36 @@ class Faulty
|
|
|
99
129
|
raise NotImplementedError
|
|
100
130
|
end
|
|
101
131
|
|
|
132
|
+
# Reserve an exclusive run for this circuit
|
|
133
|
+
#
|
|
134
|
+
# This is used when the circuit is half-open and the test run is being
|
|
135
|
+
# attempted. We need to make sure only a single run is allowed.
|
|
136
|
+
#
|
|
137
|
+
# The backend should store reserved_at and use it to serve future status
|
|
138
|
+
# requests. When setting reserved_at, the backend should atomically
|
|
139
|
+
# compare any existing value using previous_reserved_at. This ensures
|
|
140
|
+
# that mutltiple parallel processes can't reserve the circuit.
|
|
141
|
+
#
|
|
142
|
+
# The return value is the caller's signal to proceed with the half-open
|
|
143
|
+
# test run, not a strict report of whether atomic acquisition succeeded.
|
|
144
|
+
# Atomic backends should return `true` only when the CAS against
|
|
145
|
+
# `previous_reserved_at` succeeds. Non-atomic backends, no-op backends
|
|
146
|
+
# ({Null}), or wrappers that fail open ({FaultTolerantProxy}) may always
|
|
147
|
+
# return `true`; the caller will proceed at the cost of allowing
|
|
148
|
+
# duplicate half-open test runs.
|
|
149
|
+
#
|
|
150
|
+
# @param circuit [Circuit] The circuit to reserve
|
|
151
|
+
# @param reserved_at [Float] The timestamp of this reservation, from
|
|
152
|
+
# {Faulty.current_time}
|
|
153
|
+
# @param previous_reserved_at [Float, nil] The last known value of
|
|
154
|
+
# reserved_at, or nil for the first reservation in a new open cycle.
|
|
155
|
+
# Can be used to compare-and-set.
|
|
156
|
+
# @return [Boolean] True if the caller may proceed with the half-open
|
|
157
|
+
# test run; false if another caller already holds the reservation.
|
|
158
|
+
def reserve(circuit, reserved_at, previous_reserved_at)
|
|
159
|
+
raise NotImplementedError
|
|
160
|
+
end
|
|
161
|
+
|
|
102
162
|
# Lock the circuit in a given state
|
|
103
163
|
#
|
|
104
164
|
# No concurrency gurantees are provided for locking
|
|
@@ -41,11 +41,12 @@ class Faulty
|
|
|
41
41
|
# The internal object for storing a circuit
|
|
42
42
|
#
|
|
43
43
|
# @private
|
|
44
|
-
MemoryCircuit = Struct.new(:state, :runs, :opened_at, :lock, :options) do
|
|
44
|
+
MemoryCircuit = Struct.new(:state, :runs, :opened_at, :reserved_at, :lock, :options) do
|
|
45
45
|
def initialize
|
|
46
46
|
self.state = Concurrent::Atom.new(:closed)
|
|
47
47
|
self.runs = Concurrent::MVar.new([], dup_on_deref: true)
|
|
48
48
|
self.opened_at = Concurrent::Atom.new(nil)
|
|
49
|
+
self.reserved_at = Concurrent::Atom.new(nil)
|
|
49
50
|
self.lock = nil
|
|
50
51
|
end
|
|
51
52
|
|
|
@@ -61,6 +62,7 @@ class Faulty
|
|
|
61
62
|
state: state.value,
|
|
62
63
|
lock: lock,
|
|
63
64
|
opened_at: opened_at.value,
|
|
65
|
+
reserved_at: reserved_at.value,
|
|
64
66
|
options: circuit_options
|
|
65
67
|
)
|
|
66
68
|
end
|
|
@@ -139,7 +141,19 @@ class Faulty
|
|
|
139
141
|
def close(circuit)
|
|
140
142
|
memory = fetch(circuit)
|
|
141
143
|
memory.runs.modify { |_old| [] }
|
|
142
|
-
memory.state.compare_and_set(:open, :closed)
|
|
144
|
+
closed = memory.state.compare_and_set(:open, :closed)
|
|
145
|
+
memory.reserved_at.reset(nil) if closed
|
|
146
|
+
closed
|
|
147
|
+
end
|
|
148
|
+
|
|
149
|
+
# Reserve an exclusive run for this circuit
|
|
150
|
+
#
|
|
151
|
+
# @see Interface#reserve
|
|
152
|
+
# @param (see Interface#reserve)
|
|
153
|
+
# @return (see Interface#reserve)
|
|
154
|
+
def reserve(circuit, reserved_at, previous_reserved_at)
|
|
155
|
+
memory = fetch(circuit)
|
|
156
|
+
memory.reserved_at.compare_and_set(previous_reserved_at, reserved_at)
|
|
143
157
|
end
|
|
144
158
|
|
|
145
159
|
# Lock a circuit open or closed
|
data/lib/faulty/storage/null.rb
CHANGED
|
@@ -46,6 +46,12 @@ class Faulty
|
|
|
46
46
|
true
|
|
47
47
|
end
|
|
48
48
|
|
|
49
|
+
# @param (see Interface#reserve)
|
|
50
|
+
# @return (see Interface#reserve)
|
|
51
|
+
def reserve(_circuit, _reserved_at, _previous_reserved_at)
|
|
52
|
+
true
|
|
53
|
+
end
|
|
54
|
+
|
|
49
55
|
# @param (see Interface#lock)
|
|
50
56
|
# @return (see Interface#lock)
|
|
51
57
|
def lock(_circuit, _state)
|
data/lib/faulty/storage/redis.rb
CHANGED
|
@@ -174,11 +174,25 @@ class Faulty
|
|
|
174
174
|
result = watch_exec(key, ['open']) do |m|
|
|
175
175
|
m.set(key, 'closed', ex: ex)
|
|
176
176
|
m.del(entries_key(circuit.name))
|
|
177
|
+
m.del(reserved_at_key(circuit.name))
|
|
177
178
|
end
|
|
178
179
|
|
|
179
180
|
result && result[0] == 'OK'
|
|
180
181
|
end
|
|
181
182
|
|
|
183
|
+
# Reserve an exclusive run for this circuit
|
|
184
|
+
#
|
|
185
|
+
# @see Interface#reserve
|
|
186
|
+
# @param (see Interface#reserve)
|
|
187
|
+
# @return (see Interface#reserve)
|
|
188
|
+
def reserve(circuit, reserved_at, previous_reserved_at)
|
|
189
|
+
key = reserved_at_key(circuit.name)
|
|
190
|
+
result = watch_exec(key, [previous_reserved_at&.to_s]) do |m|
|
|
191
|
+
m.set(key, reserved_at, ex: options.circuit_ttl)
|
|
192
|
+
end
|
|
193
|
+
result && result[0] == 'OK'
|
|
194
|
+
end
|
|
195
|
+
|
|
182
196
|
# Lock a circuit open or closed
|
|
183
197
|
#
|
|
184
198
|
# The circuit_ttl does not apply to locks
|
|
@@ -210,6 +224,7 @@ class Faulty
|
|
|
210
224
|
r.del(
|
|
211
225
|
entries_key(name),
|
|
212
226
|
opened_at_key(name),
|
|
227
|
+
reserved_at_key(name),
|
|
213
228
|
lock_key(name),
|
|
214
229
|
options_key(name)
|
|
215
230
|
)
|
|
@@ -228,20 +243,11 @@ class Faulty
|
|
|
228
243
|
futures[:state] = r.get(state_key(circuit.name))
|
|
229
244
|
futures[:lock] = r.get(lock_key(circuit.name))
|
|
230
245
|
futures[:opened_at] = r.get(opened_at_key(circuit.name))
|
|
246
|
+
futures[:reserved_at] = r.get(reserved_at_key(circuit.name))
|
|
231
247
|
futures[:entries] = r.lrange(entries_key(circuit.name), 0, -1)
|
|
232
248
|
end
|
|
233
249
|
|
|
234
|
-
|
|
235
|
-
opened_at = futures[:opened_at].value ? Float(futures[:opened_at].value) : nil
|
|
236
|
-
opened_at = Faulty.current_time - options.circuit_ttl if state == :open && opened_at.nil?
|
|
237
|
-
|
|
238
|
-
Faulty::Status.from_entries(
|
|
239
|
-
map_entries(futures[:entries].value),
|
|
240
|
-
state: state,
|
|
241
|
-
lock: futures[:lock].value&.to_sym,
|
|
242
|
-
opened_at: opened_at,
|
|
243
|
-
options: circuit.options
|
|
244
|
-
)
|
|
250
|
+
build_status(circuit, futures)
|
|
245
251
|
end
|
|
246
252
|
|
|
247
253
|
# Get the circuit history up to `max_sample_size`
|
|
@@ -285,6 +291,26 @@ class Faulty
|
|
|
285
291
|
|
|
286
292
|
private
|
|
287
293
|
|
|
294
|
+
# Build a {Status} from the redis values fetched by {#status}
|
|
295
|
+
def build_status(circuit, futures)
|
|
296
|
+
state = futures[:state].value&.to_sym || :closed
|
|
297
|
+
opened_at = parse_float(futures[:opened_at].value)
|
|
298
|
+
opened_at = Faulty.current_time - options.circuit_ttl if state == :open && opened_at.nil?
|
|
299
|
+
|
|
300
|
+
Faulty::Status.from_entries(
|
|
301
|
+
map_entries(futures[:entries].value),
|
|
302
|
+
state: state,
|
|
303
|
+
lock: futures[:lock].value&.to_sym,
|
|
304
|
+
opened_at: opened_at,
|
|
305
|
+
reserved_at: parse_float(futures[:reserved_at].value),
|
|
306
|
+
options: circuit.options
|
|
307
|
+
)
|
|
308
|
+
end
|
|
309
|
+
|
|
310
|
+
def parse_float(value)
|
|
311
|
+
value ? Float(value) : nil
|
|
312
|
+
end
|
|
313
|
+
|
|
288
314
|
# Generate a key from its parts
|
|
289
315
|
#
|
|
290
316
|
# @return [String] The key
|
|
@@ -321,6 +347,11 @@ class Faulty
|
|
|
321
347
|
ckey(circuit_name, 'opened_at')
|
|
322
348
|
end
|
|
323
349
|
|
|
350
|
+
# @return [String] The key for circuit reserved_at
|
|
351
|
+
def reserved_at_key(circuit_name)
|
|
352
|
+
ckey(circuit_name, 'reserved_at')
|
|
353
|
+
end
|
|
354
|
+
|
|
324
355
|
# Get the current key to add circuit names to
|
|
325
356
|
def list_key
|
|
326
357
|
key('list', current_list_block)
|
data/lib/faulty/version.rb
CHANGED
data/lib/faulty.rb
CHANGED
|
@@ -125,9 +125,11 @@ class Faulty
|
|
|
125
125
|
# The current time
|
|
126
126
|
#
|
|
127
127
|
# Used by Faulty wherever the current time is needed. Can be overridden
|
|
128
|
-
# for testing
|
|
128
|
+
# for testing. Returned as a Float (Unix epoch seconds with sub-second
|
|
129
|
+
# precision) so it can be stored in numeric Redis fields and compared
|
|
130
|
+
# against other timestamps without conversion.
|
|
129
131
|
#
|
|
130
|
-
# @return [
|
|
132
|
+
# @return [Float] The current time as a Unix timestamp
|
|
131
133
|
def current_time
|
|
132
134
|
Time.now.to_f
|
|
133
135
|
end
|
metadata
CHANGED
|
@@ -1,14 +1,14 @@
|
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
|
2
2
|
name: faulty
|
|
3
3
|
version: !ruby/object:Gem::Version
|
|
4
|
-
version: 0.
|
|
4
|
+
version: 0.13.0
|
|
5
5
|
platform: ruby
|
|
6
6
|
authors:
|
|
7
7
|
- Justin Howard
|
|
8
8
|
autorequire:
|
|
9
9
|
bindir: bin
|
|
10
10
|
cert_chain: []
|
|
11
|
-
date: 2026-05-
|
|
11
|
+
date: 2026-05-14 00:00:00.000000000 Z
|
|
12
12
|
dependencies:
|
|
13
13
|
- !ruby/object:Gem::Dependency
|
|
14
14
|
name: concurrent-ruby
|
|
@@ -152,7 +152,7 @@ licenses:
|
|
|
152
152
|
metadata:
|
|
153
153
|
rubygems_mfa_required: 'true'
|
|
154
154
|
changelog_uri: https://github.com/ParentSquare/faulty/blob/master/CHANGELOG.md
|
|
155
|
-
documentation_uri: https://www.rubydoc.info/gems/faulty/0.
|
|
155
|
+
documentation_uri: https://www.rubydoc.info/gems/faulty/0.13.0
|
|
156
156
|
post_install_message:
|
|
157
157
|
rdoc_options: []
|
|
158
158
|
require_paths:
|