redeye-breaker 0.4.0 → 0.5.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +55 -8
- package/dist/circuit-breaker.d.ts +81 -3
- package/dist/circuit-breaker.d.ts.map +1 -1
- package/dist/circuit-breaker.js +189 -5
- package/dist/circuit-breaker.js.map +1 -1
- package/dist/store.d.ts +64 -1
- package/dist/store.d.ts.map +1 -1
- package/dist/stores/redis-store.d.ts +41 -8
- package/dist/stores/redis-store.d.ts.map +1 -1
- package/dist/stores/redis-store.js +324 -84
- package/dist/stores/redis-store.js.map +1 -1
- package/dist/types.d.ts +24 -0
- package/dist/types.d.ts.map +1 -1
- package/dist/types.js +1 -0
- package/dist/types.js.map +1 -1
- package/package.json +2 -1
package/README.md
CHANGED
|
@@ -104,7 +104,7 @@ Every process pointed at the same Redis instance (and using the same operation n
|
|
|
104
104
|
|
|
105
105
|
### Bring your own store
|
|
106
106
|
|
|
107
|
-
`RedisStore` implements a `Store` interface with two required methods and
|
|
107
|
+
`RedisStore` implements a `Store` interface with two required methods and seven *optional* ones:
|
|
108
108
|
|
|
109
109
|
```ts
|
|
110
110
|
export interface Store {
|
|
@@ -115,19 +115,62 @@ export interface Store {
|
|
|
115
115
|
// Optional — implement these for the reliability guarantees below.
|
|
116
116
|
// Without them, redeye falls back to best-effort semantics and logs a
|
|
117
117
|
// one-time warning telling you exactly what's degraded.
|
|
118
|
-
recordFailureAtomic?(key: string, opts: { ttlSeconds: number; failureThreshold: number; now: number }): Promise<CircuitBreakerState>;
|
|
119
|
-
recordOutcomeAtomic?(key: string, opts: { success: boolean; decay: number; minimumCalls: number; errorRateThreshold: number; ttlSeconds: number; now: number }): Promise<CircuitBreakerState & { openedNow: boolean }>;
|
|
118
|
+
recordFailureAtomic?(key: string, opts: { ttlSeconds: number; failureThreshold: number; now: number; operation: string }): Promise<CircuitBreakerState>;
|
|
119
|
+
recordOutcomeAtomic?(key: string, opts: { success: boolean; decay: number; minimumCalls: number; errorRateThreshold: number; ttlSeconds: number; now: number; operation: string }): Promise<CircuitBreakerState & { openedNow: boolean }>;
|
|
120
120
|
claimTrial?(key: string, ttlSeconds: number): Promise<string | null>;
|
|
121
121
|
releaseTrial?(key: string, token: string): Promise<void>;
|
|
122
|
+
|
|
123
|
+
// Optional — only needed for the closed-state local cache (`localCache`, see below).
|
|
124
|
+
closeAtomic?(key: string, eventsKey: string, opts: { ttlSeconds: number; operation: string }): Promise<CircuitBreakerState>;
|
|
125
|
+
reopenTrialFailureAtomic?(key: string, eventsKey: string, opts: { ttlSeconds: number; failureThreshold: number; now: number; operation: string }): Promise<CircuitBreakerState>;
|
|
126
|
+
subscribeTransitions?(eventsKey: string, handler: (event: TransitionEvent | null) => void): Promise<() => void>;
|
|
122
127
|
}
|
|
123
128
|
```
|
|
124
129
|
|
|
125
|
-
- `recordFailureAtomic` (`strategy: 'consecutive'`) should increment the failure count and decide `isOpen` in one atomic round trip (a Lua script in Redis, a conditional update in DynamoDB, etc.).
|
|
126
|
-
- `recordOutcomeAtomic` (`strategy: 'errorRate'`) should fold one call's outcome into the EWMA rate and decide `isOpen` in one atomic round trip — called on every closed-phase call, not just failures.
|
|
130
|
+
- `recordFailureAtomic` (`strategy: 'consecutive'`) should increment the failure count and decide `isOpen` in one atomic round trip (a Lua script in Redis, a conditional update in DynamoDB, etc.). Should also bump a monotonic `version` counter and publish a transition event to `eventsKey` (see below) when — and only when — this call flips `isOpen`; a plain sub-threshold increment isn't a transition and shouldn't publish one.
|
|
131
|
+
- `recordOutcomeAtomic` (`strategy: 'errorRate'`) should fold one call's outcome into the EWMA rate and decide `isOpen` in one atomic round trip — called on every closed-phase call, not just failures. Same `version`/publish rule as above, keyed off an `isOpen` flip.
|
|
127
132
|
- `claimTrial` should be a conditional "create if absent" write (`SET key token NX EX ttl` in Redis, with `token` a fresh unique value per call, e.g. a UUID) — it's what makes half-open recovery exclusive to one caller instead of a free-for-all. Return the token you wrote if this call won the claim, or `null` if the key was already taken.
|
|
128
133
|
- `releaseTrial` should be a compare-and-delete (`if GET key == token then DEL key` in Redis, a conditional delete elsewhere), not an unconditional delete. This is what stops a trial that outran its TTL (see [item 11](#whats-still-a-fundamental-tradeoff-not-a-bug)) from deleting a *different* instance's newer claim on the same key when it finally gets around to releasing its own now-stale one. If omitted (or if a call has no confirmed ownership token to release at all — e.g. `claimTrial` itself errored), redeye logs a one-time warning and leaves the claim for its TTL to expire naturally, rather than risk an unconditional delete that could destroy a claim it never confirmed was its own.
|
|
134
|
+
- `closeAtomic` should close `key`, bumping `version` and publishing a transition, **but only if there was anything to clear** (it was open, or had accumulated sub-threshold failures) — a true no-op, no write at all, when the state was already clean. This is what lets the healthy-path write-skip (see [limitation item 1](#whats-still-a-fundamental-tradeoff-not-a-bug)) happen atomically server-side instead of relying on the caller's own possibly-stale view.
|
|
135
|
+
- `reopenTrialFailureAtomic` should reopen `key` after a failed half-open trial — `isOpen: true`, `openCount + 1`, `failures` raised to at least `failureThreshold`, `lastFailure: now`, `version + 1` — and always publish a transition (a failed trial is always a real transition, unlike `closeAtomic`).
|
|
136
|
+
- `subscribeTransitions` should subscribe to a stream of transition events at `eventsKey` — **one shared stream per store/prefix, not one per operation** — and invoke `handler` with each decoded event, or with `null` whenever the underlying subscription drops or errors (callers must then distrust every locally cached entry until it's re-verified by a real read; implementations should retry their own subscription with backoff, not expect the caller to re-subscribe). Returns an unsubscribe function.
|
|
137
|
+
|
|
138
|
+
Implement the two required methods against Memcached, DynamoDB, your own cache wrapper, etc., and you have a working (best-effort) distributed breaker. Add `recordFailureAtomic`/`recordOutcomeAtomic`/`claimTrial`/`releaseTrial` when that store supports a real atomic increment and a real conditional write, and you get the full reliability guarantees. Add `closeAtomic`/`reopenTrialFailureAtomic`/`subscribeTransitions` on top of that if you also want the [closed-state local cache](#local-caching-hybrid-mode) — every capability is independently optional, so skip any subset that doesn't fit your store.
|
|
139
|
+
|
|
140
|
+
## Local caching (hybrid mode)
|
|
141
|
+
|
|
142
|
+
Everything up to this point reads the store on every gate check (a genuine read every closed-path call; a cached, 0-read rejection once open — see [limitation item 1](#whats-still-a-fundamental-tradeoff-not-a-bug)). `localCache` goes one step further, opt-in, only for `RedisStore` (or any store implementing `subscribeTransitions`): it caches the **closed** state locally too, invalidated push-style instead of polling, so a healthy call can skip the store read entirely, not just the write.
|
|
143
|
+
|
|
144
|
+
```ts
|
|
145
|
+
const breaker = new CircuitBreaker({
|
|
146
|
+
failureThreshold: 5,
|
|
147
|
+
resetTimeout: 60_000,
|
|
148
|
+
store,
|
|
149
|
+
localCache: { staleToleranceMs: 100 }, // opt-in; default staleToleranceMs is 100 once the object is present
|
|
150
|
+
});
|
|
151
|
+
```
|
|
129
152
|
|
|
130
|
-
|
|
153
|
+
**What's cached, and how it's invalidated.** Per `operation`, redeye keeps `{ state, version, fetchedAt, trusted }` in memory. A gate check allows immediately, with zero store reads, when the entry is `trusted`, closed (`!state.isOpen`), and fresher than `staleToleranceMs`. Three independent mechanisms keep this from going stale silently:
|
|
154
|
+
|
|
155
|
+
- **Push invalidation.** `subscribeTransitions` delivers a decoded event the instant any instance's write flips `isOpen` or clears real accumulated state (never on a plain sub-threshold counter update — those aren't transitions and don't invalidate the cache, by design, since they can't affect the *closed* fast path this cache serves). A higher `version` than what's cached overwrites the entry immediately.
|
|
156
|
+
- **Fallback poll.** The existing local-mode monitor interval doubles as a missed-message safety net in distributed mode: any cached entry older than 5s gets one authoritative `get` and is reconciled by `version`. In steady state (no missed messages) this is the *only* background traffic the cache generates — one `GET` per stale operation per sweep, not per call.
|
|
157
|
+
- **Your own writes are free.** The state `recordFailureAtomic`/`recordOutcomeAtomic`/`closeAtomic`/`reopenTrialFailureAtomic` hand back already updates this instance's own cache — the instance that causes a transition never has a stale window on it, regardless of stream or poll timing.
|
|
158
|
+
|
|
159
|
+
**The fail-open window, as a formula, not an adjective.** This is the one place in redeye that's biased toward availability over freshness on the *closed* side (everywhere else — the open-state cache, `failOpenOnStoreError: false`, etc. — is fail-closed or symmetric by default). If the breaker trips elsewhere in the fleet, a call landing on this instance's still-fresh cached-closed entry can pass through before that instance learns about it. The bound is exactly:
|
|
160
|
+
|
|
161
|
+
```
|
|
162
|
+
worst-case fail-open window = staleToleranceMs + transition-delivery latency
|
|
163
|
+
```
|
|
164
|
+
|
|
165
|
+
— and it's a hard ceiling, not a typical case: the cache stops trusting itself once `staleToleranceMs` elapses regardless of whether the transition event ever arrives (that's what makes the fallback poll a correctness backstop and not just a nice-to-have), and delivery latency in the common case is one Redis stream round trip, not a poll cycle. Set `staleToleranceMs` to whatever window you're comfortable a downstream outage could go un-noticed on this one path — `100`ms (the default) is a reasonable starting point for most services; push it lower if even a hundred milliseconds of continued traffic against a freshly-tripped dependency is unacceptable, at the cost of the cache being useful for a shorter fraction of the healthy path.
|
|
166
|
+
|
|
167
|
+
**Compare this to the open-state cache deliberately.** `openCacheRefreshMs` never risks an extra call through — staleness there can only make a rejection *late*, not wrong, because it always falls through to a real read right at the real eligibility instant (see limitation item 1). `localCache`'s closed-side cache is the opposite shape: staleness there can make an *allow* wrong for a bounded window, in exchange for the healthy path's read too. That's the real tradeoff you're opting into, not a bug to route around — the same one Envoy's outlier detection makes by default (cache state locally with a bounded TTL, accept staleness, skip the round trip), just off by default here and explicit about the bound when you turn it on.
|
|
168
|
+
|
|
169
|
+
**The version/absence rule, and mixed-fleet rolling deploys.** `CircuitBreakerState.version` is a monotonic counter that increments only on a real transition — its *absence* (not `0`) means "written by a library version that predates `localCache`, provenance unknown." A cache entry built from a version-less read is marked `trusted: false` and never used for the fast path; the very next call for that operation performs a real read instead, forever, until a version-bearing write appears. This is what makes a rolling deploy safe: instances running an old version keep working exactly as before (they don't publish transitions, and their writes lack `version`), and new instances simply never trust a versionless observation rather than guessing. Nothing about this requires deploying in any particular order.
|
|
170
|
+
|
|
171
|
+
**Strategy asymmetry, stated plainly.** `consecutive` is the strategy this cache is built for: combined with the healthy-path write skip, a run of successes against a warm cache is a genuine 0-store-op healthy path. `errorRate` cannot get the same benefit — `recordOutcomeAtomic` must fold *every* outcome (success or failure) into the EWMA to keep the rate meaningful, so it floors at 1 write per call no matter what's cached. `localCache` still saves the *read* for `errorRate` the same way it does for `consecutive`; it just can't save the write, structurally, not as an oversight.
|
|
172
|
+
|
|
173
|
+
**Requires `subscribeTransitions`.** Without it (a custom `Store` that doesn't implement the capability), `localCache` is a no-op: one-time warning, and behavior is identical to not setting it at all. `RedisStore` implements it via a dedicated (`duplicate()`d) blocking connection running `XREAD BLOCK` against the shared `circuit_breaker:events` stream, reconnecting with backoff on its own if that connection drops — see the `Store` interface docs above for the reconnect contract.
|
|
131
174
|
|
|
132
175
|
## API
|
|
133
176
|
|
|
@@ -167,6 +210,7 @@ redeye can't stop you from doing this, but it can flag it: `execute`/`recordFail
|
|
|
167
210
|
| `logger` | no-op | `{ warn(msg), log(msg) }` — plug in your own logger |
|
|
168
211
|
| `maxOperations` | none | Logs a one-time warning once the breaker has seen more than this many distinct `operation` names — see the cardinality note above. Unset: no limit, no warning. |
|
|
169
212
|
| `openCacheRefreshMs` | `2000` | Distributed mode only: while `operation` is known-open, reject locally without a store read, re-verifying against the store at most once per this many ms (to catch an early close by another instance's trial, or a manual `reset()`). Never delays trial eligibility — see [limitation item 1](#whats-still-a-fundamental-tradeoff-not-a-bug). `0` disables the cache (a store read on every call, the pre-cache behavior). |
|
|
213
|
+
| `localCache` | none (disabled) | Distributed mode, opt-in, requires a store implementing `subscribeTransitions`: `{ staleToleranceMs?: number }` (default `100` once the object is present) — caches CLOSED state locally, invalidated push-style, so a healthy call can skip the store read too, not just the write. See [Local caching (hybrid mode)](#local-caching-hybrid-mode) for the fail-open window this trades for that. |
|
|
170
214
|
|
|
171
215
|
### Store unavailability: fail-open vs fail-closed
|
|
172
216
|
|
|
@@ -204,7 +248,7 @@ If your store *doesn't* implement the atomic method your chosen strategy needs (
|
|
|
204
248
|
|
|
205
249
|
### Split-brain: what's prevented, and what isn't
|
|
206
250
|
|
|
207
|
-
There's no split-brain among your app instances in the classic sense, because instances never hold independent authoritative state to
|
|
251
|
+
There's no split-brain among your app instances in the classic sense, because instances never hold independent authoritative state *to write from* — every state-changing decision goes through the store, and the two local caches (`openCacheRefreshMs`, `localCache`) exist only to skip *reading* it before an already-known-safe decision, never to substitute for it. The store is the only "brain"; instances are just readers/writers of it, some of the time from a recent, bounded-staleness memory of what they last read instead of asking again. Two mechanisms enforce agreement on the common path:
|
|
208
252
|
|
|
209
253
|
- **Atomic writes.** `recordFailureAtomic`/`recordOutcomeAtomic` run as one Lua script — GET, decode, increment, decide, SET — inside Redis's single-threaded execution, so two instances failing at the same instant can't both read the same count and both write the same increment.
|
|
210
254
|
- **Exclusive trial claim.** `claimTrial` is a `SET ... NX`, which Redis resolves atomically, so exactly one instance ever runs the half-open recovery probe — never two instances simultaneously deciding they're the one testing recovery.
|
|
@@ -221,7 +265,8 @@ So the design doesn't eliminate split-brain so much as route around it: it pushe
|
|
|
221
265
|
|
|
222
266
|
These are true of any distributed circuit breaker, rate limiter, or lock built on a shared store — not gaps specific to redeye:
|
|
223
267
|
|
|
224
|
-
1. **Distributed mode
|
|
268
|
+
1. **Distributed mode's per-call store cost is opt-in-able down to a bound you choose, not fixed — but "zero" always means "zero, at a stated staleness price," never "zero, for free."** By default, every `execute()` call does a gate read (`store.get`) before deciding whether to proceed; what happens after differs by strategy — `errorRate` writes (`recordOutcomeAtomic`) on every closed-phase call, success or failure, since the rate needs both to be meaningful (this never goes away, at any cache setting — see below), while `consecutive` only writes when the gate's own read observed accumulated state to clear (an open breaker, or `failures > 0`), making a run of healthy successes 1 read and 0 writes per call by default, not 1-and-1. **Once the breaker is open, rejections stop costing a read at all**, unconditionally, no opt-in required: the gate caches the open state locally (`lastFailure`, `openCount`, and the same jittered `expiresAt` the eligibility check itself uses) and rejects straight from that cache, re-verifying against the store only once per `openCacheRefreshMs` (default 2000ms), and always falling through to a real read right at `expiresAt` — so this cache can delay a rejection's freshness but can never delay a trial's eligibility, and never risks letting an extra call through. That's the fail-*closed*-biased half of the story: staleness there can only make things more conservative.
|
|
269
|
+
The closed, healthy path is the other half, and it's where a real tradeoff lives: **`localCache` (opt-in) is redeye's answer to exactly the caching-with-bounded-staleness tradeoff Envoy's outlier detection makes by default** — cache state locally, accept a bounded staleness window, skip the round trip. Off by default (redeye chooses per-call freshness there until you say otherwise), and explicit about the bound when you turn it on: the worst case is `staleToleranceMs + transition-delivery latency`, a formula, not an adjective — see [Local caching (hybrid mode)](#local-caching-hybrid-mode) for exactly what that means and what it doesn't cover (`errorRate`'s write floor, in particular, which `localCache` cannot remove — only `consecutive` reaches a genuine 0-store-op healthy path). A resolving half-open trial always adds a round trip to claim it and another to release its claim, at any cache setting, on either strategy — that path is never cacheable, and rightly so. Local mode has none of this cost at all — see [Local mode vs. distributed mode](#local-mode-vs-distributed-mode-which-do-you-need) if you're not sure you need shared state to begin with.
|
|
225
270
|
2. **Correctness depends on trusting your store.** If Redis evicts a breaker key early under memory pressure (e.g. `maxmemory-policy allkeys-lru` with `circuit_breaker:*` competing for space with everything else), the breaker can reset earlier than intended. Mitigate by giving circuit-breaker keys their own keyspace/DB with a policy that doesn't evict them, or by monitoring `onStoreError` and Redis memory pressure directly. This is not fixable in-library — it's an operational configuration matter.
|
|
226
271
|
3. **Distributed timing is sensitive to clock skew.** Backoff and jitter windows are computed by comparing each instance's local `Date.now()` against a `lastFailure` timestamp written by whichever instance recorded it. With reasonably synchronized clocks (standard NTP) this is a non-issue; on a fleet with significant clock drift, instances can disagree by that drift amount about exactly when a trial becomes eligible. This affects *timing* only — the half-open claim mechanism (`claimTrial`) still guarantees exactly one trial runs regardless of skew, so skew cannot cause a thundering herd, only a slightly early or late trial attempt.
|
|
227
272
|
4. **`recordFailure`/`recordSuccess`/`canExecuteAsync` don't participate in trial claiming.** Only `execute()` claims and releases the half-open trial slot. If you build your own call flow around the manual API instead of `execute()`, you lose the single-trial guarantee and get the old "everyone retries once elapsed" behavior for that flow. Prefer `execute()`.
|
|
@@ -252,6 +297,8 @@ docker compose down
|
|
|
252
297
|
|
|
253
298
|
It connects to `REDIS_URL` (default `redis://localhost:6379`).
|
|
254
299
|
|
|
300
|
+
`npm run bench` runs `bench/index.js` against a real Redis, comparing total store round trips and p50/p99 latency across three configurations: no caching, today's defaults (`openCacheRefreshMs`), and `localCache` on top. Same `REDIS_URL`; start Redis first the same way as the integration suite. See [BENCHMARK.md](BENCHMARK.md) for a full run's results and how to read them.
|
|
301
|
+
|
|
255
302
|
## Contributing
|
|
256
303
|
|
|
257
304
|
See [CONTRIBUTING.md](CONTRIBUTING.md). Changes are tracked in [CHANGELOG.md](CHANGELOG.md).
|
|
@@ -1,11 +1,11 @@
|
|
|
1
1
|
import { Store } from './store';
|
|
2
|
-
import { CircuitBreakerState, CircuitMetrics } from './types';
|
|
2
|
+
import { CircuitBreakerState, CircuitMetrics, TransitionEvent } from './types';
|
|
3
3
|
export type CircuitState = 'closed' | 'open' | 'half-open';
|
|
4
4
|
export interface BreakerLogger {
|
|
5
5
|
warn(message: string): void;
|
|
6
6
|
log(message: string): void;
|
|
7
7
|
}
|
|
8
|
-
export type { CircuitBreakerState, CircuitMetrics };
|
|
8
|
+
export type { CircuitBreakerState, CircuitMetrics, TransitionEvent };
|
|
9
9
|
/**
|
|
10
10
|
* Thrown in distributed mode when the backing store is unreachable and
|
|
11
11
|
* `failOpenOnStoreError` is `false`. Distinguishable from a downstream
|
|
@@ -122,6 +122,32 @@ export interface CircuitBreakerOptions {
|
|
|
122
122
|
* Default: 2000.
|
|
123
123
|
*/
|
|
124
124
|
openCacheRefreshMs?: number;
|
|
125
|
+
/**
|
|
126
|
+
* Distributed mode, opt-in. Serves gate decisions for CLOSED state from a
|
|
127
|
+
* local cache while it's fresher than `staleToleranceMs`, invalidated
|
|
128
|
+
* push-style via the store's transition stream (`Store.subscribeTransitions`)
|
|
129
|
+
* instead of polling — so a healthy call can skip the store read entirely,
|
|
130
|
+
* not just the write Release 1 already skips.
|
|
131
|
+
*
|
|
132
|
+
* This only ever short-circuits an *allow*: a cache entry observed open
|
|
133
|
+
* always defers to the open-state cache (`openCacheRefreshMs`) and a real
|
|
134
|
+
* read, which remains authoritative for all rejection handling. Requires
|
|
135
|
+
* a store implementing `subscribeTransitions`; without one, this option
|
|
136
|
+
* is a no-op (one-time warning) and behavior is unchanged from not
|
|
137
|
+
* setting it at all.
|
|
138
|
+
*
|
|
139
|
+
* The worst case is a bounded fail-open window: if the breaker trips
|
|
140
|
+
* elsewhere in the fleet, a calls still landing on a cached entry can be
|
|
141
|
+
* let through for up to `staleToleranceMs` plus however long the
|
|
142
|
+
* transition event takes to arrive — never longer, since the cache entry
|
|
143
|
+
* expires against `staleToleranceMs` on its own regardless of whether the
|
|
144
|
+
* event ever arrives. See the README's local caching section for the
|
|
145
|
+
* full fail-open/fail-closed asymmetry this creates versus the
|
|
146
|
+
* (fail-closed-biased) open-state cache.
|
|
147
|
+
*/
|
|
148
|
+
localCache?: {
|
|
149
|
+
staleToleranceMs?: number;
|
|
150
|
+
};
|
|
125
151
|
}
|
|
126
152
|
/**
|
|
127
153
|
* A circuit breaker with an optional distributed mode.
|
|
@@ -159,6 +185,22 @@ export declare class CircuitBreaker {
|
|
|
159
185
|
* an early close by another instance or a manual `reset()`.
|
|
160
186
|
*/
|
|
161
187
|
private readonly cachedOpen;
|
|
188
|
+
/** Fixed, unprefixed key for the shared transition-event stream — one per store/prefix, not one per operation. `RedisStore` applies its own `keyPrefix` the same way it does for every other key this class hands it. */
|
|
189
|
+
private readonly eventsKey;
|
|
190
|
+
/**
|
|
191
|
+
* Distributed mode only, `localCache` opt-in: a local, push-invalidated
|
|
192
|
+
* record of the last known state for `operation`, used to serve gate
|
|
193
|
+
* decisions for CLOSED state with zero store reads while fresh and
|
|
194
|
+
* trusted. `trusted: false` means the last write we saw for this entry
|
|
195
|
+
* had no `version` (unknown provenance — an older library version wrote
|
|
196
|
+
* it) or the transition subscription itself dropped; either way, the
|
|
197
|
+
* gate must not use it and instead performs a real read next time,
|
|
198
|
+
* which re-establishes trust.
|
|
199
|
+
*/
|
|
200
|
+
private readonly closedCache;
|
|
201
|
+
/** True once `subscribeTransitions` has actually succeeded; false (the closed cache is never consulted) if `localCache` wasn't set, the store lacks the capability, or the initial subscribe attempt failed. */
|
|
202
|
+
private localCacheActive;
|
|
203
|
+
private unsubscribeTransitions?;
|
|
162
204
|
private readonly failures;
|
|
163
205
|
private readonly lastFailureTime;
|
|
164
206
|
private readonly openOperations;
|
|
@@ -169,7 +211,33 @@ export declare class CircuitBreaker {
|
|
|
169
211
|
private readonly sampleCounts;
|
|
170
212
|
private readonly monitorHandle;
|
|
171
213
|
constructor(options?: CircuitBreakerOptions);
|
|
172
|
-
/**
|
|
214
|
+
/**
|
|
215
|
+
* Attempts to establish the closed-state cache's push invalidation via
|
|
216
|
+
* `Store.subscribeTransitions`. Best-effort and one-shot: if the store
|
|
217
|
+
* doesn't implement it, or the initial subscribe attempt itself throws,
|
|
218
|
+
* `localCache` stays disabled for this breaker's lifetime (a one-time
|
|
219
|
+
* warning either way) — reconnects *after* a successful subscription are
|
|
220
|
+
* the store implementation's own responsibility (see the interface doc).
|
|
221
|
+
*/
|
|
222
|
+
private setupLocalCache;
|
|
223
|
+
/**
|
|
224
|
+
* `null` means the subscription's connection dropped or errored: every
|
|
225
|
+
* cached entry may now be missing events, so all of them are marked
|
|
226
|
+
* untrusted until their next authoritative (real-read) refresh — the
|
|
227
|
+
* fallback poll in `monitor()` is what eventually forces that refresh if
|
|
228
|
+
* no call happens to trigger one first.
|
|
229
|
+
*/
|
|
230
|
+
private handleTransitionEvent;
|
|
231
|
+
/**
|
|
232
|
+
* Refreshes the closed-state cache entry for `operation` with a fresh
|
|
233
|
+
* authoritative read, whether the write came from this instance (already
|
|
234
|
+
* has the resulting state in hand) or a real gate read. A no-op if
|
|
235
|
+
* `localCache` isn't active. Never regresses a cache entry that a newer
|
|
236
|
+
* transition event has already superseded, since a real read can race a
|
|
237
|
+
* push notification and land after it.
|
|
238
|
+
*/
|
|
239
|
+
private updateClosedCache;
|
|
240
|
+
/** Stops the local-mode monitor interval and any transition-event subscription. Call this when the breaker is no longer needed. */
|
|
173
241
|
destroy(): void;
|
|
174
242
|
execute<T>(operation: string, fn: () => Promise<T>): Promise<T>;
|
|
175
243
|
/**
|
|
@@ -240,6 +308,15 @@ export declare class CircuitBreaker {
|
|
|
240
308
|
private key;
|
|
241
309
|
private trialKey;
|
|
242
310
|
private stateTtlSeconds;
|
|
311
|
+
/**
|
|
312
|
+
* Same generous, non-load-bearing safety-net TTL as `stateTtlSeconds`, for
|
|
313
|
+
* the one call site (`reopenTrialFailureAtomic`) that must pick a TTL
|
|
314
|
+
* *before* knowing the resulting `openCount` the atomic script itself
|
|
315
|
+
* computes server-side. Sized off `maxResetTimeout` — the ceiling
|
|
316
|
+
* `backoffCapped` can ever reach at any `openCount` — so it's always at
|
|
317
|
+
* least as generous as the true per-`openCount` value would be.
|
|
318
|
+
*/
|
|
319
|
+
private maxStateTtlSeconds;
|
|
243
320
|
private trialTtlSeconds;
|
|
244
321
|
private reportStoreError;
|
|
245
322
|
private warnMissingCapability;
|
|
@@ -302,6 +379,7 @@ export declare class CircuitBreaker {
|
|
|
302
379
|
* breaker open with no way to ever attempt another trial.
|
|
303
380
|
*/
|
|
304
381
|
private monitor;
|
|
382
|
+
private reconcileClosedCache;
|
|
305
383
|
private executeWithTimeout;
|
|
306
384
|
}
|
|
307
385
|
//# sourceMappingURL=circuit-breaker.d.ts.map
|
|
@@ -1 +1 @@
|
|
|
1
|
-
{"version":3,"file":"circuit-breaker.d.ts","sourceRoot":"","sources":["../src/circuit-breaker.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,KAAK,EAAE,MAAM,SAAS,CAAC;AAChC,OAAO,EAAE,mBAAmB,EAAE,cAAc,EAA8B,MAAM,SAAS,CAAC;
|
|
1
|
+
{"version":3,"file":"circuit-breaker.d.ts","sourceRoot":"","sources":["../src/circuit-breaker.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,KAAK,EAAE,MAAM,SAAS,CAAC;AAChC,OAAO,EAAE,mBAAmB,EAAE,cAAc,EAA8B,eAAe,EAAE,MAAM,SAAS,CAAC;AAE3G,MAAM,MAAM,YAAY,GAAG,QAAQ,GAAG,MAAM,GAAG,WAAW,CAAC;AAE3D,MAAM,WAAW,aAAa;IAC5B,IAAI,CAAC,OAAO,EAAE,MAAM,GAAG,IAAI,CAAC;IAC5B,GAAG,CAAC,OAAO,EAAE,MAAM,GAAG,IAAI,CAAC;CAC5B;AAED,YAAY,EAAE,mBAAmB,EAAE,cAAc,EAAE,eAAe,EAAE,CAAC;AAErE;;;;;GAKG;AACH,qBAAa,qBAAsB,SAAQ,KAAK;IAC9C,QAAQ,CAAC,KAAK,EAAE,OAAO,CAAC;gBAEZ,SAAS,EAAE,MAAM,EAAE,KAAK,EAAE,OAAO;CAK9C;AAED,MAAM,WAAW,qBAAqB;IACpC;;;;;;;;;;;OAWG;IACH,QAAQ,CAAC,EAAE,aAAa,GAAG,WAAW,CAAC;IACvC,wGAAwG;IACxG,gBAAgB,CAAC,EAAE,MAAM,CAAC;IAC1B;;;OAGG;IACH,kBAAkB,CAAC,EAAE,MAAM,CAAC;IAC5B;;;;OAIG;IACH,YAAY,CAAC,EAAE,MAAM,CAAC;IACtB;;;;OAIG;IACH,cAAc,CAAC,EAAE,MAAM,CAAC;IACxB,mGAAmG;IACnG,YAAY,CAAC,EAAE,MAAM,CAAC;IACtB,2FAA2F;IAC3F,eAAe,CAAC,EAAE,MAAM,CAAC;IACzB,qIAAqI;IACrI,OAAO,CAAC,EAAE,MAAM,CAAC;IACjB;;;;;;OAMG;IACH,YAAY,CAAC,EAAE,MAAM,CAAC;IACtB;;;;OAIG;IACH,iBAAiB,CAAC,EAAE,MAAM,CAAC;IAC3B,oFAAoF;IACpF,eAAe,CAAC,EAAE,MAAM,CAAC;IACzB;;;;OAIG;IACH,MAAM,CAAC,EAAE,MAAM,CAAC;IAChB,aAAa,CAAC,EAAE,CAAC,KAAK,EAAE,YAAY,EAAE,SAAS,EAAE,MAAM,KAAK,IAAI,CAAC;IACjE;;;;;OAKG;IACH,KAAK,CAAC,EAAE,KAAK,CAAC;IACd;;;;;;;;;;;;;;OAcG;IACH,oBAAoB,CAAC,EAAE,OAAO,CAAC;IAC/B,YAAY,CAAC,EAAE,CAAC,KAAK,EAAE,OAAO,EAAE,SAAS,EAAE,MAAM,KAAK,IAAI,CAAC;IAC3D,MAAM,CAAC,EAAE,aAAa,CAAC;IACvB;;;;;;;OAOG;IACH,aAAa,CAAC,EAAE,MAAM,CAAC;IACvB;;;;;;;;OAQG;IACH,kBAAkB,CAAC,EAAE,MAAM,CAAC;IAC5B;;;;;;;;;;;;;;;;;;;;;;OAsBG;IACH,UAAU,CAAC,EAAE;QAAE,gBAAgB,CAAC,EAAE,MAAM,CAAA;KAAE,CAAC;CAC5C;AAqCD;;;;;;;;;;;;;;GAcG;AACH,qBAAa,cAAc;IACzB,OAAO,CAAC,QAAQ,CAAC,OAAO,CAuBtB;IAEF,OAAO,CAAC,QAAQ,CAAC,WAAW,CAAU;IACtC,OAAO,CAAC,QAAQ,CAAC,kBAAkB,CAAqB;IACxD,OAAO,CAAC,iCAAiC,CAAS;IAClD,OAAO,CAAC,QAAQ,CAAC,eAAe,CAAqB;IACrD,OAAO,CAAC,mBAAmB,CAAS;IACpC,OAAO,CAAC,0BAA0B,CAAS;IAC3C,OAAO,CAAC,QAAQ,CAAC,OAAO,CAAqC;IAC7D,OAAO,CAAC,QAAQ,CAAC,mBAAmB,CAAgF;IACpH;;;;;;;;;OASG;IACH,OAAO,CAAC,QAAQ,CAAC,UAAU,CAAyG;IAEpI,yNAAyN;IACzN,OAAO,CAAC,QAAQ,CAAC,SAAS,CAA4B;IACtD;;;;;;;;;OASG;IACH,OAAO,CAAC,QAAQ,CAAC,WAAW,CAA2G;IACvI,gNAAgN;IAChN,OAAO,CAAC,gBAAgB,CAAS;IACjC,OAAO,CAAC,sBAAsB,CAAC,CAAa;IAG5C,OAAO,CAAC,QAAQ,CAAC,QAAQ,CAA6B;IACtD,OAAO,CAAC,QAAQ,CAAC,eAAe,CAA6B;IAC7D,OAAO,CAAC,QAAQ,CAAC,cAAc,CAAqB;IACpD,OAAO,CAAC,QAAQ,CAAC,UAAU,CAA6B;IACxD,OAAO,CAAC,QAAQ,CAAC,eAAe,CAAqB;IACrD,OAAO,CAAC,QAAQ,CAAC,cAAc,CAA6B;IAC5D,OAAO,CAAC,QAAQ,CAAC,UAAU,CAA6B;IACxD,OAAO,CAAC,QAAQ,CAAC,YAAY,CAA6B;IAE1D,OAAO,CAAC,QAAQ,CAAC,aAAa,CAAiC;gBAEnD,OAAO,GAAE,qBAA0B;IAmC/C;;;;;;;OAOG;YACW,eAAe;IAkB7B;;;;;;OAMG;IACH,OAAO,CAAC,qBAAqB;IAe7B;;;;;;;OAOG;IACH,OAAO,CAAC,iBAAiB;IAiBzB,mIAAmI;IACnI,OAAO,IAAI,IAAI;IAKT,OAAO,CAAC,CAAC,EAAE,SAAS,EAAE,MAAM,EAAE,EAAE,EAAE,MAAM,OAAO,CAAC,CAAC,CAAC,GAAG,OAAO,CAAC,CAAC,CAAC;IAKrE;;;;;;OAMG;IACH,OAAO,CAAC,kBAAkB;IAsB1B;;;;;;;;OAQG;IACH,UAAU,CAAC,SAAS,EAAE,MAAM,GAAG,OAAO;IAiBtC;;;;;OAKG;IACG,eAAe,CAAC,SAAS,EAAE,MAAM,GAAG,OAAO,CAAC,OAAO,CAAC;IAM1D,aAAa,CAAC,SAAS,EAAE,MAAM,GAAG,IAAI;IAetC,aAAa,CAAC,SAAS,EAAE,MAAM,GAAG,IAAI;IAStC;;;;;;;;;OASG;YACW,yBAAyB;IAQjC,QAAQ,CAAC,SAAS,EAAE,MAAM,GAAG,OAAO,CAAC,mBAAmB,CAAC;IAYzD,KAAK,CAAC,SAAS,EAAE,MAAM,GAAG,OAAO,CAAC,IAAI,CAAC;IAiB7C,wPAAwP;IACxP,UAAU,CAAC,SAAS,EAAE,MAAM,GAAG,cAAc;IAK7C,aAAa,IAAI,MAAM,CAAC,MAAM,EAAE,cAAc,CAAC;IAM/C,OAAO,CAAC,UAAU;IAWlB,OAAO,CAAC,aAAa;IAKrB;;;;;;;;;OASG;IACH,OAAO,CAAC,qBAAqB;IAe7B;;;;;;;OAOG;IACH,OAAO,CAAC,cAAc;IAatB,OAAO,CAAC,GAAG;IAIX,OAAO,CAAC,QAAQ;IAIhB,OAAO,CAAC,eAAe;IAUvB;;;;;;;OAOG;IACH,OAAO,CAAC,kBAAkB;IAI1B,OAAO,CAAC,eAAe;IAIvB,OAAO,CAAC,gBAAgB;IAMxB,OAAO,CAAC,qBAAqB;YAUf,OAAO;YASP,OAAO;IAQrB,qFAAqF;YACvE,WAAW;IAQzB,4NAA4N;YAC9M,YAAY;IAQ1B;;;;;;;;;;;;;;;;OAgBG;YACW,gBAAgB;YAahB,gBAAgB;IAyB9B;;;;;OAKG;YACW,eAAe;YAsEf,kBAAkB;YAsDlB,gCAAgC;IAkD9C;;;;OAIG;YACW,wBAAwB;YAuDxB,iCAAiC;YAyCjC,YAAY;IAkC1B,gFAAgF;IAChF,OAAO,CAAC,aAAa;IAcrB,oFAAoF;IACpF,OAAO,CAAC,cAAc;IAuBtB,OAAO,CAAC,0BAA0B;IAclC,kIAAkI;IAClI,OAAO,CAAC,sBAAsB;IAkB9B,OAAO,CAAC,2BAA2B;IAUnC,OAAO,CAAC,UAAU;IAiBlB;;;;;;OAMG;IACH,OAAO,CAAC,OAAO;YA6BD,oBAAoB;IAOlC,OAAO,CAAC,kBAAkB;CAiB3B"}
|
package/dist/circuit-breaker.js
CHANGED
|
@@ -52,6 +52,21 @@ class CircuitBreaker {
|
|
|
52
52
|
* an early close by another instance or a manual `reset()`.
|
|
53
53
|
*/
|
|
54
54
|
this.cachedOpen = new Map();
|
|
55
|
+
/** Fixed, unprefixed key for the shared transition-event stream — one per store/prefix, not one per operation. `RedisStore` applies its own `keyPrefix` the same way it does for every other key this class hands it. */
|
|
56
|
+
this.eventsKey = 'circuit_breaker:events';
|
|
57
|
+
/**
|
|
58
|
+
* Distributed mode only, `localCache` opt-in: a local, push-invalidated
|
|
59
|
+
* record of the last known state for `operation`, used to serve gate
|
|
60
|
+
* decisions for CLOSED state with zero store reads while fresh and
|
|
61
|
+
* trusted. `trusted: false` means the last write we saw for this entry
|
|
62
|
+
* had no `version` (unknown provenance — an older library version wrote
|
|
63
|
+
* it) or the transition subscription itself dropped; either way, the
|
|
64
|
+
* gate must not use it and instead performs a real read next time,
|
|
65
|
+
* which re-establishes trust.
|
|
66
|
+
*/
|
|
67
|
+
this.closedCache = new Map();
|
|
68
|
+
/** True once `subscribeTransitions` has actually succeeded; false (the closed cache is never consulted) if `localCache` wasn't set, the store lacks the capability, or the initial subscribe attempt failed. */
|
|
69
|
+
this.localCacheActive = false;
|
|
55
70
|
// local mode state
|
|
56
71
|
this.failures = new Map();
|
|
57
72
|
this.lastFailureTime = new Map();
|
|
@@ -82,14 +97,90 @@ class CircuitBreaker {
|
|
|
82
97
|
logger: options.logger ?? noopLogger,
|
|
83
98
|
maxOperations: options.maxOperations,
|
|
84
99
|
openCacheRefreshMs: options.openCacheRefreshMs ?? 2000,
|
|
100
|
+
localCache: options.localCache ? { staleToleranceMs: options.localCache.staleToleranceMs ?? 100 } : undefined,
|
|
85
101
|
};
|
|
86
102
|
this.distributed = !!options.store;
|
|
87
103
|
this.monitorHandle = setInterval(() => this.monitor(), this.options.monitorInterval);
|
|
88
104
|
this.monitorHandle.unref?.();
|
|
105
|
+
if (this.distributed && this.options.localCache) {
|
|
106
|
+
void this.setupLocalCache();
|
|
107
|
+
}
|
|
89
108
|
}
|
|
90
|
-
/**
|
|
109
|
+
/**
|
|
110
|
+
* Attempts to establish the closed-state cache's push invalidation via
|
|
111
|
+
* `Store.subscribeTransitions`. Best-effort and one-shot: if the store
|
|
112
|
+
* doesn't implement it, or the initial subscribe attempt itself throws,
|
|
113
|
+
* `localCache` stays disabled for this breaker's lifetime (a one-time
|
|
114
|
+
* warning either way) — reconnects *after* a successful subscription are
|
|
115
|
+
* the store implementation's own responsibility (see the interface doc).
|
|
116
|
+
*/
|
|
117
|
+
async setupLocalCache() {
|
|
118
|
+
const store = this.options.store;
|
|
119
|
+
if (!store.subscribeTransitions) {
|
|
120
|
+
this.options.logger.warn('localCache is set but the store does not implement subscribeTransitions(); the closed-state cache stays disabled (falling back to open-cache-only behavior). See README.');
|
|
121
|
+
return;
|
|
122
|
+
}
|
|
123
|
+
try {
|
|
124
|
+
this.unsubscribeTransitions = await store.subscribeTransitions(this.eventsKey, (event) => this.handleTransitionEvent(event));
|
|
125
|
+
this.localCacheActive = true;
|
|
126
|
+
}
|
|
127
|
+
catch (error) {
|
|
128
|
+
this.options.logger.warn(`localCache is set but subscribing to transition events failed; the closed-state cache stays disabled: ${error?.message ?? error}`);
|
|
129
|
+
}
|
|
130
|
+
}
|
|
131
|
+
/**
|
|
132
|
+
* `null` means the subscription's connection dropped or errored: every
|
|
133
|
+
* cached entry may now be missing events, so all of them are marked
|
|
134
|
+
* untrusted until their next authoritative (real-read) refresh — the
|
|
135
|
+
* fallback poll in `monitor()` is what eventually forces that refresh if
|
|
136
|
+
* no call happens to trigger one first.
|
|
137
|
+
*/
|
|
138
|
+
handleTransitionEvent(event) {
|
|
139
|
+
if (event === null) {
|
|
140
|
+
for (const entry of this.closedCache.values())
|
|
141
|
+
entry.trusted = false;
|
|
142
|
+
return;
|
|
143
|
+
}
|
|
144
|
+
const existing = this.closedCache.get(event.operation);
|
|
145
|
+
if (existing && event.version <= existing.version)
|
|
146
|
+
return; // stale/duplicate delivery -- ignore
|
|
147
|
+
this.closedCache.set(event.operation, {
|
|
148
|
+
state: { ...types_1.CLOSED_STATE, isOpen: event.isOpen, lastFailure: event.lastFailure, openCount: event.openCount, version: event.version },
|
|
149
|
+
version: event.version,
|
|
150
|
+
fetchedAt: Date.now(),
|
|
151
|
+
trusted: true,
|
|
152
|
+
});
|
|
153
|
+
}
|
|
154
|
+
/**
|
|
155
|
+
* Refreshes the closed-state cache entry for `operation` with a fresh
|
|
156
|
+
* authoritative read, whether the write came from this instance (already
|
|
157
|
+
* has the resulting state in hand) or a real gate read. A no-op if
|
|
158
|
+
* `localCache` isn't active. Never regresses a cache entry that a newer
|
|
159
|
+
* transition event has already superseded, since a real read can race a
|
|
160
|
+
* push notification and land after it.
|
|
161
|
+
*/
|
|
162
|
+
updateClosedCache(operation, state) {
|
|
163
|
+
if (!this.localCacheActive)
|
|
164
|
+
return;
|
|
165
|
+
// null (never written) is unambiguously clean, same as CLOSED_STATE's
|
|
166
|
+
// own version: 0 -- distrust is for a *stored* object that predates the
|
|
167
|
+
// version field, not for "nothing has ever been written here."
|
|
168
|
+
const trusted = state === null || state.version !== undefined;
|
|
169
|
+
const version = state?.version ?? 0;
|
|
170
|
+
const existing = this.closedCache.get(operation);
|
|
171
|
+
if (trusted && existing?.trusted && version < existing.version)
|
|
172
|
+
return; // don't regress past a newer stream event
|
|
173
|
+
this.closedCache.set(operation, {
|
|
174
|
+
state: state ?? { ...types_1.CLOSED_STATE },
|
|
175
|
+
version,
|
|
176
|
+
fetchedAt: Date.now(),
|
|
177
|
+
trusted,
|
|
178
|
+
});
|
|
179
|
+
}
|
|
180
|
+
/** Stops the local-mode monitor interval and any transition-event subscription. Call this when the breaker is no longer needed. */
|
|
91
181
|
destroy() {
|
|
92
182
|
clearInterval(this.monitorHandle);
|
|
183
|
+
this.unsubscribeTransitions?.();
|
|
93
184
|
}
|
|
94
185
|
async execute(operation, fn) {
|
|
95
186
|
this.checkOperationName(operation);
|
|
@@ -212,6 +303,7 @@ class CircuitBreaker {
|
|
|
212
303
|
await this.safeDelTrial(operation);
|
|
213
304
|
this.cachedResetTimeouts.delete(operation);
|
|
214
305
|
this.cachedOpen.delete(operation);
|
|
306
|
+
this.closedCache.delete(operation);
|
|
215
307
|
if (prior?.isOpen) {
|
|
216
308
|
this.options.onStateChange?.('closed', operation);
|
|
217
309
|
}
|
|
@@ -303,6 +395,17 @@ class CircuitBreaker {
|
|
|
303
395
|
// to vary a generous cleanup TTL.
|
|
304
396
|
return Math.max(1, Math.ceil((this.backoffCapped(openCount) * 3) / 1000));
|
|
305
397
|
}
|
|
398
|
+
/**
|
|
399
|
+
* Same generous, non-load-bearing safety-net TTL as `stateTtlSeconds`, for
|
|
400
|
+
* the one call site (`reopenTrialFailureAtomic`) that must pick a TTL
|
|
401
|
+
* *before* knowing the resulting `openCount` the atomic script itself
|
|
402
|
+
* computes server-side. Sized off `maxResetTimeout` — the ceiling
|
|
403
|
+
* `backoffCapped` can ever reach at any `openCount` — so it's always at
|
|
404
|
+
* least as generous as the true per-`openCount` value would be.
|
|
405
|
+
*/
|
|
406
|
+
maxStateTtlSeconds() {
|
|
407
|
+
return Math.max(1, Math.ceil((this.options.maxResetTimeout * 3) / 1000));
|
|
408
|
+
}
|
|
306
409
|
trialTtlSeconds() {
|
|
307
410
|
return Math.max(1, Math.ceil(this.options.trialTimeout / 1000));
|
|
308
411
|
}
|
|
@@ -384,9 +487,29 @@ class CircuitBreaker {
|
|
|
384
487
|
}
|
|
385
488
|
}
|
|
386
489
|
async closeDistributed(operation) {
|
|
490
|
+
if (this.options.store.closeAtomic) {
|
|
491
|
+
try {
|
|
492
|
+
const next = await this.options.store.closeAtomic(this.key(operation), this.eventsKey, {
|
|
493
|
+
ttlSeconds: this.stateTtlSeconds(0),
|
|
494
|
+
operation,
|
|
495
|
+
});
|
|
496
|
+
this.cachedResetTimeouts.delete(operation);
|
|
497
|
+
this.cachedOpen.delete(operation);
|
|
498
|
+
this.updateClosedCache(operation, next);
|
|
499
|
+
return;
|
|
500
|
+
}
|
|
501
|
+
catch (error) {
|
|
502
|
+
this.reportStoreError(error, operation);
|
|
503
|
+
// fall through to non-atomic fallback below
|
|
504
|
+
}
|
|
505
|
+
}
|
|
506
|
+
else {
|
|
507
|
+
this.warnMissingCapability('closeAtomic');
|
|
508
|
+
}
|
|
387
509
|
await this.safeSet(operation, { ...types_1.CLOSED_STATE }, this.stateTtlSeconds(0));
|
|
388
510
|
this.cachedResetTimeouts.delete(operation);
|
|
389
511
|
this.cachedOpen.delete(operation);
|
|
512
|
+
this.updateClosedCache(operation, { ...types_1.CLOSED_STATE });
|
|
390
513
|
}
|
|
391
514
|
/**
|
|
392
515
|
* Determines whether a call is allowed through in distributed mode.
|
|
@@ -395,6 +518,15 @@ class CircuitBreaker {
|
|
|
395
518
|
* `canExecuteAsync`) is read-only and never claims.
|
|
396
519
|
*/
|
|
397
520
|
async gateDistributed(operation, claim) {
|
|
521
|
+
if (this.localCacheActive) {
|
|
522
|
+
const entry = this.closedCache.get(operation);
|
|
523
|
+
if (entry && entry.trusted && !entry.state.isOpen && Date.now() - entry.fetchedAt < this.options.localCache.staleToleranceMs) {
|
|
524
|
+
return { allowed: true, isTrial: false, assumeClean: true };
|
|
525
|
+
}
|
|
526
|
+
// entry.state.isOpen, stale, untrusted, or absent -- defer entirely to
|
|
527
|
+
// the open-state cache / a real read below, which remain
|
|
528
|
+
// authoritative for all open/rejection handling.
|
|
529
|
+
}
|
|
398
530
|
const cached = this.cachedOpen.get(operation);
|
|
399
531
|
if (cached && this.options.openCacheRefreshMs > 0) {
|
|
400
532
|
const now = Date.now();
|
|
@@ -421,6 +553,7 @@ class CircuitBreaker {
|
|
|
421
553
|
return { allowed: true, isTrial: false };
|
|
422
554
|
throw new StoreUnavailableError(operation, error);
|
|
423
555
|
}
|
|
556
|
+
this.updateClosedCache(operation, state); // every real read refreshes the closed cache too, whatever it finds
|
|
424
557
|
if (!state?.isOpen) {
|
|
425
558
|
this.cachedOpen.delete(operation);
|
|
426
559
|
return { allowed: true, isTrial: false, observedState: state };
|
|
@@ -474,15 +607,19 @@ class CircuitBreaker {
|
|
|
474
607
|
else {
|
|
475
608
|
// consecutive strategy, non-trial success: only write if the gate
|
|
476
609
|
// positively observed dirty state. If the read errored
|
|
477
|
-
// (observedState undefined), write anyway — we can't know.
|
|
610
|
+
// (observedState undefined), write anyway — we can't know. A
|
|
611
|
+
// closed-cache fast-pathed gate (`assumeClean`) skips the write
|
|
612
|
+
// too, accepting the same bounded staleness the 0-read cache hit
|
|
613
|
+
// itself already accepted -- this is what gets `consecutive` to a
|
|
614
|
+
// genuine 0-round-trip healthy path.
|
|
478
615
|
//
|
|
479
616
|
// Race analysis: if a concurrent failure lands between our gate
|
|
480
|
-
// read and this skipped write, not writing preserves
|
|
481
|
-
// count — strictly safer than the old unconditional
|
|
617
|
+
// read (or cache hit) and this skipped write, not writing preserves
|
|
618
|
+
// that failure count — strictly safer than the old unconditional
|
|
482
619
|
// `closeDistributed`, which could erase a legitimate concurrent
|
|
483
620
|
// increment by overwriting it with CLOSED_STATE.
|
|
484
621
|
const s = gate.observedState;
|
|
485
|
-
const observedClean = s !== undefined && (s === null || (!s.isOpen && s.failures === 0));
|
|
622
|
+
const observedClean = gate.assumeClean || (s !== undefined && (s === null || (!s.isOpen && s.failures === 0)));
|
|
486
623
|
if (!observedClean) {
|
|
487
624
|
await this.closeDistributed(operation);
|
|
488
625
|
}
|
|
@@ -511,10 +648,12 @@ class CircuitBreaker {
|
|
|
511
648
|
ttlSeconds: this.stateTtlSeconds(0),
|
|
512
649
|
failureThreshold: this.options.failureThreshold,
|
|
513
650
|
now: Date.now(),
|
|
651
|
+
operation,
|
|
514
652
|
});
|
|
515
653
|
this.options.logger.warn(`Recorded failure for operation: ${operation}, total failures: ${next.failures}`);
|
|
516
654
|
if (next.isOpen)
|
|
517
655
|
this.cacheOpenState(operation, next);
|
|
656
|
+
this.updateClosedCache(operation, next);
|
|
518
657
|
if (next.isOpen && next.failures === this.options.failureThreshold) {
|
|
519
658
|
this.options.logger.warn(`Circuit breaker opened for operation: ${operation}`);
|
|
520
659
|
this.options.onStateChange?.('open', operation);
|
|
@@ -546,6 +685,7 @@ class CircuitBreaker {
|
|
|
546
685
|
await this.safeSet(operation, next, this.stateTtlSeconds(0));
|
|
547
686
|
if (next.isOpen)
|
|
548
687
|
this.cacheOpenState(operation, next);
|
|
688
|
+
this.updateClosedCache(operation, next);
|
|
549
689
|
this.options.logger.warn(`Recorded failure for operation: ${operation}, total failures: ${next.failures}`);
|
|
550
690
|
if (next.isOpen && !current.isOpen) {
|
|
551
691
|
this.options.logger.warn(`Circuit breaker opened for operation: ${operation}`);
|
|
@@ -567,9 +707,11 @@ class CircuitBreaker {
|
|
|
567
707
|
errorRateThreshold: this.options.errorRateThreshold,
|
|
568
708
|
ttlSeconds: this.stateTtlSeconds(0),
|
|
569
709
|
now: Date.now(),
|
|
710
|
+
operation,
|
|
570
711
|
});
|
|
571
712
|
if (next.isOpen)
|
|
572
713
|
this.cacheOpenState(operation, next);
|
|
714
|
+
this.updateClosedCache(operation, next);
|
|
573
715
|
if (next.openedNow) {
|
|
574
716
|
this.options.logger.warn(`Circuit breaker opened for operation: ${operation} (error rate ${(next.errorRate * 100).toFixed(1)}% over ${next.sampleCount} calls)`);
|
|
575
717
|
this.options.onStateChange?.('open', operation);
|
|
@@ -600,12 +742,35 @@ class CircuitBreaker {
|
|
|
600
742
|
await this.safeSet(operation, next, this.stateTtlSeconds(0));
|
|
601
743
|
if (isOpen)
|
|
602
744
|
this.cacheOpenState(operation, next);
|
|
745
|
+
this.updateClosedCache(operation, next);
|
|
603
746
|
if (isOpen && !current.isOpen) {
|
|
604
747
|
this.options.logger.warn(`Circuit breaker opened for operation: ${operation} (error rate ${(errorRate * 100).toFixed(1)}% over ${sampleCount} calls)`);
|
|
605
748
|
this.options.onStateChange?.('open', operation);
|
|
606
749
|
}
|
|
607
750
|
}
|
|
608
751
|
async reopenAfterFailedTrialDistributed(operation) {
|
|
752
|
+
if (this.options.store.reopenTrialFailureAtomic) {
|
|
753
|
+
try {
|
|
754
|
+
const next = await this.options.store.reopenTrialFailureAtomic(this.key(operation), this.eventsKey, {
|
|
755
|
+
ttlSeconds: this.maxStateTtlSeconds(),
|
|
756
|
+
failureThreshold: this.options.failureThreshold,
|
|
757
|
+
now: Date.now(),
|
|
758
|
+
operation,
|
|
759
|
+
});
|
|
760
|
+
this.cacheOpenState(operation, next);
|
|
761
|
+
this.updateClosedCache(operation, next);
|
|
762
|
+
this.options.logger.warn(`Half-open trial failed for operation: ${operation}; backing off (attempt ${next.openCount + 1})`);
|
|
763
|
+
this.options.onStateChange?.('open', operation);
|
|
764
|
+
return;
|
|
765
|
+
}
|
|
766
|
+
catch (error) {
|
|
767
|
+
this.reportStoreError(error, operation);
|
|
768
|
+
// fall through to non-atomic fallback below
|
|
769
|
+
}
|
|
770
|
+
}
|
|
771
|
+
else {
|
|
772
|
+
this.warnMissingCapability('reopenTrialFailureAtomic');
|
|
773
|
+
}
|
|
609
774
|
const current = (await this.safeGet(operation)) ?? { ...types_1.CLOSED_STATE, isOpen: true };
|
|
610
775
|
const openCount = (current.openCount ?? 0) + 1;
|
|
611
776
|
const next = {
|
|
@@ -618,6 +783,7 @@ class CircuitBreaker {
|
|
|
618
783
|
};
|
|
619
784
|
await this.safeSet(operation, next, this.stateTtlSeconds(openCount));
|
|
620
785
|
this.cacheOpenState(operation, next); // this instance authored the open state; cache it
|
|
786
|
+
this.updateClosedCache(operation, next);
|
|
621
787
|
this.options.logger.warn(`Half-open trial failed for operation: ${operation}; backing off (attempt ${openCount + 1})`);
|
|
622
788
|
this.options.onStateChange?.('open', operation);
|
|
623
789
|
}
|
|
@@ -760,6 +926,24 @@ class CircuitBreaker {
|
|
|
760
926
|
this.options.logger.warn(`Circuit breaker trial for operation: ${operation} appears stuck (wrapped function never settled); force-releasing after ${staleBound}ms. Consider configuring a timeout.`);
|
|
761
927
|
}
|
|
762
928
|
}
|
|
929
|
+
// Missed-message safety net for the closed-state cache: transitions are
|
|
930
|
+
// normally learned about push-style via subscribeTransitions, but a
|
|
931
|
+
// dropped stream message (or a subscription gap) would otherwise leave
|
|
932
|
+
// a cache entry stale forever. In steady state this is the only
|
|
933
|
+
// background traffic the closed-state cache generates -- one GET per
|
|
934
|
+
// stale operation per monitor tick, not per call.
|
|
935
|
+
if (this.distributed && this.localCacheActive) {
|
|
936
|
+
const pollAgeMs = 5000;
|
|
937
|
+
for (const [operation, entry] of this.closedCache.entries()) {
|
|
938
|
+
if (now - entry.fetchedAt > pollAgeMs) {
|
|
939
|
+
void this.reconcileClosedCache(operation);
|
|
940
|
+
}
|
|
941
|
+
}
|
|
942
|
+
}
|
|
943
|
+
}
|
|
944
|
+
async reconcileClosedCache(operation) {
|
|
945
|
+
const state = await this.safeGet(operation);
|
|
946
|
+
this.updateClosedCache(operation, state);
|
|
763
947
|
}
|
|
764
948
|
// ---- shared -------------------------------------------------------------
|
|
765
949
|
executeWithTimeout(fn) {
|