redeye-breaker 0.1.0 → 0.2.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +68 -14
- package/dist/circuit-breaker.d.ts +44 -2
- package/dist/circuit-breaker.d.ts.map +1 -1
- package/dist/circuit-breaker.js +114 -19
- package/dist/circuit-breaker.js.map +1 -1
- package/dist/store.d.ts +25 -5
- package/dist/store.d.ts.map +1 -1
- package/dist/stores/redis-store.d.ts +19 -5
- package/dist/stores/redis-store.d.ts.map +1 -1
- package/dist/stores/redis-store.js +36 -7
- package/dist/stores/redis-store.js.map +1 -1
- package/package.json +4 -2
package/README.md
CHANGED
|
@@ -1,5 +1,9 @@
|
|
|
1
1
|
# redeye
|
|
2
2
|
|
|
3
|
+
[](https://github.com/laurells/redeye/actions/workflows/ci.yml)
|
|
4
|
+
[](https://www.npmjs.com/package/redeye-breaker)
|
|
5
|
+
[](LICENSE)
|
|
6
|
+
|
|
3
7
|
A circuit breaker for Node.js with an optional **distributed mode**: instead of tracking failures only in the memory of one process, breaker state lives in Redis (or any store you plug in), so a failure burst against a downstream dependency trips the breaker for *every* instance sharing that store — not just the one that saw the failures.
|
|
4
8
|
|
|
5
9
|
When paired with `RedisStore`, redeye is a *reliable* distributed circuit breaker, not just a best-effort one: failure counting is atomic (a Redis Lua script, not a racy get-then-set), and recovery goes through a real half-open state where exactly one instance gets to try the dependency again while everyone else stays blocked — the two properties most local-only or naively-distributed circuit breakers skip.
|
|
@@ -35,6 +39,15 @@ const result = await breaker.execute('payment-gateway', () => callPaymentGateway
|
|
|
35
39
|
|
|
36
40
|
Local mode gets real half-open (single in-flight trial) and backoff+jitter too — it's single-process, so there's no atomicity concern to begin with.
|
|
37
41
|
|
|
42
|
+
### Local mode vs. distributed mode: which do you need?
|
|
43
|
+
|
|
44
|
+
Local mode is not the "lesser" option — for a lot of services, N independent per-instance breakers is the right answer, not a compromise:
|
|
45
|
+
|
|
46
|
+
- **Use local mode** when each instance tripping independently is fine, or even preferable: instances see meaningfully different traffic, the dependency is instance-local anyway (a sidecar, a per-instance cache), or you'd rather each instance protect itself than take on a new piece of shared infrastructure (the store) that can itself fail. Zero dependencies, zero added latency per call, nothing extra to operate.
|
|
47
|
+
- **Use distributed mode** when a failure burst against a *shared* downstream dependency should trip the breaker for the whole fleet at once — e.g. a payment gateway or third-party API, where 5 of 6 instances continuing to hammer a dependency the 6th just found dead is actively harmful (wasted capacity, a worse incident, a slower recovery signal). This costs something real in return: a store round trip on every gated decision (see [item 1](#whats-still-a-fundamental-tradeoff-not-a-bug) below), and a new dependency whose own outages now shape the breaker's behavior (see [Total coordination-layer outage](#total-coordination-layer-outage-the-store-itself-is-down)).
|
|
48
|
+
|
|
49
|
+
If you're not sure, start local — it's strictly cheaper, and nothing about switching to distributed mode later changes your call sites.
|
|
50
|
+
|
|
38
51
|
### Two tripping strategies
|
|
39
52
|
|
|
40
53
|
```ts
|
|
@@ -87,9 +100,11 @@ await breaker.execute('payment-gateway', () => callPaymentGateway());
|
|
|
87
100
|
|
|
88
101
|
Every process pointed at the same Redis instance (and using the same operation name / key prefix) shares breaker state, with atomic counting and single-trial recovery.
|
|
89
102
|
|
|
103
|
+
`RedisStore` stores state under `{keyPrefix}circuit_breaker:{operation}`, plus `{keyPrefix}circuit_breaker:{operation}:trial` while a half-open trial is in flight. `keyPrefix` defaults to `''` (no prefix) — set one (as above) if you share a Redis instance/DB across services and want your keys namespaced.
|
|
104
|
+
|
|
90
105
|
### Bring your own store
|
|
91
106
|
|
|
92
|
-
`RedisStore` implements a `Store` interface with two required methods and
|
|
107
|
+
`RedisStore` implements a `Store` interface with two required methods and four *optional* ones:
|
|
93
108
|
|
|
94
109
|
```ts
|
|
95
110
|
export interface Store {
|
|
@@ -102,20 +117,22 @@ export interface Store {
|
|
|
102
117
|
// one-time warning telling you exactly what's degraded.
|
|
103
118
|
recordFailureAtomic?(key: string, opts: { ttlSeconds: number; failureThreshold: number; now: number }): Promise<CircuitBreakerState>;
|
|
104
119
|
recordOutcomeAtomic?(key: string, opts: { success: boolean; decay: number; minimumCalls: number; errorRateThreshold: number; ttlSeconds: number; now: number }): Promise<CircuitBreakerState & { openedNow: boolean }>;
|
|
105
|
-
claimTrial?(key: string, ttlSeconds: number): Promise<
|
|
120
|
+
claimTrial?(key: string, ttlSeconds: number): Promise<string | null>;
|
|
121
|
+
releaseTrial?(key: string, token: string): Promise<void>;
|
|
106
122
|
}
|
|
107
123
|
```
|
|
108
124
|
|
|
109
125
|
- `recordFailureAtomic` (`strategy: 'consecutive'`) should increment the failure count and decide `isOpen` in one atomic round trip (a Lua script in Redis, a conditional update in DynamoDB, etc.).
|
|
110
126
|
- `recordOutcomeAtomic` (`strategy: 'errorRate'`) should fold one call's outcome into the EWMA rate and decide `isOpen` in one atomic round trip — called on every closed-phase call, not just failures.
|
|
111
|
-
- `claimTrial` should be a conditional "create if absent" write (`SET key
|
|
127
|
+
- `claimTrial` should be a conditional "create if absent" write (`SET key token NX EX ttl` in Redis, with `token` a fresh unique value per call, e.g. a UUID) — it's what makes half-open recovery exclusive to one caller instead of a free-for-all. Return the token you wrote if this call won the claim, or `null` if the key was already taken.
|
|
128
|
+
- `releaseTrial` should be a compare-and-delete (`if GET key == token then DEL key` in Redis, a conditional delete elsewhere), not an unconditional delete. This is what stops a trial that outran its TTL (see [item 11](#whats-still-a-fundamental-tradeoff-not-a-bug)) from deleting a *different* instance's newer claim on the same key when it finally gets around to releasing its own now-stale one. If omitted (or if a call has no confirmed ownership token to release at all — e.g. `claimTrial` itself errored), redeye logs a one-time warning and leaves the claim for its TTL to expire naturally, rather than risk an unconditional delete that could destroy a claim it never confirmed was its own.
|
|
112
129
|
|
|
113
130
|
Implement the two required methods against Memcached, DynamoDB, your own cache wrapper, etc., and you have a working (best-effort) distributed breaker. Add the optional methods relevant to the strategy you use when that store supports a real atomic increment and a real conditional write, and you get the full reliability guarantees.
|
|
114
131
|
|
|
115
132
|
## API
|
|
116
133
|
|
|
117
134
|
- `execute(operation, fn)` — runs `fn` if the breaker allows it (closed, or this call won the half-open trial); throws immediately without calling `fn` otherwise. Records the outcome automatically.
|
|
118
|
-
- `canExecute(operation)` — synchronous, read-only, best-effort check. **Always returns `true` in distributed mode** (it can't await the store) — use `canExecuteAsync` there. Never claims a trial slot.
|
|
135
|
+
- `canExecute(operation)` — synchronous, read-only, best-effort check. **Always returns `true` in distributed mode** (it can't await the store) — use `canExecuteAsync` there. Never claims a trial slot. Logs a one-time warning the first time it's called in distributed mode, since a `true` there doesn't reflect real breaker state.
|
|
119
136
|
- `canExecuteAsync(operation)` — async, store-aware, read-only check. Honors `failOpenOnStoreError`. Never claims a trial slot — see the TOCTOU note below.
|
|
120
137
|
- `recordFailure(operation)` / `recordSuccess(operation)` — manually record an outcome without routing the call through `execute`. Note: these bypass the half-open trial-claim mechanism entirely (see [limitations](#reliability-model--limitations)) — prefer `execute`.
|
|
121
138
|
- `getState(operation)` — returns `{ failures, lastFailure, isOpen, openCount, errorRate, sampleCount }`. `openCount` is how many consecutive half-open trials have failed since the breaker last fully closed (drives backoff). `errorRate`/`sampleCount` are only meaningful under `strategy: 'errorRate'`.
|
|
@@ -123,6 +140,8 @@ Implement the two required methods against Memcached, DynamoDB, your own cache w
|
|
|
123
140
|
- `reset(operation)` — manually closes the breaker for an operation.
|
|
124
141
|
- `destroy()` — stops the internal monitor interval. Call this when a breaker instance is no longer needed (e.g. in tests, or on service shutdown) to avoid leaking a timer.
|
|
125
142
|
|
|
143
|
+
`operation` should be a small, fixed set of names (`'payment-gateway'`, `'fraud-api'`, one per dependency) — not something you interpolate per-request values into (a tenant ID, a URL, a user ID). Every distinct `operation` string gets its own entry in several per-instance maps (metrics, local-mode failure/backoff state, an internal jitter cache) that are never evicted, so a dynamic operation name is an unbounded memory leak, the same footgun as labeling a Prometheus metric with a high-cardinality value.
|
|
144
|
+
|
|
126
145
|
## Options
|
|
127
146
|
|
|
128
147
|
| Option | Default | Description |
|
|
@@ -141,7 +160,7 @@ Implement the two required methods against Memcached, DynamoDB, your own cache w
|
|
|
141
160
|
| `timeout` | none | optional per-call timeout in ms |
|
|
142
161
|
| `store` | none | enables distributed mode when provided |
|
|
143
162
|
| `failOpenOnStoreError` | `true` | see [Store unavailability](#store-unavailability-fail-open-vs-fail-closed) |
|
|
144
|
-
| `onStateChange` | none | `(state: 'open' \| 'closed', operation: string) => void` |
|
|
163
|
+
| `onStateChange` | none | `(state: 'open' \| 'half-open' \| 'closed', operation: string) => void` — `'half-open'` fires exactly when a caller claims the single half-open trial slot. In distributed mode this is a per-process callback, not a fleet-wide broadcast — see [Observability](#what-redeye-actually-solves-with-redisstore-or-any-store-implementing-the-matching-optional-methods) below for which instance actually sees it. |
|
|
145
164
|
| `onStoreError` | none | `(error: unknown, operation: string) => void` — fired whenever a store read/write fails, in addition to being logged |
|
|
146
165
|
| `logger` | no-op | `{ warn(msg), log(msg) }` — plug in your own logger |
|
|
147
166
|
|
|
@@ -154,31 +173,62 @@ If the store itself throws (Redis is down, times out, network partition, etc.),
|
|
|
154
173
|
|
|
155
174
|
Writes back to the store are **always** best-effort — a write failure is logged and reported via `onStoreError`, but never thrown, and never overrides the real result of a call that already happened.
|
|
156
175
|
|
|
176
|
+
### Total coordination-layer outage (the store itself is down)
|
|
177
|
+
|
|
178
|
+
Same `failOpenOnStoreError` branch as above, just hit by every instance at once instead of one partitioned instance — redeye doesn't promote a backup coordinator or fall back to per-instance local tracking as a substitute:
|
|
179
|
+
|
|
180
|
+
- **`true` (default):** every instance lets every call through — no breaking happens anywhere, fleet-wide, for as long as the outage lasts.
|
|
181
|
+
- **`false`:** every instance blocks every call with `StoreUnavailableError`, fleet-wide, including requests that would have succeeded.
|
|
182
|
+
|
|
183
|
+
Nothing is buffered or replayed: outcomes during the outage are never recorded, and the breaker has no memory of the blackout once the store recovers. There's also no back-off from hitting a store known to be down — every guarded call keeps retrying it, so pair `failOpenOnStoreError: false` with fail-fast client options (`ioredis`'s `maxRetriesPerRequest`, `enableOfflineQueue: false`) if you don't want calls queuing on the client's own reconnect logic. Recovery is instant and automatic on the next successful read — no warm-up, no manual reset.
|
|
184
|
+
|
|
185
|
+
High availability of the store itself (Sentinel, Cluster) is your responsibility, configured on the client you hand to `RedisStore` — whatever failover consistency that setup provides is inherited as-is (see [Split-brain](#split-brain-whats-prevented-and-what-isnt)).
|
|
186
|
+
|
|
157
187
|
## Reliability model & limitations
|
|
158
188
|
|
|
159
189
|
### What redeye actually solves (with `RedisStore`, or any store implementing the matching optional methods)
|
|
160
190
|
|
|
161
191
|
- **Exact counting under concurrent failure, for both strategies.** `recordFailureAtomic` (consecutive) and `recordOutcomeAtomic` (errorRate) each run as a single Lua script inside Redis's single-threaded execution — two instances updating at the same instant cannot race and lose an update the way a plain get-then-set would.
|
|
162
192
|
- **Catches flapping, not just hard drops.** `strategy: 'errorRate'` trips on a smoothed failure *rate* once enough samples are seen, so a dependency that succeeds 1 in 5 requests (an 80% failure rate) still trips the breaker even though it never fails N times consecutively — a case `'consecutive'` structurally cannot catch, by design (see [Two tripping strategies](#two-tripping-strategies)).
|
|
163
|
-
- **Real half-open, single-trial recovery.** When the reset window elapses, callers don't all rush in — one caller atomically claims the trial slot (`claimTrial`, a conditional write) and the rest stay blocked (`Circuit breaker is open`) until that trial resolves. This is the classic Hystrix-style half-open state, not "everyone tries at once." Applies to both strategies.
|
|
193
|
+
- **Real half-open, single-trial recovery.** When the reset window elapses, callers don't all rush in — one caller atomically claims the trial slot (`claimTrial`, a conditional write) and the rest stay blocked (`Circuit breaker is open`) until that trial resolves. This is the classic Hystrix-style half-open state, not "everyone tries at once." Applies to both strategies. A single successful trial fully closes the breaker — redeye does not support requiring N consecutive successful trials before closing; that's intentionally out of scope for now, not an oversight.
|
|
164
194
|
- **Exponential backoff with jitter.** A dependency that keeps failing its trial is retried less often each time (`resetTimeout * backoffMultiplier ^ openCount`, capped at `maxResetTimeout`), and the exact retry moment is jittered per-instance so a cluster doesn't hammer a recovering dependency in lockstep.
|
|
165
195
|
- **No hard dependency on precise TTL timing for correctness.** The gating decision is based on elapsed time since the last recorded failure, not "did the key vanish yet" — the store's TTL is a generous safety-net for cleanup, not the primary mechanism. (Early eviction is still possible — see below — but it degrades gracefully instead of being load-bearing.)
|
|
166
196
|
- **Store outages are a distinct, handled condition**, not silently misattributed as the protected operation failing (see fail-open/fail-closed above).
|
|
167
|
-
- **Observability**: per-instance call/success/failure/rejection/store-error counters via `getMetrics`, plus `onStateChange` and `onStoreError` hooks.
|
|
197
|
+
- **Observability**: per-instance call/success/failure/rejection/store-error counters via `getMetrics`, plus `onStateChange` and `onStoreError` hooks. In distributed mode, `onStateChange` is invoked locally by whichever instance's own call happens to trigger a given transition — not broadcast to the rest of the fleet. Concretely: `'open'` fires only on the instance whose write is the one that crosses the threshold (or whose failed trial reopens it); `'half-open'` fires only on the instance that wins the trial claim; `'closed'` fires only on the instance that closes it (via a successful trial, or a manual `recordSuccess()`/`reset()` call). Every other instance sharing that operation still sees the new state reflected in their own gating decisions immediately — they just never get their own `onStateChange` call for a transition they didn't personally cause. If you need a fleet-wide notification (e.g. to page on-call once, not once per instance that happens to notice), de-duplicate downstream of the hook, or drive alerting off the store directly instead.
|
|
168
198
|
|
|
169
199
|
If your store *doesn't* implement the atomic method your chosen strategy needs (`recordFailureAtomic` or `recordOutcomeAtomic`) or `claimTrial`, redeye logs a one-time warning per missing capability and falls back to a non-atomic get-then-set — it still works, just without the atomicity/exclusivity guarantee for that piece.
|
|
170
200
|
|
|
201
|
+
### Split-brain: what's prevented, and what isn't
|
|
202
|
+
|
|
203
|
+
There's no split-brain among your app instances in the classic sense, because instances never hold independent authoritative state to disagree over — every instance reads the store fresh before every single decision (`gateDistributed`) instead of trusting a locally cached opinion. The store is the only "brain"; instances are just readers/writers of it. Two mechanisms enforce agreement on the common path:
|
|
204
|
+
|
|
205
|
+
- **Atomic writes.** `recordFailureAtomic`/`recordOutcomeAtomic` run as one Lua script — GET, decode, increment, decide, SET — inside Redis's single-threaded execution, so two instances failing at the same instant can't both read the same count and both write the same increment.
|
|
206
|
+
- **Exclusive trial claim.** `claimTrial` is a `SET ... NX`, which Redis resolves atomically, so exactly one instance ever runs the half-open recovery probe — never two instances simultaneously deciding they're the one testing recovery.
|
|
207
|
+
|
|
208
|
+
So the design doesn't eliminate split-brain so much as route around it: it pushes the single-arbiter requirement onto the store and never lets an instance act on stale local state instead of asking the store. That said, there are real, narrow windows where instances *can* still disagree — each one is a deliberate availability-over-consistency choice, not a hidden gap:
|
|
209
|
+
|
|
210
|
+
- **An instance partitioned from the store fails open independently** (item 9 below) — during the partition, that one instance's view of the breaker can genuinely diverge from the rest of the fleet's.
|
|
211
|
+
- **A `claimTrial` error can rarely produce two "winners"** (item 10 below) — a narrow window traded for never permanently wedging the breaker open.
|
|
212
|
+
- **A slow trial outliving its claim's TTL is the *more likely* way to get two concurrent trials** (item 11 below) — routine for a slowly-recovering dependency, not rare like item 10.
|
|
213
|
+
- **The store's own replication/failover consistency is inherited, not solved** (item 12 below) — this library adds no consensus layer on top of whatever your Redis deployment already guarantees.
|
|
214
|
+
- **Clock skew causes timing disagreement, not state disagreement** (item 3 below) — it can shift exactly when a trial becomes eligible, but it cannot cause a double-trial, since that's arbitrated by the store, not by comparing clocks.
|
|
215
|
+
|
|
171
216
|
### What's still a fundamental tradeoff, not a bug
|
|
172
217
|
|
|
173
218
|
These are true of any distributed circuit breaker, rate limiter, or lock built on a shared store — not gaps specific to redeye:
|
|
174
219
|
|
|
175
|
-
1. **
|
|
176
|
-
2. **
|
|
177
|
-
3.
|
|
178
|
-
4.
|
|
179
|
-
5. **
|
|
180
|
-
6. **
|
|
181
|
-
7.
|
|
220
|
+
1. **Distributed mode costs two Redis round trips per guarded call, not zero.** Every `execute()` call does a gate read (`store.get`) before deciding whether to proceed, then a state write afterward — `closeDistributed`/`recordFailureAtomic` for `consecutive` (yes, every success writes too, not just failures, since closing is unconditional), `recordOutcomeAtomic` for `errorRate` (writes on every closed-phase call, success or failure, since the rate needs both to be meaningful). A resolving half-open trial adds a third round trip to release its claim. On a latency-sensitive path this is added tail latency, and Redis load that scales linearly with request volume — the opposite tradeoff from something like Envoy's outlier detection, which caches state locally with a short TTL and accepts bounded staleness instead of paying a store round trip per call. redeye chose per-call freshness over that; it's a reasonable choice for many services, but measure it before putting this in front of your highest-QPS call. Local mode has none of this cost — see [Local mode vs. distributed mode](#local-mode-vs-distributed-mode-which-do-you-need) if you're not sure you need shared state at all.
|
|
221
|
+
2. **Correctness depends on trusting your store.** If Redis evicts a breaker key early under memory pressure (e.g. `maxmemory-policy allkeys-lru` with `circuit_breaker:*` competing for space with everything else), the breaker can reset earlier than intended. Mitigate by giving circuit-breaker keys their own keyspace/DB with a policy that doesn't evict them, or by monitoring `onStoreError` and Redis memory pressure directly. This is not fixable in-library — it's an operational configuration matter.
|
|
222
|
+
3. **Distributed timing is sensitive to clock skew.** Backoff and jitter windows are computed by comparing each instance's local `Date.now()` against a `lastFailure` timestamp written by whichever instance recorded it. With reasonably synchronized clocks (standard NTP) this is a non-issue; on a fleet with significant clock drift, instances can disagree by that drift amount about exactly when a trial becomes eligible. This affects *timing* only — the half-open claim mechanism (`claimTrial`) still guarantees exactly one trial runs regardless of skew, so skew cannot cause a thundering herd, only a slightly early or late trial attempt.
|
|
223
|
+
4. **`recordFailure`/`recordSuccess`/`canExecuteAsync` don't participate in trial claiming.** Only `execute()` claims and releases the half-open trial slot. If you build your own call flow around the manual API instead of `execute()`, you lose the single-trial guarantee and get the old "everyone retries once elapsed" behavior for that flow. Prefer `execute()`.
|
|
224
|
+
5. **One strategy/threshold/backoff policy per breaker instance**, applied uniformly to every `operation` string passed to it. Use separate `CircuitBreaker` instances for dependencies that need different policies — and separate `RedisStore` key prefixes if two breakers share an operation name with different strategies (their state shapes are incompatible and will corrupt each other under the same key).
|
|
225
|
+
6. **Local mode has no cross-restart persistence**, by design — it's explicitly the zero-dependency option. Use distributed mode if breaker state needs to survive a process restart.
|
|
226
|
+
7. **Metrics are per-instance**, not automatically aggregated across your fleet — the same as any Prometheus counter you'd scrape per-pod. Aggregate them in your own metrics backend if you need a fleet-wide view.
|
|
227
|
+
8. **`errorRate` is EWMA-smoothed, not a precise sliding window.** It approximates "failure rate over roughly the last ~`1/(1-decay)` calls," not an exact count over an exact window — good enough to catch flapping dependencies, but the exact trip point for a given call sequence is a function of the decay math, not literally "N of the last M calls."
|
|
228
|
+
9. **A partitioned instance fails open on its own, independently of the rest of the fleet.** If an instance's own call to the store errors (that instance can't reach Redis, but others can), it doesn't consult anyone else — it applies `failOpenOnStoreError` locally (default `true`). For the duration of that partition, the isolated instance treats the breaker as closed while the rest of the fleet, still able to reach the store, correctly sees it open. This is scoped to exactly the partitioned instance for exactly the duration of the partition, and it's a deliberate choice: favoring that one instance staying available over it blocking calls based on a store it can't even confirm the state of.
|
|
229
|
+
10. **A `claimTrial` error can rarely let two instances both believe they won the trial.** If the `claimTrial` call itself throws (not "no key", an actual error), the gate does not block the caller — it fails open on the trial too (`allowed: true, isTrial: true`), because refusing here would mean a single store blip at exactly the wrong moment could permanently wedge the breaker open (no one could ever claim the trial again). If two instances hit that specific error in the same narrow window, both can proceed with a trial call. This trades a rare double-trial for never risking permanent lockout.
|
|
230
|
+
11. **The trial-claim TTL expiring mid-flight is a more likely double-trial window than item 10, and it's by design.** `claimTrial`'s key has a TTL of `trialTimeout` (default `min(timeout ?? 10000, resetTimeout)`). If the trial call itself runs *longer* than that TTL, the claim expires while the call is still in flight, and another instance can then claim a second trial against the same recovering dependency — this is the intended anti-wedging mechanism (nothing should be able to hold the exclusive slot forever), not a bug. It's also the routine case, not the rare one: a slowly-recovering dependency whose first trial takes 12 seconds against a 10-second default `trialTimeout` will hit this on every recovery attempt, not occasionally. Set `trialTimeout` comfortably above your dependency's real p99 recovery-check latency if avoiding this matters to you. **What this does *not* do, strictly bounded by the release mechanism:** cascade into a third concurrent trial. `claimTrial` returns a per-call ownership token, and releasing a claim is always either a compare-and-delete against that exact token or nothing at all — never an unconditional delete — so the original caller's own expired claim, once it finally gets released, can never destroy whichever *other* instance's claim now legitimately occupies the key. That makes this limitation "double trial, strictly bounded to two" rather than "double trial, possibly cascading further."
|
|
231
|
+
12. **`RedisStore` inherits Redis's own replication/failover consistency — it doesn't add a consensus layer on top.** A single Redis instance has nothing to split-brain over. But if you run Sentinel or Cluster behind it, a failover with asynchronous replication can lose the last few writes, and reading from a lagging replica can return stale state. `RedisStore` doesn't issue `WAIT` or otherwise wait for replica acknowledgment — it trusts whatever consistency guarantees your Redis deployment itself provides.
|
|
182
232
|
|
|
183
233
|
### What redeye deliberately does not try to be
|
|
184
234
|
|
|
@@ -198,6 +248,10 @@ docker compose down
|
|
|
198
248
|
|
|
199
249
|
It connects to `REDIS_URL` (default `redis://localhost:6379`).
|
|
200
250
|
|
|
251
|
+
## Contributing
|
|
252
|
+
|
|
253
|
+
See [CONTRIBUTING.md](CONTRIBUTING.md). Changes are tracked in [CHANGELOG.md](CHANGELOG.md).
|
|
254
|
+
|
|
201
255
|
## License
|
|
202
256
|
|
|
203
257
|
MIT
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
import { Store } from './store';
|
|
2
2
|
import { CircuitBreakerState, CircuitMetrics } from './types';
|
|
3
|
-
export type CircuitState = 'closed' | 'open';
|
|
3
|
+
export type CircuitState = 'closed' | 'open' | 'half-open';
|
|
4
4
|
export interface BreakerLogger {
|
|
5
5
|
warn(message: string): void;
|
|
6
6
|
log(message: string): void;
|
|
@@ -123,7 +123,9 @@ export declare class CircuitBreaker {
|
|
|
123
123
|
private readonly options;
|
|
124
124
|
private readonly distributed;
|
|
125
125
|
private readonly warnedCapabilities;
|
|
126
|
+
private warnedCanExecuteInDistributedMode;
|
|
126
127
|
private readonly metrics;
|
|
128
|
+
private readonly cachedResetTimeouts;
|
|
127
129
|
private readonly failures;
|
|
128
130
|
private readonly lastFailureTime;
|
|
129
131
|
private readonly openOperations;
|
|
@@ -152,12 +154,34 @@ export declare class CircuitBreaker {
|
|
|
152
154
|
canExecuteAsync(operation: string): Promise<boolean>;
|
|
153
155
|
recordFailure(operation: string): void;
|
|
154
156
|
recordSuccess(operation: string): void;
|
|
157
|
+
/**
|
|
158
|
+
* Used by the manual `recordSuccess()` API only — `execute()`'s own
|
|
159
|
+
* success paths already know locally whether they're closing from a real
|
|
160
|
+
* trial (and fire `onStateChange` directly) or from an already-closed
|
|
161
|
+
* state (where firing would be spurious), so they call `closeDistributed`
|
|
162
|
+
* without this extra read. `recordSuccess()` has no such context — the
|
|
163
|
+
* caller might invoke it on a breaker that's actually open — so it pays
|
|
164
|
+
* for one extra read to detect a real transition, matching what `reset()`
|
|
165
|
+
* already does.
|
|
166
|
+
*/
|
|
167
|
+
private closeDistributedAndNotify;
|
|
155
168
|
getState(operation: string): Promise<CircuitBreakerState>;
|
|
156
169
|
reset(operation: string): Promise<void>;
|
|
157
170
|
/** Per-instance call counters for `operation`. In distributed mode this reflects only calls this instance handled — export it via your own metrics system and aggregate across instances, the same way you would any per-process Prometheus counter. */
|
|
158
171
|
getMetrics(operation: string): CircuitMetrics;
|
|
159
172
|
getAllMetrics(): Record<string, CircuitMetrics>;
|
|
160
173
|
private metricsFor;
|
|
174
|
+
private backoffCapped;
|
|
175
|
+
/**
|
|
176
|
+
* Jitter is rolled once per open "episode" — identified by (openCount,
|
|
177
|
+
* lastFailure), which only change when the breaker opens or a trial fails
|
|
178
|
+
* — and cached, not re-rolled on every gate check. Re-rolling per call
|
|
179
|
+
* would let callers keep sampling until a roll happens to pass, which
|
|
180
|
+
* systematically biases the effective earliest trial time toward
|
|
181
|
+
* `capped * (1 - jitter)` instead of spreading it around the nominal
|
|
182
|
+
* timeout, and could flip a call between "blocked" and "eligible" and
|
|
183
|
+
* back on consecutive checks milliseconds apart.
|
|
184
|
+
*/
|
|
161
185
|
private effectiveResetTimeout;
|
|
162
186
|
private key;
|
|
163
187
|
private trialKey;
|
|
@@ -169,8 +193,26 @@ export declare class CircuitBreaker {
|
|
|
169
193
|
private safeSet;
|
|
170
194
|
/** Deletes the main state key for `operation`. Best-effort: logged, never thrown. */
|
|
171
195
|
private safeDelMain;
|
|
172
|
-
/** Deletes the trial-claim key for `operation
|
|
196
|
+
/** Deletes the trial-claim key for `operation` unconditionally, releasing it early instead of waiting for its TTL. Only for a full reset/wipe — see `safeReleaseTrial` for releasing a specific held claim. Best-effort. */
|
|
173
197
|
private safeDelTrial;
|
|
198
|
+
/**
|
|
199
|
+
* Releases a trial claim this call actually won, early instead of
|
|
200
|
+
* waiting for its TTL. Uses a compare-and-delete (`Store.releaseTrial`)
|
|
201
|
+
* so a trial that outran its TTL can't delete a *different* instance's
|
|
202
|
+
* newer claim on the same key (see README limitation on trial-TTL
|
|
203
|
+
* expiry).
|
|
204
|
+
*
|
|
205
|
+
* Deliberately does *not* fall back to an unconditional delete when
|
|
206
|
+
* there's no token (the gate failed open on the trial without a
|
|
207
|
+
* confirmed claim — e.g. `claimTrial` itself errored) or the store lacks
|
|
208
|
+
* `releaseTrial` (an incomplete custom `Store`) — an unconditional delete
|
|
209
|
+
* in either case would recreate the exact bug this token scheme exists to
|
|
210
|
+
* prevent: deleting a claim we never confirmed is ours, which may by now
|
|
211
|
+
* belong to a different instance. The worst case for doing nothing is the
|
|
212
|
+
* slot staying occupied up to `trialTimeout` longer than it needs to —
|
|
213
|
+
* bounded and safe. Best-effort: logged, never thrown.
|
|
214
|
+
*/
|
|
215
|
+
private safeReleaseTrial;
|
|
174
216
|
private closeDistributed;
|
|
175
217
|
/**
|
|
176
218
|
* Determines whether a call is allowed through in distributed mode.
|
|
@@ -1 +1 @@
|
|
|
1
|
-
{"version":3,"file":"circuit-breaker.d.ts","sourceRoot":"","sources":["../src/circuit-breaker.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,KAAK,EAAE,MAAM,SAAS,CAAC;AAChC,OAAO,EAAE,mBAAmB,EAAE,cAAc,EAA8B,MAAM,SAAS,CAAC;AAE1F,MAAM,MAAM,YAAY,GAAG,QAAQ,GAAG,MAAM,CAAC;
|
|
1
|
+
{"version":3,"file":"circuit-breaker.d.ts","sourceRoot":"","sources":["../src/circuit-breaker.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,KAAK,EAAE,MAAM,SAAS,CAAC;AAChC,OAAO,EAAE,mBAAmB,EAAE,cAAc,EAA8B,MAAM,SAAS,CAAC;AAE1F,MAAM,MAAM,YAAY,GAAG,QAAQ,GAAG,MAAM,GAAG,WAAW,CAAC;AAE3D,MAAM,WAAW,aAAa;IAC5B,IAAI,CAAC,OAAO,EAAE,MAAM,GAAG,IAAI,CAAC;IAC5B,GAAG,CAAC,OAAO,EAAE,MAAM,GAAG,IAAI,CAAC;CAC5B;AAED,YAAY,EAAE,mBAAmB,EAAE,cAAc,EAAE,CAAC;AAEpD;;;;;GAKG;AACH,qBAAa,qBAAsB,SAAQ,KAAK;IAC9C,QAAQ,CAAC,KAAK,EAAE,OAAO,CAAC;gBAEZ,SAAS,EAAE,MAAM,EAAE,KAAK,EAAE,OAAO;CAK9C;AAED,MAAM,WAAW,qBAAqB;IACpC;;;;;;;;;;;OAWG;IACH,QAAQ,CAAC,EAAE,aAAa,GAAG,WAAW,CAAC;IACvC,wGAAwG;IACxG,gBAAgB,CAAC,EAAE,MAAM,CAAC;IAC1B;;;OAGG;IACH,kBAAkB,CAAC,EAAE,MAAM,CAAC;IAC5B;;;;OAIG;IACH,YAAY,CAAC,EAAE,MAAM,CAAC;IACtB;;;;OAIG;IACH,cAAc,CAAC,EAAE,MAAM,CAAC;IACxB,mGAAmG;IACnG,YAAY,CAAC,EAAE,MAAM,CAAC;IACtB,2FAA2F;IAC3F,eAAe,CAAC,EAAE,MAAM,CAAC;IACzB,qIAAqI;IACrI,OAAO,CAAC,EAAE,MAAM,CAAC;IACjB;;;;;;OAMG;IACH,YAAY,CAAC,EAAE,MAAM,CAAC;IACtB;;;;OAIG;IACH,iBAAiB,CAAC,EAAE,MAAM,CAAC;IAC3B,oFAAoF;IACpF,eAAe,CAAC,EAAE,MAAM,CAAC;IACzB;;;;OAIG;IACH,MAAM,CAAC,EAAE,MAAM,CAAC;IAChB,aAAa,CAAC,EAAE,CAAC,KAAK,EAAE,YAAY,EAAE,SAAS,EAAE,MAAM,KAAK,IAAI,CAAC;IACjE;;;;;OAKG;IACH,KAAK,CAAC,EAAE,KAAK,CAAC;IACd;;;;;;;;;;;;;;OAcG;IACH,oBAAoB,CAAC,EAAE,OAAO,CAAC;IAC/B,YAAY,CAAC,EAAE,CAAC,KAAK,EAAE,OAAO,EAAE,SAAS,EAAE,MAAM,KAAK,IAAI,CAAC;IAC3D,MAAM,CAAC,EAAE,aAAa,CAAC;CACxB;AAoBD;;;;;;;;;;;;;;GAcG;AACH,qBAAa,cAAc;IACzB,OAAO,CAAC,QAAQ,CAAC,OAAO,CAatB;IAEF,OAAO,CAAC,QAAQ,CAAC,WAAW,CAAU;IACtC,OAAO,CAAC,QAAQ,CAAC,kBAAkB,CAAqB;IACxD,OAAO,CAAC,iCAAiC,CAAS;IAClD,OAAO,CAAC,QAAQ,CAAC,OAAO,CAAqC;IAC7D,OAAO,CAAC,QAAQ,CAAC,mBAAmB,CAAgF;IAGpH,OAAO,CAAC,QAAQ,CAAC,QAAQ,CAA6B;IACtD,OAAO,CAAC,QAAQ,CAAC,eAAe,CAA6B;IAC7D,OAAO,CAAC,QAAQ,CAAC,cAAc,CAAqB;IACpD,OAAO,CAAC,QAAQ,CAAC,UAAU,CAA6B;IACxD,OAAO,CAAC,QAAQ,CAAC,eAAe,CAAqB;IACrD,OAAO,CAAC,QAAQ,CAAC,cAAc,CAA6B;IAC5D,OAAO,CAAC,QAAQ,CAAC,UAAU,CAA6B;IACxD,OAAO,CAAC,QAAQ,CAAC,YAAY,CAA6B;IAE1D,OAAO,CAAC,QAAQ,CAAC,aAAa,CAAiC;gBAEnD,OAAO,GAAE,qBAA0B;IA4B/C,6FAA6F;IAC7F,OAAO,IAAI,IAAI;IAIT,OAAO,CAAC,CAAC,EAAE,SAAS,EAAE,MAAM,EAAE,EAAE,EAAE,MAAM,OAAO,CAAC,CAAC,CAAC,GAAG,OAAO,CAAC,CAAC,CAAC;IAIrE;;;;OAIG;IACH,UAAU,CAAC,SAAS,EAAE,MAAM,GAAG,OAAO;IAatC;;;;;OAKG;IACG,eAAe,CAAC,SAAS,EAAE,MAAM,GAAG,OAAO,CAAC,OAAO,CAAC;IAM1D,aAAa,CAAC,SAAS,EAAE,MAAM,GAAG,IAAI;IActC,aAAa,CAAC,SAAS,EAAE,MAAM,GAAG,IAAI;IAQtC;;;;;;;;;OASG;YACW,yBAAyB;IAQjC,QAAQ,CAAC,SAAS,EAAE,MAAM,GAAG,OAAO,CAAC,mBAAmB,CAAC;IAYzD,KAAK,CAAC,SAAS,EAAE,MAAM,GAAG,OAAO,CAAC,IAAI,CAAC;IAe7C,wPAAwP;IACxP,UAAU,CAAC,SAAS,EAAE,MAAM,GAAG,cAAc;IAK7C,aAAa,IAAI,MAAM,CAAC,MAAM,EAAE,cAAc,CAAC;IAM/C,OAAO,CAAC,UAAU;IAWlB,OAAO,CAAC,aAAa;IAKrB;;;;;;;;;OASG;IACH,OAAO,CAAC,qBAAqB;IAiB7B,OAAO,CAAC,GAAG;IAIX,OAAO,CAAC,QAAQ;IAIhB,OAAO,CAAC,eAAe;IAUvB,OAAO,CAAC,eAAe;IAIvB,OAAO,CAAC,gBAAgB;IAMxB,OAAO,CAAC,qBAAqB;YAQf,OAAO;YASP,OAAO;IAQrB,qFAAqF;YACvE,WAAW;IAQzB,4NAA4N;YAC9M,YAAY;IAQ1B;;;;;;;;;;;;;;;;OAgBG;YACW,gBAAgB;YAahB,gBAAgB;IAK9B;;;;;OAKG;YACW,eAAe;YAsCf,kBAAkB;YAqClB,gCAAgC;IA6C9C;;;;OAIG;YACW,wBAAwB;YAkDxB,iCAAiC;YAkBjC,YAAY;IAkC1B,gFAAgF;IAChF,OAAO,CAAC,aAAa;IAcrB,oFAAoF;IACpF,OAAO,CAAC,cAAc;IAuBtB,OAAO,CAAC,0BAA0B;IAclC,kIAAkI;IAClI,OAAO,CAAC,sBAAsB;IAkB9B,OAAO,CAAC,2BAA2B;IAUnC,OAAO,CAAC,UAAU;IAiBlB;;;;;;OAMG;IACH,OAAO,CAAC,OAAO;IAgBf,OAAO,CAAC,kBAAkB;CAiB3B"}
|
package/dist/circuit-breaker.js
CHANGED
|
@@ -35,7 +35,9 @@ const noopLogger = { warn: () => { }, log: () => { } };
|
|
|
35
35
|
class CircuitBreaker {
|
|
36
36
|
constructor(options = {}) {
|
|
37
37
|
this.warnedCapabilities = new Set();
|
|
38
|
+
this.warnedCanExecuteInDistributedMode = false;
|
|
38
39
|
this.metrics = new Map();
|
|
40
|
+
this.cachedResetTimeouts = new Map();
|
|
39
41
|
// local mode state
|
|
40
42
|
this.failures = new Map();
|
|
41
43
|
this.lastFailureTime = new Map();
|
|
@@ -82,7 +84,14 @@ class CircuitBreaker {
|
|
|
82
84
|
* `true` — use `canExecuteAsync` there.
|
|
83
85
|
*/
|
|
84
86
|
canExecute(operation) {
|
|
85
|
-
|
|
87
|
+
if (this.distributed) {
|
|
88
|
+
if (!this.warnedCanExecuteInDistributedMode) {
|
|
89
|
+
this.warnedCanExecuteInDistributedMode = true;
|
|
90
|
+
this.options.logger.warn('canExecute() always returns true in distributed mode (it cannot await the store) and does not reflect real breaker state; use canExecuteAsync() instead.');
|
|
91
|
+
}
|
|
92
|
+
return true;
|
|
93
|
+
}
|
|
94
|
+
return this.peekLocalGate(operation);
|
|
86
95
|
}
|
|
87
96
|
/**
|
|
88
97
|
* Async, store-aware, read-only check (never claims a trial slot — use
|
|
@@ -114,12 +123,29 @@ class CircuitBreaker {
|
|
|
114
123
|
}
|
|
115
124
|
recordSuccess(operation) {
|
|
116
125
|
if (this.distributed) {
|
|
117
|
-
void this.
|
|
126
|
+
void this.closeDistributedAndNotify(operation);
|
|
118
127
|
}
|
|
119
128
|
else {
|
|
120
129
|
this.resetLocal(operation);
|
|
121
130
|
}
|
|
122
131
|
}
|
|
132
|
+
/**
|
|
133
|
+
* Used by the manual `recordSuccess()` API only — `execute()`'s own
|
|
134
|
+
* success paths already know locally whether they're closing from a real
|
|
135
|
+
* trial (and fire `onStateChange` directly) or from an already-closed
|
|
136
|
+
* state (where firing would be spurious), so they call `closeDistributed`
|
|
137
|
+
* without this extra read. `recordSuccess()` has no such context — the
|
|
138
|
+
* caller might invoke it on a breaker that's actually open — so it pays
|
|
139
|
+
* for one extra read to detect a real transition, matching what `reset()`
|
|
140
|
+
* already does.
|
|
141
|
+
*/
|
|
142
|
+
async closeDistributedAndNotify(operation) {
|
|
143
|
+
const prior = await this.safeGet(operation);
|
|
144
|
+
await this.closeDistributed(operation);
|
|
145
|
+
if (prior?.isOpen) {
|
|
146
|
+
this.options.onStateChange?.('closed', operation);
|
|
147
|
+
}
|
|
148
|
+
}
|
|
123
149
|
async getState(operation) {
|
|
124
150
|
if (this.distributed) {
|
|
125
151
|
return (await this.safeGet(operation)) ?? { ...types_1.CLOSED_STATE };
|
|
@@ -136,6 +162,7 @@ class CircuitBreaker {
|
|
|
136
162
|
const prior = await this.safeGet(operation);
|
|
137
163
|
await this.safeDelMain(operation);
|
|
138
164
|
await this.safeDelTrial(operation);
|
|
165
|
+
this.cachedResetTimeouts.delete(operation);
|
|
139
166
|
if (prior?.isOpen) {
|
|
140
167
|
this.options.onStateChange?.('closed', operation);
|
|
141
168
|
}
|
|
@@ -165,13 +192,33 @@ class CircuitBreaker {
|
|
|
165
192
|
return m;
|
|
166
193
|
}
|
|
167
194
|
// ---- backoff / jitter -------------------------------------------------
|
|
168
|
-
|
|
195
|
+
backoffCapped(openCount) {
|
|
169
196
|
const backed = this.options.resetTimeout * Math.pow(this.options.backoffMultiplier, openCount);
|
|
170
|
-
|
|
171
|
-
|
|
172
|
-
|
|
173
|
-
|
|
174
|
-
|
|
197
|
+
return Math.min(backed, this.options.maxResetTimeout);
|
|
198
|
+
}
|
|
199
|
+
/**
|
|
200
|
+
* Jitter is rolled once per open "episode" — identified by (openCount,
|
|
201
|
+
* lastFailure), which only change when the breaker opens or a trial fails
|
|
202
|
+
* — and cached, not re-rolled on every gate check. Re-rolling per call
|
|
203
|
+
* would let callers keep sampling until a roll happens to pass, which
|
|
204
|
+
* systematically biases the effective earliest trial time toward
|
|
205
|
+
* `capped * (1 - jitter)` instead of spreading it around the nominal
|
|
206
|
+
* timeout, and could flip a call between "blocked" and "eligible" and
|
|
207
|
+
* back on consecutive checks milliseconds apart.
|
|
208
|
+
*/
|
|
209
|
+
effectiveResetTimeout(operation, openCount, lastFailure) {
|
|
210
|
+
const cached = this.cachedResetTimeouts.get(operation);
|
|
211
|
+
if (cached && cached.openCount === openCount && cached.lastFailure === lastFailure) {
|
|
212
|
+
return cached.value;
|
|
213
|
+
}
|
|
214
|
+
const capped = this.backoffCapped(openCount);
|
|
215
|
+
let value = capped;
|
|
216
|
+
if (this.options.jitter) {
|
|
217
|
+
const delta = capped * this.options.jitter;
|
|
218
|
+
value = capped + (Math.random() * 2 - 1) * delta;
|
|
219
|
+
}
|
|
220
|
+
this.cachedResetTimeouts.set(operation, { openCount, lastFailure, value });
|
|
221
|
+
return value;
|
|
175
222
|
}
|
|
176
223
|
// ---- distributed mode -----------------------------------------------
|
|
177
224
|
key(operation) {
|
|
@@ -184,8 +231,10 @@ class CircuitBreaker {
|
|
|
184
231
|
// A generous safety-net TTL, decoupled from the actual gating decision
|
|
185
232
|
// (which uses lastFailure + effectiveResetTimeout below) so early
|
|
186
233
|
// eviction degrades gracefully rather than being the sole correctness
|
|
187
|
-
// mechanism. See README limitation on store TTL trust.
|
|
188
|
-
|
|
234
|
+
// mechanism. See README limitation on store TTL trust. Uses the
|
|
235
|
+
// unjittered bound — jitter exists to spread out gating decisions, not
|
|
236
|
+
// to vary a generous cleanup TTL.
|
|
237
|
+
return Math.max(1, Math.ceil((this.backoffCapped(openCount) * 3) / 1000));
|
|
189
238
|
}
|
|
190
239
|
trialTtlSeconds() {
|
|
191
240
|
return Math.max(1, Math.ceil(this.options.trialTimeout / 1000));
|
|
@@ -227,7 +276,7 @@ class CircuitBreaker {
|
|
|
227
276
|
this.reportStoreError(error, operation);
|
|
228
277
|
}
|
|
229
278
|
}
|
|
230
|
-
/** Deletes the trial-claim key for `operation
|
|
279
|
+
/** Deletes the trial-claim key for `operation` unconditionally, releasing it early instead of waiting for its TTL. Only for a full reset/wipe — see `safeReleaseTrial` for releasing a specific held claim. Best-effort. */
|
|
231
280
|
async safeDelTrial(operation) {
|
|
232
281
|
try {
|
|
233
282
|
await this.options.store.del(this.trialKey(operation));
|
|
@@ -236,8 +285,40 @@ class CircuitBreaker {
|
|
|
236
285
|
this.reportStoreError(error, operation);
|
|
237
286
|
}
|
|
238
287
|
}
|
|
288
|
+
/**
|
|
289
|
+
* Releases a trial claim this call actually won, early instead of
|
|
290
|
+
* waiting for its TTL. Uses a compare-and-delete (`Store.releaseTrial`)
|
|
291
|
+
* so a trial that outran its TTL can't delete a *different* instance's
|
|
292
|
+
* newer claim on the same key (see README limitation on trial-TTL
|
|
293
|
+
* expiry).
|
|
294
|
+
*
|
|
295
|
+
* Deliberately does *not* fall back to an unconditional delete when
|
|
296
|
+
* there's no token (the gate failed open on the trial without a
|
|
297
|
+
* confirmed claim — e.g. `claimTrial` itself errored) or the store lacks
|
|
298
|
+
* `releaseTrial` (an incomplete custom `Store`) — an unconditional delete
|
|
299
|
+
* in either case would recreate the exact bug this token scheme exists to
|
|
300
|
+
* prevent: deleting a claim we never confirmed is ours, which may by now
|
|
301
|
+
* belong to a different instance. The worst case for doing nothing is the
|
|
302
|
+
* slot staying occupied up to `trialTimeout` longer than it needs to —
|
|
303
|
+
* bounded and safe. Best-effort: logged, never thrown.
|
|
304
|
+
*/
|
|
305
|
+
async safeReleaseTrial(operation, token) {
|
|
306
|
+
if (!token)
|
|
307
|
+
return;
|
|
308
|
+
if (!this.options.store.releaseTrial) {
|
|
309
|
+
this.warnMissingCapability('releaseTrial');
|
|
310
|
+
return;
|
|
311
|
+
}
|
|
312
|
+
try {
|
|
313
|
+
await this.options.store.releaseTrial(this.trialKey(operation), token);
|
|
314
|
+
}
|
|
315
|
+
catch (error) {
|
|
316
|
+
this.reportStoreError(error, operation);
|
|
317
|
+
}
|
|
318
|
+
}
|
|
239
319
|
async closeDistributed(operation) {
|
|
240
320
|
await this.safeSet(operation, { ...types_1.CLOSED_STATE }, this.stateTtlSeconds(0));
|
|
321
|
+
this.cachedResetTimeouts.delete(operation);
|
|
241
322
|
}
|
|
242
323
|
/**
|
|
243
324
|
* Determines whether a call is allowed through in distributed mode.
|
|
@@ -258,7 +339,7 @@ class CircuitBreaker {
|
|
|
258
339
|
}
|
|
259
340
|
if (!state?.isOpen)
|
|
260
341
|
return { allowed: true, isTrial: false };
|
|
261
|
-
const effective = this.effectiveResetTimeout(state.openCount ?? 0);
|
|
342
|
+
const effective = this.effectiveResetTimeout(operation, state.openCount ?? 0, state.lastFailure);
|
|
262
343
|
if (Date.now() - state.lastFailure < effective) {
|
|
263
344
|
return { allowed: false, isTrial: false };
|
|
264
345
|
}
|
|
@@ -266,18 +347,22 @@ class CircuitBreaker {
|
|
|
266
347
|
return { allowed: true, isTrial: false };
|
|
267
348
|
if (this.options.store.claimTrial) {
|
|
268
349
|
try {
|
|
269
|
-
const
|
|
270
|
-
|
|
350
|
+
const token = await this.options.store.claimTrial(this.trialKey(operation), this.trialTtlSeconds());
|
|
351
|
+
if (token)
|
|
352
|
+
this.options.onStateChange?.('half-open', operation);
|
|
353
|
+
return { allowed: token !== null, isTrial: token !== null, trialToken: token ?? undefined };
|
|
271
354
|
}
|
|
272
355
|
catch (error) {
|
|
273
356
|
this.reportStoreError(error, operation);
|
|
274
357
|
if (!this.options.failOpenOnStoreError)
|
|
275
358
|
throw new StoreUnavailableError(operation, error);
|
|
276
359
|
// Can't confirm exclusivity — fail open on the trial itself rather than deadlocking forever.
|
|
360
|
+
this.options.onStateChange?.('half-open', operation);
|
|
277
361
|
return { allowed: true, isTrial: true };
|
|
278
362
|
}
|
|
279
363
|
}
|
|
280
364
|
this.warnMissingCapability('claimTrial');
|
|
365
|
+
this.options.onStateChange?.('half-open', operation);
|
|
281
366
|
return { allowed: true, isTrial: true };
|
|
282
367
|
}
|
|
283
368
|
async executeDistributed(operation, fn) {
|
|
@@ -293,7 +378,8 @@ class CircuitBreaker {
|
|
|
293
378
|
metrics.totalSuccesses++;
|
|
294
379
|
if (gate.isTrial) {
|
|
295
380
|
await this.closeDistributed(operation);
|
|
296
|
-
await this.
|
|
381
|
+
await this.safeReleaseTrial(operation, gate.trialToken);
|
|
382
|
+
this.options.onStateChange?.('closed', operation);
|
|
297
383
|
}
|
|
298
384
|
else if (this.options.strategy === 'errorRate') {
|
|
299
385
|
await this.recordOutcomeDistributed(operation, true);
|
|
@@ -307,7 +393,7 @@ class CircuitBreaker {
|
|
|
307
393
|
metrics.totalFailures++;
|
|
308
394
|
if (gate.isTrial) {
|
|
309
395
|
await this.reopenAfterFailedTrialDistributed(operation);
|
|
310
|
-
await this.
|
|
396
|
+
await this.safeReleaseTrial(operation, gate.trialToken);
|
|
311
397
|
}
|
|
312
398
|
else if (this.options.strategy === 'errorRate') {
|
|
313
399
|
await this.recordOutcomeDistributed(operation, false);
|
|
@@ -346,7 +432,12 @@ class CircuitBreaker {
|
|
|
346
432
|
failures: current.failures + 1,
|
|
347
433
|
lastFailure: Date.now(),
|
|
348
434
|
isOpen: current.failures + 1 >= this.options.failureThreshold,
|
|
349
|
-
|
|
435
|
+
// Preserve, don't reset: a closed-phase failure can't normally happen
|
|
436
|
+
// while the breaker is open, but under failOpenOnStoreError a store
|
|
437
|
+
// blip on the gate read can let a call proceed while state.isOpen is
|
|
438
|
+
// still true elsewhere, and this write must not wipe accumulated
|
|
439
|
+
// backoff if that call then fails and lands here.
|
|
440
|
+
openCount: current.openCount ?? 0,
|
|
350
441
|
errorRate: current.errorRate ?? 0,
|
|
351
442
|
sampleCount: current.sampleCount ?? 0,
|
|
352
443
|
};
|
|
@@ -419,6 +510,7 @@ class CircuitBreaker {
|
|
|
419
510
|
};
|
|
420
511
|
await this.safeSet(operation, next, this.stateTtlSeconds(openCount));
|
|
421
512
|
this.options.logger.warn(`Half-open trial failed for operation: ${operation}; backing off (attempt ${openCount + 1})`);
|
|
513
|
+
this.options.onStateChange?.('open', operation);
|
|
422
514
|
}
|
|
423
515
|
// ---- local mode -------------------------------------------------------
|
|
424
516
|
async executeLocal(operation, fn) {
|
|
@@ -463,7 +555,7 @@ class CircuitBreaker {
|
|
|
463
555
|
return true;
|
|
464
556
|
const openCount = this.openCounts.get(operation) ?? 0;
|
|
465
557
|
const lastFailure = this.lastFailureTime.get(operation) ?? 0;
|
|
466
|
-
const effective = this.effectiveResetTimeout(openCount);
|
|
558
|
+
const effective = this.effectiveResetTimeout(operation, openCount, lastFailure);
|
|
467
559
|
if (Date.now() - lastFailure < effective) {
|
|
468
560
|
this.options.logger.warn(`Circuit breaker open for operation: ${operation}`);
|
|
469
561
|
return false;
|
|
@@ -476,7 +568,7 @@ class CircuitBreaker {
|
|
|
476
568
|
return { allowed: true, isTrial: false };
|
|
477
569
|
const openCount = this.openCounts.get(operation) ?? 0;
|
|
478
570
|
const lastFailure = this.lastFailureTime.get(operation) ?? 0;
|
|
479
|
-
const effective = this.effectiveResetTimeout(openCount);
|
|
571
|
+
const effective = this.effectiveResetTimeout(operation, openCount, lastFailure);
|
|
480
572
|
if (Date.now() - lastFailure < effective) {
|
|
481
573
|
this.options.logger.warn(`Circuit breaker open for operation: ${operation}`);
|
|
482
574
|
return { allowed: false, isTrial: false };
|
|
@@ -487,6 +579,7 @@ class CircuitBreaker {
|
|
|
487
579
|
}
|
|
488
580
|
this.trialInProgress.add(operation);
|
|
489
581
|
this.trialClaimedAt.set(operation, Date.now());
|
|
582
|
+
this.options.onStateChange?.('half-open', operation);
|
|
490
583
|
return { allowed: true, isTrial: true };
|
|
491
584
|
}
|
|
492
585
|
recordFailureLocalCounting(operation) {
|
|
@@ -523,6 +616,7 @@ class CircuitBreaker {
|
|
|
523
616
|
this.openCounts.set(operation, openCount);
|
|
524
617
|
this.lastFailureTime.set(operation, Date.now());
|
|
525
618
|
this.options.logger.warn(`Half-open trial failed for operation: ${operation}; backing off (attempt ${openCount + 1})`);
|
|
619
|
+
this.options.onStateChange?.('open', operation);
|
|
526
620
|
}
|
|
527
621
|
resetLocal(operation) {
|
|
528
622
|
const wasOpen = this.openOperations.has(operation);
|
|
@@ -534,6 +628,7 @@ class CircuitBreaker {
|
|
|
534
628
|
this.trialClaimedAt.delete(operation);
|
|
535
629
|
this.errorRates.delete(operation);
|
|
536
630
|
this.sampleCounts.delete(operation);
|
|
631
|
+
this.cachedResetTimeouts.delete(operation);
|
|
537
632
|
if (wasOpen) {
|
|
538
633
|
this.options.onStateChange?.('closed', operation);
|
|
539
634
|
}
|
|
@@ -1 +1 @@
|
|
|
1
|
-
{"version":3,"file":"circuit-breaker.js","sourceRoot":"","sources":["../src/circuit-breaker.ts"],"names":[],"mappings":";;;AACA,mCAA0F;AAW1F;;;;;GAKG;AACH,MAAa,qBAAsB,SAAQ,KAAK;IAG9C,YAAY,SAAiB,EAAE,KAAc;QAC3C,KAAK,CAAC,oDAAoD,SAAS,EAAE,CAAC,CAAC;QACvE,IAAI,CAAC,IAAI,GAAG,uBAAuB,CAAC;QACpC,IAAI,CAAC,KAAK,GAAG,KAAK,CAAC;IACrB,CAAC;CACF;AARD,sDAQC;AA2FD,MAAM,UAAU,GAAkB,EAAE,IAAI,EAAE,GAAG,EAAE,GAAE,CAAC,EAAE,GAAG,EAAE,GAAG,EAAE,GAAE,CAAC,EAAE,CAAC;AAQpE;;;;;;;;;;;;;;GAcG;AACH,MAAa,cAAc;IAgCzB,YAAY,UAAiC,EAAE;QAf9B,uBAAkB,GAAG,IAAI,GAAG,EAAU,CAAC;QACvC,YAAO,GAAG,IAAI,GAAG,EAA0B,CAAC;QAE7D,mBAAmB;QACF,aAAQ,GAAG,IAAI,GAAG,EAAkB,CAAC;QACrC,oBAAe,GAAG,IAAI,GAAG,EAAkB,CAAC;QAC5C,mBAAc,GAAG,IAAI,GAAG,EAAU,CAAC;QACnC,eAAU,GAAG,IAAI,GAAG,EAAkB,CAAC;QACvC,oBAAe,GAAG,IAAI,GAAG,EAAU,CAAC;QACpC,mBAAc,GAAG,IAAI,GAAG,EAAkB,CAAC;QAC3C,eAAU,GAAG,IAAI,GAAG,EAAkB,CAAC;QACvC,iBAAY,GAAG,IAAI,GAAG,EAAkB,CAAC;QAKxD,MAAM,YAAY,GAAG,OAAO,CAAC,YAAY,IAAI,KAAK,CAAC;QACnD,IAAI,CAAC,OAAO,GAAG;YACb,QAAQ,EAAE,OAAO,CAAC,QAAQ,IAAI,aAAa;YAC3C,gBAAgB,EAAE,OAAO,CAAC,gBAAgB,IAAI,CAAC;YAC/C,kBAAkB,EAAE,OAAO,CAAC,kBAAkB,IAAI,GAAG;YACrD,YAAY,EAAE,OAAO,CAAC,YAAY,IAAI,EAAE;YACxC,cAAc,EAAE,OAAO,CAAC,cAAc,IAAI,GAAG;YAC7C,YAAY;YACZ,eAAe,EAAE,OAAO,CAAC,eAAe,IAAI,IAAI;YAChD,OAAO,EAAE,OAAO,CAAC,OAAO;YACxB,YAAY,EAAE,OAAO,CAAC,YAAY,IAAI,IAAI,CAAC,GAAG,CAAC,OAAO,CAAC,OAAO,IAAI,KAAK,EAAE,YAAY,CAAC;YACtF,iBAAiB,EAAE,OAAO,CAAC,iBAAiB,IAAI,CAAC;YACjD,eAAe,EAAE,OAAO,CAAC,eAAe,IAAI,YAAY,GAAG,CAAC;YAC5D,MAAM,EAAE,OAAO,CAAC,MAAM,IAAI,GAAG;YAC7B,aAAa,EAAE,OAAO,CAAC,aAAa;YACpC,KAAK,EAAE,OAAO,CAAC,KAAK;YACpB,oBAAoB,EAAE,OAAO,CAAC,oBAAoB,IAAI,IAAI;YAC1D,YAAY,EAAE,OAAO,CAAC,YAAY;YAClC,MAAM,EAAE,OAAO,CAAC,MAAM,IAAI,UAAU;SACrC,CAAC;QAEF,IAAI,CAAC,WAAW,GAAG,CAAC,CAAC,OAAO,CAAC,KAAK,CAAC;QAEnC,IAAI,CAAC,aAAa,GAAG,WAAW,CAAC,GAAG,EAAE,CAAC,IAAI,CAAC,OAAO,EAAE,EAAE,IAAI,CAAC,OAAO,CAAC,eAAe,CAAC,CAAC;QACrF,IAAI,CAAC,aAAa,CAAC,KAAK,EAAE,EAAE,CAAC;IAC/B,CAAC;IAED,6FAA6F;IAC7F,OAAO;QACL,aAAa,CAAC,IAAI,CAAC,aAAa,CAAC,CAAC;IACpC,CAAC;IAED,KAAK,CAAC,OAAO,CAAI,SAAiB,EAAE,EAAoB;QACtD,OAAO,IAAI,CAAC,WAAW,CAAC,CAAC,CAAC,IAAI,CAAC,kBAAkB,CAAC,SAAS,EAAE,EAAE,CAAC,CAAC,CAAC,CAAC,IAAI,CAAC,YAAY,CAAC,SAAS,EAAE,EAAE,CAAC,CAAC;IACtG,CAAC;IAED;;;;OAIG;IACH,UAAU,CAAC,SAAiB;QAC1B,OAAO,IAAI,CAAC,WAAW,CAAC,CAAC,CAAC,IAAI,CAAC,CAAC,CAAC,IAAI,CAAC,aAAa,CAAC,SAAS,CAAC,CAAC;IACjE,CAAC;IAED;;;;;OAKG;IACH,KAAK,CAAC,eAAe,CAAC,SAAiB;QACrC,IAAI,CAAC,IAAI,CAAC,WAAW;YAAE,OAAO,IAAI,CAAC,aAAa,CAAC,SAAS,CAAC,CAAC;QAC5D,MAAM,IAAI,GAAG,MAAM,IAAI,CAAC,eAAe,CAAC,SAAS,EAAE,KAAK,CAAC,CAAC;QAC1D,OAAO,IAAI,CAAC,OAAO,CAAC;IACtB,CAAC;IAED,aAAa,CAAC,SAAiB;QAC7B,IAAI,IAAI,CAAC,WAAW,EAAE,CAAC;YACrB,IAAI,IAAI,CAAC,OAAO,CAAC,QAAQ,KAAK,WAAW,EAAE,CAAC;gBAC1C,KAAK,IAAI,CAAC,wBAAwB,CAAC,SAAS,EAAE,KAAK,CAAC,CAAC;YACvD,CAAC;iBAAM,CAAC;gBACN,KAAK,IAAI,CAAC,gCAAgC,CAAC,SAAS,CAAC,CAAC;YACxD,CAAC;QACH,CAAC;aAAM,IAAI,IAAI,CAAC,OAAO,CAAC,QAAQ,KAAK,WAAW,EAAE,CAAC;YACjD,IAAI,CAAC,sBAAsB,CAAC,SAAS,EAAE,KAAK,CAAC,CAAC;QAChD,CAAC;aAAM,CAAC;YACN,IAAI,CAAC,0BAA0B,CAAC,SAAS,CAAC,CAAC;QAC7C,CAAC;IACH,CAAC;IAED,aAAa,CAAC,SAAiB;QAC7B,IAAI,IAAI,CAAC,WAAW,EAAE,CAAC;YACrB,KAAK,IAAI,CAAC,gBAAgB,CAAC,SAAS,CAAC,CAAC;QACxC,CAAC;aAAM,CAAC;YACN,IAAI,CAAC,UAAU,CAAC,SAAS,CAAC,CAAC;QAC7B,CAAC;IACH,CAAC;IAED,KAAK,CAAC,QAAQ,CAAC,SAAiB;QAC9B,IAAI,IAAI,CAAC,WAAW,EAAE,CAAC;YACrB,OAAO,CAAC,MAAM,IAAI,CAAC,OAAO,CAAC,SAAS,CAAC,CAAC,IAAI,EAAE,GAAG,oBAAY,EAAE,CAAC;QAChE,CAAC;QACD,MAAM,QAAQ,GAAG,IAAI,CAAC,QAAQ,CAAC,GAAG,CAAC,SAAS,CAAC,IAAI,CAAC,CAAC;QACnD,MAAM,WAAW,GAAG,IAAI,CAAC,eAAe,CAAC,GAAG,CAAC,SAAS,CAAC,IAAI,CAAC,CAAC;QAC7D,MAAM,SAAS,GAAG,IAAI,CAAC,UAAU,CAAC,GAAG,CAAC,SAAS,CAAC,IAAI,CAAC,CAAC;QACtD,MAAM,SAAS,GAAG,IAAI,CAAC,UAAU,CAAC,GAAG,CAAC,SAAS,CAAC,IAAI,CAAC,CAAC;QACtD,MAAM,WAAW,GAAG,IAAI,CAAC,YAAY,CAAC,GAAG,CAAC,SAAS,CAAC,IAAI,CAAC,CAAC;QAC1D,OAAO,EAAE,QAAQ,EAAE,WAAW,EAAE,MAAM,EAAE,IAAI,CAAC,cAAc,CAAC,GAAG,CAAC,SAAS,CAAC,EAAE,SAAS,EAAE,SAAS,EAAE,WAAW,EAAE,CAAC;IAClH,CAAC;IAED,KAAK,CAAC,KAAK,CAAC,SAAiB;QAC3B,IAAI,IAAI,CAAC,WAAW,EAAE,CAAC;YACrB,MAAM,KAAK,GAAG,MAAM,IAAI,CAAC,OAAO,CAAC,SAAS,CAAC,CAAC;YAC5C,MAAM,IAAI,CAAC,WAAW,CAAC,SAAS,CAAC,CAAC;YAClC,MAAM,IAAI,CAAC,YAAY,CAAC,SAAS,CAAC,CAAC;YACnC,IAAI,KAAK,EAAE,MAAM,EAAE,CAAC;gBAClB,IAAI,CAAC,OAAO,CAAC,aAAa,EAAE,CAAC,QAAQ,EAAE,SAAS,CAAC,CAAC;YACpD,CAAC;YACD,IAAI,CAAC,OAAO,CAAC,MAAM,CAAC,GAAG,CAAC,wCAAwC,SAAS,EAAE,CAAC,CAAC;QAC/E,CAAC;aAAM,CAAC;YACN,IAAI,CAAC,UAAU,CAAC,SAAS,CAAC,CAAC;QAC7B,CAAC;IACH,CAAC;IAED,wPAAwP;IACxP,UAAU,CAAC,SAAiB;QAC1B,MAAM,CAAC,GAAG,IAAI,CAAC,OAAO,CAAC,GAAG,CAAC,SAAS,CAAC,CAAC;QACtC,OAAO,CAAC,CAAC,CAAC,CAAC,EAAE,GAAG,CAAC,EAAE,CAAC,CAAC,CAAC,IAAA,oBAAY,GAAE,CAAC;IACvC,CAAC;IAED,aAAa;QACX,MAAM,MAAM,GAAmC,EAAE,CAAC;QAClD,KAAK,MAAM,CAAC,SAAS,EAAE,CAAC,CAAC,IAAI,IAAI,CAAC,OAAO,CAAC,OAAO,EAAE;YAAE,MAAM,CAAC,SAAS,CAAC,GAAG,EAAE,GAAG,CAAC,EAAE,CAAC;QAClF,OAAO,MAAM,CAAC;IAChB,CAAC;IAEO,UAAU,CAAC,SAAiB;QAClC,IAAI,CAAC,GAAG,IAAI,CAAC,OAAO,CAAC,GAAG,CAAC,SAAS,CAAC,CAAC;QACpC,IAAI,CAAC,CAAC,EAAE,CAAC;YACP,CAAC,GAAG,IAAA,oBAAY,GAAE,CAAC;YACnB,IAAI,CAAC,OAAO,CAAC,GAAG,CAAC,SAAS,EAAE,CAAC,CAAC,CAAC;QACjC,CAAC;QACD,OAAO,CAAC,CAAC;IACX,CAAC;IAED,0EAA0E;IAElE,qBAAqB,CAAC,SAAiB;QAC7C,MAAM,MAAM,GAAG,IAAI,CAAC,OAAO,CAAC,YAAY,GAAG,IAAI,CAAC,GAAG,CAAC,IAAI,CAAC,OAAO,CAAC,iBAAiB,EAAE,SAAS,CAAC,CAAC;QAC/F,MAAM,MAAM,GAAG,IAAI,CAAC,GAAG,CAAC,MAAM,EAAE,IAAI,CAAC,OAAO,CAAC,eAAe,CAAC,CAAC;QAC9D,IAAI,CAAC,IAAI,CAAC,OAAO,CAAC,MAAM;YAAE,OAAO,MAAM,CAAC;QACxC,MAAM,KAAK,GAAG,MAAM,GAAG,IAAI,CAAC,OAAO,CAAC,MAAM,CAAC;QAC3C,OAAO,MAAM,GAAG,CAAC,IAAI,CAAC,MAAM,EAAE,GAAG,CAAC,GAAG,CAAC,CAAC,GAAG,KAAK,CAAC;IAClD,CAAC;IAED,wEAAwE;IAEhE,GAAG,CAAC,SAAiB;QAC3B,OAAO,mBAAmB,SAAS,EAAE,CAAC;IACxC,CAAC;IAEO,QAAQ,CAAC,SAAiB;QAChC,OAAO,mBAAmB,SAAS,QAAQ,CAAC;IAC9C,CAAC;IAEO,eAAe,CAAC,SAAiB;QACvC,uEAAuE;QACvE,kEAAkE;QAClE,sEAAsE;QACtE,uDAAuD;QACvD,OAAO,IAAI,CAAC,GAAG,CAAC,CAAC,EAAE,IAAI,CAAC,IAAI,CAAC,CAAC,IAAI,CAAC,qBAAqB,CAAC,SAAS,CAAC,GAAG,CAAC,CAAC,GAAG,IAAI,CAAC,CAAC,CAAC;IACpF,CAAC;IAEO,eAAe;QACrB,OAAO,IAAI,CAAC,GAAG,CAAC,CAAC,EAAE,IAAI,CAAC,IAAI,CAAC,IAAI,CAAC,OAAO,CAAC,YAAY,GAAG,IAAI,CAAC,CAAC,CAAC;IAClE,CAAC;IAEO,gBAAgB,CAAC,KAAc,EAAE,SAAiB;QACxD,IAAI,CAAC,UAAU,CAAC,SAAS,CAAC,CAAC,gBAAgB,EAAE,CAAC;QAC9C,IAAI,CAAC,OAAO,CAAC,MAAM,CAAC,IAAI,CAAC,8CAA8C,SAAS,KAAM,KAAe,EAAE,OAAO,IAAI,KAAK,EAAE,CAAC,CAAC;QAC3H,IAAI,CAAC,OAAO,CAAC,YAAY,EAAE,CAAC,KAAK,EAAE,SAAS,CAAC,CAAC;IAChD,CAAC;IAEO,qBAAqB,CAAC,UAAwE;QACpG,IAAI,IAAI,CAAC,kBAAkB,CAAC,GAAG,CAAC,UAAU,CAAC;YAAE,OAAO;QACpD,IAAI,CAAC,kBAAkB,CAAC,GAAG,CAAC,UAAU,CAAC,CAAC;QACxC,IAAI,CAAC,OAAO,CAAC,MAAM,CAAC,IAAI,CACtB,4BAA4B,UAAU,oGAAoG,CAC3I,CAAC;IACJ,CAAC;IAEO,KAAK,CAAC,OAAO,CAAC,SAAiB;QACrC,IAAI,CAAC;YACH,OAAO,MAAM,IAAI,CAAC,OAAO,CAAC,KAAM,CAAC,GAAG,CAAsB,IAAI,CAAC,GAAG,CAAC,SAAS,CAAC,CAAC,CAAC;QACjF,CAAC;QAAC,OAAO,KAAK,EAAE,CAAC;YACf,IAAI,CAAC,gBAAgB,CAAC,KAAK,EAAE,SAAS,CAAC,CAAC;YACxC,OAAO,IAAI,CAAC;QACd,CAAC;IACH,CAAC;IAEO,KAAK,CAAC,OAAO,CAAC,SAAiB,EAAE,KAA0B,EAAE,UAAkB;QACrF,IAAI,CAAC;YACH,MAAM,IAAI,CAAC,OAAO,CAAC,KAAM,CAAC,GAAG,CAAC,IAAI,CAAC,GAAG,CAAC,SAAS,CAAC,EAAE,KAAK,EAAE,UAAU,CAAC,CAAC;QACxE,CAAC;QAAC,OAAO,KAAK,EAAE,CAAC;YACf,IAAI,CAAC,gBAAgB,CAAC,KAAK,EAAE,SAAS,CAAC,CAAC;QAC1C,CAAC;IACH,CAAC;IAED,qFAAqF;IAC7E,KAAK,CAAC,WAAW,CAAC,SAAiB;QACzC,IAAI,CAAC;YACH,MAAM,IAAI,CAAC,OAAO,CAAC,KAAM,CAAC,GAAG,CAAC,IAAI,CAAC,GAAG,CAAC,SAAS,CAAC,CAAC,CAAC;QACrD,CAAC;QAAC,OAAO,KAAK,EAAE,CAAC;YACf,IAAI,CAAC,gBAAgB,CAAC,KAAK,EAAE,SAAS,CAAC,CAAC;QAC1C,CAAC;IACH,CAAC;IAED,mHAAmH;IAC3G,KAAK,CAAC,YAAY,CAAC,SAAiB;QAC1C,IAAI,CAAC;YACH,MAAM,IAAI,CAAC,OAAO,CAAC,KAAM,CAAC,GAAG,CAAC,IAAI,CAAC,QAAQ,CAAC,SAAS,CAAC,CAAC,CAAC;QAC1D,CAAC;QAAC,OAAO,KAAK,EAAE,CAAC;YACf,IAAI,CAAC,gBAAgB,CAAC,KAAK,EAAE,SAAS,CAAC,CAAC;QAC1C,CAAC;IACH,CAAC;IAEO,KAAK,CAAC,gBAAgB,CAAC,SAAiB;QAC9C,MAAM,IAAI,CAAC,OAAO,CAAC,SAAS,EAAE,EAAE,GAAG,oBAAY,EAAE,EAAE,IAAI,CAAC,eAAe,CAAC,CAAC,CAAC,CAAC,CAAC;IAC9E,CAAC;IAED;;;;;OAKG;IACK,KAAK,CAAC,eAAe,CAAC,SAAiB,EAAE,KAAc;QAC7D,IAAI,KAAiC,CAAC;QACtC,IAAI,CAAC;YACH,KAAK,GAAG,MAAM,IAAI,CAAC,OAAO,CAAC,KAAM,CAAC,GAAG,CAAsB,IAAI,CAAC,GAAG,CAAC,SAAS,CAAC,CAAC,CAAC;QAClF,CAAC;QAAC,OAAO,KAAK,EAAE,CAAC;YACf,IAAI,CAAC,gBAAgB,CAAC,KAAK,EAAE,SAAS,CAAC,CAAC;YACxC,IAAI,IAAI,CAAC,OAAO,CAAC,oBAAoB;gBAAE,OAAO,EAAE,OAAO,EAAE,IAAI,EAAE,OAAO,EAAE,KAAK,EAAE,CAAC;YAChF,MAAM,IAAI,qBAAqB,CAAC,SAAS,EAAE,KAAK,CAAC,CAAC;QACpD,CAAC;QAED,IAAI,CAAC,KAAK,EAAE,MAAM;YAAE,OAAO,EAAE,OAAO,EAAE,IAAI,EAAE,OAAO,EAAE,KAAK,EAAE,CAAC;QAE7D,MAAM,SAAS,GAAG,IAAI,CAAC,qBAAqB,CAAC,KAAK,CAAC,SAAS,IAAI,CAAC,CAAC,CAAC;QACnE,IAAI,IAAI,CAAC,GAAG,EAAE,GAAG,KAAK,CAAC,WAAW,GAAG,SAAS,EAAE,CAAC;YAC/C,OAAO,EAAE,OAAO,EAAE,KAAK,EAAE,OAAO,EAAE,KAAK,EAAE,CAAC;QAC5C,CAAC;QAED,IAAI,CAAC,KAAK;YAAE,OAAO,EAAE,OAAO,EAAE,IAAI,EAAE,OAAO,EAAE,KAAK,EAAE,CAAC;QAErD,IAAI,IAAI,CAAC,OAAO,CAAC,KAAM,CAAC,UAAU,EAAE,CAAC;YACnC,IAAI,CAAC;gBACH,MAAM,OAAO,GAAG,MAAM,IAAI,CAAC,OAAO,CAAC,KAAM,CAAC,UAAU,CAAC,IAAI,CAAC,QAAQ,CAAC,SAAS,CAAC,EAAE,IAAI,CAAC,eAAe,EAAE,CAAC,CAAC;gBACvG,OAAO,EAAE,OAAO,EAAE,OAAO,EAAE,OAAO,EAAE,OAAO,EAAE,CAAC;YAChD,CAAC;YAAC,OAAO,KAAK,EAAE,CAAC;gBACf,IAAI,CAAC,gBAAgB,CAAC,KAAK,EAAE,SAAS,CAAC,CAAC;gBACxC,IAAI,CAAC,IAAI,CAAC,OAAO,CAAC,oBAAoB;oBAAE,MAAM,IAAI,qBAAqB,CAAC,SAAS,EAAE,KAAK,CAAC,CAAC;gBAC1F,6FAA6F;gBAC7F,OAAO,EAAE,OAAO,EAAE,IAAI,EAAE,OAAO,EAAE,IAAI,EAAE,CAAC;YAC1C,CAAC;QACH,CAAC;QAED,IAAI,CAAC,qBAAqB,CAAC,YAAY,CAAC,CAAC;QACzC,OAAO,EAAE,OAAO,EAAE,IAAI,EAAE,OAAO,EAAE,IAAI,EAAE,CAAC;IAC1C,CAAC;IAEO,KAAK,CAAC,kBAAkB,CAAI,SAAiB,EAAE,EAAoB;QACzE,MAAM,OAAO,GAAG,IAAI,CAAC,UAAU,CAAC,SAAS,CAAC,CAAC;QAC3C,OAAO,CAAC,UAAU,EAAE,CAAC;QAErB,MAAM,IAAI,GAAG,MAAM,IAAI,CAAC,eAAe,CAAC,SAAS,EAAE,IAAI,CAAC,CAAC;QACzD,IAAI,CAAC,IAAI,CAAC,OAAO,EAAE,CAAC;YAClB,OAAO,CAAC,eAAe,EAAE,CAAC;YAC1B,MAAM,IAAI,KAAK,CAAC,0CAA0C,SAAS,EAAE,CAAC,CAAC;QACzE,CAAC;QAED,IAAI,CAAC;YACH,MAAM,MAAM,GAAG,MAAM,IAAI,CAAC,kBAAkB,CAAC,EAAE,CAAC,CAAC;YACjD,OAAO,CAAC,cAAc,EAAE,CAAC;YACzB,IAAI,IAAI,CAAC,OAAO,EAAE,CAAC;gBACjB,MAAM,IAAI,CAAC,gBAAgB,CAAC,SAAS,CAAC,CAAC;gBACvC,MAAM,IAAI,CAAC,YAAY,CAAC,SAAS,CAAC,CAAC;YACrC,CAAC;iBAAM,IAAI,IAAI,CAAC,OAAO,CAAC,QAAQ,KAAK,WAAW,EAAE,CAAC;gBACjD,MAAM,IAAI,CAAC,wBAAwB,CAAC,SAAS,EAAE,IAAI,CAAC,CAAC;YACvD,CAAC;iBAAM,CAAC;gBACN,MAAM,IAAI,CAAC,gBAAgB,CAAC,SAAS,CAAC,CAAC;YACzC,CAAC;YACD,OAAO,MAAM,CAAC;QAChB,CAAC;QAAC,OAAO,KAAK,EAAE,CAAC;YACf,OAAO,CAAC,aAAa,EAAE,CAAC;YACxB,IAAI,IAAI,CAAC,OAAO,EAAE,CAAC;gBACjB,MAAM,IAAI,CAAC,iCAAiC,CAAC,SAAS,CAAC,CAAC;gBACxD,MAAM,IAAI,CAAC,YAAY,CAAC,SAAS,CAAC,CAAC;YACrC,CAAC;iBAAM,IAAI,IAAI,CAAC,OAAO,CAAC,QAAQ,KAAK,WAAW,EAAE,CAAC;gBACjD,MAAM,IAAI,CAAC,wBAAwB,CAAC,SAAS,EAAE,KAAK,CAAC,CAAC;YACxD,CAAC;iBAAM,CAAC;gBACN,MAAM,IAAI,CAAC,gCAAgC,CAAC,SAAS,CAAC,CAAC;YACzD,CAAC;YACD,MAAM,KAAK,CAAC;QACd,CAAC;IACH,CAAC;IAEO,KAAK,CAAC,gCAAgC,CAAC,SAAiB;QAC9D,IAAI,IAAI,CAAC,OAAO,CAAC,KAAM,CAAC,mBAAmB,EAAE,CAAC;YAC5C,IAAI,CAAC;gBACH,MAAM,IAAI,GAAG,MAAM,IAAI,CAAC,OAAO,CAAC,KAAM,CAAC,mBAAmB,CAAC,IAAI,CAAC,GAAG,CAAC,SAAS,CAAC,EAAE;oBAC9E,UAAU,EAAE,IAAI,CAAC,eAAe,CAAC,CAAC,CAAC;oBACnC,gBAAgB,EAAE,IAAI,CAAC,OAAO,CAAC,gBAAgB;oBAC/C,GAAG,EAAE,IAAI,CAAC,GAAG,EAAE;iBAChB,CAAC,CAAC;gBACH,IAAI,CAAC,OAAO,CAAC,MAAM,CAAC,IAAI,CAAC,mCAAmC,SAAS,qBAAqB,IAAI,CAAC,QAAQ,EAAE,CAAC,CAAC;gBAC3G,IAAI,IAAI,CAAC,MAAM,IAAI,IAAI,CAAC,QAAQ,KAAK,IAAI,CAAC,OAAO,CAAC,gBAAgB,EAAE,CAAC;oBACnE,IAAI,CAAC,OAAO,CAAC,MAAM,CAAC,IAAI,CAAC,yCAAyC,SAAS,EAAE,CAAC,CAAC;oBAC/E,IAAI,CAAC,OAAO,CAAC,aAAa,EAAE,CAAC,MAAM,EAAE,SAAS,CAAC,CAAC;gBAClD,CAAC;gBACD,OAAO;YACT,CAAC;YAAC,OAAO,KAAK,EAAE,CAAC;gBACf,IAAI,CAAC,gBAAgB,CAAC,KAAK,EAAE,SAAS,CAAC,CAAC;gBACxC,4CAA4C;YAC9C,CAAC;QACH,CAAC;aAAM,CAAC;YACN,IAAI,CAAC,qBAAqB,CAAC,qBAAqB,CAAC,CAAC;QACpD,CAAC;QAED,MAAM,OAAO,GAAG,CAAC,MAAM,IAAI,CAAC,OAAO,CAAC,SAAS,CAAC,CAAC,IAAI,EAAE,GAAG,oBAAY,EAAE,CAAC;QACvE,MAAM,IAAI,GAAwB;YAChC,QAAQ,EAAE,OAAO,CAAC,QAAQ,GAAG,CAAC;YAC9B,WAAW,EAAE,IAAI,CAAC,GAAG,EAAE;YACvB,MAAM,EAAE,OAAO,CAAC,QAAQ,GAAG,CAAC,IAAI,IAAI,CAAC,OAAO,CAAC,gBAAgB;YAC7D,SAAS,EAAE,CAAC;YACZ,SAAS,EAAE,OAAO,CAAC,SAAS,IAAI,CAAC;YACjC,WAAW,EAAE,OAAO,CAAC,WAAW,IAAI,CAAC;SACtC,CAAC;QACF,MAAM,IAAI,CAAC,OAAO,CAAC,SAAS,EAAE,IAAI,EAAE,IAAI,CAAC,eAAe,CAAC,CAAC,CAAC,CAAC,CAAC;QAC7D,IAAI,CAAC,OAAO,CAAC,MAAM,CAAC,IAAI,CAAC,mCAAmC,SAAS,qBAAqB,IAAI,CAAC,QAAQ,EAAE,CAAC,CAAC;QAE3G,IAAI,IAAI,CAAC,MAAM,IAAI,CAAC,OAAO,CAAC,MAAM,EAAE,CAAC;YACnC,IAAI,CAAC,OAAO,CAAC,MAAM,CAAC,IAAI,CAAC,yCAAyC,SAAS,EAAE,CAAC,CAAC;YAC/E,IAAI,CAAC,OAAO,CAAC,aAAa,EAAE,CAAC,MAAM,EAAE,SAAS,CAAC,CAAC;QAClD,CAAC;IACH,CAAC;IAED;;;;OAIG;IACK,KAAK,CAAC,wBAAwB,CAAC,SAAiB,EAAE,OAAgB;QACxE,IAAI,IAAI,CAAC,OAAO,CAAC,KAAM,CAAC,mBAAmB,EAAE,CAAC;YAC5C,IAAI,CAAC;gBACH,MAAM,IAAI,GAAG,MAAM,IAAI,CAAC,OAAO,CAAC,KAAM,CAAC,mBAAmB,CAAC,IAAI,CAAC,GAAG,CAAC,SAAS,CAAC,EAAE;oBAC9E,OAAO;oBACP,KAAK,EAAE,IAAI,CAAC,OAAO,CAAC,cAAc;oBAClC,YAAY,EAAE,IAAI,CAAC,OAAO,CAAC,YAAY;oBACvC,kBAAkB,EAAE,IAAI,CAAC,OAAO,CAAC,kBAAkB;oBACnD,UAAU,EAAE,IAAI,CAAC,eAAe,CAAC,CAAC,CAAC;oBACnC,GAAG,EAAE,IAAI,CAAC,GAAG,EAAE;iBAChB,CAAC,CAAC;gBACH,IAAI,IAAI,CAAC,SAAS,EAAE,CAAC;oBACnB,IAAI,CAAC,OAAO,CAAC,MAAM,CAAC,IAAI,CACtB,yCAAyC,SAAS,gBAAgB,CAAC,IAAI,CAAC,SAAS,GAAG,GAAG,CAAC,CAAC,OAAO,CAAC,CAAC,CAAC,UAAU,IAAI,CAAC,WAAW,SAAS,CACvI,CAAC;oBACF,IAAI,CAAC,OAAO,CAAC,aAAa,EAAE,CAAC,MAAM,EAAE,SAAS,CAAC,CAAC;gBAClD,CAAC;gBACD,OAAO;YACT,CAAC;YAAC,OAAO,KAAK,EAAE,CAAC;gBACf,IAAI,CAAC,gBAAgB,CAAC,KAAK,EAAE,SAAS,CAAC,CAAC;gBACxC,4CAA4C;YAC9C,CAAC;QACH,CAAC;aAAM,CAAC;YACN,IAAI,CAAC,qBAAqB,CAAC,qBAAqB,CAAC,CAAC;QACpD,CAAC;QAED,MAAM,OAAO,GAAG,CAAC,MAAM,IAAI,CAAC,OAAO,CAAC,SAAS,CAAC,CAAC,IAAI,EAAE,GAAG,oBAAY,EAAE,CAAC;QACvE,MAAM,KAAK,GAAG,IAAI,CAAC,OAAO,CAAC,cAAc,CAAC;QAC1C,MAAM,SAAS,GAAG,CAAC,OAAO,CAAC,SAAS,IAAI,CAAC,CAAC,GAAG,KAAK,GAAG,CAAC,OAAO,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,GAAG,CAAC,CAAC,GAAG,KAAK,CAAC,CAAC;QACrF,MAAM,WAAW,GAAG,CAAC,OAAO,CAAC,WAAW,IAAI,CAAC,CAAC,GAAG,CAAC,CAAC;QACnD,MAAM,MAAM,GAAG,WAAW,IAAI,IAAI,CAAC,OAAO,CAAC,YAAY,IAAI,SAAS,IAAI,IAAI,CAAC,OAAO,CAAC,kBAAkB,CAAC;QAExG,MAAM,IAAI,GAAwB;YAChC,QAAQ,EAAE,OAAO,CAAC,QAAQ;YAC1B,WAAW,EAAE,IAAI,CAAC,GAAG,EAAE;YACvB,MAAM;YACN,SAAS,EAAE,OAAO,CAAC,SAAS,IAAI,CAAC;YACjC,SAAS;YACT,WAAW;SACZ,CAAC;QACF,MAAM,IAAI,CAAC,OAAO,CAAC,SAAS,EAAE,IAAI,EAAE,IAAI,CAAC,eAAe,CAAC,CAAC,CAAC,CAAC,CAAC;QAE7D,IAAI,MAAM,IAAI,CAAC,OAAO,CAAC,MAAM,EAAE,CAAC;YAC9B,IAAI,CAAC,OAAO,CAAC,MAAM,CAAC,IAAI,CACtB,yCAAyC,SAAS,gBAAgB,CAAC,SAAS,GAAG,GAAG,CAAC,CAAC,OAAO,CAAC,CAAC,CAAC,UAAU,WAAW,SAAS,CAC7H,CAAC;YACF,IAAI,CAAC,OAAO,CAAC,aAAa,EAAE,CAAC,MAAM,EAAE,SAAS,CAAC,CAAC;QAClD,CAAC;IACH,CAAC;IAEO,KAAK,CAAC,iCAAiC,CAAC,SAAiB;QAC/D,MAAM,OAAO,GAAG,CAAC,MAAM,IAAI,CAAC,OAAO,CAAC,SAAS,CAAC,CAAC,IAAI,EAAE,GAAG,oBAAY,EAAE,MAAM,EAAE,IAAI,EAAE,CAAC;QACrF,MAAM,SAAS,GAAG,CAAC,OAAO,CAAC,SAAS,IAAI,CAAC,CAAC,GAAG,CAAC,CAAC;QAC/C,MAAM,IAAI,GAAwB;YAChC,QAAQ,EAAE,IAAI,CAAC,GAAG,CAAC,OAAO,CAAC,QAAQ,EAAE,IAAI,CAAC,OAAO,CAAC,gBAAgB,CAAC;YACnE,WAAW,EAAE,IAAI,CAAC,GAAG,EAAE;YACvB,MAAM,EAAE,IAAI;YACZ,SAAS;YACT,SAAS,EAAE,OAAO,CAAC,SAAS,IAAI,CAAC;YACjC,WAAW,EAAE,OAAO,CAAC,WAAW,IAAI,CAAC;SACtC,CAAC;QACF,MAAM,IAAI,CAAC,OAAO,CAAC,SAAS,EAAE,IAAI,EAAE,IAAI,CAAC,eAAe,CAAC,SAAS,CAAC,CAAC,CAAC;QACrE,IAAI,CAAC,OAAO,CAAC,MAAM,CAAC,IAAI,CAAC,yCAAyC,SAAS,0BAA0B,SAAS,GAAG,CAAC,GAAG,CAAC,CAAC;IACzH,CAAC;IAED,0EAA0E;IAElE,KAAK,CAAC,YAAY,CAAI,SAAiB,EAAE,EAAoB;QACnE,MAAM,OAAO,GAAG,IAAI,CAAC,UAAU,CAAC,SAAS,CAAC,CAAC;QAC3C,OAAO,CAAC,UAAU,EAAE,CAAC;QAErB,MAAM,IAAI,GAAG,IAAI,CAAC,cAAc,CAAC,SAAS,CAAC,CAAC;QAC5C,IAAI,CAAC,IAAI,CAAC,OAAO,EAAE,CAAC;YAClB,OAAO,CAAC,eAAe,EAAE,CAAC;YAC1B,MAAM,IAAI,KAAK,CAAC,0CAA0C,SAAS,EAAE,CAAC,CAAC;QACzE,CAAC;QAED,IAAI,CAAC;YACH,MAAM,MAAM,GAAG,MAAM,IAAI,CAAC,kBAAkB,CAAC,EAAE,CAAC,CAAC;YACjD,OAAO,CAAC,cAAc,EAAE,CAAC;YACzB,IAAI,IAAI,CAAC,OAAO,EAAE,CAAC;gBACjB,IAAI,CAAC,UAAU,CAAC,SAAS,CAAC,CAAC;YAC7B,CAAC;iBAAM,IAAI,IAAI,CAAC,OAAO,CAAC,QAAQ,KAAK,WAAW,EAAE,CAAC;gBACjD,IAAI,CAAC,sBAAsB,CAAC,SAAS,EAAE,IAAI,CAAC,CAAC;YAC/C,CAAC;iBAAM,CAAC;gBACN,IAAI,CAAC,UAAU,CAAC,SAAS,CAAC,CAAC;YAC7B,CAAC;YACD,OAAO,MAAM,CAAC;QAChB,CAAC;QAAC,OAAO,KAAK,EAAE,CAAC;YACf,OAAO,CAAC,aAAa,EAAE,CAAC;YACxB,IAAI,IAAI,CAAC,OAAO,EAAE,CAAC;gBACjB,IAAI,CAAC,2BAA2B,CAAC,SAAS,CAAC,CAAC;YAC9C,CAAC;iBAAM,IAAI,IAAI,CAAC,OAAO,CAAC,QAAQ,KAAK,WAAW,EAAE,CAAC;gBACjD,IAAI,CAAC,sBAAsB,CAAC,SAAS,EAAE,KAAK,CAAC,CAAC;YAChD,CAAC;iBAAM,CAAC;gBACN,IAAI,CAAC,0BAA0B,CAAC,SAAS,CAAC,CAAC;YAC7C,CAAC;YACD,MAAM,KAAK,CAAC;QACd,CAAC;IACH,CAAC;IAED,gFAAgF;IACxE,aAAa,CAAC,SAAiB;QACrC,IAAI,CAAC,IAAI,CAAC,cAAc,CAAC,GAAG,CAAC,SAAS,CAAC;YAAE,OAAO,IAAI,CAAC;QAErD,MAAM,SAAS,GAAG,IAAI,CAAC,UAAU,CAAC,GAAG,CAAC,SAAS,CAAC,IAAI,CAAC,CAAC;QACtD,MAAM,WAAW,GAAG,IAAI,CAAC,eAAe,CAAC,GAAG,CAAC,SAAS,CAAC,IAAI,CAAC,CAAC;QAC7D,MAAM,SAAS,GAAG,IAAI,CAAC,qBAAqB,CAAC,SAAS,CAAC,CAAC;QAExD,IAAI,IAAI,CAAC,GAAG,EAAE,GAAG,WAAW,GAAG,SAAS,EAAE,CAAC;YACzC,IAAI,CAAC,OAAO,CAAC,MAAM,CAAC,IAAI,CAAC,uCAAuC,SAAS,EAAE,CAAC,CAAC;YAC7E,OAAO,KAAK,CAAC;QACf,CAAC;QACD,OAAO,IAAI,CAAC;IACd,CAAC;IAED,oFAAoF;IAC5E,cAAc,CAAC,SAAiB;QACtC,IAAI,CAAC,IAAI,CAAC,cAAc,CAAC,GAAG,CAAC,SAAS,CAAC;YAAE,OAAO,EAAE,OAAO,EAAE,IAAI,EAAE,OAAO,EAAE,KAAK,EAAE,CAAC;QAElF,MAAM,SAAS,GAAG,IAAI,CAAC,UAAU,CAAC,GAAG,CAAC,SAAS,CAAC,IAAI,CAAC,CAAC;QACtD,MAAM,WAAW,GAAG,IAAI,CAAC,eAAe,CAAC,GAAG,CAAC,SAAS,CAAC,IAAI,CAAC,CAAC;QAC7D,MAAM,SAAS,GAAG,IAAI,CAAC,qBAAqB,CAAC,SAAS,CAAC,CAAC;QAExD,IAAI,IAAI,CAAC,GAAG,EAAE,GAAG,WAAW,GAAG,SAAS,EAAE,CAAC;YACzC,IAAI,CAAC,OAAO,CAAC,MAAM,CAAC,IAAI,CAAC,uCAAuC,SAAS,EAAE,CAAC,CAAC;YAC7E,OAAO,EAAE,OAAO,EAAE,KAAK,EAAE,OAAO,EAAE,KAAK,EAAE,CAAC;QAC5C,CAAC;QAED,IAAI,IAAI,CAAC,eAAe,CAAC,GAAG,CAAC,SAAS,CAAC,EAAE,CAAC;YACxC,oFAAoF;YACpF,OAAO,EAAE,OAAO,EAAE,KAAK,EAAE,OAAO,EAAE,KAAK,EAAE,CAAC;QAC5C,CAAC;QAED,IAAI,CAAC,eAAe,CAAC,GAAG,CAAC,SAAS,CAAC,CAAC;QACpC,IAAI,CAAC,cAAc,CAAC,GAAG,CAAC,SAAS,EAAE,IAAI,CAAC,GAAG,EAAE,CAAC,CAAC;QAC/C,OAAO,EAAE,OAAO,EAAE,IAAI,EAAE,OAAO,EAAE,IAAI,EAAE,CAAC;IAC1C,CAAC;IAEO,0BAA0B,CAAC,SAAiB;QAClD,MAAM,QAAQ,GAAG,CAAC,IAAI,CAAC,QAAQ,CAAC,GAAG,CAAC,SAAS,CAAC,IAAI,CAAC,CAAC,GAAG,CAAC,CAAC;QACzD,IAAI,CAAC,QAAQ,CAAC,GAAG,CAAC,SAAS,EAAE,QAAQ,CAAC,CAAC;QACvC,IAAI,CAAC,eAAe,CAAC,GAAG,CAAC,SAAS,EAAE,IAAI,CAAC,GAAG,EAAE,CAAC,CAAC;QAChD,IAAI,CAAC,OAAO,CAAC,MAAM,CAAC,IAAI,CAAC,mCAAmC,SAAS,qBAAqB,QAAQ,EAAE,CAAC,CAAC;QAEtG,IAAI,QAAQ,IAAI,IAAI,CAAC,OAAO,CAAC,gBAAgB,IAAI,CAAC,IAAI,CAAC,cAAc,CAAC,GAAG,CAAC,SAAS,CAAC,EAAE,CAAC;YACrF,IAAI,CAAC,cAAc,CAAC,GAAG,CAAC,SAAS,CAAC,CAAC;YACnC,IAAI,CAAC,UAAU,CAAC,GAAG,CAAC,SAAS,EAAE,CAAC,CAAC,CAAC;YAClC,IAAI,CAAC,OAAO,CAAC,MAAM,CAAC,IAAI,CAAC,yCAAyC,SAAS,EAAE,CAAC,CAAC;YAC/E,IAAI,CAAC,OAAO,CAAC,aAAa,EAAE,CAAC,MAAM,EAAE,SAAS,CAAC,CAAC;QAClD,CAAC;IACH,CAAC;IAED,kIAAkI;IAC1H,sBAAsB,CAAC,SAAiB,EAAE,OAAgB;QAChE,MAAM,KAAK,GAAG,IAAI,CAAC,OAAO,CAAC,cAAc,CAAC;QAC1C,MAAM,IAAI,GAAG,CAAC,IAAI,CAAC,UAAU,CAAC,GAAG,CAAC,SAAS,CAAC,IAAI,CAAC,CAAC,GAAG,KAAK,GAAG,CAAC,OAAO,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,GAAG,CAAC,CAAC,GAAG,KAAK,CAAC,CAAC;QAC7F,MAAM,KAAK,GAAG,CAAC,IAAI,CAAC,YAAY,CAAC,GAAG,CAAC,SAAS,CAAC,IAAI,CAAC,CAAC,GAAG,CAAC,CAAC;QAC1D,IAAI,CAAC,UAAU,CAAC,GAAG,CAAC,SAAS,EAAE,IAAI,CAAC,CAAC;QACrC,IAAI,CAAC,YAAY,CAAC,GAAG,CAAC,SAAS,EAAE,KAAK,CAAC,CAAC;QAExC,IAAI,KAAK,IAAI,IAAI,CAAC,OAAO,CAAC,YAAY,IAAI,IAAI,IAAI,IAAI,CAAC,OAAO,CAAC,kBAAkB,IAAI,CAAC,IAAI,CAAC,cAAc,CAAC,GAAG,CAAC,SAAS,CAAC,EAAE,CAAC;YACzH,IAAI,CAAC,cAAc,CAAC,GAAG,CAAC,SAAS,CAAC,CAAC;YACnC,IAAI,CAAC,UAAU,CAAC,GAAG,CAAC,SAAS,EAAE,CAAC,CAAC,CAAC;YAClC,IAAI,CAAC,eAAe,CAAC,GAAG,CAAC,SAAS,EAAE,IAAI,CAAC,GAAG,EAAE,CAAC,CAAC;YAChD,IAAI,CAAC,OAAO,CAAC,MAAM,CAAC,IAAI,CACtB,yCAAyC,SAAS,gBAAgB,CAAC,IAAI,GAAG,GAAG,CAAC,CAAC,OAAO,CAAC,CAAC,CAAC,UAAU,KAAK,SAAS,CAClH,CAAC;YACF,IAAI,CAAC,OAAO,CAAC,aAAa,EAAE,CAAC,MAAM,EAAE,SAAS,CAAC,CAAC;QAClD,CAAC;IACH,CAAC;IAEO,2BAA2B,CAAC,SAAiB;QACnD,IAAI,CAAC,eAAe,CAAC,MAAM,CAAC,SAAS,CAAC,CAAC;QACvC,IAAI,CAAC,cAAc,CAAC,MAAM,CAAC,SAAS,CAAC,CAAC;QACtC,MAAM,SAAS,GAAG,CAAC,IAAI,CAAC,UAAU,CAAC,GAAG,CAAC,SAAS,CAAC,IAAI,CAAC,CAAC,GAAG,CAAC,CAAC;QAC5D,IAAI,CAAC,UAAU,CAAC,GAAG,CAAC,SAAS,EAAE,SAAS,CAAC,CAAC;QAC1C,IAAI,CAAC,eAAe,CAAC,GAAG,CAAC,SAAS,EAAE,IAAI,CAAC,GAAG,EAAE,CAAC,CAAC;QAChD,IAAI,CAAC,OAAO,CAAC,MAAM,CAAC,IAAI,CAAC,yCAAyC,SAAS,0BAA0B,SAAS,GAAG,CAAC,GAAG,CAAC,CAAC;IACzH,CAAC;IAEO,UAAU,CAAC,SAAiB;QAClC,MAAM,OAAO,GAAG,IAAI,CAAC,cAAc,CAAC,GAAG,CAAC,SAAS,CAAC,CAAC;QACnD,IAAI,CAAC,QAAQ,CAAC,MAAM,CAAC,SAAS,CAAC,CAAC;QAChC,IAAI,CAAC,eAAe,CAAC,MAAM,CAAC,SAAS,CAAC,CAAC;QACvC,IAAI,CAAC,cAAc,CAAC,MAAM,CAAC,SAAS,CAAC,CAAC;QACtC,IAAI,CAAC,UAAU,CAAC,MAAM,CAAC,SAAS,CAAC,CAAC;QAClC,IAAI,CAAC,eAAe,CAAC,MAAM,CAAC,SAAS,CAAC,CAAC;QACvC,IAAI,CAAC,cAAc,CAAC,MAAM,CAAC,SAAS,CAAC,CAAC;QACtC,IAAI,CAAC,UAAU,CAAC,MAAM,CAAC,SAAS,CAAC,CAAC;QAClC,IAAI,CAAC,YAAY,CAAC,MAAM,CAAC,SAAS,CAAC,CAAC;QACpC,IAAI,OAAO,EAAE,CAAC;YACZ,IAAI,CAAC,OAAO,CAAC,aAAa,EAAE,CAAC,QAAQ,EAAE,SAAS,CAAC,CAAC;QACpD,CAAC;QACD,IAAI,CAAC,OAAO,CAAC,MAAM,CAAC,GAAG,CAAC,wCAAwC,SAAS,EAAE,CAAC,CAAC;IAC/E,CAAC;IAED;;;;;;OAMG;IACK,OAAO;QACb,MAAM,GAAG,GAAG,IAAI,CAAC,GAAG,EAAE,CAAC;QACvB,MAAM,UAAU,GAAG,IAAI,CAAC,GAAG,CAAC,IAAI,CAAC,OAAO,CAAC,YAAY,EAAE,IAAI,CAAC,GAAG,CAAC,CAAC;QACjE,KAAK,MAAM,CAAC,SAAS,EAAE,SAAS,CAAC,IAAI,IAAI,CAAC,cAAc,CAAC,OAAO,EAAE,EAAE,CAAC;YACnE,IAAI,GAAG,GAAG,SAAS,GAAG,UAAU,EAAE,CAAC;gBACjC,IAAI,CAAC,eAAe,CAAC,MAAM,CAAC,SAAS,CAAC,CAAC;gBACvC,IAAI,CAAC,cAAc,CAAC,MAAM,CAAC,SAAS,CAAC,CAAC;gBACtC,IAAI,CAAC,OAAO,CAAC,MAAM,CAAC,IAAI,CACtB,wCAAwC,SAAS,0EAA0E,UAAU,qCAAqC,CAC3K,CAAC;YACJ,CAAC;QACH,CAAC;IACH,CAAC;IAED,4EAA4E;IAEpE,kBAAkB,CAAI,EAAoB;QAChD,IAAI,CAAC,IAAI,CAAC,OAAO,CAAC,OAAO;YAAE,OAAO,EAAE,EAAE,CAAC;QAEvC,OAAO,IAAI,OAAO,CAAI,CAAC,OAAO,EAAE,MAAM,EAAE,EAAE;YACxC,MAAM,KAAK,GAAG,UAAU,CAAC,GAAG,EAAE,CAAC,MAAM,CAAC,IAAI,KAAK,CAAC,6BAA6B,IAAI,CAAC,OAAO,CAAC,OAAO,IAAI,CAAC,CAAC,EAAE,IAAI,CAAC,OAAO,CAAC,OAAO,CAAC,CAAC;YAC/H,EAAE,EAAE,CAAC,IAAI,CACP,CAAC,KAAK,EAAE,EAAE;gBACR,YAAY,CAAC,KAAK,CAAC,CAAC;gBACpB,OAAO,CAAC,KAAK,CAAC,CAAC;YACjB,CAAC,EACD,CAAC,KAAK,EAAE,EAAE;gBACR,YAAY,CAAC,KAAK,CAAC,CAAC;gBACpB,MAAM,CAAC,KAAK,CAAC,CAAC;YAChB,CAAC,CACF,CAAC;QACJ,CAAC,CAAC,CAAC;IACL,CAAC;CACF;AA3lBD,wCA2lBC"}
|
|
1
|
+
{"version":3,"file":"circuit-breaker.js","sourceRoot":"","sources":["../src/circuit-breaker.ts"],"names":[],"mappings":";;;AACA,mCAA0F;AAW1F;;;;;GAKG;AACH,MAAa,qBAAsB,SAAQ,KAAK;IAG9C,YAAY,SAAiB,EAAE,KAAc;QAC3C,KAAK,CAAC,oDAAoD,SAAS,EAAE,CAAC,CAAC;QACvE,IAAI,CAAC,IAAI,GAAG,uBAAuB,CAAC;QACpC,IAAI,CAAC,KAAK,GAAG,KAAK,CAAC;IACrB,CAAC;CACF;AARD,sDAQC;AA2FD,MAAM,UAAU,GAAkB,EAAE,IAAI,EAAE,GAAG,EAAE,GAAE,CAAC,EAAE,GAAG,EAAE,GAAG,EAAE,GAAE,CAAC,EAAE,CAAC;AAkBpE;;;;;;;;;;;;;;GAcG;AACH,MAAa,cAAc;IAkCzB,YAAY,UAAiC,EAAE;QAjB9B,uBAAkB,GAAG,IAAI,GAAG,EAAU,CAAC;QAChD,sCAAiC,GAAG,KAAK,CAAC;QACjC,YAAO,GAAG,IAAI,GAAG,EAA0B,CAAC;QAC5C,wBAAmB,GAAG,IAAI,GAAG,EAAqE,CAAC;QAEpH,mBAAmB;QACF,aAAQ,GAAG,IAAI,GAAG,EAAkB,CAAC;QACrC,oBAAe,GAAG,IAAI,GAAG,EAAkB,CAAC;QAC5C,mBAAc,GAAG,IAAI,GAAG,EAAU,CAAC;QACnC,eAAU,GAAG,IAAI,GAAG,EAAkB,CAAC;QACvC,oBAAe,GAAG,IAAI,GAAG,EAAU,CAAC;QACpC,mBAAc,GAAG,IAAI,GAAG,EAAkB,CAAC;QAC3C,eAAU,GAAG,IAAI,GAAG,EAAkB,CAAC;QACvC,iBAAY,GAAG,IAAI,GAAG,EAAkB,CAAC;QAKxD,MAAM,YAAY,GAAG,OAAO,CAAC,YAAY,IAAI,KAAK,CAAC;QACnD,IAAI,CAAC,OAAO,GAAG;YACb,QAAQ,EAAE,OAAO,CAAC,QAAQ,IAAI,aAAa;YAC3C,gBAAgB,EAAE,OAAO,CAAC,gBAAgB,IAAI,CAAC;YAC/C,kBAAkB,EAAE,OAAO,CAAC,kBAAkB,IAAI,GAAG;YACrD,YAAY,EAAE,OAAO,CAAC,YAAY,IAAI,EAAE;YACxC,cAAc,EAAE,OAAO,CAAC,cAAc,IAAI,GAAG;YAC7C,YAAY;YACZ,eAAe,EAAE,OAAO,CAAC,eAAe,IAAI,IAAI;YAChD,OAAO,EAAE,OAAO,CAAC,OAAO;YACxB,YAAY,EAAE,OAAO,CAAC,YAAY,IAAI,IAAI,CAAC,GAAG,CAAC,OAAO,CAAC,OAAO,IAAI,KAAK,EAAE,YAAY,CAAC;YACtF,iBAAiB,EAAE,OAAO,CAAC,iBAAiB,IAAI,CAAC;YACjD,eAAe,EAAE,OAAO,CAAC,eAAe,IAAI,YAAY,GAAG,CAAC;YAC5D,MAAM,EAAE,OAAO,CAAC,MAAM,IAAI,GAAG;YAC7B,aAAa,EAAE,OAAO,CAAC,aAAa;YACpC,KAAK,EAAE,OAAO,CAAC,KAAK;YACpB,oBAAoB,EAAE,OAAO,CAAC,oBAAoB,IAAI,IAAI;YAC1D,YAAY,EAAE,OAAO,CAAC,YAAY;YAClC,MAAM,EAAE,OAAO,CAAC,MAAM,IAAI,UAAU;SACrC,CAAC;QAEF,IAAI,CAAC,WAAW,GAAG,CAAC,CAAC,OAAO,CAAC,KAAK,CAAC;QAEnC,IAAI,CAAC,aAAa,GAAG,WAAW,CAAC,GAAG,EAAE,CAAC,IAAI,CAAC,OAAO,EAAE,EAAE,IAAI,CAAC,OAAO,CAAC,eAAe,CAAC,CAAC;QACrF,IAAI,CAAC,aAAa,CAAC,KAAK,EAAE,EAAE,CAAC;IAC/B,CAAC;IAED,6FAA6F;IAC7F,OAAO;QACL,aAAa,CAAC,IAAI,CAAC,aAAa,CAAC,CAAC;IACpC,CAAC;IAED,KAAK,CAAC,OAAO,CAAI,SAAiB,EAAE,EAAoB;QACtD,OAAO,IAAI,CAAC,WAAW,CAAC,CAAC,CAAC,IAAI,CAAC,kBAAkB,CAAC,SAAS,EAAE,EAAE,CAAC,CAAC,CAAC,CAAC,IAAI,CAAC,YAAY,CAAC,SAAS,EAAE,EAAE,CAAC,CAAC;IACtG,CAAC;IAED;;;;OAIG;IACH,UAAU,CAAC,SAAiB;QAC1B,IAAI,IAAI,CAAC,WAAW,EAAE,CAAC;YACrB,IAAI,CAAC,IAAI,CAAC,iCAAiC,EAAE,CAAC;gBAC5C,IAAI,CAAC,iCAAiC,GAAG,IAAI,CAAC;gBAC9C,IAAI,CAAC,OAAO,CAAC,MAAM,CAAC,IAAI,CACtB,0JAA0J,CAC3J,CAAC;YACJ,CAAC;YACD,OAAO,IAAI,CAAC;QACd,CAAC;QACD,OAAO,IAAI,CAAC,aAAa,CAAC,SAAS,CAAC,CAAC;IACvC,CAAC;IAED;;;;;OAKG;IACH,KAAK,CAAC,eAAe,CAAC,SAAiB;QACrC,IAAI,CAAC,IAAI,CAAC,WAAW;YAAE,OAAO,IAAI,CAAC,aAAa,CAAC,SAAS,CAAC,CAAC;QAC5D,MAAM,IAAI,GAAG,MAAM,IAAI,CAAC,eAAe,CAAC,SAAS,EAAE,KAAK,CAAC,CAAC;QAC1D,OAAO,IAAI,CAAC,OAAO,CAAC;IACtB,CAAC;IAED,aAAa,CAAC,SAAiB;QAC7B,IAAI,IAAI,CAAC,WAAW,EAAE,CAAC;YACrB,IAAI,IAAI,CAAC,OAAO,CAAC,QAAQ,KAAK,WAAW,EAAE,CAAC;gBAC1C,KAAK,IAAI,CAAC,wBAAwB,CAAC,SAAS,EAAE,KAAK,CAAC,CAAC;YACvD,CAAC;iBAAM,CAAC;gBACN,KAAK,IAAI,CAAC,gCAAgC,CAAC,SAAS,CAAC,CAAC;YACxD,CAAC;QACH,CAAC;aAAM,IAAI,IAAI,CAAC,OAAO,CAAC,QAAQ,KAAK,WAAW,EAAE,CAAC;YACjD,IAAI,CAAC,sBAAsB,CAAC,SAAS,EAAE,KAAK,CAAC,CAAC;QAChD,CAAC;aAAM,CAAC;YACN,IAAI,CAAC,0BAA0B,CAAC,SAAS,CAAC,CAAC;QAC7C,CAAC;IACH,CAAC;IAED,aAAa,CAAC,SAAiB;QAC7B,IAAI,IAAI,CAAC,WAAW,EAAE,CAAC;YACrB,KAAK,IAAI,CAAC,yBAAyB,CAAC,SAAS,CAAC,CAAC;QACjD,CAAC;aAAM,CAAC;YACN,IAAI,CAAC,UAAU,CAAC,SAAS,CAAC,CAAC;QAC7B,CAAC;IACH,CAAC;IAED;;;;;;;;;OASG;IACK,KAAK,CAAC,yBAAyB,CAAC,SAAiB;QACvD,MAAM,KAAK,GAAG,MAAM,IAAI,CAAC,OAAO,CAAC,SAAS,CAAC,CAAC;QAC5C,MAAM,IAAI,CAAC,gBAAgB,CAAC,SAAS,CAAC,CAAC;QACvC,IAAI,KAAK,EAAE,MAAM,EAAE,CAAC;YAClB,IAAI,CAAC,OAAO,CAAC,aAAa,EAAE,CAAC,QAAQ,EAAE,SAAS,CAAC,CAAC;QACpD,CAAC;IACH,CAAC;IAED,KAAK,CAAC,QAAQ,CAAC,SAAiB;QAC9B,IAAI,IAAI,CAAC,WAAW,EAAE,CAAC;YACrB,OAAO,CAAC,MAAM,IAAI,CAAC,OAAO,CAAC,SAAS,CAAC,CAAC,IAAI,EAAE,GAAG,oBAAY,EAAE,CAAC;QAChE,CAAC;QACD,MAAM,QAAQ,GAAG,IAAI,CAAC,QAAQ,CAAC,GAAG,CAAC,SAAS,CAAC,IAAI,CAAC,CAAC;QACnD,MAAM,WAAW,GAAG,IAAI,CAAC,eAAe,CAAC,GAAG,CAAC,SAAS,CAAC,IAAI,CAAC,CAAC;QAC7D,MAAM,SAAS,GAAG,IAAI,CAAC,UAAU,CAAC,GAAG,CAAC,SAAS,CAAC,IAAI,CAAC,CAAC;QACtD,MAAM,SAAS,GAAG,IAAI,CAAC,UAAU,CAAC,GAAG,CAAC,SAAS,CAAC,IAAI,CAAC,CAAC;QACtD,MAAM,WAAW,GAAG,IAAI,CAAC,YAAY,CAAC,GAAG,CAAC,SAAS,CAAC,IAAI,CAAC,CAAC;QAC1D,OAAO,EAAE,QAAQ,EAAE,WAAW,EAAE,MAAM,EAAE,IAAI,CAAC,cAAc,CAAC,GAAG,CAAC,SAAS,CAAC,EAAE,SAAS,EAAE,SAAS,EAAE,WAAW,EAAE,CAAC;IAClH,CAAC;IAED,KAAK,CAAC,KAAK,CAAC,SAAiB;QAC3B,IAAI,IAAI,CAAC,WAAW,EAAE,CAAC;YACrB,MAAM,KAAK,GAAG,MAAM,IAAI,CAAC,OAAO,CAAC,SAAS,CAAC,CAAC;YAC5C,MAAM,IAAI,CAAC,WAAW,CAAC,SAAS,CAAC,CAAC;YAClC,MAAM,IAAI,CAAC,YAAY,CAAC,SAAS,CAAC,CAAC;YACnC,IAAI,CAAC,mBAAmB,CAAC,MAAM,CAAC,SAAS,CAAC,CAAC;YAC3C,IAAI,KAAK,EAAE,MAAM,EAAE,CAAC;gBAClB,IAAI,CAAC,OAAO,CAAC,aAAa,EAAE,CAAC,QAAQ,EAAE,SAAS,CAAC,CAAC;YACpD,CAAC;YACD,IAAI,CAAC,OAAO,CAAC,MAAM,CAAC,GAAG,CAAC,wCAAwC,SAAS,EAAE,CAAC,CAAC;QAC/E,CAAC;aAAM,CAAC;YACN,IAAI,CAAC,UAAU,CAAC,SAAS,CAAC,CAAC;QAC7B,CAAC;IACH,CAAC;IAED,wPAAwP;IACxP,UAAU,CAAC,SAAiB;QAC1B,MAAM,CAAC,GAAG,IAAI,CAAC,OAAO,CAAC,GAAG,CAAC,SAAS,CAAC,CAAC;QACtC,OAAO,CAAC,CAAC,CAAC,CAAC,EAAE,GAAG,CAAC,EAAE,CAAC,CAAC,CAAC,IAAA,oBAAY,GAAE,CAAC;IACvC,CAAC;IAED,aAAa;QACX,MAAM,MAAM,GAAmC,EAAE,CAAC;QAClD,KAAK,MAAM,CAAC,SAAS,EAAE,CAAC,CAAC,IAAI,IAAI,CAAC,OAAO,CAAC,OAAO,EAAE;YAAE,MAAM,CAAC,SAAS,CAAC,GAAG,EAAE,GAAG,CAAC,EAAE,CAAC;QAClF,OAAO,MAAM,CAAC;IAChB,CAAC;IAEO,UAAU,CAAC,SAAiB;QAClC,IAAI,CAAC,GAAG,IAAI,CAAC,OAAO,CAAC,GAAG,CAAC,SAAS,CAAC,CAAC;QACpC,IAAI,CAAC,CAAC,EAAE,CAAC;YACP,CAAC,GAAG,IAAA,oBAAY,GAAE,CAAC;YACnB,IAAI,CAAC,OAAO,CAAC,GAAG,CAAC,SAAS,EAAE,CAAC,CAAC,CAAC;QACjC,CAAC;QACD,OAAO,CAAC,CAAC;IACX,CAAC;IAED,0EAA0E;IAElE,aAAa,CAAC,SAAiB;QACrC,MAAM,MAAM,GAAG,IAAI,CAAC,OAAO,CAAC,YAAY,GAAG,IAAI,CAAC,GAAG,CAAC,IAAI,CAAC,OAAO,CAAC,iBAAiB,EAAE,SAAS,CAAC,CAAC;QAC/F,OAAO,IAAI,CAAC,GAAG,CAAC,MAAM,EAAE,IAAI,CAAC,OAAO,CAAC,eAAe,CAAC,CAAC;IACxD,CAAC;IAED;;;;;;;;;OASG;IACK,qBAAqB,CAAC,SAAiB,EAAE,SAAiB,EAAE,WAAmB;QACrF,MAAM,MAAM,GAAG,IAAI,CAAC,mBAAmB,CAAC,GAAG,CAAC,SAAS,CAAC,CAAC;QACvD,IAAI,MAAM,IAAI,MAAM,CAAC,SAAS,KAAK,SAAS,IAAI,MAAM,CAAC,WAAW,KAAK,WAAW,EAAE,CAAC;YACnF,OAAO,MAAM,CAAC,KAAK,CAAC;QACtB,CAAC;QACD,MAAM,MAAM,GAAG,IAAI,CAAC,aAAa,CAAC,SAAS,CAAC,CAAC;QAC7C,IAAI,KAAK,GAAG,MAAM,CAAC;QACnB,IAAI,IAAI,CAAC,OAAO,CAAC,MAAM,EAAE,CAAC;YACxB,MAAM,KAAK,GAAG,MAAM,GAAG,IAAI,CAAC,OAAO,CAAC,MAAM,CAAC;YAC3C,KAAK,GAAG,MAAM,GAAG,CAAC,IAAI,CAAC,MAAM,EAAE,GAAG,CAAC,GAAG,CAAC,CAAC,GAAG,KAAK,CAAC;QACnD,CAAC;QACD,IAAI,CAAC,mBAAmB,CAAC,GAAG,CAAC,SAAS,EAAE,EAAE,SAAS,EAAE,WAAW,EAAE,KAAK,EAAE,CAAC,CAAC;QAC3E,OAAO,KAAK,CAAC;IACf,CAAC;IAED,wEAAwE;IAEhE,GAAG,CAAC,SAAiB;QAC3B,OAAO,mBAAmB,SAAS,EAAE,CAAC;IACxC,CAAC;IAEO,QAAQ,CAAC,SAAiB;QAChC,OAAO,mBAAmB,SAAS,QAAQ,CAAC;IAC9C,CAAC;IAEO,eAAe,CAAC,SAAiB;QACvC,uEAAuE;QACvE,kEAAkE;QAClE,sEAAsE;QACtE,gEAAgE;QAChE,uEAAuE;QACvE,kCAAkC;QAClC,OAAO,IAAI,CAAC,GAAG,CAAC,CAAC,EAAE,IAAI,CAAC,IAAI,CAAC,CAAC,IAAI,CAAC,aAAa,CAAC,SAAS,CAAC,GAAG,CAAC,CAAC,GAAG,IAAI,CAAC,CAAC,CAAC;IAC5E,CAAC;IAEO,eAAe;QACrB,OAAO,IAAI,CAAC,GAAG,CAAC,CAAC,EAAE,IAAI,CAAC,IAAI,CAAC,IAAI,CAAC,OAAO,CAAC,YAAY,GAAG,IAAI,CAAC,CAAC,CAAC;IAClE,CAAC;IAEO,gBAAgB,CAAC,KAAc,EAAE,SAAiB;QACxD,IAAI,CAAC,UAAU,CAAC,SAAS,CAAC,CAAC,gBAAgB,EAAE,CAAC;QAC9C,IAAI,CAAC,OAAO,CAAC,MAAM,CAAC,IAAI,CAAC,8CAA8C,SAAS,KAAM,KAAe,EAAE,OAAO,IAAI,KAAK,EAAE,CAAC,CAAC;QAC3H,IAAI,CAAC,OAAO,CAAC,YAAY,EAAE,CAAC,KAAK,EAAE,SAAS,CAAC,CAAC;IAChD,CAAC;IAEO,qBAAqB,CAAC,UAAyF;QACrH,IAAI,IAAI,CAAC,kBAAkB,CAAC,GAAG,CAAC,UAAU,CAAC;YAAE,OAAO;QACpD,IAAI,CAAC,kBAAkB,CAAC,GAAG,CAAC,UAAU,CAAC,CAAC;QACxC,IAAI,CAAC,OAAO,CAAC,MAAM,CAAC,IAAI,CACtB,4BAA4B,UAAU,oGAAoG,CAC3I,CAAC;IACJ,CAAC;IAEO,KAAK,CAAC,OAAO,CAAC,SAAiB;QACrC,IAAI,CAAC;YACH,OAAO,MAAM,IAAI,CAAC,OAAO,CAAC,KAAM,CAAC,GAAG,CAAsB,IAAI,CAAC,GAAG,CAAC,SAAS,CAAC,CAAC,CAAC;QACjF,CAAC;QAAC,OAAO,KAAK,EAAE,CAAC;YACf,IAAI,CAAC,gBAAgB,CAAC,KAAK,EAAE,SAAS,CAAC,CAAC;YACxC,OAAO,IAAI,CAAC;QACd,CAAC;IACH,CAAC;IAEO,KAAK,CAAC,OAAO,CAAC,SAAiB,EAAE,KAA0B,EAAE,UAAkB;QACrF,IAAI,CAAC;YACH,MAAM,IAAI,CAAC,OAAO,CAAC,KAAM,CAAC,GAAG,CAAC,IAAI,CAAC,GAAG,CAAC,SAAS,CAAC,EAAE,KAAK,EAAE,UAAU,CAAC,CAAC;QACxE,CAAC;QAAC,OAAO,KAAK,EAAE,CAAC;YACf,IAAI,CAAC,gBAAgB,CAAC,KAAK,EAAE,SAAS,CAAC,CAAC;QAC1C,CAAC;IACH,CAAC;IAED,qFAAqF;IAC7E,KAAK,CAAC,WAAW,CAAC,SAAiB;QACzC,IAAI,CAAC;YACH,MAAM,IAAI,CAAC,OAAO,CAAC,KAAM,CAAC,GAAG,CAAC,IAAI,CAAC,GAAG,CAAC,SAAS,CAAC,CAAC,CAAC;QACrD,CAAC;QAAC,OAAO,KAAK,EAAE,CAAC;YACf,IAAI,CAAC,gBAAgB,CAAC,KAAK,EAAE,SAAS,CAAC,CAAC;QAC1C,CAAC;IACH,CAAC;IAED,4NAA4N;IACpN,KAAK,CAAC,YAAY,CAAC,SAAiB;QAC1C,IAAI,CAAC;YACH,MAAM,IAAI,CAAC,OAAO,CAAC,KAAM,CAAC,GAAG,CAAC,IAAI,CAAC,QAAQ,CAAC,SAAS,CAAC,CAAC,CAAC;QAC1D,CAAC;QAAC,OAAO,KAAK,EAAE,CAAC;YACf,IAAI,CAAC,gBAAgB,CAAC,KAAK,EAAE,SAAS,CAAC,CAAC;QAC1C,CAAC;IACH,CAAC;IAED;;;;;;;;;;;;;;;;OAgBG;IACK,KAAK,CAAC,gBAAgB,CAAC,SAAiB,EAAE,KAAyB;QACzE,IAAI,CAAC,KAAK;YAAE,OAAO;QACnB,IAAI,CAAC,IAAI,CAAC,OAAO,CAAC,KAAM,CAAC,YAAY,EAAE,CAAC;YACtC,IAAI,CAAC,qBAAqB,CAAC,cAAc,CAAC,CAAC;YAC3C,OAAO;QACT,CAAC;QACD,IAAI,CAAC;YACH,MAAM,IAAI,CAAC,OAAO,CAAC,KAAM,CAAC,YAAY,CAAC,IAAI,CAAC,QAAQ,CAAC,SAAS,CAAC,EAAE,KAAK,CAAC,CAAC;QAC1E,CAAC;QAAC,OAAO,KAAK,EAAE,CAAC;YACf,IAAI,CAAC,gBAAgB,CAAC,KAAK,EAAE,SAAS,CAAC,CAAC;QAC1C,CAAC;IACH,CAAC;IAEO,KAAK,CAAC,gBAAgB,CAAC,SAAiB;QAC9C,MAAM,IAAI,CAAC,OAAO,CAAC,SAAS,EAAE,EAAE,GAAG,oBAAY,EAAE,EAAE,IAAI,CAAC,eAAe,CAAC,CAAC,CAAC,CAAC,CAAC;QAC5E,IAAI,CAAC,mBAAmB,CAAC,MAAM,CAAC,SAAS,CAAC,CAAC;IAC7C,CAAC;IAED;;;;;OAKG;IACK,KAAK,CAAC,eAAe,CAAC,SAAiB,EAAE,KAAc;QAC7D,IAAI,KAAiC,CAAC;QACtC,IAAI,CAAC;YACH,KAAK,GAAG,MAAM,IAAI,CAAC,OAAO,CAAC,KAAM,CAAC,GAAG,CAAsB,IAAI,CAAC,GAAG,CAAC,SAAS,CAAC,CAAC,CAAC;QAClF,CAAC;QAAC,OAAO,KAAK,EAAE,CAAC;YACf,IAAI,CAAC,gBAAgB,CAAC,KAAK,EAAE,SAAS,CAAC,CAAC;YACxC,IAAI,IAAI,CAAC,OAAO,CAAC,oBAAoB;gBAAE,OAAO,EAAE,OAAO,EAAE,IAAI,EAAE,OAAO,EAAE,KAAK,EAAE,CAAC;YAChF,MAAM,IAAI,qBAAqB,CAAC,SAAS,EAAE,KAAK,CAAC,CAAC;QACpD,CAAC;QAED,IAAI,CAAC,KAAK,EAAE,MAAM;YAAE,OAAO,EAAE,OAAO,EAAE,IAAI,EAAE,OAAO,EAAE,KAAK,EAAE,CAAC;QAE7D,MAAM,SAAS,GAAG,IAAI,CAAC,qBAAqB,CAAC,SAAS,EAAE,KAAK,CAAC,SAAS,IAAI,CAAC,EAAE,KAAK,CAAC,WAAW,CAAC,CAAC;QACjG,IAAI,IAAI,CAAC,GAAG,EAAE,GAAG,KAAK,CAAC,WAAW,GAAG,SAAS,EAAE,CAAC;YAC/C,OAAO,EAAE,OAAO,EAAE,KAAK,EAAE,OAAO,EAAE,KAAK,EAAE,CAAC;QAC5C,CAAC;QAED,IAAI,CAAC,KAAK;YAAE,OAAO,EAAE,OAAO,EAAE,IAAI,EAAE,OAAO,EAAE,KAAK,EAAE,CAAC;QAErD,IAAI,IAAI,CAAC,OAAO,CAAC,KAAM,CAAC,UAAU,EAAE,CAAC;YACnC,IAAI,CAAC;gBACH,MAAM,KAAK,GAAG,MAAM,IAAI,CAAC,OAAO,CAAC,KAAM,CAAC,UAAU,CAAC,IAAI,CAAC,QAAQ,CAAC,SAAS,CAAC,EAAE,IAAI,CAAC,eAAe,EAAE,CAAC,CAAC;gBACrG,IAAI,KAAK;oBAAE,IAAI,CAAC,OAAO,CAAC,aAAa,EAAE,CAAC,WAAW,EAAE,SAAS,CAAC,CAAC;gBAChE,OAAO,EAAE,OAAO,EAAE,KAAK,KAAK,IAAI,EAAE,OAAO,EAAE,KAAK,KAAK,IAAI,EAAE,UAAU,EAAE,KAAK,IAAI,SAAS,EAAE,CAAC;YAC9F,CAAC;YAAC,OAAO,KAAK,EAAE,CAAC;gBACf,IAAI,CAAC,gBAAgB,CAAC,KAAK,EAAE,SAAS,CAAC,CAAC;gBACxC,IAAI,CAAC,IAAI,CAAC,OAAO,CAAC,oBAAoB;oBAAE,MAAM,IAAI,qBAAqB,CAAC,SAAS,EAAE,KAAK,CAAC,CAAC;gBAC1F,6FAA6F;gBAC7F,IAAI,CAAC,OAAO,CAAC,aAAa,EAAE,CAAC,WAAW,EAAE,SAAS,CAAC,CAAC;gBACrD,OAAO,EAAE,OAAO,EAAE,IAAI,EAAE,OAAO,EAAE,IAAI,EAAE,CAAC;YAC1C,CAAC;QACH,CAAC;QAED,IAAI,CAAC,qBAAqB,CAAC,YAAY,CAAC,CAAC;QACzC,IAAI,CAAC,OAAO,CAAC,aAAa,EAAE,CAAC,WAAW,EAAE,SAAS,CAAC,CAAC;QACrD,OAAO,EAAE,OAAO,EAAE,IAAI,EAAE,OAAO,EAAE,IAAI,EAAE,CAAC;IAC1C,CAAC;IAEO,KAAK,CAAC,kBAAkB,CAAI,SAAiB,EAAE,EAAoB;QACzE,MAAM,OAAO,GAAG,IAAI,CAAC,UAAU,CAAC,SAAS,CAAC,CAAC;QAC3C,OAAO,CAAC,UAAU,EAAE,CAAC;QAErB,MAAM,IAAI,GAAG,MAAM,IAAI,CAAC,eAAe,CAAC,SAAS,EAAE,IAAI,CAAC,CAAC;QACzD,IAAI,CAAC,IAAI,CAAC,OAAO,EAAE,CAAC;YAClB,OAAO,CAAC,eAAe,EAAE,CAAC;YAC1B,MAAM,IAAI,KAAK,CAAC,0CAA0C,SAAS,EAAE,CAAC,CAAC;QACzE,CAAC;QAED,IAAI,CAAC;YACH,MAAM,MAAM,GAAG,MAAM,IAAI,CAAC,kBAAkB,CAAC,EAAE,CAAC,CAAC;YACjD,OAAO,CAAC,cAAc,EAAE,CAAC;YACzB,IAAI,IAAI,CAAC,OAAO,EAAE,CAAC;gBACjB,MAAM,IAAI,CAAC,gBAAgB,CAAC,SAAS,CAAC,CAAC;gBACvC,MAAM,IAAI,CAAC,gBAAgB,CAAC,SAAS,EAAE,IAAI,CAAC,UAAU,CAAC,CAAC;gBACxD,IAAI,CAAC,OAAO,CAAC,aAAa,EAAE,CAAC,QAAQ,EAAE,SAAS,CAAC,CAAC;YACpD,CAAC;iBAAM,IAAI,IAAI,CAAC,OAAO,CAAC,QAAQ,KAAK,WAAW,EAAE,CAAC;gBACjD,MAAM,IAAI,CAAC,wBAAwB,CAAC,SAAS,EAAE,IAAI,CAAC,CAAC;YACvD,CAAC;iBAAM,CAAC;gBACN,MAAM,IAAI,CAAC,gBAAgB,CAAC,SAAS,CAAC,CAAC;YACzC,CAAC;YACD,OAAO,MAAM,CAAC;QAChB,CAAC;QAAC,OAAO,KAAK,EAAE,CAAC;YACf,OAAO,CAAC,aAAa,EAAE,CAAC;YACxB,IAAI,IAAI,CAAC,OAAO,EAAE,CAAC;gBACjB,MAAM,IAAI,CAAC,iCAAiC,CAAC,SAAS,CAAC,CAAC;gBACxD,MAAM,IAAI,CAAC,gBAAgB,CAAC,SAAS,EAAE,IAAI,CAAC,UAAU,CAAC,CAAC;YAC1D,CAAC;iBAAM,IAAI,IAAI,CAAC,OAAO,CAAC,QAAQ,KAAK,WAAW,EAAE,CAAC;gBACjD,MAAM,IAAI,CAAC,wBAAwB,CAAC,SAAS,EAAE,KAAK,CAAC,CAAC;YACxD,CAAC;iBAAM,CAAC;gBACN,MAAM,IAAI,CAAC,gCAAgC,CAAC,SAAS,CAAC,CAAC;YACzD,CAAC;YACD,MAAM,KAAK,CAAC;QACd,CAAC;IACH,CAAC;IAEO,KAAK,CAAC,gCAAgC,CAAC,SAAiB;QAC9D,IAAI,IAAI,CAAC,OAAO,CAAC,KAAM,CAAC,mBAAmB,EAAE,CAAC;YAC5C,IAAI,CAAC;gBACH,MAAM,IAAI,GAAG,MAAM,IAAI,CAAC,OAAO,CAAC,KAAM,CAAC,mBAAmB,CAAC,IAAI,CAAC,GAAG,CAAC,SAAS,CAAC,EAAE;oBAC9E,UAAU,EAAE,IAAI,CAAC,eAAe,CAAC,CAAC,CAAC;oBACnC,gBAAgB,EAAE,IAAI,CAAC,OAAO,CAAC,gBAAgB;oBAC/C,GAAG,EAAE,IAAI,CAAC,GAAG,EAAE;iBAChB,CAAC,CAAC;gBACH,IAAI,CAAC,OAAO,CAAC,MAAM,CAAC,IAAI,CAAC,mCAAmC,SAAS,qBAAqB,IAAI,CAAC,QAAQ,EAAE,CAAC,CAAC;gBAC3G,IAAI,IAAI,CAAC,MAAM,IAAI,IAAI,CAAC,QAAQ,KAAK,IAAI,CAAC,OAAO,CAAC,gBAAgB,EAAE,CAAC;oBACnE,IAAI,CAAC,OAAO,CAAC,MAAM,CAAC,IAAI,CAAC,yCAAyC,SAAS,EAAE,CAAC,CAAC;oBAC/E,IAAI,CAAC,OAAO,CAAC,aAAa,EAAE,CAAC,MAAM,EAAE,SAAS,CAAC,CAAC;gBAClD,CAAC;gBACD,OAAO;YACT,CAAC;YAAC,OAAO,KAAK,EAAE,CAAC;gBACf,IAAI,CAAC,gBAAgB,CAAC,KAAK,EAAE,SAAS,CAAC,CAAC;gBACxC,4CAA4C;YAC9C,CAAC;QACH,CAAC;aAAM,CAAC;YACN,IAAI,CAAC,qBAAqB,CAAC,qBAAqB,CAAC,CAAC;QACpD,CAAC;QAED,MAAM,OAAO,GAAG,CAAC,MAAM,IAAI,CAAC,OAAO,CAAC,SAAS,CAAC,CAAC,IAAI,EAAE,GAAG,oBAAY,EAAE,CAAC;QACvE,MAAM,IAAI,GAAwB;YAChC,QAAQ,EAAE,OAAO,CAAC,QAAQ,GAAG,CAAC;YAC9B,WAAW,EAAE,IAAI,CAAC,GAAG,EAAE;YACvB,MAAM,EAAE,OAAO,CAAC,QAAQ,GAAG,CAAC,IAAI,IAAI,CAAC,OAAO,CAAC,gBAAgB;YAC7D,sEAAsE;YACtE,oEAAoE;YACpE,qEAAqE;YACrE,iEAAiE;YACjE,kDAAkD;YAClD,SAAS,EAAE,OAAO,CAAC,SAAS,IAAI,CAAC;YACjC,SAAS,EAAE,OAAO,CAAC,SAAS,IAAI,CAAC;YACjC,WAAW,EAAE,OAAO,CAAC,WAAW,IAAI,CAAC;SACtC,CAAC;QACF,MAAM,IAAI,CAAC,OAAO,CAAC,SAAS,EAAE,IAAI,EAAE,IAAI,CAAC,eAAe,CAAC,CAAC,CAAC,CAAC,CAAC;QAC7D,IAAI,CAAC,OAAO,CAAC,MAAM,CAAC,IAAI,CAAC,mCAAmC,SAAS,qBAAqB,IAAI,CAAC,QAAQ,EAAE,CAAC,CAAC;QAE3G,IAAI,IAAI,CAAC,MAAM,IAAI,CAAC,OAAO,CAAC,MAAM,EAAE,CAAC;YACnC,IAAI,CAAC,OAAO,CAAC,MAAM,CAAC,IAAI,CAAC,yCAAyC,SAAS,EAAE,CAAC,CAAC;YAC/E,IAAI,CAAC,OAAO,CAAC,aAAa,EAAE,CAAC,MAAM,EAAE,SAAS,CAAC,CAAC;QAClD,CAAC;IACH,CAAC;IAED;;;;OAIG;IACK,KAAK,CAAC,wBAAwB,CAAC,SAAiB,EAAE,OAAgB;QACxE,IAAI,IAAI,CAAC,OAAO,CAAC,KAAM,CAAC,mBAAmB,EAAE,CAAC;YAC5C,IAAI,CAAC;gBACH,MAAM,IAAI,GAAG,MAAM,IAAI,CAAC,OAAO,CAAC,KAAM,CAAC,mBAAmB,CAAC,IAAI,CAAC,GAAG,CAAC,SAAS,CAAC,EAAE;oBAC9E,OAAO;oBACP,KAAK,EAAE,IAAI,CAAC,OAAO,CAAC,cAAc;oBAClC,YAAY,EAAE,IAAI,CAAC,OAAO,CAAC,YAAY;oBACvC,kBAAkB,EAAE,IAAI,CAAC,OAAO,CAAC,kBAAkB;oBACnD,UAAU,EAAE,IAAI,CAAC,eAAe,CAAC,CAAC,CAAC;oBACnC,GAAG,EAAE,IAAI,CAAC,GAAG,EAAE;iBAChB,CAAC,CAAC;gBACH,IAAI,IAAI,CAAC,SAAS,EAAE,CAAC;oBACnB,IAAI,CAAC,OAAO,CAAC,MAAM,CAAC,IAAI,CACtB,yCAAyC,SAAS,gBAAgB,CAAC,IAAI,CAAC,SAAS,GAAG,GAAG,CAAC,CAAC,OAAO,CAAC,CAAC,CAAC,UAAU,IAAI,CAAC,WAAW,SAAS,CACvI,CAAC;oBACF,IAAI,CAAC,OAAO,CAAC,aAAa,EAAE,CAAC,MAAM,EAAE,SAAS,CAAC,CAAC;gBAClD,CAAC;gBACD,OAAO;YACT,CAAC;YAAC,OAAO,KAAK,EAAE,CAAC;gBACf,IAAI,CAAC,gBAAgB,CAAC,KAAK,EAAE,SAAS,CAAC,CAAC;gBACxC,4CAA4C;YAC9C,CAAC;QACH,CAAC;aAAM,CAAC;YACN,IAAI,CAAC,qBAAqB,CAAC,qBAAqB,CAAC,CAAC;QACpD,CAAC;QAED,MAAM,OAAO,GAAG,CAAC,MAAM,IAAI,CAAC,OAAO,CAAC,SAAS,CAAC,CAAC,IAAI,EAAE,GAAG,oBAAY,EAAE,CAAC;QACvE,MAAM,KAAK,GAAG,IAAI,CAAC,OAAO,CAAC,cAAc,CAAC;QAC1C,MAAM,SAAS,GAAG,CAAC,OAAO,CAAC,SAAS,IAAI,CAAC,CAAC,GAAG,KAAK,GAAG,CAAC,OAAO,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,GAAG,CAAC,CAAC,GAAG,KAAK,CAAC,CAAC;QACrF,MAAM,WAAW,GAAG,CAAC,OAAO,CAAC,WAAW,IAAI,CAAC,CAAC,GAAG,CAAC,CAAC;QACnD,MAAM,MAAM,GAAG,WAAW,IAAI,IAAI,CAAC,OAAO,CAAC,YAAY,IAAI,SAAS,IAAI,IAAI,CAAC,OAAO,CAAC,kBAAkB,CAAC;QAExG,MAAM,IAAI,GAAwB;YAChC,QAAQ,EAAE,OAAO,CAAC,QAAQ;YAC1B,WAAW,EAAE,IAAI,CAAC,GAAG,EAAE;YACvB,MAAM;YACN,SAAS,EAAE,OAAO,CAAC,SAAS,IAAI,CAAC;YACjC,SAAS;YACT,WAAW;SACZ,CAAC;QACF,MAAM,IAAI,CAAC,OAAO,CAAC,SAAS,EAAE,IAAI,EAAE,IAAI,CAAC,eAAe,CAAC,CAAC,CAAC,CAAC,CAAC;QAE7D,IAAI,MAAM,IAAI,CAAC,OAAO,CAAC,MAAM,EAAE,CAAC;YAC9B,IAAI,CAAC,OAAO,CAAC,MAAM,CAAC,IAAI,CACtB,yCAAyC,SAAS,gBAAgB,CAAC,SAAS,GAAG,GAAG,CAAC,CAAC,OAAO,CAAC,CAAC,CAAC,UAAU,WAAW,SAAS,CAC7H,CAAC;YACF,IAAI,CAAC,OAAO,CAAC,aAAa,EAAE,CAAC,MAAM,EAAE,SAAS,CAAC,CAAC;QAClD,CAAC;IACH,CAAC;IAEO,KAAK,CAAC,iCAAiC,CAAC,SAAiB;QAC/D,MAAM,OAAO,GAAG,CAAC,MAAM,IAAI,CAAC,OAAO,CAAC,SAAS,CAAC,CAAC,IAAI,EAAE,GAAG,oBAAY,EAAE,MAAM,EAAE,IAAI,EAAE,CAAC;QACrF,MAAM,SAAS,GAAG,CAAC,OAAO,CAAC,SAAS,IAAI,CAAC,CAAC,GAAG,CAAC,CAAC;QAC/C,MAAM,IAAI,GAAwB;YAChC,QAAQ,EAAE,IAAI,CAAC,GAAG,CAAC,OAAO,CAAC,QAAQ,EAAE,IAAI,CAAC,OAAO,CAAC,gBAAgB,CAAC;YACnE,WAAW,EAAE,IAAI,CAAC,GAAG,EAAE;YACvB,MAAM,EAAE,IAAI;YACZ,SAAS;YACT,SAAS,EAAE,OAAO,CAAC,SAAS,IAAI,CAAC;YACjC,WAAW,EAAE,OAAO,CAAC,WAAW,IAAI,CAAC;SACtC,CAAC;QACF,MAAM,IAAI,CAAC,OAAO,CAAC,SAAS,EAAE,IAAI,EAAE,IAAI,CAAC,eAAe,CAAC,SAAS,CAAC,CAAC,CAAC;QACrE,IAAI,CAAC,OAAO,CAAC,MAAM,CAAC,IAAI,CAAC,yCAAyC,SAAS,0BAA0B,SAAS,GAAG,CAAC,GAAG,CAAC,CAAC;QACvH,IAAI,CAAC,OAAO,CAAC,aAAa,EAAE,CAAC,MAAM,EAAE,SAAS,CAAC,CAAC;IAClD,CAAC;IAED,0EAA0E;IAElE,KAAK,CAAC,YAAY,CAAI,SAAiB,EAAE,EAAoB;QACnE,MAAM,OAAO,GAAG,IAAI,CAAC,UAAU,CAAC,SAAS,CAAC,CAAC;QAC3C,OAAO,CAAC,UAAU,EAAE,CAAC;QAErB,MAAM,IAAI,GAAG,IAAI,CAAC,cAAc,CAAC,SAAS,CAAC,CAAC;QAC5C,IAAI,CAAC,IAAI,CAAC,OAAO,EAAE,CAAC;YAClB,OAAO,CAAC,eAAe,EAAE,CAAC;YAC1B,MAAM,IAAI,KAAK,CAAC,0CAA0C,SAAS,EAAE,CAAC,CAAC;QACzE,CAAC;QAED,IAAI,CAAC;YACH,MAAM,MAAM,GAAG,MAAM,IAAI,CAAC,kBAAkB,CAAC,EAAE,CAAC,CAAC;YACjD,OAAO,CAAC,cAAc,EAAE,CAAC;YACzB,IAAI,IAAI,CAAC,OAAO,EAAE,CAAC;gBACjB,IAAI,CAAC,UAAU,CAAC,SAAS,CAAC,CAAC;YAC7B,CAAC;iBAAM,IAAI,IAAI,CAAC,OAAO,CAAC,QAAQ,KAAK,WAAW,EAAE,CAAC;gBACjD,IAAI,CAAC,sBAAsB,CAAC,SAAS,EAAE,IAAI,CAAC,CAAC;YAC/C,CAAC;iBAAM,CAAC;gBACN,IAAI,CAAC,UAAU,CAAC,SAAS,CAAC,CAAC;YAC7B,CAAC;YACD,OAAO,MAAM,CAAC;QAChB,CAAC;QAAC,OAAO,KAAK,EAAE,CAAC;YACf,OAAO,CAAC,aAAa,EAAE,CAAC;YACxB,IAAI,IAAI,CAAC,OAAO,EAAE,CAAC;gBACjB,IAAI,CAAC,2BAA2B,CAAC,SAAS,CAAC,CAAC;YAC9C,CAAC;iBAAM,IAAI,IAAI,CAAC,OAAO,CAAC,QAAQ,KAAK,WAAW,EAAE,CAAC;gBACjD,IAAI,CAAC,sBAAsB,CAAC,SAAS,EAAE,KAAK,CAAC,CAAC;YAChD,CAAC;iBAAM,CAAC;gBACN,IAAI,CAAC,0BAA0B,CAAC,SAAS,CAAC,CAAC;YAC7C,CAAC;YACD,MAAM,KAAK,CAAC;QACd,CAAC;IACH,CAAC;IAED,gFAAgF;IACxE,aAAa,CAAC,SAAiB;QACrC,IAAI,CAAC,IAAI,CAAC,cAAc,CAAC,GAAG,CAAC,SAAS,CAAC;YAAE,OAAO,IAAI,CAAC;QAErD,MAAM,SAAS,GAAG,IAAI,CAAC,UAAU,CAAC,GAAG,CAAC,SAAS,CAAC,IAAI,CAAC,CAAC;QACtD,MAAM,WAAW,GAAG,IAAI,CAAC,eAAe,CAAC,GAAG,CAAC,SAAS,CAAC,IAAI,CAAC,CAAC;QAC7D,MAAM,SAAS,GAAG,IAAI,CAAC,qBAAqB,CAAC,SAAS,EAAE,SAAS,EAAE,WAAW,CAAC,CAAC;QAEhF,IAAI,IAAI,CAAC,GAAG,EAAE,GAAG,WAAW,GAAG,SAAS,EAAE,CAAC;YACzC,IAAI,CAAC,OAAO,CAAC,MAAM,CAAC,IAAI,CAAC,uCAAuC,SAAS,EAAE,CAAC,CAAC;YAC7E,OAAO,KAAK,CAAC;QACf,CAAC;QACD,OAAO,IAAI,CAAC;IACd,CAAC;IAED,oFAAoF;IAC5E,cAAc,CAAC,SAAiB;QACtC,IAAI,CAAC,IAAI,CAAC,cAAc,CAAC,GAAG,CAAC,SAAS,CAAC;YAAE,OAAO,EAAE,OAAO,EAAE,IAAI,EAAE,OAAO,EAAE,KAAK,EAAE,CAAC;QAElF,MAAM,SAAS,GAAG,IAAI,CAAC,UAAU,CAAC,GAAG,CAAC,SAAS,CAAC,IAAI,CAAC,CAAC;QACtD,MAAM,WAAW,GAAG,IAAI,CAAC,eAAe,CAAC,GAAG,CAAC,SAAS,CAAC,IAAI,CAAC,CAAC;QAC7D,MAAM,SAAS,GAAG,IAAI,CAAC,qBAAqB,CAAC,SAAS,EAAE,SAAS,EAAE,WAAW,CAAC,CAAC;QAEhF,IAAI,IAAI,CAAC,GAAG,EAAE,GAAG,WAAW,GAAG,SAAS,EAAE,CAAC;YACzC,IAAI,CAAC,OAAO,CAAC,MAAM,CAAC,IAAI,CAAC,uCAAuC,SAAS,EAAE,CAAC,CAAC;YAC7E,OAAO,EAAE,OAAO,EAAE,KAAK,EAAE,OAAO,EAAE,KAAK,EAAE,CAAC;QAC5C,CAAC;QAED,IAAI,IAAI,CAAC,eAAe,CAAC,GAAG,CAAC,SAAS,CAAC,EAAE,CAAC;YACxC,oFAAoF;YACpF,OAAO,EAAE,OAAO,EAAE,KAAK,EAAE,OAAO,EAAE,KAAK,EAAE,CAAC;QAC5C,CAAC;QAED,IAAI,CAAC,eAAe,CAAC,GAAG,CAAC,SAAS,CAAC,CAAC;QACpC,IAAI,CAAC,cAAc,CAAC,GAAG,CAAC,SAAS,EAAE,IAAI,CAAC,GAAG,EAAE,CAAC,CAAC;QAC/C,IAAI,CAAC,OAAO,CAAC,aAAa,EAAE,CAAC,WAAW,EAAE,SAAS,CAAC,CAAC;QACrD,OAAO,EAAE,OAAO,EAAE,IAAI,EAAE,OAAO,EAAE,IAAI,EAAE,CAAC;IAC1C,CAAC;IAEO,0BAA0B,CAAC,SAAiB;QAClD,MAAM,QAAQ,GAAG,CAAC,IAAI,CAAC,QAAQ,CAAC,GAAG,CAAC,SAAS,CAAC,IAAI,CAAC,CAAC,GAAG,CAAC,CAAC;QACzD,IAAI,CAAC,QAAQ,CAAC,GAAG,CAAC,SAAS,EAAE,QAAQ,CAAC,CAAC;QACvC,IAAI,CAAC,eAAe,CAAC,GAAG,CAAC,SAAS,EAAE,IAAI,CAAC,GAAG,EAAE,CAAC,CAAC;QAChD,IAAI,CAAC,OAAO,CAAC,MAAM,CAAC,IAAI,CAAC,mCAAmC,SAAS,qBAAqB,QAAQ,EAAE,CAAC,CAAC;QAEtG,IAAI,QAAQ,IAAI,IAAI,CAAC,OAAO,CAAC,gBAAgB,IAAI,CAAC,IAAI,CAAC,cAAc,CAAC,GAAG,CAAC,SAAS,CAAC,EAAE,CAAC;YACrF,IAAI,CAAC,cAAc,CAAC,GAAG,CAAC,SAAS,CAAC,CAAC;YACnC,IAAI,CAAC,UAAU,CAAC,GAAG,CAAC,SAAS,EAAE,CAAC,CAAC,CAAC;YAClC,IAAI,CAAC,OAAO,CAAC,MAAM,CAAC,IAAI,CAAC,yCAAyC,SAAS,EAAE,CAAC,CAAC;YAC/E,IAAI,CAAC,OAAO,CAAC,aAAa,EAAE,CAAC,MAAM,EAAE,SAAS,CAAC,CAAC;QAClD,CAAC;IACH,CAAC;IAED,kIAAkI;IAC1H,sBAAsB,CAAC,SAAiB,EAAE,OAAgB;QAChE,MAAM,KAAK,GAAG,IAAI,CAAC,OAAO,CAAC,cAAc,CAAC;QAC1C,MAAM,IAAI,GAAG,CAAC,IAAI,CAAC,UAAU,CAAC,GAAG,CAAC,SAAS,CAAC,IAAI,CAAC,CAAC,GAAG,KAAK,GAAG,CAAC,OAAO,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,GAAG,CAAC,CAAC,GAAG,KAAK,CAAC,CAAC;QAC7F,MAAM,KAAK,GAAG,CAAC,IAAI,CAAC,YAAY,CAAC,GAAG,CAAC,SAAS,CAAC,IAAI,CAAC,CAAC,GAAG,CAAC,CAAC;QAC1D,IAAI,CAAC,UAAU,CAAC,GAAG,CAAC,SAAS,EAAE,IAAI,CAAC,CAAC;QACrC,IAAI,CAAC,YAAY,CAAC,GAAG,CAAC,SAAS,EAAE,KAAK,CAAC,CAAC;QAExC,IAAI,KAAK,IAAI,IAAI,CAAC,OAAO,CAAC,YAAY,IAAI,IAAI,IAAI,IAAI,CAAC,OAAO,CAAC,kBAAkB,IAAI,CAAC,IAAI,CAAC,cAAc,CAAC,GAAG,CAAC,SAAS,CAAC,EAAE,CAAC;YACzH,IAAI,CAAC,cAAc,CAAC,GAAG,CAAC,SAAS,CAAC,CAAC;YACnC,IAAI,CAAC,UAAU,CAAC,GAAG,CAAC,SAAS,EAAE,CAAC,CAAC,CAAC;YAClC,IAAI,CAAC,eAAe,CAAC,GAAG,CAAC,SAAS,EAAE,IAAI,CAAC,GAAG,EAAE,CAAC,CAAC;YAChD,IAAI,CAAC,OAAO,CAAC,MAAM,CAAC,IAAI,CACtB,yCAAyC,SAAS,gBAAgB,CAAC,IAAI,GAAG,GAAG,CAAC,CAAC,OAAO,CAAC,CAAC,CAAC,UAAU,KAAK,SAAS,CAClH,CAAC;YACF,IAAI,CAAC,OAAO,CAAC,aAAa,EAAE,CAAC,MAAM,EAAE,SAAS,CAAC,CAAC;QAClD,CAAC;IACH,CAAC;IAEO,2BAA2B,CAAC,SAAiB;QACnD,IAAI,CAAC,eAAe,CAAC,MAAM,CAAC,SAAS,CAAC,CAAC;QACvC,IAAI,CAAC,cAAc,CAAC,MAAM,CAAC,SAAS,CAAC,CAAC;QACtC,MAAM,SAAS,GAAG,CAAC,IAAI,CAAC,UAAU,CAAC,GAAG,CAAC,SAAS,CAAC,IAAI,CAAC,CAAC,GAAG,CAAC,CAAC;QAC5D,IAAI,CAAC,UAAU,CAAC,GAAG,CAAC,SAAS,EAAE,SAAS,CAAC,CAAC;QAC1C,IAAI,CAAC,eAAe,CAAC,GAAG,CAAC,SAAS,EAAE,IAAI,CAAC,GAAG,EAAE,CAAC,CAAC;QAChD,IAAI,CAAC,OAAO,CAAC,MAAM,CAAC,IAAI,CAAC,yCAAyC,SAAS,0BAA0B,SAAS,GAAG,CAAC,GAAG,CAAC,CAAC;QACvH,IAAI,CAAC,OAAO,CAAC,aAAa,EAAE,CAAC,MAAM,EAAE,SAAS,CAAC,CAAC;IAClD,CAAC;IAEO,UAAU,CAAC,SAAiB;QAClC,MAAM,OAAO,GAAG,IAAI,CAAC,cAAc,CAAC,GAAG,CAAC,SAAS,CAAC,CAAC;QACnD,IAAI,CAAC,QAAQ,CAAC,MAAM,CAAC,SAAS,CAAC,CAAC;QAChC,IAAI,CAAC,eAAe,CAAC,MAAM,CAAC,SAAS,CAAC,CAAC;QACvC,IAAI,CAAC,cAAc,CAAC,MAAM,CAAC,SAAS,CAAC,CAAC;QACtC,IAAI,CAAC,UAAU,CAAC,MAAM,CAAC,SAAS,CAAC,CAAC;QAClC,IAAI,CAAC,eAAe,CAAC,MAAM,CAAC,SAAS,CAAC,CAAC;QACvC,IAAI,CAAC,cAAc,CAAC,MAAM,CAAC,SAAS,CAAC,CAAC;QACtC,IAAI,CAAC,UAAU,CAAC,MAAM,CAAC,SAAS,CAAC,CAAC;QAClC,IAAI,CAAC,YAAY,CAAC,MAAM,CAAC,SAAS,CAAC,CAAC;QACpC,IAAI,CAAC,mBAAmB,CAAC,MAAM,CAAC,SAAS,CAAC,CAAC;QAC3C,IAAI,OAAO,EAAE,CAAC;YACZ,IAAI,CAAC,OAAO,CAAC,aAAa,EAAE,CAAC,QAAQ,EAAE,SAAS,CAAC,CAAC;QACpD,CAAC;QACD,IAAI,CAAC,OAAO,CAAC,MAAM,CAAC,GAAG,CAAC,wCAAwC,SAAS,EAAE,CAAC,CAAC;IAC/E,CAAC;IAED;;;;;;OAMG;IACK,OAAO;QACb,MAAM,GAAG,GAAG,IAAI,CAAC,GAAG,EAAE,CAAC;QACvB,MAAM,UAAU,GAAG,IAAI,CAAC,GAAG,CAAC,IAAI,CAAC,OAAO,CAAC,YAAY,EAAE,IAAI,CAAC,GAAG,CAAC,CAAC;QACjE,KAAK,MAAM,CAAC,SAAS,EAAE,SAAS,CAAC,IAAI,IAAI,CAAC,cAAc,CAAC,OAAO,EAAE,EAAE,CAAC;YACnE,IAAI,GAAG,GAAG,SAAS,GAAG,UAAU,EAAE,CAAC;gBACjC,IAAI,CAAC,eAAe,CAAC,MAAM,CAAC,SAAS,CAAC,CAAC;gBACvC,IAAI,CAAC,cAAc,CAAC,MAAM,CAAC,SAAS,CAAC,CAAC;gBACtC,IAAI,CAAC,OAAO,CAAC,MAAM,CAAC,IAAI,CACtB,wCAAwC,SAAS,0EAA0E,UAAU,qCAAqC,CAC3K,CAAC;YACJ,CAAC;QACH,CAAC;IACH,CAAC;IAED,4EAA4E;IAEpE,kBAAkB,CAAI,EAAoB;QAChD,IAAI,CAAC,IAAI,CAAC,OAAO,CAAC,OAAO;YAAE,OAAO,EAAE,EAAE,CAAC;QAEvC,OAAO,IAAI,OAAO,CAAI,CAAC,OAAO,EAAE,MAAM,EAAE,EAAE;YACxC,MAAM,KAAK,GAAG,UAAU,CAAC,GAAG,EAAE,CAAC,MAAM,CAAC,IAAI,KAAK,CAAC,6BAA6B,IAAI,CAAC,OAAO,CAAC,OAAO,IAAI,CAAC,CAAC,EAAE,IAAI,CAAC,OAAO,CAAC,OAAO,CAAC,CAAC;YAC/H,EAAE,EAAE,CAAC,IAAI,CACP,CAAC,KAAK,EAAE,EAAE;gBACR,YAAY,CAAC,KAAK,CAAC,CAAC;gBACpB,OAAO,CAAC,KAAK,CAAC,CAAC;YACjB,CAAC,EACD,CAAC,KAAK,EAAE,EAAE;gBACR,YAAY,CAAC,KAAK,CAAC,CAAC;gBACpB,MAAM,CAAC,KAAK,CAAC,CAAC;YAChB,CAAC,CACF,CAAC;QACJ,CAAC,CAAC,CAAC;IACL,CAAC;CACF;AA7rBD,wCA6rBC"}
|
package/dist/store.d.ts
CHANGED
|
@@ -18,16 +18,36 @@ export interface Store {
|
|
|
18
18
|
}): Promise<CircuitBreakerState>;
|
|
19
19
|
/**
|
|
20
20
|
* Atomically claims a short-lived, exclusive "trial" slot for `key`
|
|
21
|
-
* (conceptually a SET-if-not-exists). Returns
|
|
22
|
-
*
|
|
23
|
-
*
|
|
24
|
-
*
|
|
21
|
+
* (conceptually a SET-if-not-exists). Returns an ownership token (an
|
|
22
|
+
* opaque string) if this call won the claim, or `null` if someone else
|
|
23
|
+
* already holds it. Used to implement half-open: only the winner gets to
|
|
24
|
+
* make the trial call while a breaker is transitioning from open back to
|
|
25
|
+
* closed; everyone else stays blocked until the trial resolves.
|
|
26
|
+
*
|
|
27
|
+
* Release the claim via `releaseTrial` with this same token when the
|
|
28
|
+
* trial resolves — never delete the key unconditionally. If the trial
|
|
29
|
+
* outran its TTL, another instance may have since claimed a *new* trial
|
|
30
|
+
* on the same key (see README limitation on trial-TTL expiry, which is
|
|
31
|
+
* by design); an unconditional delete at that point would destroy that
|
|
32
|
+
* other instance's active claim instead of the caller's own expired one.
|
|
25
33
|
*
|
|
26
34
|
* If omitted, the breaker falls back to letting every caller through
|
|
27
35
|
* once the reset window elapses (a thundering herd on recovery — see
|
|
28
36
|
* README).
|
|
29
37
|
*/
|
|
30
|
-
claimTrial?(key: string, ttlSeconds: number): Promise<
|
|
38
|
+
claimTrial?(key: string, ttlSeconds: number): Promise<string | null>;
|
|
39
|
+
/**
|
|
40
|
+
* Releases a trial slot claimed via `claimTrial`, but only if `token`
|
|
41
|
+
* still matches what's currently stored under `key` — a compare-and-
|
|
42
|
+
* delete, not an unconditional delete. This is what prevents a trial
|
|
43
|
+
* that outran its TTL from later deleting a *different* instance's newer
|
|
44
|
+
* claim on the same key.
|
|
45
|
+
*
|
|
46
|
+
* If omitted, the breaker logs a one-time warning and does not delete the
|
|
47
|
+
* key at all — claims are then released only by TTL expiry, so recovery
|
|
48
|
+
* after a resolved trial can lag by up to the trial TTL.
|
|
49
|
+
*/
|
|
50
|
+
releaseTrial?(key: string, token: string): Promise<void>;
|
|
31
51
|
/**
|
|
32
52
|
* `strategy: 'errorRate'` only. Atomically folds one call's outcome into
|
|
33
53
|
* an EWMA failure-rate estimate and decides `isOpen` in a single round
|
package/dist/store.d.ts.map
CHANGED
|
@@ -1 +1 @@
|
|
|
1
|
-
{"version":3,"file":"store.d.ts","sourceRoot":"","sources":["../src/store.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,mBAAmB,EAAE,MAAM,SAAS,CAAC;AAE9C,MAAM,WAAW,KAAK;IACpB,GAAG,CAAC,CAAC,EAAE,GAAG,EAAE,MAAM,GAAG,OAAO,CAAC,CAAC,GAAG,IAAI,CAAC,CAAC;IACvC,GAAG,CAAC,CAAC,EAAE,GAAG,EAAE,MAAM,EAAE,KAAK,EAAE,CAAC,EAAE,UAAU,EAAE,MAAM,GAAG,OAAO,CAAC,IAAI,CAAC,CAAC;IACjE,GAAG,CAAC,GAAG,EAAE,MAAM,GAAG,OAAO,CAAC,IAAI,CAAC,CAAC;IAEhC;;;;;;;OAOG;IACH,mBAAmB,CAAC,CAClB,GAAG,EAAE,MAAM,EACX,IAAI,EAAE;QAAE,UAAU,EAAE,MAAM,CAAC;QAAC,gBAAgB,EAAE,MAAM,CAAC;QAAC,GAAG,EAAE,MAAM,CAAA;KAAE,GAClE,OAAO,CAAC,mBAAmB,CAAC,CAAC;IAEhC
|
|
1
|
+
{"version":3,"file":"store.d.ts","sourceRoot":"","sources":["../src/store.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,mBAAmB,EAAE,MAAM,SAAS,CAAC;AAE9C,MAAM,WAAW,KAAK;IACpB,GAAG,CAAC,CAAC,EAAE,GAAG,EAAE,MAAM,GAAG,OAAO,CAAC,CAAC,GAAG,IAAI,CAAC,CAAC;IACvC,GAAG,CAAC,CAAC,EAAE,GAAG,EAAE,MAAM,EAAE,KAAK,EAAE,CAAC,EAAE,UAAU,EAAE,MAAM,GAAG,OAAO,CAAC,IAAI,CAAC,CAAC;IACjE,GAAG,CAAC,GAAG,EAAE,MAAM,GAAG,OAAO,CAAC,IAAI,CAAC,CAAC;IAEhC;;;;;;;OAOG;IACH,mBAAmB,CAAC,CAClB,GAAG,EAAE,MAAM,EACX,IAAI,EAAE;QAAE,UAAU,EAAE,MAAM,CAAC;QAAC,gBAAgB,EAAE,MAAM,CAAC;QAAC,GAAG,EAAE,MAAM,CAAA;KAAE,GAClE,OAAO,CAAC,mBAAmB,CAAC,CAAC;IAEhC;;;;;;;;;;;;;;;;;;OAkBG;IACH,UAAU,CAAC,CAAC,GAAG,EAAE,MAAM,EAAE,UAAU,EAAE,MAAM,GAAG,OAAO,CAAC,MAAM,GAAG,IAAI,CAAC,CAAC;IAErE;;;;;;;;;;OAUG;IACH,YAAY,CAAC,CAAC,GAAG,EAAE,MAAM,EAAE,KAAK,EAAE,MAAM,GAAG,OAAO,CAAC,IAAI,CAAC,CAAC;IAEzD;;;;;;;;;;;;OAYG;IACH,mBAAmB,CAAC,CAClB,GAAG,EAAE,MAAM,EACX,IAAI,EAAE;QAAE,OAAO,EAAE,OAAO,CAAC;QAAC,KAAK,EAAE,MAAM,CAAC;QAAC,YAAY,EAAE,MAAM,CAAC;QAAC,kBAAkB,EAAE,MAAM,CAAC;QAAC,UAAU,EAAE,MAAM,CAAC;QAAC,GAAG,EAAE,MAAM,CAAA;KAAE,GAC3H,OAAO,CAAC,mBAAmB,GAAG;QAAE,SAAS,EAAE,OAAO,CAAA;KAAE,CAAC,CAAC;CAC1D"}
|
|
@@ -1,15 +1,28 @@
|
|
|
1
|
-
import type { Redis } from 'ioredis';
|
|
1
|
+
import type { Redis, Result } from 'ioredis';
|
|
2
2
|
import { Store } from '../store';
|
|
3
3
|
import { CircuitBreakerState } from '../types';
|
|
4
|
+
declare module 'ioredis' {
|
|
5
|
+
interface RedisCommander<Context> {
|
|
6
|
+
redeyeRecordFailure(key: string, ttlSeconds: number, failureThreshold: number, now: number): Result<string, Context>;
|
|
7
|
+
redeyeRecordOutcome(key: string, decay: number, isFailure: number, minimumCalls: number, errorRateThreshold: number, ttlSeconds: number, now: number): Result<string, Context>;
|
|
8
|
+
redeyeReleaseTrial(key: string, token: string): Result<number, Context>;
|
|
9
|
+
}
|
|
10
|
+
}
|
|
4
11
|
/**
|
|
5
12
|
* Redis-backed Store implementation. Import this from
|
|
6
13
|
* `redeye/redis-store` so the core package has no hard dependency on
|
|
7
14
|
* `ioredis` — only consumers who use RedisStore need it installed.
|
|
8
15
|
*
|
|
9
16
|
* Implements all three optional atomic capabilities (`recordFailureAtomic`
|
|
10
|
-
* and `recordOutcomeAtomic` via Lua scripts, `claimTrial` via
|
|
11
|
-
*
|
|
12
|
-
*
|
|
17
|
+
* and `recordOutcomeAtomic` via Lua scripts, `claimTrial`/`releaseTrial` via
|
|
18
|
+
* a tokened `SET ... NX` and compare-and-delete), so breakers backed by
|
|
19
|
+
* RedisStore get exact counting and real single-trial half-open behavior
|
|
20
|
+
* under both strategies instead of best-effort fallbacks.
|
|
21
|
+
*
|
|
22
|
+
* Scripts are registered once per Redis client via `defineCommand`, so
|
|
23
|
+
* ioredis sends a cached script hash (`EVALSHA`) on every call instead of
|
|
24
|
+
* the full script body, with automatic fallback to `EVAL` if the script
|
|
25
|
+
* cache is ever flushed.
|
|
13
26
|
*/
|
|
14
27
|
export declare class RedisStore implements Store {
|
|
15
28
|
private readonly redis;
|
|
@@ -35,6 +48,7 @@ export declare class RedisStore implements Store {
|
|
|
35
48
|
}): Promise<CircuitBreakerState & {
|
|
36
49
|
openedNow: boolean;
|
|
37
50
|
}>;
|
|
38
|
-
claimTrial(key: string, ttlSeconds: number): Promise<
|
|
51
|
+
claimTrial(key: string, ttlSeconds: number): Promise<string | null>;
|
|
52
|
+
releaseTrial(key: string, token: string): Promise<void>;
|
|
39
53
|
}
|
|
40
54
|
//# sourceMappingURL=redis-store.d.ts.map
|
|
@@ -1 +1 @@
|
|
|
1
|
-
{"version":3,"file":"redis-store.d.ts","sourceRoot":"","sources":["../../src/stores/redis-store.ts"],"names":[],"mappings":"AAAA,OAAO,KAAK,EAAE,KAAK,EAAE,MAAM,SAAS,CAAC;
|
|
1
|
+
{"version":3,"file":"redis-store.d.ts","sourceRoot":"","sources":["../../src/stores/redis-store.ts"],"names":[],"mappings":"AAAA,OAAO,KAAK,EAAE,KAAK,EAAE,MAAM,EAAE,MAAM,SAAS,CAAC;AAE7C,OAAO,EAAE,KAAK,EAAE,MAAM,UAAU,CAAC;AACjC,OAAO,EAAE,mBAAmB,EAAE,MAAM,UAAU,CAAC;AAE/C,OAAO,QAAQ,SAAS,CAAC;IACvB,UAAU,cAAc,CAAC,OAAO;QAC9B,mBAAmB,CACjB,GAAG,EAAE,MAAM,EACX,UAAU,EAAE,MAAM,EAClB,gBAAgB,EAAE,MAAM,EACxB,GAAG,EAAE,MAAM,GACV,MAAM,CAAC,MAAM,EAAE,OAAO,CAAC,CAAC;QAC3B,mBAAmB,CACjB,GAAG,EAAE,MAAM,EACX,KAAK,EAAE,MAAM,EACb,SAAS,EAAE,MAAM,EACjB,YAAY,EAAE,MAAM,EACpB,kBAAkB,EAAE,MAAM,EAC1B,UAAU,EAAE,MAAM,EAClB,GAAG,EAAE,MAAM,GACV,MAAM,CAAC,MAAM,EAAE,OAAO,CAAC,CAAC;QAC3B,kBAAkB,CAAC,GAAG,EAAE,MAAM,EAAE,KAAK,EAAE,MAAM,GAAG,MAAM,CAAC,MAAM,EAAE,OAAO,CAAC,CAAC;KACzE;CACF;AA0GD;;;;;;;;;;;;;;;GAeG;AACH,qBAAa,UAAW,YAAW,KAAK;IAG1B,OAAO,CAAC,QAAQ,CAAC,KAAK;IAFlC,OAAO,CAAC,QAAQ,CAAC,MAAM,CAAS;gBAEH,KAAK,EAAE,KAAK,EAAE,OAAO,GAAE;QAAE,SAAS,CAAC,EAAE,MAAM,CAAA;KAAO;IAOzE,GAAG,CAAC,CAAC,EAAE,GAAG,EAAE,MAAM,GAAG,OAAO,CAAC,CAAC,GAAG,IAAI,CAAC;IAMtC,GAAG,CAAC,CAAC,EAAE,GAAG,EAAE,MAAM,EAAE,KAAK,EAAE,CAAC,EAAE,UAAU,EAAE,MAAM,GAAG,OAAO,CAAC,IAAI,CAAC;IAIhE,GAAG,CAAC,GAAG,EAAE,MAAM,GAAG,OAAO,CAAC,IAAI,CAAC;IAI/B,mBAAmB,CACvB,GAAG,EAAE,MAAM,EACX,IAAI,EAAE;QAAE,UAAU,EAAE,MAAM,CAAC;QAAC,gBAAgB,EAAE,MAAM,CAAC;QAAC,GAAG,EAAE,MAAM,CAAA;KAAE,GAClE,OAAO,CAAC,mBAAmB,CAAC;IAUzB,mBAAmB,CACvB,GAAG,EAAE,MAAM,EACX,IAAI,EAAE;QAAE,OAAO,EAAE,OAAO,CAAC;QAAC,KAAK,EAAE,MAAM,CAAC;QAAC,YAAY,EAAE,MAAM,CAAC;QAAC,kBAAkB,EAAE,MAAM,CAAC;QAAC,UAAU,EAAE,MAAM,CAAC;QAAC,GAAG,EAAE,MAAM,CAAA;KAAE,GAC3H,OAAO,CAAC,mBAAmB,GAAG;QAAE,SAAS,EAAE,OAAO,CAAA;KAAE,CAAC;IAalD,UAAU,CAAC,GAAG,EAAE,MAAM,EAAE,UAAU,EAAE,MAAM,GAAG,OAAO,CAAC,MAAM,GAAG,IAAI,CAAC;IAMnE,YAAY,CAAC,GAAG,EAAE,MAAM,EAAE,KAAK,EAAE,MAAM,GAAG,OAAO,CAAC,IAAI,CAAC;CAG9D"}
|
|
@@ -1,6 +1,7 @@
|
|
|
1
1
|
"use strict";
|
|
2
2
|
Object.defineProperty(exports, "__esModule", { value: true });
|
|
3
3
|
exports.RedisStore = void 0;
|
|
4
|
+
const node_crypto_1 = require("node:crypto");
|
|
4
5
|
/**
|
|
5
6
|
* Atomically increments the stored failure count and decides `isOpen` in one
|
|
6
7
|
* round trip: GET -> decode -> increment -> compute isOpen -> SET, all inside
|
|
@@ -87,20 +88,44 @@ for k, v in pairs(state) do result[k] = v end
|
|
|
87
88
|
result.openedNow = isOpen and not wasOpen
|
|
88
89
|
return cjson.encode(result)
|
|
89
90
|
`;
|
|
91
|
+
/**
|
|
92
|
+
* Compare-and-delete: releases a trial claim only if `token` still matches
|
|
93
|
+
* what's stored. Without this, releasing with a plain DEL can destroy a
|
|
94
|
+
* *different* instance's newer claim if the original trial outran its TTL
|
|
95
|
+
* and someone else has since claimed the key (see README limitation on
|
|
96
|
+
* trial-TTL expiry) — a stale release would otherwise cascade into an extra
|
|
97
|
+
* concurrent trial beyond the one that scenario already accepts by design.
|
|
98
|
+
*/
|
|
99
|
+
const RELEASE_TRIAL_SCRIPT = `
|
|
100
|
+
if redis.call('GET', KEYS[1]) == ARGV[1] then
|
|
101
|
+
return redis.call('DEL', KEYS[1])
|
|
102
|
+
else
|
|
103
|
+
return 0
|
|
104
|
+
end
|
|
105
|
+
`;
|
|
90
106
|
/**
|
|
91
107
|
* Redis-backed Store implementation. Import this from
|
|
92
108
|
* `redeye/redis-store` so the core package has no hard dependency on
|
|
93
109
|
* `ioredis` — only consumers who use RedisStore need it installed.
|
|
94
110
|
*
|
|
95
111
|
* Implements all three optional atomic capabilities (`recordFailureAtomic`
|
|
96
|
-
* and `recordOutcomeAtomic` via Lua scripts, `claimTrial` via
|
|
97
|
-
*
|
|
98
|
-
*
|
|
112
|
+
* and `recordOutcomeAtomic` via Lua scripts, `claimTrial`/`releaseTrial` via
|
|
113
|
+
* a tokened `SET ... NX` and compare-and-delete), so breakers backed by
|
|
114
|
+
* RedisStore get exact counting and real single-trial half-open behavior
|
|
115
|
+
* under both strategies instead of best-effort fallbacks.
|
|
116
|
+
*
|
|
117
|
+
* Scripts are registered once per Redis client via `defineCommand`, so
|
|
118
|
+
* ioredis sends a cached script hash (`EVALSHA`) on every call instead of
|
|
119
|
+
* the full script body, with automatic fallback to `EVAL` if the script
|
|
120
|
+
* cache is ever flushed.
|
|
99
121
|
*/
|
|
100
122
|
class RedisStore {
|
|
101
123
|
constructor(redis, options = {}) {
|
|
102
124
|
this.redis = redis;
|
|
103
125
|
this.prefix = options.keyPrefix ?? '';
|
|
126
|
+
redis.defineCommand('redeyeRecordFailure', { numberOfKeys: 1, lua: RECORD_FAILURE_SCRIPT });
|
|
127
|
+
redis.defineCommand('redeyeRecordOutcome', { numberOfKeys: 1, lua: RECORD_OUTCOME_SCRIPT });
|
|
128
|
+
redis.defineCommand('redeyeReleaseTrial', { numberOfKeys: 1, lua: RELEASE_TRIAL_SCRIPT });
|
|
104
129
|
}
|
|
105
130
|
async get(key) {
|
|
106
131
|
const raw = await this.redis.get(this.prefix + key);
|
|
@@ -115,16 +140,20 @@ class RedisStore {
|
|
|
115
140
|
await this.redis.del(this.prefix + key);
|
|
116
141
|
}
|
|
117
142
|
async recordFailureAtomic(key, opts) {
|
|
118
|
-
const raw =
|
|
143
|
+
const raw = await this.redis.redeyeRecordFailure(this.prefix + key, Math.max(1, Math.ceil(opts.ttlSeconds)), opts.failureThreshold, opts.now);
|
|
119
144
|
return JSON.parse(raw);
|
|
120
145
|
}
|
|
121
146
|
async recordOutcomeAtomic(key, opts) {
|
|
122
|
-
const raw =
|
|
147
|
+
const raw = await this.redis.redeyeRecordOutcome(this.prefix + key, opts.decay, opts.success ? 0 : 1, opts.minimumCalls, opts.errorRateThreshold, Math.max(1, Math.ceil(opts.ttlSeconds)), opts.now);
|
|
123
148
|
return JSON.parse(raw);
|
|
124
149
|
}
|
|
125
150
|
async claimTrial(key, ttlSeconds) {
|
|
126
|
-
const
|
|
127
|
-
|
|
151
|
+
const token = (0, node_crypto_1.randomUUID)();
|
|
152
|
+
const result = await this.redis.set(this.prefix + key, token, 'EX', Math.max(1, Math.ceil(ttlSeconds)), 'NX');
|
|
153
|
+
return result === 'OK' ? token : null;
|
|
154
|
+
}
|
|
155
|
+
async releaseTrial(key, token) {
|
|
156
|
+
await this.redis.redeyeReleaseTrial(this.prefix + key, token);
|
|
128
157
|
}
|
|
129
158
|
}
|
|
130
159
|
exports.RedisStore = RedisStore;
|
|
@@ -1 +1 @@
|
|
|
1
|
-
{"version":3,"file":"redis-store.js","sourceRoot":"","sources":["../../src/stores/redis-store.ts"],"names":[],"mappings":";;;
|
|
1
|
+
{"version":3,"file":"redis-store.js","sourceRoot":"","sources":["../../src/stores/redis-store.ts"],"names":[],"mappings":";;;AACA,6CAAyC;AAyBzC;;;;;;GAMG;AACH,MAAM,qBAAqB,GAAG;;;;;;;;;;;;;;;;;;;;;;;;;;;CA2B7B,CAAC;AAEF;;;;;;;GAOG;AACH,MAAM,qBAAqB,GAAG;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;CA0C7B,CAAC;AAEF;;;;;;;GAOG;AACH,MAAM,oBAAoB,GAAG;;;;;;CAM5B,CAAC;AAEF;;;;;;;;;;;;;;;GAeG;AACH,MAAa,UAAU;IAGrB,YAA6B,KAAY,EAAE,UAAkC,EAAE;QAAlD,UAAK,GAAL,KAAK,CAAO;QACvC,IAAI,CAAC,MAAM,GAAG,OAAO,CAAC,SAAS,IAAI,EAAE,CAAC;QACtC,KAAK,CAAC,aAAa,CAAC,qBAAqB,EAAE,EAAE,YAAY,EAAE,CAAC,EAAE,GAAG,EAAE,qBAAqB,EAAE,CAAC,CAAC;QAC5F,KAAK,CAAC,aAAa,CAAC,qBAAqB,EAAE,EAAE,YAAY,EAAE,CAAC,EAAE,GAAG,EAAE,qBAAqB,EAAE,CAAC,CAAC;QAC5F,KAAK,CAAC,aAAa,CAAC,oBAAoB,EAAE,EAAE,YAAY,EAAE,CAAC,EAAE,GAAG,EAAE,oBAAoB,EAAE,CAAC,CAAC;IAC5F,CAAC;IAED,KAAK,CAAC,GAAG,CAAI,GAAW;QACtB,MAAM,GAAG,GAAG,MAAM,IAAI,CAAC,KAAK,CAAC,GAAG,CAAC,IAAI,CAAC,MAAM,GAAG,GAAG,CAAC,CAAC;QACpD,IAAI,GAAG,KAAK,IAAI;YAAE,OAAO,IAAI,CAAC;QAC9B,OAAO,IAAI,CAAC,KAAK,CAAC,GAAG,CAAM,CAAC;IAC9B,CAAC;IAED,KAAK,CAAC,GAAG,CAAI,GAAW,EAAE,KAAQ,EAAE,UAAkB;QACpD,MAAM,IAAI,CAAC,KAAK,CAAC,GAAG,CAAC,IAAI,CAAC,MAAM,GAAG,GAAG,EAAE,IAAI,CAAC,SAAS,CAAC,KAAK,CAAC,EAAE,IAAI,EAAE,IAAI,CAAC,GAAG,CAAC,CAAC,EAAE,IAAI,CAAC,IAAI,CAAC,UAAU,CAAC,CAAC,CAAC,CAAC;IAC3G,CAAC;IAED,KAAK,CAAC,GAAG,CAAC,GAAW;QACnB,MAAM,IAAI,CAAC,KAAK,CAAC,GAAG,CAAC,IAAI,CAAC,MAAM,GAAG,GAAG,CAAC,CAAC;IAC1C,CAAC;IAED,KAAK,CAAC,mBAAmB,CACvB,GAAW,EACX,IAAmE;QAEnE,MAAM,GAAG,GAAG,MAAM,IAAI,CAAC,KAAK,CAAC,mBAAmB,CAC9C,IAAI,CAAC,MAAM,GAAG,GAAG,EACjB,IAAI,CAAC,GAAG,CAAC,CAAC,EAAE,IAAI,CAAC,IAAI,CAAC,IAAI,CAAC,UAAU,CAAC,CAAC,EACvC,IAAI,CAAC,gBAAgB,EACrB,IAAI,CAAC,GAAG,CACT,CAAC;QACF,OAAO,IAAI,CAAC,KAAK,CAAC,GAAG,CAAwB,CAAC;IAChD,CAAC;IAED,KAAK,CAAC,mBAAmB,CACvB,GAAW,EACX,IAA4H;QAE5H,MAAM,GAAG,GAAG,MAAM,IAAI,CAAC,KAAK,CAAC,mBAAmB,CAC9C,IAAI,CAAC,MAAM,GAAG,GAAG,EACjB,IAAI,CAAC,KAAK,EACV,IAAI,CAAC,OAAO,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EACpB,IAAI,CAAC,YAAY,EACjB,IAAI,CAAC,kBAAkB,EACvB,IAAI,CAAC,GAAG,CAAC,CAAC,EAAE,IAAI,CAAC,IAAI,CAAC,IAAI,CAAC,UAAU,CAAC,CAAC,EACvC,IAAI,CAAC,GAAG,CACT,CAAC;QACF,OAAO,IAAI,CAAC,KAAK,CAAC,GAAG,CAAiD,CAAC;IACzE,CAAC;IAED,KAAK,CAAC,UAAU,CAAC,GAAW,EAAE,UAAkB;QAC9C,MAAM,KAAK,GAAG,IAAA,wBAAU,GAAE,CAAC;QAC3B,MAAM,MAAM,GAAG,MAAM,IAAI,CAAC,KAAK,CAAC,GAAG,CAAC,IAAI,CAAC,MAAM,GAAG,GAAG,EAAE,KAAK,EAAE,IAAI,EAAE,IAAI,CAAC,GAAG,CAAC,CAAC,EAAE,IAAI,CAAC,IAAI,CAAC,UAAU,CAAC,CAAC,EAAE,IAAI,CAAC,CAAC;QAC9G,OAAO,MAAM,KAAK,IAAI,CAAC,CAAC,CAAC,KAAK,CAAC,CAAC,CAAC,IAAI,CAAC;IACxC,CAAC;IAED,KAAK,CAAC,YAAY,CAAC,GAAW,EAAE,KAAa;QAC3C,MAAM,IAAI,CAAC,KAAK,CAAC,kBAAkB,CAAC,IAAI,CAAC,MAAM,GAAG,GAAG,EAAE,KAAK,CAAC,CAAC;IAChE,CAAC;CACF;AA9DD,gCA8DC"}
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "redeye-breaker",
|
|
3
|
-
"version": "0.
|
|
3
|
+
"version": "0.2.0",
|
|
4
4
|
"description": "A circuit breaker for Node.js with an optional Redis-backed distributed mode, so breaker state can be shared across multiple app instances instead of being tracked per-process.",
|
|
5
5
|
"main": "dist/index.js",
|
|
6
6
|
"types": "dist/index.d.ts",
|
|
@@ -19,7 +19,9 @@
|
|
|
19
19
|
},
|
|
20
20
|
"typesVersions": {
|
|
21
21
|
"*": {
|
|
22
|
-
"redis-store": [
|
|
22
|
+
"redis-store": [
|
|
23
|
+
"dist/stores/redis-store.d.ts"
|
|
24
|
+
]
|
|
23
25
|
}
|
|
24
26
|
},
|
|
25
27
|
"scripts": {
|