redeye-breaker 0.3.0 → 0.4.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +261 -260
- package/dist/circuit-breaker.d.ts +36 -2
- package/dist/circuit-breaker.d.ts.map +1 -1
- package/dist/circuit-breaker.js +92 -11
- package/dist/circuit-breaker.js.map +1 -1
- package/package.json +71 -71
package/README.md
CHANGED
|
@@ -1,260 +1,261 @@
|
|
|
1
|
-
# redeye
|
|
2
|
-
|
|
3
|
-
[](https://github.com/laurells/redeye/actions/workflows/ci.yml)
|
|
4
|
-
[](https://www.npmjs.com/package/redeye-breaker)
|
|
5
|
-
[](LICENSE)
|
|
6
|
-
|
|
7
|
-
A circuit breaker for Node.js with an optional **distributed mode**: instead of tracking failures only in the memory of one process, breaker state lives in Redis (or any store you plug in), so a failure burst against a downstream dependency trips the breaker for *every* instance sharing that store — not just the one that saw the failures.
|
|
8
|
-
|
|
9
|
-
When paired with `RedisStore`, redeye is a *reliable* distributed circuit breaker, not just a best-effort one: failure counting is atomic (a Redis Lua script, not a racy get-then-set), and recovery goes through a real half-open state where exactly one instance gets to try the dependency again while everyone else stays blocked — the two properties most local-only or naively-distributed circuit breakers skip.
|
|
10
|
-
|
|
11
|
-
**Read [Reliability model & limitations](#reliability-model--limitations) before using this for anything load-bearing.** A handful of tradeoffs are fundamental to any distributed system (trusting your store, tolerating clock skew) and no library can engineer those away — that section is honest about exactly which ones those are, and which ones redeye actually solves.
|
|
12
|
-
|
|
13
|
-
## Install
|
|
14
|
-
|
|
15
|
-
```bash
|
|
16
|
-
npm install redeye-breaker
|
|
17
|
-
```
|
|
18
|
-
|
|
19
|
-
Redis support is an optional peer dependency — only needed if you use `RedisStore`:
|
|
20
|
-
|
|
21
|
-
```bash
|
|
22
|
-
npm install ioredis
|
|
23
|
-
```
|
|
24
|
-
|
|
25
|
-
## Usage
|
|
26
|
-
|
|
27
|
-
### Local mode (default, no dependencies)
|
|
28
|
-
|
|
29
|
-
```ts
|
|
30
|
-
import { CircuitBreaker } from 'redeye-breaker';
|
|
31
|
-
|
|
32
|
-
const breaker = new CircuitBreaker({
|
|
33
|
-
failureThreshold: 5,
|
|
34
|
-
resetTimeout: 60_000,
|
|
35
|
-
});
|
|
36
|
-
|
|
37
|
-
const result = await breaker.execute('payment-gateway', () => callPaymentGateway());
|
|
38
|
-
```
|
|
39
|
-
|
|
40
|
-
Local mode gets real half-open (single in-flight trial) and backoff+jitter too — it's single-process, so there's no atomicity concern to begin with.
|
|
41
|
-
|
|
42
|
-
### Local mode vs. distributed mode: which do you need?
|
|
43
|
-
|
|
44
|
-
Local mode is not the "lesser" option — for a lot of services, N independent per-instance breakers is the right answer, not a compromise:
|
|
45
|
-
|
|
46
|
-
- **Use local mode** when each instance tripping independently is fine, or even preferable: instances see meaningfully different traffic, the dependency is instance-local anyway (a sidecar, a per-instance cache), or you'd rather each instance protect itself than take on a new piece of shared infrastructure (the store) that can itself fail. Zero dependencies, zero added latency per call, nothing extra to operate.
|
|
47
|
-
- **Use distributed mode** when a failure burst against a *shared* downstream dependency should trip the breaker for the whole fleet at once — e.g. a payment gateway or third-party API, where 5 of 6 instances continuing to hammer a dependency the 6th just found dead is actively harmful (wasted capacity, a worse incident, a slower recovery signal). This costs something real in return: a store round trip on every gated decision (see [item 1](#whats-still-a-fundamental-tradeoff-not-a-bug) below), and a new dependency whose own outages now shape the breaker's behavior (see [Total coordination-layer outage](#total-coordination-layer-outage-the-store-itself-is-down)).
|
|
48
|
-
|
|
49
|
-
If you're not sure, start local — it's strictly cheaper, and nothing about switching to distributed mode later changes your call sites.
|
|
50
|
-
|
|
51
|
-
### Two tripping strategies
|
|
52
|
-
|
|
53
|
-
```ts
|
|
54
|
-
// Default: trips after N failures in a row. Good for hard drops (a
|
|
55
|
-
// dependency going fully down). Resets to zero on any success, so it
|
|
56
|
-
// will not catch a dependency that's merely degraded.
|
|
57
|
-
new CircuitBreaker({ strategy: 'consecutive', failureThreshold: 5 });
|
|
58
|
-
|
|
59
|
-
// Trips when an EWMA-smoothed failure rate crosses a threshold, once
|
|
60
|
-
// enough samples have been seen. Catches flapping/degrading dependencies
|
|
61
|
-
// a consecutive-failure breaker structurally cannot: e.g. an API that
|
|
62
|
-
// fails 4 out of every 5 requests (80% failure rate) never fails twice
|
|
63
|
-
// in a row in that exact pattern, so 'consecutive' never trips — but
|
|
64
|
-
// 'errorRate' does, because it looks at the rate, not the streak.
|
|
65
|
-
new CircuitBreaker({
|
|
66
|
-
strategy: 'errorRate',
|
|
67
|
-
errorRateThreshold: 0.5, // open at >= 50% failure rate
|
|
68
|
-
minimumCalls: 10, // ...but only once we've seen at least 10 calls
|
|
69
|
-
errorRateDecay: 0.9, // how much weight recent calls get vs. history
|
|
70
|
-
});
|
|
71
|
-
```
|
|
72
|
-
|
|
73
|
-
Pick `consecutive` when you mainly care about clean outages, `errorRate` when the dependency is more likely to degrade than to go fully dark. You can run two breaker *instances* with different strategies over the same logical operation for both kinds of protection at once — in distributed mode, give each its own `RedisStore` `keyPrefix` (or the two breakers will read/write the same Redis key with incompatible state shapes and corrupt each other).
|
|
74
|
-
|
|
75
|
-
### Distributed mode (Redis-backed, fully atomic)
|
|
76
|
-
|
|
77
|
-
```ts
|
|
78
|
-
import { CircuitBreaker } from 'redeye-breaker';
|
|
79
|
-
import { RedisStore } from 'redeye-breaker/redis-store';
|
|
80
|
-
import Redis from 'ioredis';
|
|
81
|
-
|
|
82
|
-
const redis = new Redis(process.env.REDIS_URL);
|
|
83
|
-
const store = new RedisStore(redis, { keyPrefix: 'myapp:' });
|
|
84
|
-
|
|
85
|
-
const breaker = new CircuitBreaker({
|
|
86
|
-
failureThreshold: 5,
|
|
87
|
-
resetTimeout: 60_000,
|
|
88
|
-
store, // <- presence of a store is what enables distributed mode
|
|
89
|
-
onStateChange: (state, operation) => {
|
|
90
|
-
console.warn(`circuit breaker for ${operation} is now ${state}`);
|
|
91
|
-
},
|
|
92
|
-
onStoreError: (error, operation) => {
|
|
93
|
-
// Redis unreachable, timed out, etc. — see "Store unavailability" below.
|
|
94
|
-
console.error(`circuit breaker store error for ${operation}`, error);
|
|
95
|
-
},
|
|
96
|
-
});
|
|
97
|
-
|
|
98
|
-
await breaker.execute('payment-gateway', () => callPaymentGateway());
|
|
99
|
-
```
|
|
100
|
-
|
|
101
|
-
Every process pointed at the same Redis instance (and using the same operation name / key prefix) shares breaker state, with atomic counting and single-trial recovery.
|
|
102
|
-
|
|
103
|
-
`RedisStore` stores state under `{keyPrefix}circuit_breaker:{operation}`, plus `{keyPrefix}circuit_breaker:{operation}:trial` while a half-open trial is in flight. `keyPrefix` defaults to `''` (no prefix) — set one (as above) if you share a Redis instance/DB across services and want your keys namespaced.
|
|
104
|
-
|
|
105
|
-
### Bring your own store
|
|
106
|
-
|
|
107
|
-
`RedisStore` implements a `Store` interface with two required methods and four *optional* ones:
|
|
108
|
-
|
|
109
|
-
```ts
|
|
110
|
-
export interface Store {
|
|
111
|
-
get<T>(key: string): Promise<T | null>;
|
|
112
|
-
set<T>(key: string, value: T, ttlSeconds: number): Promise<void>;
|
|
113
|
-
del(key: string): Promise<void>;
|
|
114
|
-
|
|
115
|
-
// Optional — implement these for the reliability guarantees below.
|
|
116
|
-
// Without them, redeye falls back to best-effort semantics and logs a
|
|
117
|
-
// one-time warning telling you exactly what's degraded.
|
|
118
|
-
recordFailureAtomic?(key: string, opts: { ttlSeconds: number; failureThreshold: number; now: number }): Promise<CircuitBreakerState>;
|
|
119
|
-
recordOutcomeAtomic?(key: string, opts: { success: boolean; decay: number; minimumCalls: number; errorRateThreshold: number; ttlSeconds: number; now: number }): Promise<CircuitBreakerState & { openedNow: boolean }>;
|
|
120
|
-
claimTrial?(key: string, ttlSeconds: number): Promise<string | null>;
|
|
121
|
-
releaseTrial?(key: string, token: string): Promise<void>;
|
|
122
|
-
}
|
|
123
|
-
```
|
|
124
|
-
|
|
125
|
-
- `recordFailureAtomic` (`strategy: 'consecutive'`) should increment the failure count and decide `isOpen` in one atomic round trip (a Lua script in Redis, a conditional update in DynamoDB, etc.).
|
|
126
|
-
- `recordOutcomeAtomic` (`strategy: 'errorRate'`) should fold one call's outcome into the EWMA rate and decide `isOpen` in one atomic round trip — called on every closed-phase call, not just failures.
|
|
127
|
-
- `claimTrial` should be a conditional "create if absent" write (`SET key token NX EX ttl` in Redis, with `token` a fresh unique value per call, e.g. a UUID) — it's what makes half-open recovery exclusive to one caller instead of a free-for-all. Return the token you wrote if this call won the claim, or `null` if the key was already taken.
|
|
128
|
-
- `releaseTrial` should be a compare-and-delete (`if GET key == token then DEL key` in Redis, a conditional delete elsewhere), not an unconditional delete. This is what stops a trial that outran its TTL (see [item 11](#whats-still-a-fundamental-tradeoff-not-a-bug)) from deleting a *different* instance's newer claim on the same key when it finally gets around to releasing its own now-stale one. If omitted (or if a call has no confirmed ownership token to release at all — e.g. `claimTrial` itself errored), redeye logs a one-time warning and leaves the claim for its TTL to expire naturally, rather than risk an unconditional delete that could destroy a claim it never confirmed was its own.
|
|
129
|
-
|
|
130
|
-
Implement the two required methods against Memcached, DynamoDB, your own cache wrapper, etc., and you have a working (best-effort) distributed breaker. Add the optional methods relevant to the strategy you use when that store supports a real atomic increment and a real conditional write, and you get the full reliability guarantees.
|
|
131
|
-
|
|
132
|
-
## API
|
|
133
|
-
|
|
134
|
-
- `execute(operation, fn)` — runs `fn` if the breaker allows it (closed, or this call won the half-open trial); throws immediately without calling `fn` otherwise. Records the outcome automatically.
|
|
135
|
-
- `canExecute(operation)` — synchronous, read-only, best-effort check.
|
|
136
|
-
- `canExecuteAsync(operation)` — async, store-aware, read-only check. Honors `failOpenOnStoreError`. Never claims a trial slot — see the TOCTOU note below.
|
|
137
|
-
- `recordFailure(operation)` / `recordSuccess(operation)` — manually record an outcome without routing the call through `execute`. Note: these bypass the half-open trial-claim mechanism entirely (see [limitations](#reliability-model--limitations)) — prefer `execute`.
|
|
138
|
-
- `getState(operation)` — returns `{ failures, lastFailure, isOpen, openCount, errorRate, sampleCount }`. `openCount` is how many consecutive half-open trials have failed since the breaker last fully closed (drives backoff). `errorRate`/`sampleCount` are only meaningful under `strategy: 'errorRate'`.
|
|
139
|
-
- `getMetrics(operation)` / `getAllMetrics()` — per-instance counters: `{ totalCalls, totalSuccesses, totalFailures, totalRejections, totalStoreErrors }`.
|
|
140
|
-
- `reset(operation)` — manually closes the breaker for an operation.
|
|
141
|
-
- `destroy()` — stops the internal monitor interval. Call this when a breaker instance is no longer needed (e.g. in tests, or on service shutdown) to avoid leaking a timer.
|
|
142
|
-
|
|
143
|
-
`operation` should be a small, fixed set of names (`'payment-gateway'`, `'fraud-api'`, one per dependency) — not something you interpolate per-request values into (a tenant ID, a URL, a user ID). Every distinct `operation` string gets its own entry in several per-instance maps (metrics, local-mode failure/backoff state, an internal jitter cache) that are never evicted, so a dynamic operation name is an unbounded memory leak, the same footgun as labeling a Prometheus metric with a high-cardinality value.
|
|
144
|
-
|
|
145
|
-
redeye can't stop you from doing this, but it can flag it: `execute`/`recordFailure`/`recordSuccess` log a one-time warning the first time they see an `operation` name over 100 characters or containing `/` (both common signs of an interpolated URL or ID), and — if you set `maxOperations` — a second one-time warning once the breaker has seen more distinct operation names than that. Neither check ever rejects a call; they're a smoke alarm, not enforcement.
|
|
146
|
-
|
|
147
|
-
## Options
|
|
148
|
-
|
|
149
|
-
| Option | Default | Description |
|
|
150
|
-
|---|---|---|
|
|
151
|
-
| `strategy` | `'consecutive'` | `'consecutive'` or `'errorRate'` — see [Two tripping strategies](#two-tripping-strategies) |
|
|
152
|
-
| `failureThreshold` | `5` | `'consecutive'` only: failures in a row before the breaker opens |
|
|
153
|
-
| `errorRateThreshold` | `0.5` | `'errorRate'` only: failure rate (0-1) at or above which the breaker opens |
|
|
154
|
-
| `minimumCalls` | `10` | `'errorRate'` only: minimum samples before the rate can trip the breaker |
|
|
155
|
-
| `errorRateDecay` | `0.9` | `'errorRate'` only: EWMA decay factor — closer to 1 weighs history more heavily |
|
|
156
|
-
| `resetTimeout` | `60000` | ms to stay open before allowing a half-open trial |
|
|
157
|
-
| `backoffMultiplier` | `2` | Multiplies `resetTimeout` on each failed trial (`1` disables backoff) |
|
|
158
|
-
| `maxResetTimeout` | `resetTimeout * 8` | Ceiling for the backed-off reset timeout |
|
|
159
|
-
| `jitter` | `0.1` | Randomizes the effective reset timeout by ±this fraction, so instances don't all retry in lockstep |
|
|
160
|
-
| `trialTimeout` | `min(timeout ?? 10000, resetTimeout)` | Max ms a claimed half-open trial may run before its claim is released/expires |
|
|
161
|
-
| `monitorInterval` | `5000` | ms between local-mode sweeps for a trial that never settled (safety net) |
|
|
162
|
-
| `timeout` | none | optional per-call timeout in ms |
|
|
163
|
-
| `store` | none | enables distributed mode when provided |
|
|
164
|
-
| `failOpenOnStoreError` | `true` | see [Store unavailability](#store-unavailability-fail-open-vs-fail-closed) |
|
|
165
|
-
| `onStateChange` | none | `(state: 'open' \| 'half-open' \| 'closed', operation: string) => void` — `'half-open'` fires exactly when a caller claims the single half-open trial slot. In distributed mode this is a per-process callback, not a fleet-wide broadcast — see [Observability](#what-redeye-actually-solves-with-redisstore-or-any-store-implementing-the-matching-optional-methods) below for which instance actually sees it. |
|
|
166
|
-
| `onStoreError` | none | `(error: unknown, operation: string) => void` — fired whenever a store read/write fails, in addition to being logged |
|
|
167
|
-
| `logger` | no-op | `{ warn(msg), log(msg) }` — plug in your own logger |
|
|
168
|
-
| `maxOperations` | none | Logs a one-time warning once the breaker has seen more than this many distinct `operation` names — see the cardinality note above. Unset: no limit, no warning. |
|
|
169
|
-
|
|
170
|
-
|
|
171
|
-
|
|
172
|
-
|
|
173
|
-
|
|
174
|
-
|
|
175
|
-
- **`
|
|
176
|
-
|
|
177
|
-
|
|
178
|
-
|
|
179
|
-
|
|
180
|
-
|
|
181
|
-
|
|
182
|
-
|
|
183
|
-
|
|
184
|
-
- **`
|
|
185
|
-
|
|
186
|
-
|
|
187
|
-
|
|
188
|
-
|
|
189
|
-
|
|
190
|
-
|
|
191
|
-
|
|
192
|
-
|
|
193
|
-
|
|
194
|
-
|
|
195
|
-
- **
|
|
196
|
-
- **
|
|
197
|
-
- **
|
|
198
|
-
- **
|
|
199
|
-
- **
|
|
200
|
-
- **
|
|
201
|
-
|
|
202
|
-
|
|
203
|
-
|
|
204
|
-
|
|
205
|
-
|
|
206
|
-
|
|
207
|
-
|
|
208
|
-
|
|
209
|
-
- **
|
|
210
|
-
|
|
211
|
-
|
|
212
|
-
|
|
213
|
-
|
|
214
|
-
- **
|
|
215
|
-
- **A
|
|
216
|
-
- **
|
|
217
|
-
- **
|
|
218
|
-
|
|
219
|
-
|
|
220
|
-
|
|
221
|
-
|
|
222
|
-
|
|
223
|
-
|
|
224
|
-
|
|
225
|
-
|
|
226
|
-
|
|
227
|
-
|
|
228
|
-
|
|
229
|
-
|
|
230
|
-
|
|
231
|
-
|
|
232
|
-
|
|
233
|
-
|
|
234
|
-
|
|
235
|
-
|
|
236
|
-
|
|
237
|
-
|
|
238
|
-
|
|
239
|
-
|
|
240
|
-
|
|
241
|
-
|
|
242
|
-
|
|
243
|
-
|
|
244
|
-
|
|
245
|
-
|
|
246
|
-
|
|
247
|
-
|
|
248
|
-
|
|
249
|
-
|
|
250
|
-
|
|
251
|
-
|
|
252
|
-
|
|
253
|
-
|
|
254
|
-
|
|
255
|
-
|
|
256
|
-
|
|
257
|
-
|
|
258
|
-
|
|
259
|
-
|
|
260
|
-
|
|
1
|
+
# redeye
|
|
2
|
+
|
|
3
|
+
[](https://github.com/laurells/redeye/actions/workflows/ci.yml)
|
|
4
|
+
[](https://www.npmjs.com/package/redeye-breaker)
|
|
5
|
+
[](LICENSE)
|
|
6
|
+
|
|
7
|
+
A circuit breaker for Node.js with an optional **distributed mode**: instead of tracking failures only in the memory of one process, breaker state lives in Redis (or any store you plug in), so a failure burst against a downstream dependency trips the breaker for *every* instance sharing that store — not just the one that saw the failures.
|
|
8
|
+
|
|
9
|
+
When paired with `RedisStore`, redeye is a *reliable* distributed circuit breaker, not just a best-effort one: failure counting is atomic (a Redis Lua script, not a racy get-then-set), and recovery goes through a real half-open state where exactly one instance gets to try the dependency again while everyone else stays blocked — the two properties most local-only or naively-distributed circuit breakers skip.
|
|
10
|
+
|
|
11
|
+
**Read [Reliability model & limitations](#reliability-model--limitations) before using this for anything load-bearing.** A handful of tradeoffs are fundamental to any distributed system (trusting your store, tolerating clock skew) and no library can engineer those away — that section is honest about exactly which ones those are, and which ones redeye actually solves.
|
|
12
|
+
|
|
13
|
+
## Install
|
|
14
|
+
|
|
15
|
+
```bash
|
|
16
|
+
npm install redeye-breaker
|
|
17
|
+
```
|
|
18
|
+
|
|
19
|
+
Redis support is an optional peer dependency — only needed if you use `RedisStore`:
|
|
20
|
+
|
|
21
|
+
```bash
|
|
22
|
+
npm install ioredis
|
|
23
|
+
```
|
|
24
|
+
|
|
25
|
+
## Usage
|
|
26
|
+
|
|
27
|
+
### Local mode (default, no dependencies)
|
|
28
|
+
|
|
29
|
+
```ts
|
|
30
|
+
import { CircuitBreaker } from 'redeye-breaker';
|
|
31
|
+
|
|
32
|
+
const breaker = new CircuitBreaker({
|
|
33
|
+
failureThreshold: 5,
|
|
34
|
+
resetTimeout: 60_000,
|
|
35
|
+
});
|
|
36
|
+
|
|
37
|
+
const result = await breaker.execute('payment-gateway', () => callPaymentGateway());
|
|
38
|
+
```
|
|
39
|
+
|
|
40
|
+
Local mode gets real half-open (single in-flight trial) and backoff+jitter too — it's single-process, so there's no atomicity concern to begin with.
|
|
41
|
+
|
|
42
|
+
### Local mode vs. distributed mode: which do you need?
|
|
43
|
+
|
|
44
|
+
Local mode is not the "lesser" option — for a lot of services, N independent per-instance breakers is the right answer, not a compromise:
|
|
45
|
+
|
|
46
|
+
- **Use local mode** when each instance tripping independently is fine, or even preferable: instances see meaningfully different traffic, the dependency is instance-local anyway (a sidecar, a per-instance cache), or you'd rather each instance protect itself than take on a new piece of shared infrastructure (the store) that can itself fail. Zero dependencies, zero added latency per call, nothing extra to operate.
|
|
47
|
+
- **Use distributed mode** when a failure burst against a *shared* downstream dependency should trip the breaker for the whole fleet at once — e.g. a payment gateway or third-party API, where 5 of 6 instances continuing to hammer a dependency the 6th just found dead is actively harmful (wasted capacity, a worse incident, a slower recovery signal). This costs something real in return: a store round trip on every gated decision (see [item 1](#whats-still-a-fundamental-tradeoff-not-a-bug) below), and a new dependency whose own outages now shape the breaker's behavior (see [Total coordination-layer outage](#total-coordination-layer-outage-the-store-itself-is-down)).
|
|
48
|
+
|
|
49
|
+
If you're not sure, start local — it's strictly cheaper, and nothing about switching to distributed mode later changes your call sites.
|
|
50
|
+
|
|
51
|
+
### Two tripping strategies
|
|
52
|
+
|
|
53
|
+
```ts
|
|
54
|
+
// Default: trips after N failures in a row. Good for hard drops (a
|
|
55
|
+
// dependency going fully down). Resets to zero on any success, so it
|
|
56
|
+
// will not catch a dependency that's merely degraded.
|
|
57
|
+
new CircuitBreaker({ strategy: 'consecutive', failureThreshold: 5 });
|
|
58
|
+
|
|
59
|
+
// Trips when an EWMA-smoothed failure rate crosses a threshold, once
|
|
60
|
+
// enough samples have been seen. Catches flapping/degrading dependencies
|
|
61
|
+
// a consecutive-failure breaker structurally cannot: e.g. an API that
|
|
62
|
+
// fails 4 out of every 5 requests (80% failure rate) never fails twice
|
|
63
|
+
// in a row in that exact pattern, so 'consecutive' never trips — but
|
|
64
|
+
// 'errorRate' does, because it looks at the rate, not the streak.
|
|
65
|
+
new CircuitBreaker({
|
|
66
|
+
strategy: 'errorRate',
|
|
67
|
+
errorRateThreshold: 0.5, // open at >= 50% failure rate
|
|
68
|
+
minimumCalls: 10, // ...but only once we've seen at least 10 calls
|
|
69
|
+
errorRateDecay: 0.9, // how much weight recent calls get vs. history
|
|
70
|
+
});
|
|
71
|
+
```
|
|
72
|
+
|
|
73
|
+
Pick `consecutive` when you mainly care about clean outages, `errorRate` when the dependency is more likely to degrade than to go fully dark. You can run two breaker *instances* with different strategies over the same logical operation for both kinds of protection at once — in distributed mode, give each its own `RedisStore` `keyPrefix` (or the two breakers will read/write the same Redis key with incompatible state shapes and corrupt each other).
|
|
74
|
+
|
|
75
|
+
### Distributed mode (Redis-backed, fully atomic)
|
|
76
|
+
|
|
77
|
+
```ts
|
|
78
|
+
import { CircuitBreaker } from 'redeye-breaker';
|
|
79
|
+
import { RedisStore } from 'redeye-breaker/redis-store';
|
|
80
|
+
import Redis from 'ioredis';
|
|
81
|
+
|
|
82
|
+
const redis = new Redis(process.env.REDIS_URL);
|
|
83
|
+
const store = new RedisStore(redis, { keyPrefix: 'myapp:' });
|
|
84
|
+
|
|
85
|
+
const breaker = new CircuitBreaker({
|
|
86
|
+
failureThreshold: 5,
|
|
87
|
+
resetTimeout: 60_000,
|
|
88
|
+
store, // <- presence of a store is what enables distributed mode
|
|
89
|
+
onStateChange: (state, operation) => {
|
|
90
|
+
console.warn(`circuit breaker for ${operation} is now ${state}`);
|
|
91
|
+
},
|
|
92
|
+
onStoreError: (error, operation) => {
|
|
93
|
+
// Redis unreachable, timed out, etc. — see "Store unavailability" below.
|
|
94
|
+
console.error(`circuit breaker store error for ${operation}`, error);
|
|
95
|
+
},
|
|
96
|
+
});
|
|
97
|
+
|
|
98
|
+
await breaker.execute('payment-gateway', () => callPaymentGateway());
|
|
99
|
+
```
|
|
100
|
+
|
|
101
|
+
Every process pointed at the same Redis instance (and using the same operation name / key prefix) shares breaker state, with atomic counting and single-trial recovery.
|
|
102
|
+
|
|
103
|
+
`RedisStore` stores state under `{keyPrefix}circuit_breaker:{operation}`, plus `{keyPrefix}circuit_breaker:{operation}:trial` while a half-open trial is in flight. `keyPrefix` defaults to `''` (no prefix) — set one (as above) if you share a Redis instance/DB across services and want your keys namespaced.
|
|
104
|
+
|
|
105
|
+
### Bring your own store
|
|
106
|
+
|
|
107
|
+
`RedisStore` implements a `Store` interface with two required methods and four *optional* ones:
|
|
108
|
+
|
|
109
|
+
```ts
|
|
110
|
+
export interface Store {
|
|
111
|
+
get<T>(key: string): Promise<T | null>;
|
|
112
|
+
set<T>(key: string, value: T, ttlSeconds: number): Promise<void>;
|
|
113
|
+
del(key: string): Promise<void>;
|
|
114
|
+
|
|
115
|
+
// Optional — implement these for the reliability guarantees below.
|
|
116
|
+
// Without them, redeye falls back to best-effort semantics and logs a
|
|
117
|
+
// one-time warning telling you exactly what's degraded.
|
|
118
|
+
recordFailureAtomic?(key: string, opts: { ttlSeconds: number; failureThreshold: number; now: number }): Promise<CircuitBreakerState>;
|
|
119
|
+
recordOutcomeAtomic?(key: string, opts: { success: boolean; decay: number; minimumCalls: number; errorRateThreshold: number; ttlSeconds: number; now: number }): Promise<CircuitBreakerState & { openedNow: boolean }>;
|
|
120
|
+
claimTrial?(key: string, ttlSeconds: number): Promise<string | null>;
|
|
121
|
+
releaseTrial?(key: string, token: string): Promise<void>;
|
|
122
|
+
}
|
|
123
|
+
```
|
|
124
|
+
|
|
125
|
+
- `recordFailureAtomic` (`strategy: 'consecutive'`) should increment the failure count and decide `isOpen` in one atomic round trip (a Lua script in Redis, a conditional update in DynamoDB, etc.).
|
|
126
|
+
- `recordOutcomeAtomic` (`strategy: 'errorRate'`) should fold one call's outcome into the EWMA rate and decide `isOpen` in one atomic round trip — called on every closed-phase call, not just failures.
|
|
127
|
+
- `claimTrial` should be a conditional "create if absent" write (`SET key token NX EX ttl` in Redis, with `token` a fresh unique value per call, e.g. a UUID) — it's what makes half-open recovery exclusive to one caller instead of a free-for-all. Return the token you wrote if this call won the claim, or `null` if the key was already taken.
|
|
128
|
+
- `releaseTrial` should be a compare-and-delete (`if GET key == token then DEL key` in Redis, a conditional delete elsewhere), not an unconditional delete. This is what stops a trial that outran its TTL (see [item 11](#whats-still-a-fundamental-tradeoff-not-a-bug)) from deleting a *different* instance's newer claim on the same key when it finally gets around to releasing its own now-stale one. If omitted (or if a call has no confirmed ownership token to release at all — e.g. `claimTrial` itself errored), redeye logs a one-time warning and leaves the claim for its TTL to expire naturally, rather than risk an unconditional delete that could destroy a claim it never confirmed was its own.
|
|
129
|
+
|
|
130
|
+
Implement the two required methods against Memcached, DynamoDB, your own cache wrapper, etc., and you have a working (best-effort) distributed breaker. Add the optional methods relevant to the strategy you use when that store supports a real atomic increment and a real conditional write, and you get the full reliability guarantees.
|
|
131
|
+
|
|
132
|
+
## API
|
|
133
|
+
|
|
134
|
+
- `execute(operation, fn)` — runs `fn` if the breaker allows it (closed, or this call won the half-open trial); throws immediately without calling `fn` otherwise. Records the outcome automatically.
|
|
135
|
+
- `canExecute(operation)` — synchronous, read-only, best-effort check. In distributed mode it can't await the store, so it can only consult the local open-state cache (see `openCacheRefreshMs` below): a `false` is a real, cached-open result, but a `true` is still only advisory — it means either nothing is cached open, or nothing has been cached yet, not a confirmed closed state — use `canExecuteAsync` for that. Never claims a trial slot. Logs a one-time warning the first time it's called in distributed mode, explaining that asymmetry.
|
|
136
|
+
- `canExecuteAsync(operation)` — async, store-aware, read-only check. Honors `failOpenOnStoreError`. Never claims a trial slot — see the TOCTOU note below.
|
|
137
|
+
- `recordFailure(operation)` / `recordSuccess(operation)` — manually record an outcome without routing the call through `execute`. Note: these bypass the half-open trial-claim mechanism entirely (see [limitations](#reliability-model--limitations)) — prefer `execute`.
|
|
138
|
+
- `getState(operation)` — returns `{ failures, lastFailure, isOpen, openCount, errorRate, sampleCount }`. `openCount` is how many consecutive half-open trials have failed since the breaker last fully closed (drives backoff). `errorRate`/`sampleCount` are only meaningful under `strategy: 'errorRate'`.
|
|
139
|
+
- `getMetrics(operation)` / `getAllMetrics()` — per-instance counters: `{ totalCalls, totalSuccesses, totalFailures, totalRejections, totalStoreErrors }`.
|
|
140
|
+
- `reset(operation)` — manually closes the breaker for an operation.
|
|
141
|
+
- `destroy()` — stops the internal monitor interval. Call this when a breaker instance is no longer needed (e.g. in tests, or on service shutdown) to avoid leaking a timer.
|
|
142
|
+
|
|
143
|
+
`operation` should be a small, fixed set of names (`'payment-gateway'`, `'fraud-api'`, one per dependency) — not something you interpolate per-request values into (a tenant ID, a URL, a user ID). Every distinct `operation` string gets its own entry in several per-instance maps (metrics, local-mode failure/backoff state, an internal jitter cache) that are never evicted, so a dynamic operation name is an unbounded memory leak, the same footgun as labeling a Prometheus metric with a high-cardinality value.
|
|
144
|
+
|
|
145
|
+
redeye can't stop you from doing this, but it can flag it: `execute`/`recordFailure`/`recordSuccess` log a one-time warning the first time they see an `operation` name over 100 characters or containing `/` (both common signs of an interpolated URL or ID), and — if you set `maxOperations` — a second one-time warning once the breaker has seen more distinct operation names than that. Neither check ever rejects a call; they're a smoke alarm, not enforcement.
|
|
146
|
+
|
|
147
|
+
## Options
|
|
148
|
+
|
|
149
|
+
| Option | Default | Description |
|
|
150
|
+
|---|---|---|
|
|
151
|
+
| `strategy` | `'consecutive'` | `'consecutive'` or `'errorRate'` — see [Two tripping strategies](#two-tripping-strategies) |
|
|
152
|
+
| `failureThreshold` | `5` | `'consecutive'` only: failures in a row before the breaker opens |
|
|
153
|
+
| `errorRateThreshold` | `0.5` | `'errorRate'` only: failure rate (0-1) at or above which the breaker opens |
|
|
154
|
+
| `minimumCalls` | `10` | `'errorRate'` only: minimum samples before the rate can trip the breaker |
|
|
155
|
+
| `errorRateDecay` | `0.9` | `'errorRate'` only: EWMA decay factor — closer to 1 weighs history more heavily |
|
|
156
|
+
| `resetTimeout` | `60000` | ms to stay open before allowing a half-open trial |
|
|
157
|
+
| `backoffMultiplier` | `2` | Multiplies `resetTimeout` on each failed trial (`1` disables backoff) |
|
|
158
|
+
| `maxResetTimeout` | `resetTimeout * 8` | Ceiling for the backed-off reset timeout |
|
|
159
|
+
| `jitter` | `0.1` | Randomizes the effective reset timeout by ±this fraction, so instances don't all retry in lockstep |
|
|
160
|
+
| `trialTimeout` | `min(timeout ?? 10000, resetTimeout)` | Max ms a claimed half-open trial may run before its claim is released/expires |
|
|
161
|
+
| `monitorInterval` | `5000` | ms between local-mode sweeps for a trial that never settled (safety net) |
|
|
162
|
+
| `timeout` | none | optional per-call timeout in ms |
|
|
163
|
+
| `store` | none | enables distributed mode when provided |
|
|
164
|
+
| `failOpenOnStoreError` | `true` | see [Store unavailability](#store-unavailability-fail-open-vs-fail-closed) |
|
|
165
|
+
| `onStateChange` | none | `(state: 'open' \| 'half-open' \| 'closed', operation: string) => void` — `'half-open'` fires exactly when a caller claims the single half-open trial slot. In distributed mode this is a per-process callback, not a fleet-wide broadcast — see [Observability](#what-redeye-actually-solves-with-redisstore-or-any-store-implementing-the-matching-optional-methods) below for which instance actually sees it. |
|
|
166
|
+
| `onStoreError` | none | `(error: unknown, operation: string) => void` — fired whenever a store read/write fails, in addition to being logged |
|
|
167
|
+
| `logger` | no-op | `{ warn(msg), log(msg) }` — plug in your own logger |
|
|
168
|
+
| `maxOperations` | none | Logs a one-time warning once the breaker has seen more than this many distinct `operation` names — see the cardinality note above. Unset: no limit, no warning. |
|
|
169
|
+
| `openCacheRefreshMs` | `2000` | Distributed mode only: while `operation` is known-open, reject locally without a store read, re-verifying against the store at most once per this many ms (to catch an early close by another instance's trial, or a manual `reset()`). Never delays trial eligibility — see [limitation item 1](#whats-still-a-fundamental-tradeoff-not-a-bug). `0` disables the cache (a store read on every call, the pre-cache behavior). |
|
|
170
|
+
|
|
171
|
+
### Store unavailability: fail-open vs fail-closed
|
|
172
|
+
|
|
173
|
+
If the store itself throws (Redis is down, times out, network partition, etc.), redeye does **not** treat that the same as the protected operation failing — it's a distinct condition, controlled by `failOpenOnStoreError`:
|
|
174
|
+
|
|
175
|
+
- **`true` (default)** — treat the breaker as closed and let the call through. A store outage degrades you to "no circuit-breaker protection," not "every call blocked."
|
|
176
|
+
- **`false`** — throw `StoreUnavailableError` (exported) without calling the wrapped function at all. Appropriate when the protected operation is expensive, dangerous to retry blindly, or would itself add load to whatever's already causing the outage.
|
|
177
|
+
|
|
178
|
+
Writes back to the store are **always** best-effort — a write failure is logged and reported via `onStoreError`, but never thrown, and never overrides the real result of a call that already happened.
|
|
179
|
+
|
|
180
|
+
### Total coordination-layer outage (the store itself is down)
|
|
181
|
+
|
|
182
|
+
Same `failOpenOnStoreError` branch as above, just hit by every instance at once instead of one partitioned instance — redeye doesn't promote a backup coordinator or fall back to per-instance local tracking as a substitute:
|
|
183
|
+
|
|
184
|
+
- **`true` (default):** every instance lets every call through — no breaking happens anywhere, fleet-wide, for as long as the outage lasts.
|
|
185
|
+
- **`false`:** every instance blocks every call with `StoreUnavailableError`, fleet-wide, including requests that would have succeeded.
|
|
186
|
+
|
|
187
|
+
Nothing is buffered or replayed: outcomes during the outage are never recorded, and the breaker has no memory of the blackout once the store recovers. There's also no back-off from hitting a store known to be down — every guarded call keeps retrying it, so pair `failOpenOnStoreError: false` with fail-fast client options (`ioredis`'s `maxRetriesPerRequest`, `enableOfflineQueue: false`) if you don't want calls queuing on the client's own reconnect logic. Recovery is instant and automatic on the next successful read — no warm-up, no manual reset.
|
|
188
|
+
|
|
189
|
+
High availability of the store itself (Sentinel, Cluster) is your responsibility, configured on the client you hand to `RedisStore` — whatever failover consistency that setup provides is inherited as-is (see [Split-brain](#split-brain-whats-prevented-and-what-isnt)).
|
|
190
|
+
|
|
191
|
+
## Reliability model & limitations
|
|
192
|
+
|
|
193
|
+
### What redeye actually solves (with `RedisStore`, or any store implementing the matching optional methods)
|
|
194
|
+
|
|
195
|
+
- **Exact counting under concurrent failure, for both strategies.** `recordFailureAtomic` (consecutive) and `recordOutcomeAtomic` (errorRate) each run as a single Lua script inside Redis's single-threaded execution — two instances updating at the same instant cannot race and lose an update the way a plain get-then-set would.
|
|
196
|
+
- **Catches flapping, not just hard drops.** `strategy: 'errorRate'` trips on a smoothed failure *rate* once enough samples are seen, so a dependency that succeeds 1 in 5 requests (an 80% failure rate) still trips the breaker even though it never fails N times consecutively — a case `'consecutive'` structurally cannot catch, by design (see [Two tripping strategies](#two-tripping-strategies)).
|
|
197
|
+
- **Real half-open, single-trial recovery.** When the reset window elapses, callers don't all rush in — one caller atomically claims the trial slot (`claimTrial`, a conditional write) and the rest stay blocked (`Circuit breaker is open`) until that trial resolves. This is the classic Hystrix-style half-open state, not "everyone tries at once." Applies to both strategies. A single successful trial fully closes the breaker — redeye does not support requiring N consecutive successful trials before closing; that's intentionally out of scope for now, not an oversight.
|
|
198
|
+
- **Exponential backoff with jitter.** A dependency that keeps failing its trial is retried less often each time (`resetTimeout * backoffMultiplier ^ openCount`, capped at `maxResetTimeout`), and the exact retry moment is jittered per-instance so a cluster doesn't hammer a recovering dependency in lockstep.
|
|
199
|
+
- **No hard dependency on precise TTL timing for correctness.** The gating decision is based on elapsed time since the last recorded failure, not "did the key vanish yet" — the store's TTL is a generous safety-net for cleanup, not the primary mechanism. (Early eviction is still possible — see below — but it degrades gracefully instead of being load-bearing.)
|
|
200
|
+
- **Store outages are a distinct, handled condition**, not silently misattributed as the protected operation failing (see fail-open/fail-closed above).
|
|
201
|
+
- **Observability**: per-instance call/success/failure/rejection/store-error counters via `getMetrics`, plus `onStateChange` and `onStoreError` hooks. In distributed mode, `onStateChange` is invoked locally by whichever instance's own call happens to trigger a given transition — not broadcast to the rest of the fleet. Concretely: `'open'` fires only on the instance whose write is the one that crosses the threshold (or whose failed trial reopens it); `'half-open'` fires only on the instance that wins the trial claim; `'closed'` fires only on the instance that closes it (via a successful trial, or a manual `recordSuccess()`/`reset()` call). Every other instance sharing that operation still sees the new state reflected in their own gating decisions immediately — they just never get their own `onStateChange` call for a transition they didn't personally cause. If you need a fleet-wide notification (e.g. to page on-call once, not once per instance that happens to notice), de-duplicate downstream of the hook, or drive alerting off the store directly instead.
|
|
202
|
+
|
|
203
|
+
If your store *doesn't* implement the atomic method your chosen strategy needs (`recordFailureAtomic` or `recordOutcomeAtomic`) or `claimTrial`, redeye logs a one-time warning per missing capability and falls back to a non-atomic get-then-set — it still works, just without the atomicity/exclusivity guarantee for that piece.
|
|
204
|
+
|
|
205
|
+
### Split-brain: what's prevented, and what isn't
|
|
206
|
+
|
|
207
|
+
There's no split-brain among your app instances in the classic sense, because instances never hold independent authoritative state to disagree over — every instance reads the store fresh before every single decision (`gateDistributed`) instead of trusting a locally cached opinion. The store is the only "brain"; instances are just readers/writers of it. Two mechanisms enforce agreement on the common path:
|
|
208
|
+
|
|
209
|
+
- **Atomic writes.** `recordFailureAtomic`/`recordOutcomeAtomic` run as one Lua script — GET, decode, increment, decide, SET — inside Redis's single-threaded execution, so two instances failing at the same instant can't both read the same count and both write the same increment.
|
|
210
|
+
- **Exclusive trial claim.** `claimTrial` is a `SET ... NX`, which Redis resolves atomically, so exactly one instance ever runs the half-open recovery probe — never two instances simultaneously deciding they're the one testing recovery.
|
|
211
|
+
|
|
212
|
+
So the design doesn't eliminate split-brain so much as route around it: it pushes the single-arbiter requirement onto the store and never lets an instance act on stale local state instead of asking the store. That said, there are real, narrow windows where instances *can* still disagree — each one is a deliberate availability-over-consistency choice, not a hidden gap:
|
|
213
|
+
|
|
214
|
+
- **An instance partitioned from the store fails open independently** (item 9 below) — during the partition, that one instance's view of the breaker can genuinely diverge from the rest of the fleet's.
|
|
215
|
+
- **A `claimTrial` error can rarely produce two "winners"** (item 10 below) — a narrow window traded for never permanently wedging the breaker open.
|
|
216
|
+
- **A slow trial outliving its claim's TTL is the *more likely* way to get two concurrent trials** (item 11 below) — routine for a slowly-recovering dependency, not rare like item 10.
|
|
217
|
+
- **The store's own replication/failover consistency is inherited, not solved** (item 12 below) — this library adds no consensus layer on top of whatever your Redis deployment already guarantees.
|
|
218
|
+
- **Clock skew causes timing disagreement, not state disagreement** (item 3 below) — it can shift exactly when a trial becomes eligible, but it cannot cause a double-trial, since that's arbitrated by the store, not by comparing clocks.
|
|
219
|
+
|
|
220
|
+
### What's still a fundamental tradeoff, not a bug
|
|
221
|
+
|
|
222
|
+
These are true of any distributed circuit breaker, rate limiter, or lock built on a shared store — not gaps specific to redeye:
|
|
223
|
+
|
|
224
|
+
1. **Distributed mode costs Redis round trips per guarded call, not zero — except while the breaker is actually open, which is exactly when that matters most.** On the closed, healthy path, every `execute()` call does a gate read (`store.get`) before deciding whether to proceed; what happens after differs by strategy — `errorRate` writes (`recordOutcomeAtomic`) on every closed-phase call, success or failure, since the rate needs both to be meaningful, while `consecutive` only writes when the gate's own read observed accumulated state to clear (an open breaker, or `failures > 0`), making a run of healthy successes 1 read and 0 writes per call, not 1-and-1 (a measured claim about the steady state, not a guarantee — a dependency flapping near the failure threshold still writes on most calls). **Once the breaker is open, rejections stop costing a read at all**: the gate caches the open state locally (`lastFailure`, `openCount`, and the same jittered `expiresAt` the eligibility check itself uses) and rejects straight from that cache, re-verifying against the store only once per `openCacheRefreshMs` (default 2000ms) — to catch an early close by another instance's trial, or a manual `reset()` — and always falling through to a real read right at `expiresAt`, so the cache can delay a rejection's freshness but can never delay a trial's eligibility. Net effect: the request-volume-scaling cost this item used to describe applies to the closed, healthy path — the load *during an incident*, when the store or its host infrastructure may itself be degraded, drops instead of scaling with traffic. A resolving half-open trial still adds a round trip to claim it and another to release its claim — that path is unaffected, and unconditional, regardless of strategy. On a latency-sensitive path the closed-path round trip is still added tail latency — the opposite tradeoff from something like Envoy's outlier detection, which caches state locally with a short TTL and accepts bounded staleness instead of paying a store round trip per call. redeye chose per-call freshness over that on the closed path; it's a reasonable choice for many services, but measure it before putting this in front of your highest-QPS call. Local mode has none of this cost — see [Local mode vs. distributed mode](#local-mode-vs-distributed-mode-which-do-you-need) if you're not sure you need shared state at all.
|
|
225
|
+
2. **Correctness depends on trusting your store.** If Redis evicts a breaker key early under memory pressure (e.g. `maxmemory-policy allkeys-lru` with `circuit_breaker:*` competing for space with everything else), the breaker can reset earlier than intended. Mitigate by giving circuit-breaker keys their own keyspace/DB with a policy that doesn't evict them, or by monitoring `onStoreError` and Redis memory pressure directly. This is not fixable in-library — it's an operational configuration matter.
|
|
226
|
+
3. **Distributed timing is sensitive to clock skew.** Backoff and jitter windows are computed by comparing each instance's local `Date.now()` against a `lastFailure` timestamp written by whichever instance recorded it. With reasonably synchronized clocks (standard NTP) this is a non-issue; on a fleet with significant clock drift, instances can disagree by that drift amount about exactly when a trial becomes eligible. This affects *timing* only — the half-open claim mechanism (`claimTrial`) still guarantees exactly one trial runs regardless of skew, so skew cannot cause a thundering herd, only a slightly early or late trial attempt.
|
|
227
|
+
4. **`recordFailure`/`recordSuccess`/`canExecuteAsync` don't participate in trial claiming.** Only `execute()` claims and releases the half-open trial slot. If you build your own call flow around the manual API instead of `execute()`, you lose the single-trial guarantee and get the old "everyone retries once elapsed" behavior for that flow. Prefer `execute()`.
|
|
228
|
+
5. **One strategy/threshold/backoff policy per breaker instance**, applied uniformly to every `operation` string passed to it. Use separate `CircuitBreaker` instances for dependencies that need different policies — and separate `RedisStore` key prefixes if two breakers share an operation name with different strategies (their state shapes are incompatible and will corrupt each other under the same key).
|
|
229
|
+
6. **Local mode has no cross-restart persistence**, by design — it's explicitly the zero-dependency option. Use distributed mode if breaker state needs to survive a process restart.
|
|
230
|
+
7. **Metrics are per-instance**, not automatically aggregated across your fleet — the same as any Prometheus counter you'd scrape per-pod. Aggregate them in your own metrics backend if you need a fleet-wide view.
|
|
231
|
+
8. **`errorRate` is EWMA-smoothed, not a precise sliding window.** It approximates "failure rate over roughly the last ~`1/(1-decay)` calls," not an exact count over an exact window — good enough to catch flapping dependencies, but the exact trip point for a given call sequence is a function of the decay math, not literally "N of the last M calls."
|
|
232
|
+
9. **A partitioned instance fails open on its own, independently of the rest of the fleet.** If an instance's own call to the store errors (that instance can't reach Redis, but others can), it doesn't consult anyone else — it applies `failOpenOnStoreError` locally (default `true`). For the duration of that partition, the isolated instance treats the breaker as closed while the rest of the fleet, still able to reach the store, correctly sees it open. This is scoped to exactly the partitioned instance for exactly the duration of the partition, and it's a deliberate choice: favoring that one instance staying available over it blocking calls based on a store it can't even confirm the state of.
|
|
233
|
+
10. **A `claimTrial` error can rarely let two instances both believe they won the trial.** If the `claimTrial` call itself throws (not "no key", an actual error), the gate does not block the caller — it fails open on the trial too (`allowed: true, isTrial: true`), because refusing here would mean a single store blip at exactly the wrong moment could permanently wedge the breaker open (no one could ever claim the trial again). If two instances hit that specific error in the same narrow window, both can proceed with a trial call. This trades a rare double-trial for never risking permanent lockout.
|
|
234
|
+
11. **The trial-claim TTL expiring mid-flight is a more likely double-trial window than item 10, and it's by design.** `claimTrial`'s key has a TTL of `trialTimeout` (default `min(timeout ?? 10000, resetTimeout)`). If the trial call itself runs *longer* than that TTL, the claim expires while the call is still in flight, and another instance can then claim a second trial against the same recovering dependency — this is the intended anti-wedging mechanism (nothing should be able to hold the exclusive slot forever), not a bug. It's also the routine case, not the rare one: a slowly-recovering dependency whose first trial takes 12 seconds against a 10-second default `trialTimeout` will hit this on every recovery attempt, not occasionally. Set `trialTimeout` comfortably above your dependency's real p99 recovery-check latency if avoiding this matters to you. **What this does *not* do, strictly bounded by the release mechanism:** cascade into a third concurrent trial. `claimTrial` returns a per-call ownership token, and releasing a claim is always either a compare-and-delete against that exact token or nothing at all — never an unconditional delete — so the original caller's own expired claim, once it finally gets released, can never destroy whichever *other* instance's claim now legitimately occupies the key. That makes this limitation "double trial, strictly bounded to two" rather than "double trial, possibly cascading further."
|
|
235
|
+
12. **`RedisStore` inherits Redis's own replication/failover consistency — it doesn't add a consensus layer on top.** A single Redis instance has nothing to split-brain over. But if you run Sentinel or Cluster behind it, a failover with asynchronous replication can lose the last few writes, and reading from a lagging replica can return stale state. `RedisStore` doesn't issue `WAIT` or otherwise wait for replica acknowledgment — it trusts whatever consistency guarantees your Redis deployment itself provides.
|
|
236
|
+
|
|
237
|
+
### What redeye deliberately does not try to be
|
|
238
|
+
|
|
239
|
+
redeye is scoped to "circuit breaking, done correctly, optionally shared via a store." It does not do retries, bulkheading, or request hedging. If you need those, compose redeye with a separate retry library, and think carefully about ordering: retries should generally happen *inside* what the breaker counts as a single call, not wrapped around it — otherwise a retry storm can trip the breaker faster than intended, or mask real failures from it entirely.
|
|
240
|
+
|
|
241
|
+
## Testing
|
|
242
|
+
|
|
243
|
+
`npm test` runs the unit suite (`test/*.spec.ts`) against in-memory fakes — no external services required.
|
|
244
|
+
|
|
245
|
+
`npm run test:integration` runs `test/*.integration.spec.ts` against a real Redis, exercising `RedisStore`'s actual Lua scripts and `SET ... NX` trial-claim logic instead of a reimplementation of them. Start Redis first:
|
|
246
|
+
|
|
247
|
+
```sh
|
|
248
|
+
docker compose up -d
|
|
249
|
+
npm run test:integration
|
|
250
|
+
docker compose down
|
|
251
|
+
```
|
|
252
|
+
|
|
253
|
+
It connects to `REDIS_URL` (default `redis://localhost:6379`).
|
|
254
|
+
|
|
255
|
+
## Contributing
|
|
256
|
+
|
|
257
|
+
See [CONTRIBUTING.md](CONTRIBUTING.md). Changes are tracked in [CHANGELOG.md](CHANGELOG.md).
|
|
258
|
+
|
|
259
|
+
## License
|
|
260
|
+
|
|
261
|
+
MIT
|
|
@@ -112,6 +112,16 @@ export interface CircuitBreakerOptions {
|
|
|
112
112
|
* indefinitely. Unset (default): no limit, no warning. See README.
|
|
113
113
|
*/
|
|
114
114
|
maxOperations?: number;
|
|
115
|
+
/**
|
|
116
|
+
* Distributed mode only: while the breaker is known-open, reject locally
|
|
117
|
+
* without a store read, re-verifying against the store at most once per
|
|
118
|
+
* this many ms (to catch an early close by another instance's trial or a
|
|
119
|
+
* manual `reset()`). Staleness here is strictly fail-closed — a stale
|
|
120
|
+
* cache entry can only cause an extra rejection, never an extra call
|
|
121
|
+
* through. `0` disables the cache entirely (a store read on every call).
|
|
122
|
+
* Default: 2000.
|
|
123
|
+
*/
|
|
124
|
+
openCacheRefreshMs?: number;
|
|
115
125
|
}
|
|
116
126
|
/**
|
|
117
127
|
* A circuit breaker with an optional distributed mode.
|
|
@@ -138,6 +148,17 @@ export declare class CircuitBreaker {
|
|
|
138
148
|
private warnedDynamicOperationName;
|
|
139
149
|
private readonly metrics;
|
|
140
150
|
private readonly cachedResetTimeouts;
|
|
151
|
+
/**
|
|
152
|
+
* Distributed mode only: a local, best-effort record that `operation` was
|
|
153
|
+
* last observed open, so a burst of calls during an incident can reject
|
|
154
|
+
* without a store round trip each time. `expiresAt` is the same instant
|
|
155
|
+
* the gate's own eligibility check uses (`lastFailure + effectiveResetTimeout`)
|
|
156
|
+
* — this cache can only make a rejection cheaper, never delay eligibility
|
|
157
|
+
* for a trial past that instant. `lastRefetch` paces how often a real read
|
|
158
|
+
* re-verifies the cache against the store (`openCacheRefreshMs`), to catch
|
|
159
|
+
* an early close by another instance or a manual `reset()`.
|
|
160
|
+
*/
|
|
161
|
+
private readonly cachedOpen;
|
|
141
162
|
private readonly failures;
|
|
142
163
|
private readonly lastFailureTime;
|
|
143
164
|
private readonly openOperations;
|
|
@@ -161,8 +182,12 @@ export declare class CircuitBreaker {
|
|
|
161
182
|
private checkOperationName;
|
|
162
183
|
/**
|
|
163
184
|
* Synchronous, best-effort, read-only check (never claims a trial slot).
|
|
164
|
-
* In distributed mode this cannot await the store, so it
|
|
165
|
-
* `
|
|
185
|
+
* In distributed mode this cannot await the store, so it can only ever
|
|
186
|
+
* consult the local open-state cache (see `openCacheRefreshMs`): a `false`
|
|
187
|
+
* is a real, cached-open result, but a `true` is still only advisory —
|
|
188
|
+
* it means either the operation isn't cached open, or nothing has been
|
|
189
|
+
* cached yet — not a confirmed closed state. Use `canExecuteAsync` for a
|
|
190
|
+
* store-verified check.
|
|
166
191
|
*/
|
|
167
192
|
canExecute(operation: string): boolean;
|
|
168
193
|
/**
|
|
@@ -203,6 +228,15 @@ export declare class CircuitBreaker {
|
|
|
203
228
|
* back on consecutive checks milliseconds apart.
|
|
204
229
|
*/
|
|
205
230
|
private effectiveResetTimeout;
|
|
231
|
+
/**
|
|
232
|
+
* Records that `operation` was just observed open (or authored as open,
|
|
233
|
+
* e.g. by this instance's own failed trial), so the next calls can reject
|
|
234
|
+
* from this cache instead of reading the store. Reuses
|
|
235
|
+
* `effectiveResetTimeout`'s per-episode jitter cache — same `expiresAt` the
|
|
236
|
+
* gate's real eligibility check uses, so the cache can never make a trial
|
|
237
|
+
* eligible later than it already would be.
|
|
238
|
+
*/
|
|
239
|
+
private cacheOpenState;
|
|
206
240
|
private key;
|
|
207
241
|
private trialKey;
|
|
208
242
|
private stateTtlSeconds;
|
|
@@ -1 +1 @@
|
|
|
1
|
-
{"version":3,"file":"circuit-breaker.d.ts","sourceRoot":"","sources":["../src/circuit-breaker.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,KAAK,EAAE,MAAM,SAAS,CAAC;AAChC,OAAO,EAAE,mBAAmB,EAAE,cAAc,EAA8B,MAAM,SAAS,CAAC;AAE1F,MAAM,MAAM,YAAY,GAAG,QAAQ,GAAG,MAAM,GAAG,WAAW,CAAC;AAE3D,MAAM,WAAW,aAAa;IAC5B,IAAI,CAAC,OAAO,EAAE,MAAM,GAAG,IAAI,CAAC;IAC5B,GAAG,CAAC,OAAO,EAAE,MAAM,GAAG,IAAI,CAAC;CAC5B;AAED,YAAY,EAAE,mBAAmB,EAAE,cAAc,EAAE,CAAC;AAEpD;;;;;GAKG;AACH,qBAAa,qBAAsB,SAAQ,KAAK;IAC9C,QAAQ,CAAC,KAAK,EAAE,OAAO,CAAC;gBAEZ,SAAS,EAAE,MAAM,EAAE,KAAK,EAAE,OAAO;CAK9C;AAED,MAAM,WAAW,qBAAqB;IACpC;;;;;;;;;;;OAWG;IACH,QAAQ,CAAC,EAAE,aAAa,GAAG,WAAW,CAAC;IACvC,wGAAwG;IACxG,gBAAgB,CAAC,EAAE,MAAM,CAAC;IAC1B;;;OAGG;IACH,kBAAkB,CAAC,EAAE,MAAM,CAAC;IAC5B;;;;OAIG;IACH,YAAY,CAAC,EAAE,MAAM,CAAC;IACtB;;;;OAIG;IACH,cAAc,CAAC,EAAE,MAAM,CAAC;IACxB,mGAAmG;IACnG,YAAY,CAAC,EAAE,MAAM,CAAC;IACtB,2FAA2F;IAC3F,eAAe,CAAC,EAAE,MAAM,CAAC;IACzB,qIAAqI;IACrI,OAAO,CAAC,EAAE,MAAM,CAAC;IACjB;;;;;;OAMG;IACH,YAAY,CAAC,EAAE,MAAM,CAAC;IACtB;;;;OAIG;IACH,iBAAiB,CAAC,EAAE,MAAM,CAAC;IAC3B,oFAAoF;IACpF,eAAe,CAAC,EAAE,MAAM,CAAC;IACzB;;;;OAIG;IACH,MAAM,CAAC,EAAE,MAAM,CAAC;IAChB,aAAa,CAAC,EAAE,CAAC,KAAK,EAAE,YAAY,EAAE,SAAS,EAAE,MAAM,KAAK,IAAI,CAAC;IACjE;;;;;OAKG;IACH,KAAK,CAAC,EAAE,KAAK,CAAC;IACd;;;;;;;;;;;;;;OAcG;IACH,oBAAoB,CAAC,EAAE,OAAO,CAAC;IAC/B,YAAY,CAAC,EAAE,CAAC,KAAK,EAAE,OAAO,EAAE,SAAS,EAAE,MAAM,KAAK,IAAI,CAAC;IAC3D,MAAM,CAAC,EAAE,aAAa,CAAC;IACvB;;;;;;;OAOG;IACH,aAAa,CAAC,EAAE,MAAM,CAAC;
|
|
1
|
+
{"version":3,"file":"circuit-breaker.d.ts","sourceRoot":"","sources":["../src/circuit-breaker.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,KAAK,EAAE,MAAM,SAAS,CAAC;AAChC,OAAO,EAAE,mBAAmB,EAAE,cAAc,EAA8B,MAAM,SAAS,CAAC;AAE1F,MAAM,MAAM,YAAY,GAAG,QAAQ,GAAG,MAAM,GAAG,WAAW,CAAC;AAE3D,MAAM,WAAW,aAAa;IAC5B,IAAI,CAAC,OAAO,EAAE,MAAM,GAAG,IAAI,CAAC;IAC5B,GAAG,CAAC,OAAO,EAAE,MAAM,GAAG,IAAI,CAAC;CAC5B;AAED,YAAY,EAAE,mBAAmB,EAAE,cAAc,EAAE,CAAC;AAEpD;;;;;GAKG;AACH,qBAAa,qBAAsB,SAAQ,KAAK;IAC9C,QAAQ,CAAC,KAAK,EAAE,OAAO,CAAC;gBAEZ,SAAS,EAAE,MAAM,EAAE,KAAK,EAAE,OAAO;CAK9C;AAED,MAAM,WAAW,qBAAqB;IACpC;;;;;;;;;;;OAWG;IACH,QAAQ,CAAC,EAAE,aAAa,GAAG,WAAW,CAAC;IACvC,wGAAwG;IACxG,gBAAgB,CAAC,EAAE,MAAM,CAAC;IAC1B;;;OAGG;IACH,kBAAkB,CAAC,EAAE,MAAM,CAAC;IAC5B;;;;OAIG;IACH,YAAY,CAAC,EAAE,MAAM,CAAC;IACtB;;;;OAIG;IACH,cAAc,CAAC,EAAE,MAAM,CAAC;IACxB,mGAAmG;IACnG,YAAY,CAAC,EAAE,MAAM,CAAC;IACtB,2FAA2F;IAC3F,eAAe,CAAC,EAAE,MAAM,CAAC;IACzB,qIAAqI;IACrI,OAAO,CAAC,EAAE,MAAM,CAAC;IACjB;;;;;;OAMG;IACH,YAAY,CAAC,EAAE,MAAM,CAAC;IACtB;;;;OAIG;IACH,iBAAiB,CAAC,EAAE,MAAM,CAAC;IAC3B,oFAAoF;IACpF,eAAe,CAAC,EAAE,MAAM,CAAC;IACzB;;;;OAIG;IACH,MAAM,CAAC,EAAE,MAAM,CAAC;IAChB,aAAa,CAAC,EAAE,CAAC,KAAK,EAAE,YAAY,EAAE,SAAS,EAAE,MAAM,KAAK,IAAI,CAAC;IACjE;;;;;OAKG;IACH,KAAK,CAAC,EAAE,KAAK,CAAC;IACd;;;;;;;;;;;;;;OAcG;IACH,oBAAoB,CAAC,EAAE,OAAO,CAAC;IAC/B,YAAY,CAAC,EAAE,CAAC,KAAK,EAAE,OAAO,EAAE,SAAS,EAAE,MAAM,KAAK,IAAI,CAAC;IAC3D,MAAM,CAAC,EAAE,aAAa,CAAC;IACvB;;;;;;;OAOG;IACH,aAAa,CAAC,EAAE,MAAM,CAAC;IACvB;;;;;;;;OAQG;IACH,kBAAkB,CAAC,EAAE,MAAM,CAAC;CAC7B;AA0BD;;;;;;;;;;;;;;GAcG;AACH,qBAAa,cAAc;IACzB,OAAO,CAAC,QAAQ,CAAC,OAAO,CAqBtB;IAEF,OAAO,CAAC,QAAQ,CAAC,WAAW,CAAU;IACtC,OAAO,CAAC,QAAQ,CAAC,kBAAkB,CAAqB;IACxD,OAAO,CAAC,iCAAiC,CAAS;IAClD,OAAO,CAAC,QAAQ,CAAC,eAAe,CAAqB;IACrD,OAAO,CAAC,mBAAmB,CAAS;IACpC,OAAO,CAAC,0BAA0B,CAAS;IAC3C,OAAO,CAAC,QAAQ,CAAC,OAAO,CAAqC;IAC7D,OAAO,CAAC,QAAQ,CAAC,mBAAmB,CAAgF;IACpH;;;;;;;;;OASG;IACH,OAAO,CAAC,QAAQ,CAAC,UAAU,CAAyG;IAGpI,OAAO,CAAC,QAAQ,CAAC,QAAQ,CAA6B;IACtD,OAAO,CAAC,QAAQ,CAAC,eAAe,CAA6B;IAC7D,OAAO,CAAC,QAAQ,CAAC,cAAc,CAAqB;IACpD,OAAO,CAAC,QAAQ,CAAC,UAAU,CAA6B;IACxD,OAAO,CAAC,QAAQ,CAAC,eAAe,CAAqB;IACrD,OAAO,CAAC,QAAQ,CAAC,cAAc,CAA6B;IAC5D,OAAO,CAAC,QAAQ,CAAC,UAAU,CAA6B;IACxD,OAAO,CAAC,QAAQ,CAAC,YAAY,CAA6B;IAE1D,OAAO,CAAC,QAAQ,CAAC,aAAa,CAAiC;gBAEnD,OAAO,GAAE,qBAA0B;IA8B/C,6FAA6F;IAC7F,OAAO,IAAI,IAAI;IAIT,OAAO,CAAC,CAAC,EAAE,SAAS,EAAE,MAAM,EAAE,EAAE,EAAE,MAAM,OAAO,CAAC,CAAC,CAAC,GAAG,OAAO,CAAC,CAAC,CAAC;IAKrE;;;;;;OAMG;IACH,OAAO,CAAC,kBAAkB;IAsB1B;;;;;;;;OAQG;IACH,UAAU,CAAC,SAAS,EAAE,MAAM,GAAG,OAAO;IAiBtC;;;;;OAKG;IACG,eAAe,CAAC,SAAS,EAAE,MAAM,GAAG,OAAO,CAAC,OAAO,CAAC;IAM1D,aAAa,CAAC,SAAS,EAAE,MAAM,GAAG,IAAI;IAetC,aAAa,CAAC,SAAS,EAAE,MAAM,GAAG,IAAI;IAStC;;;;;;;;;OASG;YACW,yBAAyB;IAQjC,QAAQ,CAAC,SAAS,EAAE,MAAM,GAAG,OAAO,CAAC,mBAAmB,CAAC;IAYzD,KAAK,CAAC,SAAS,EAAE,MAAM,GAAG,OAAO,CAAC,IAAI,CAAC;IAgB7C,wPAAwP;IACxP,UAAU,CAAC,SAAS,EAAE,MAAM,GAAG,cAAc;IAK7C,aAAa,IAAI,MAAM,CAAC,MAAM,EAAE,cAAc,CAAC;IAM/C,OAAO,CAAC,UAAU;IAWlB,OAAO,CAAC,aAAa;IAKrB;;;;;;;;;OASG;IACH,OAAO,CAAC,qBAAqB;IAe7B;;;;;;;OAOG;IACH,OAAO,CAAC,cAAc;IAatB,OAAO,CAAC,GAAG;IAIX,OAAO,CAAC,QAAQ;IAIhB,OAAO,CAAC,eAAe;IAUvB,OAAO,CAAC,eAAe;IAIvB,OAAO,CAAC,gBAAgB;IAMxB,OAAO,CAAC,qBAAqB;YAQf,OAAO;YASP,OAAO;IAQrB,qFAAqF;YACvE,WAAW;IAQzB,4NAA4N;YAC9M,YAAY;IAQ1B;;;;;;;;;;;;;;;;OAgBG;YACW,gBAAgB;YAahB,gBAAgB;IAM9B;;;;;OAKG;YACW,eAAe;YA0Df,kBAAkB;YAkDlB,gCAAgC;IA+C9C;;;;OAIG;YACW,wBAAwB;YAoDxB,iCAAiC;YAmBjC,YAAY;IAkC1B,gFAAgF;IAChF,OAAO,CAAC,aAAa;IAcrB,oFAAoF;IACpF,OAAO,CAAC,cAAc;IAuBtB,OAAO,CAAC,0BAA0B;IAclC,kIAAkI;IAClI,OAAO,CAAC,sBAAsB;IAkB9B,OAAO,CAAC,2BAA2B;IAUnC,OAAO,CAAC,UAAU;IAiBlB;;;;;;OAMG;IACH,OAAO,CAAC,OAAO;IAgBf,OAAO,CAAC,kBAAkB;CAiB3B"}
|
package/dist/circuit-breaker.js
CHANGED
|
@@ -41,6 +41,17 @@ class CircuitBreaker {
|
|
|
41
41
|
this.warnedDynamicOperationName = false;
|
|
42
42
|
this.metrics = new Map();
|
|
43
43
|
this.cachedResetTimeouts = new Map();
|
|
44
|
+
/**
|
|
45
|
+
* Distributed mode only: a local, best-effort record that `operation` was
|
|
46
|
+
* last observed open, so a burst of calls during an incident can reject
|
|
47
|
+
* without a store round trip each time. `expiresAt` is the same instant
|
|
48
|
+
* the gate's own eligibility check uses (`lastFailure + effectiveResetTimeout`)
|
|
49
|
+
* — this cache can only make a rejection cheaper, never delay eligibility
|
|
50
|
+
* for a trial past that instant. `lastRefetch` paces how often a real read
|
|
51
|
+
* re-verifies the cache against the store (`openCacheRefreshMs`), to catch
|
|
52
|
+
* an early close by another instance or a manual `reset()`.
|
|
53
|
+
*/
|
|
54
|
+
this.cachedOpen = new Map();
|
|
44
55
|
// local mode state
|
|
45
56
|
this.failures = new Map();
|
|
46
57
|
this.lastFailureTime = new Map();
|
|
@@ -70,6 +81,7 @@ class CircuitBreaker {
|
|
|
70
81
|
onStoreError: options.onStoreError,
|
|
71
82
|
logger: options.logger ?? noopLogger,
|
|
72
83
|
maxOperations: options.maxOperations,
|
|
84
|
+
openCacheRefreshMs: options.openCacheRefreshMs ?? 2000,
|
|
73
85
|
};
|
|
74
86
|
this.distributed = !!options.store;
|
|
75
87
|
this.monitorHandle = setInterval(() => this.monitor(), this.options.monitorInterval);
|
|
@@ -106,14 +118,22 @@ class CircuitBreaker {
|
|
|
106
118
|
}
|
|
107
119
|
/**
|
|
108
120
|
* Synchronous, best-effort, read-only check (never claims a trial slot).
|
|
109
|
-
* In distributed mode this cannot await the store, so it
|
|
110
|
-
* `
|
|
121
|
+
* In distributed mode this cannot await the store, so it can only ever
|
|
122
|
+
* consult the local open-state cache (see `openCacheRefreshMs`): a `false`
|
|
123
|
+
* is a real, cached-open result, but a `true` is still only advisory —
|
|
124
|
+
* it means either the operation isn't cached open, or nothing has been
|
|
125
|
+
* cached yet — not a confirmed closed state. Use `canExecuteAsync` for a
|
|
126
|
+
* store-verified check.
|
|
111
127
|
*/
|
|
112
128
|
canExecute(operation) {
|
|
113
129
|
if (this.distributed) {
|
|
114
130
|
if (!this.warnedCanExecuteInDistributedMode) {
|
|
115
131
|
this.warnedCanExecuteInDistributedMode = true;
|
|
116
|
-
this.options.logger.warn('canExecute()
|
|
132
|
+
this.options.logger.warn('canExecute() in distributed mode cannot await the store: a false reflects a real cached-open state, but a true is only advisory, not a confirmed closed state; use canExecuteAsync() to be sure.');
|
|
133
|
+
}
|
|
134
|
+
const cached = this.cachedOpen.get(operation);
|
|
135
|
+
if (cached && this.options.openCacheRefreshMs > 0 && Date.now() < cached.expiresAt) {
|
|
136
|
+
return false;
|
|
117
137
|
}
|
|
118
138
|
return true;
|
|
119
139
|
}
|
|
@@ -191,6 +211,7 @@ class CircuitBreaker {
|
|
|
191
211
|
await this.safeDelMain(operation);
|
|
192
212
|
await this.safeDelTrial(operation);
|
|
193
213
|
this.cachedResetTimeouts.delete(operation);
|
|
214
|
+
this.cachedOpen.delete(operation);
|
|
194
215
|
if (prior?.isOpen) {
|
|
195
216
|
this.options.onStateChange?.('closed', operation);
|
|
196
217
|
}
|
|
@@ -248,6 +269,24 @@ class CircuitBreaker {
|
|
|
248
269
|
this.cachedResetTimeouts.set(operation, { openCount, lastFailure, value });
|
|
249
270
|
return value;
|
|
250
271
|
}
|
|
272
|
+
/**
|
|
273
|
+
* Records that `operation` was just observed open (or authored as open,
|
|
274
|
+
* e.g. by this instance's own failed trial), so the next calls can reject
|
|
275
|
+
* from this cache instead of reading the store. Reuses
|
|
276
|
+
* `effectiveResetTimeout`'s per-episode jitter cache — same `expiresAt` the
|
|
277
|
+
* gate's real eligibility check uses, so the cache can never make a trial
|
|
278
|
+
* eligible later than it already would be.
|
|
279
|
+
*/
|
|
280
|
+
cacheOpenState(operation, state) {
|
|
281
|
+
const openCount = state.openCount ?? 0;
|
|
282
|
+
const effective = this.effectiveResetTimeout(operation, openCount, state.lastFailure);
|
|
283
|
+
this.cachedOpen.set(operation, {
|
|
284
|
+
lastFailure: state.lastFailure,
|
|
285
|
+
openCount,
|
|
286
|
+
expiresAt: state.lastFailure + effective,
|
|
287
|
+
lastRefetch: Date.now(),
|
|
288
|
+
});
|
|
289
|
+
}
|
|
251
290
|
// ---- distributed mode -----------------------------------------------
|
|
252
291
|
key(operation) {
|
|
253
292
|
return `circuit_breaker:${operation}`;
|
|
@@ -347,6 +386,7 @@ class CircuitBreaker {
|
|
|
347
386
|
async closeDistributed(operation) {
|
|
348
387
|
await this.safeSet(operation, { ...types_1.CLOSED_STATE }, this.stateTtlSeconds(0));
|
|
349
388
|
this.cachedResetTimeouts.delete(operation);
|
|
389
|
+
this.cachedOpen.delete(operation);
|
|
350
390
|
}
|
|
351
391
|
/**
|
|
352
392
|
* Determines whether a call is allowed through in distributed mode.
|
|
@@ -355,30 +395,49 @@ class CircuitBreaker {
|
|
|
355
395
|
* `canExecuteAsync`) is read-only and never claims.
|
|
356
396
|
*/
|
|
357
397
|
async gateDistributed(operation, claim) {
|
|
398
|
+
const cached = this.cachedOpen.get(operation);
|
|
399
|
+
if (cached && this.options.openCacheRefreshMs > 0) {
|
|
400
|
+
const now = Date.now();
|
|
401
|
+
if (now >= cached.expiresAt) {
|
|
402
|
+
// Eligible for a trial (or past due) — never serve a cached
|
|
403
|
+
// rejection here, or we'd delay eligibility past the real timeout.
|
|
404
|
+
this.cachedOpen.delete(operation);
|
|
405
|
+
}
|
|
406
|
+
else if (now - cached.lastRefetch < this.options.openCacheRefreshMs) {
|
|
407
|
+
return { allowed: false, isTrial: false };
|
|
408
|
+
}
|
|
409
|
+
// else: refresh window elapsed but not yet eligible — fall through to
|
|
410
|
+
// a real read, which repopulates or clears the cache below based on
|
|
411
|
+
// what it finds (an early close by another instance, most likely).
|
|
412
|
+
}
|
|
358
413
|
let state;
|
|
359
414
|
try {
|
|
360
415
|
state = await this.options.store.get(this.key(operation));
|
|
361
416
|
}
|
|
362
417
|
catch (error) {
|
|
363
418
|
this.reportStoreError(error, operation);
|
|
419
|
+
this.cachedOpen.delete(operation); // can't trust a cache we can't re-verify
|
|
364
420
|
if (this.options.failOpenOnStoreError)
|
|
365
421
|
return { allowed: true, isTrial: false };
|
|
366
422
|
throw new StoreUnavailableError(operation, error);
|
|
367
423
|
}
|
|
368
|
-
if (!state?.isOpen)
|
|
369
|
-
|
|
424
|
+
if (!state?.isOpen) {
|
|
425
|
+
this.cachedOpen.delete(operation);
|
|
426
|
+
return { allowed: true, isTrial: false, observedState: state };
|
|
427
|
+
}
|
|
370
428
|
const effective = this.effectiveResetTimeout(operation, state.openCount ?? 0, state.lastFailure);
|
|
371
429
|
if (Date.now() - state.lastFailure < effective) {
|
|
372
|
-
|
|
430
|
+
this.cacheOpenState(operation, state);
|
|
431
|
+
return { allowed: false, isTrial: false, observedState: state };
|
|
373
432
|
}
|
|
374
433
|
if (!claim)
|
|
375
|
-
return { allowed: true, isTrial: false };
|
|
434
|
+
return { allowed: true, isTrial: false, observedState: state };
|
|
376
435
|
if (this.options.store.claimTrial) {
|
|
377
436
|
try {
|
|
378
437
|
const token = await this.options.store.claimTrial(this.trialKey(operation), this.trialTtlSeconds());
|
|
379
438
|
if (token)
|
|
380
439
|
this.options.onStateChange?.('half-open', operation);
|
|
381
|
-
return { allowed: token !== null, isTrial: token !== null, trialToken: token ?? undefined };
|
|
440
|
+
return { allowed: token !== null, isTrial: token !== null, trialToken: token ?? undefined, observedState: state };
|
|
382
441
|
}
|
|
383
442
|
catch (error) {
|
|
384
443
|
this.reportStoreError(error, operation);
|
|
@@ -386,12 +445,12 @@ class CircuitBreaker {
|
|
|
386
445
|
throw new StoreUnavailableError(operation, error);
|
|
387
446
|
// Can't confirm exclusivity — fail open on the trial itself rather than deadlocking forever.
|
|
388
447
|
this.options.onStateChange?.('half-open', operation);
|
|
389
|
-
return { allowed: true, isTrial: true };
|
|
448
|
+
return { allowed: true, isTrial: true, observedState: state };
|
|
390
449
|
}
|
|
391
450
|
}
|
|
392
451
|
this.warnMissingCapability('claimTrial');
|
|
393
452
|
this.options.onStateChange?.('half-open', operation);
|
|
394
|
-
return { allowed: true, isTrial: true };
|
|
453
|
+
return { allowed: true, isTrial: true, observedState: state };
|
|
395
454
|
}
|
|
396
455
|
async executeDistributed(operation, fn) {
|
|
397
456
|
const metrics = this.metricsFor(operation);
|
|
@@ -413,7 +472,20 @@ class CircuitBreaker {
|
|
|
413
472
|
await this.recordOutcomeDistributed(operation, true);
|
|
414
473
|
}
|
|
415
474
|
else {
|
|
416
|
-
|
|
475
|
+
// consecutive strategy, non-trial success: only write if the gate
|
|
476
|
+
// positively observed dirty state. If the read errored
|
|
477
|
+
// (observedState undefined), write anyway — we can't know.
|
|
478
|
+
//
|
|
479
|
+
// Race analysis: if a concurrent failure lands between our gate
|
|
480
|
+
// read and this skipped write, not writing preserves that failure
|
|
481
|
+
// count — strictly safer than the old unconditional
|
|
482
|
+
// `closeDistributed`, which could erase a legitimate concurrent
|
|
483
|
+
// increment by overwriting it with CLOSED_STATE.
|
|
484
|
+
const s = gate.observedState;
|
|
485
|
+
const observedClean = s !== undefined && (s === null || (!s.isOpen && s.failures === 0));
|
|
486
|
+
if (!observedClean) {
|
|
487
|
+
await this.closeDistributed(operation);
|
|
488
|
+
}
|
|
417
489
|
}
|
|
418
490
|
return result;
|
|
419
491
|
}
|
|
@@ -441,6 +513,8 @@ class CircuitBreaker {
|
|
|
441
513
|
now: Date.now(),
|
|
442
514
|
});
|
|
443
515
|
this.options.logger.warn(`Recorded failure for operation: ${operation}, total failures: ${next.failures}`);
|
|
516
|
+
if (next.isOpen)
|
|
517
|
+
this.cacheOpenState(operation, next);
|
|
444
518
|
if (next.isOpen && next.failures === this.options.failureThreshold) {
|
|
445
519
|
this.options.logger.warn(`Circuit breaker opened for operation: ${operation}`);
|
|
446
520
|
this.options.onStateChange?.('open', operation);
|
|
@@ -470,6 +544,8 @@ class CircuitBreaker {
|
|
|
470
544
|
sampleCount: current.sampleCount ?? 0,
|
|
471
545
|
};
|
|
472
546
|
await this.safeSet(operation, next, this.stateTtlSeconds(0));
|
|
547
|
+
if (next.isOpen)
|
|
548
|
+
this.cacheOpenState(operation, next);
|
|
473
549
|
this.options.logger.warn(`Recorded failure for operation: ${operation}, total failures: ${next.failures}`);
|
|
474
550
|
if (next.isOpen && !current.isOpen) {
|
|
475
551
|
this.options.logger.warn(`Circuit breaker opened for operation: ${operation}`);
|
|
@@ -492,6 +568,8 @@ class CircuitBreaker {
|
|
|
492
568
|
ttlSeconds: this.stateTtlSeconds(0),
|
|
493
569
|
now: Date.now(),
|
|
494
570
|
});
|
|
571
|
+
if (next.isOpen)
|
|
572
|
+
this.cacheOpenState(operation, next);
|
|
495
573
|
if (next.openedNow) {
|
|
496
574
|
this.options.logger.warn(`Circuit breaker opened for operation: ${operation} (error rate ${(next.errorRate * 100).toFixed(1)}% over ${next.sampleCount} calls)`);
|
|
497
575
|
this.options.onStateChange?.('open', operation);
|
|
@@ -520,6 +598,8 @@ class CircuitBreaker {
|
|
|
520
598
|
sampleCount,
|
|
521
599
|
};
|
|
522
600
|
await this.safeSet(operation, next, this.stateTtlSeconds(0));
|
|
601
|
+
if (isOpen)
|
|
602
|
+
this.cacheOpenState(operation, next);
|
|
523
603
|
if (isOpen && !current.isOpen) {
|
|
524
604
|
this.options.logger.warn(`Circuit breaker opened for operation: ${operation} (error rate ${(errorRate * 100).toFixed(1)}% over ${sampleCount} calls)`);
|
|
525
605
|
this.options.onStateChange?.('open', operation);
|
|
@@ -537,6 +617,7 @@ class CircuitBreaker {
|
|
|
537
617
|
sampleCount: current.sampleCount ?? 0,
|
|
538
618
|
};
|
|
539
619
|
await this.safeSet(operation, next, this.stateTtlSeconds(openCount));
|
|
620
|
+
this.cacheOpenState(operation, next); // this instance authored the open state; cache it
|
|
540
621
|
this.options.logger.warn(`Half-open trial failed for operation: ${operation}; backing off (attempt ${openCount + 1})`);
|
|
541
622
|
this.options.onStateChange?.('open', operation);
|
|
542
623
|
}
|
|
@@ -1 +1 @@
|
|
|
1
|
-
{"version":3,"file":"circuit-breaker.js","sourceRoot":"","sources":["../src/circuit-breaker.ts"],"names":[],"mappings":";;;AACA,mCAA0F;AAW1F;;;;;GAKG;AACH,MAAa,qBAAsB,SAAQ,KAAK;IAG9C,YAAY,SAAiB,EAAE,KAAc;QAC3C,KAAK,CAAC,oDAAoD,SAAS,EAAE,CAAC,CAAC;QACvE,IAAI,CAAC,IAAI,GAAG,uBAAuB,CAAC;QACpC,IAAI,CAAC,KAAK,GAAG,KAAK,CAAC;IACrB,CAAC;CACF;AARD,sDAQC;AAoGD,MAAM,UAAU,GAAkB,EAAE,IAAI,EAAE,GAAG,EAAE,GAAE,CAAC,EAAE,GAAG,EAAE,GAAG,EAAE,GAAE,CAAC,EAAE,CAAC;AAkBpE;;;;;;;;;;;;;;GAcG;AACH,MAAa,cAAc;IA6CzB,YAAY,UAAiC,EAAE;QApB9B,uBAAkB,GAAG,IAAI,GAAG,EAAU,CAAC;QAChD,sCAAiC,GAAG,KAAK,CAAC;QACjC,oBAAe,GAAG,IAAI,GAAG,EAAU,CAAC;QAC7C,wBAAmB,GAAG,KAAK,CAAC;QAC5B,+BAA0B,GAAG,KAAK,CAAC;QAC1B,YAAO,GAAG,IAAI,GAAG,EAA0B,CAAC;QAC5C,wBAAmB,GAAG,IAAI,GAAG,EAAqE,CAAC;QAEpH,mBAAmB;QACF,aAAQ,GAAG,IAAI,GAAG,EAAkB,CAAC;QACrC,oBAAe,GAAG,IAAI,GAAG,EAAkB,CAAC;QAC5C,mBAAc,GAAG,IAAI,GAAG,EAAU,CAAC;QACnC,eAAU,GAAG,IAAI,GAAG,EAAkB,CAAC;QACvC,oBAAe,GAAG,IAAI,GAAG,EAAU,CAAC;QACpC,mBAAc,GAAG,IAAI,GAAG,EAAkB,CAAC;QAC3C,eAAU,GAAG,IAAI,GAAG,EAAkB,CAAC;QACvC,iBAAY,GAAG,IAAI,GAAG,EAAkB,CAAC;QAKxD,MAAM,YAAY,GAAG,OAAO,CAAC,YAAY,IAAI,KAAK,CAAC;QACnD,IAAI,CAAC,OAAO,GAAG;YACb,QAAQ,EAAE,OAAO,CAAC,QAAQ,IAAI,aAAa;YAC3C,gBAAgB,EAAE,OAAO,CAAC,gBAAgB,IAAI,CAAC;YAC/C,kBAAkB,EAAE,OAAO,CAAC,kBAAkB,IAAI,GAAG;YACrD,YAAY,EAAE,OAAO,CAAC,YAAY,IAAI,EAAE;YACxC,cAAc,EAAE,OAAO,CAAC,cAAc,IAAI,GAAG;YAC7C,YAAY;YACZ,eAAe,EAAE,OAAO,CAAC,eAAe,IAAI,IAAI;YAChD,OAAO,EAAE,OAAO,CAAC,OAAO;YACxB,YAAY,EAAE,OAAO,CAAC,YAAY,IAAI,IAAI,CAAC,GAAG,CAAC,OAAO,CAAC,OAAO,IAAI,KAAK,EAAE,YAAY,CAAC;YACtF,iBAAiB,EAAE,OAAO,CAAC,iBAAiB,IAAI,CAAC;YACjD,eAAe,EAAE,OAAO,CAAC,eAAe,IAAI,YAAY,GAAG,CAAC;YAC5D,MAAM,EAAE,OAAO,CAAC,MAAM,IAAI,GAAG;YAC7B,aAAa,EAAE,OAAO,CAAC,aAAa;YACpC,KAAK,EAAE,OAAO,CAAC,KAAK;YACpB,oBAAoB,EAAE,OAAO,CAAC,oBAAoB,IAAI,IAAI;YAC1D,YAAY,EAAE,OAAO,CAAC,YAAY;YAClC,MAAM,EAAE,OAAO,CAAC,MAAM,IAAI,UAAU;YACpC,aAAa,EAAE,OAAO,CAAC,aAAa;SACrC,CAAC;QAEF,IAAI,CAAC,WAAW,GAAG,CAAC,CAAC,OAAO,CAAC,KAAK,CAAC;QAEnC,IAAI,CAAC,aAAa,GAAG,WAAW,CAAC,GAAG,EAAE,CAAC,IAAI,CAAC,OAAO,EAAE,EAAE,IAAI,CAAC,OAAO,CAAC,eAAe,CAAC,CAAC;QACrF,IAAI,CAAC,aAAa,CAAC,KAAK,EAAE,EAAE,CAAC;IAC/B,CAAC;IAED,6FAA6F;IAC7F,OAAO;QACL,aAAa,CAAC,IAAI,CAAC,aAAa,CAAC,CAAC;IACpC,CAAC;IAED,KAAK,CAAC,OAAO,CAAI,SAAiB,EAAE,EAAoB;QACtD,IAAI,CAAC,kBAAkB,CAAC,SAAS,CAAC,CAAC;QACnC,OAAO,IAAI,CAAC,WAAW,CAAC,CAAC,CAAC,IAAI,CAAC,kBAAkB,CAAC,SAAS,EAAE,EAAE,CAAC,CAAC,CAAC,CAAC,IAAI,CAAC,YAAY,CAAC,SAAS,EAAE,EAAE,CAAC,CAAC;IACtG,CAAC;IAED;;;;;;OAMG;IACK,kBAAkB,CAAC,SAAiB;QAC1C,IAAI,CAAC,IAAI,CAAC,0BAA0B,IAAI,CAAC,SAAS,CAAC,MAAM,GAAG,GAAG,IAAI,SAAS,CAAC,QAAQ,CAAC,GAAG,CAAC,CAAC,EAAE,CAAC;YAC5F,IAAI,CAAC,0BAA0B,GAAG,IAAI,CAAC;YACvC,MAAM,KAAK,GAAG,SAAS,CAAC,MAAM,GAAG,GAAG,CAAC,CAAC,CAAC,GAAG,SAAS,CAAC,KAAK,CAAC,CAAC,EAAE,GAAG,CAAC,GAAG,CAAC,CAAC,CAAC,SAAS,CAAC;YACjF,IAAI,CAAC,OAAO,CAAC,MAAM,CAAC,IAAI,CACtB,mCAAmC,KAAK,oBACtC,SAAS,CAAC,MAAM,GAAG,GAAG,CAAC,CAAC,CAAC,GAAG,SAAS,CAAC,MAAM,QAAQ,CAAC,CAAC,CAAC,cACzD,uNAAuN,CACxN,CAAC;QACJ,CAAC;QAED,IAAI,IAAI,CAAC,OAAO,CAAC,aAAa,KAAK,SAAS,IAAI,CAAC,IAAI,CAAC,mBAAmB,IAAI,CAAC,IAAI,CAAC,eAAe,CAAC,GAAG,CAAC,SAAS,CAAC,EAAE,CAAC;YAClH,IAAI,CAAC,eAAe,CAAC,GAAG,CAAC,SAAS,CAAC,CAAC;YACpC,IAAI,IAAI,CAAC,eAAe,CAAC,IAAI,GAAG,IAAI,CAAC,OAAO,CAAC,aAAa,EAAE,CAAC;gBAC3D,IAAI,CAAC,mBAAmB,GAAG,IAAI,CAAC;gBAChC,IAAI,CAAC,OAAO,CAAC,MAAM,CAAC,IAAI,CACtB,4BAA4B,IAAI,CAAC,eAAe,CAAC,IAAI,uDAAuD,IAAI,CAAC,OAAO,CAAC,aAAa,8HAA8H,CACrQ,CAAC;YACJ,CAAC;QACH,CAAC;IACH,CAAC;IAED;;;;OAIG;IACH,UAAU,CAAC,SAAiB;QAC1B,IAAI,IAAI,CAAC,WAAW,EAAE,CAAC;YACrB,IAAI,CAAC,IAAI,CAAC,iCAAiC,EAAE,CAAC;gBAC5C,IAAI,CAAC,iCAAiC,GAAG,IAAI,CAAC;gBAC9C,IAAI,CAAC,OAAO,CAAC,MAAM,CAAC,IAAI,CACtB,0JAA0J,CAC3J,CAAC;YACJ,CAAC;YACD,OAAO,IAAI,CAAC;QACd,CAAC;QACD,OAAO,IAAI,CAAC,aAAa,CAAC,SAAS,CAAC,CAAC;IACvC,CAAC;IAED;;;;;OAKG;IACH,KAAK,CAAC,eAAe,CAAC,SAAiB;QACrC,IAAI,CAAC,IAAI,CAAC,WAAW;YAAE,OAAO,IAAI,CAAC,aAAa,CAAC,SAAS,CAAC,CAAC;QAC5D,MAAM,IAAI,GAAG,MAAM,IAAI,CAAC,eAAe,CAAC,SAAS,EAAE,KAAK,CAAC,CAAC;QAC1D,OAAO,IAAI,CAAC,OAAO,CAAC;IACtB,CAAC;IAED,aAAa,CAAC,SAAiB;QAC7B,IAAI,CAAC,kBAAkB,CAAC,SAAS,CAAC,CAAC;QACnC,IAAI,IAAI,CAAC,WAAW,EAAE,CAAC;YACrB,IAAI,IAAI,CAAC,OAAO,CAAC,QAAQ,KAAK,WAAW,EAAE,CAAC;gBAC1C,KAAK,IAAI,CAAC,wBAAwB,CAAC,SAAS,EAAE,KAAK,CAAC,CAAC;YACvD,CAAC;iBAAM,CAAC;gBACN,KAAK,IAAI,CAAC,gCAAgC,CAAC,SAAS,CAAC,CAAC;YACxD,CAAC;QACH,CAAC;aAAM,IAAI,IAAI,CAAC,OAAO,CAAC,QAAQ,KAAK,WAAW,EAAE,CAAC;YACjD,IAAI,CAAC,sBAAsB,CAAC,SAAS,EAAE,KAAK,CAAC,CAAC;QAChD,CAAC;aAAM,CAAC;YACN,IAAI,CAAC,0BAA0B,CAAC,SAAS,CAAC,CAAC;QAC7C,CAAC;IACH,CAAC;IAED,aAAa,CAAC,SAAiB;QAC7B,IAAI,CAAC,kBAAkB,CAAC,SAAS,CAAC,CAAC;QACnC,IAAI,IAAI,CAAC,WAAW,EAAE,CAAC;YACrB,KAAK,IAAI,CAAC,yBAAyB,CAAC,SAAS,CAAC,CAAC;QACjD,CAAC;aAAM,CAAC;YACN,IAAI,CAAC,UAAU,CAAC,SAAS,CAAC,CAAC;QAC7B,CAAC;IACH,CAAC;IAED;;;;;;;;;OASG;IACK,KAAK,CAAC,yBAAyB,CAAC,SAAiB;QACvD,MAAM,KAAK,GAAG,MAAM,IAAI,CAAC,OAAO,CAAC,SAAS,CAAC,CAAC;QAC5C,MAAM,IAAI,CAAC,gBAAgB,CAAC,SAAS,CAAC,CAAC;QACvC,IAAI,KAAK,EAAE,MAAM,EAAE,CAAC;YAClB,IAAI,CAAC,OAAO,CAAC,aAAa,EAAE,CAAC,QAAQ,EAAE,SAAS,CAAC,CAAC;QACpD,CAAC;IACH,CAAC;IAED,KAAK,CAAC,QAAQ,CAAC,SAAiB;QAC9B,IAAI,IAAI,CAAC,WAAW,EAAE,CAAC;YACrB,OAAO,CAAC,MAAM,IAAI,CAAC,OAAO,CAAC,SAAS,CAAC,CAAC,IAAI,EAAE,GAAG,oBAAY,EAAE,CAAC;QAChE,CAAC;QACD,MAAM,QAAQ,GAAG,IAAI,CAAC,QAAQ,CAAC,GAAG,CAAC,SAAS,CAAC,IAAI,CAAC,CAAC;QACnD,MAAM,WAAW,GAAG,IAAI,CAAC,eAAe,CAAC,GAAG,CAAC,SAAS,CAAC,IAAI,CAAC,CAAC;QAC7D,MAAM,SAAS,GAAG,IAAI,CAAC,UAAU,CAAC,GAAG,CAAC,SAAS,CAAC,IAAI,CAAC,CAAC;QACtD,MAAM,SAAS,GAAG,IAAI,CAAC,UAAU,CAAC,GAAG,CAAC,SAAS,CAAC,IAAI,CAAC,CAAC;QACtD,MAAM,WAAW,GAAG,IAAI,CAAC,YAAY,CAAC,GAAG,CAAC,SAAS,CAAC,IAAI,CAAC,CAAC;QAC1D,OAAO,EAAE,QAAQ,EAAE,WAAW,EAAE,MAAM,EAAE,IAAI,CAAC,cAAc,CAAC,GAAG,CAAC,SAAS,CAAC,EAAE,SAAS,EAAE,SAAS,EAAE,WAAW,EAAE,CAAC;IAClH,CAAC;IAED,KAAK,CAAC,KAAK,CAAC,SAAiB;QAC3B,IAAI,IAAI,CAAC,WAAW,EAAE,CAAC;YACrB,MAAM,KAAK,GAAG,MAAM,IAAI,CAAC,OAAO,CAAC,SAAS,CAAC,CAAC;YAC5C,MAAM,IAAI,CAAC,WAAW,CAAC,SAAS,CAAC,CAAC;YAClC,MAAM,IAAI,CAAC,YAAY,CAAC,SAAS,CAAC,CAAC;YACnC,IAAI,CAAC,mBAAmB,CAAC,MAAM,CAAC,SAAS,CAAC,CAAC;YAC3C,IAAI,KAAK,EAAE,MAAM,EAAE,CAAC;gBAClB,IAAI,CAAC,OAAO,CAAC,aAAa,EAAE,CAAC,QAAQ,EAAE,SAAS,CAAC,CAAC;YACpD,CAAC;YACD,IAAI,CAAC,OAAO,CAAC,MAAM,CAAC,GAAG,CAAC,wCAAwC,SAAS,EAAE,CAAC,CAAC;QAC/E,CAAC;aAAM,CAAC;YACN,IAAI,CAAC,UAAU,CAAC,SAAS,CAAC,CAAC;QAC7B,CAAC;IACH,CAAC;IAED,wPAAwP;IACxP,UAAU,CAAC,SAAiB;QAC1B,MAAM,CAAC,GAAG,IAAI,CAAC,OAAO,CAAC,GAAG,CAAC,SAAS,CAAC,CAAC;QACtC,OAAO,CAAC,CAAC,CAAC,CAAC,EAAE,GAAG,CAAC,EAAE,CAAC,CAAC,CAAC,IAAA,oBAAY,GAAE,CAAC;IACvC,CAAC;IAED,aAAa;QACX,MAAM,MAAM,GAAmC,EAAE,CAAC;QAClD,KAAK,MAAM,CAAC,SAAS,EAAE,CAAC,CAAC,IAAI,IAAI,CAAC,OAAO,CAAC,OAAO,EAAE;YAAE,MAAM,CAAC,SAAS,CAAC,GAAG,EAAE,GAAG,CAAC,EAAE,CAAC;QAClF,OAAO,MAAM,CAAC;IAChB,CAAC;IAEO,UAAU,CAAC,SAAiB;QAClC,IAAI,CAAC,GAAG,IAAI,CAAC,OAAO,CAAC,GAAG,CAAC,SAAS,CAAC,CAAC;QACpC,IAAI,CAAC,CAAC,EAAE,CAAC;YACP,CAAC,GAAG,IAAA,oBAAY,GAAE,CAAC;YACnB,IAAI,CAAC,OAAO,CAAC,GAAG,CAAC,SAAS,EAAE,CAAC,CAAC,CAAC;QACjC,CAAC;QACD,OAAO,CAAC,CAAC;IACX,CAAC;IAED,0EAA0E;IAElE,aAAa,CAAC,SAAiB;QACrC,MAAM,MAAM,GAAG,IAAI,CAAC,OAAO,CAAC,YAAY,GAAG,IAAI,CAAC,GAAG,CAAC,IAAI,CAAC,OAAO,CAAC,iBAAiB,EAAE,SAAS,CAAC,CAAC;QAC/F,OAAO,IAAI,CAAC,GAAG,CAAC,MAAM,EAAE,IAAI,CAAC,OAAO,CAAC,eAAe,CAAC,CAAC;IACxD,CAAC;IAED;;;;;;;;;OASG;IACK,qBAAqB,CAAC,SAAiB,EAAE,SAAiB,EAAE,WAAmB;QACrF,MAAM,MAAM,GAAG,IAAI,CAAC,mBAAmB,CAAC,GAAG,CAAC,SAAS,CAAC,CAAC;QACvD,IAAI,MAAM,IAAI,MAAM,CAAC,SAAS,KAAK,SAAS,IAAI,MAAM,CAAC,WAAW,KAAK,WAAW,EAAE,CAAC;YACnF,OAAO,MAAM,CAAC,KAAK,CAAC;QACtB,CAAC;QACD,MAAM,MAAM,GAAG,IAAI,CAAC,aAAa,CAAC,SAAS,CAAC,CAAC;QAC7C,IAAI,KAAK,GAAG,MAAM,CAAC;QACnB,IAAI,IAAI,CAAC,OAAO,CAAC,MAAM,EAAE,CAAC;YACxB,MAAM,KAAK,GAAG,MAAM,GAAG,IAAI,CAAC,OAAO,CAAC,MAAM,CAAC;YAC3C,KAAK,GAAG,MAAM,GAAG,CAAC,IAAI,CAAC,MAAM,EAAE,GAAG,CAAC,GAAG,CAAC,CAAC,GAAG,KAAK,CAAC;QACnD,CAAC;QACD,IAAI,CAAC,mBAAmB,CAAC,GAAG,CAAC,SAAS,EAAE,EAAE,SAAS,EAAE,WAAW,EAAE,KAAK,EAAE,CAAC,CAAC;QAC3E,OAAO,KAAK,CAAC;IACf,CAAC;IAED,wEAAwE;IAEhE,GAAG,CAAC,SAAiB;QAC3B,OAAO,mBAAmB,SAAS,EAAE,CAAC;IACxC,CAAC;IAEO,QAAQ,CAAC,SAAiB;QAChC,OAAO,mBAAmB,SAAS,QAAQ,CAAC;IAC9C,CAAC;IAEO,eAAe,CAAC,SAAiB;QACvC,uEAAuE;QACvE,kEAAkE;QAClE,sEAAsE;QACtE,gEAAgE;QAChE,uEAAuE;QACvE,kCAAkC;QAClC,OAAO,IAAI,CAAC,GAAG,CAAC,CAAC,EAAE,IAAI,CAAC,IAAI,CAAC,CAAC,IAAI,CAAC,aAAa,CAAC,SAAS,CAAC,GAAG,CAAC,CAAC,GAAG,IAAI,CAAC,CAAC,CAAC;IAC5E,CAAC;IAEO,eAAe;QACrB,OAAO,IAAI,CAAC,GAAG,CAAC,CAAC,EAAE,IAAI,CAAC,IAAI,CAAC,IAAI,CAAC,OAAO,CAAC,YAAY,GAAG,IAAI,CAAC,CAAC,CAAC;IAClE,CAAC;IAEO,gBAAgB,CAAC,KAAc,EAAE,SAAiB;QACxD,IAAI,CAAC,UAAU,CAAC,SAAS,CAAC,CAAC,gBAAgB,EAAE,CAAC;QAC9C,IAAI,CAAC,OAAO,CAAC,MAAM,CAAC,IAAI,CAAC,8CAA8C,SAAS,KAAM,KAAe,EAAE,OAAO,IAAI,KAAK,EAAE,CAAC,CAAC;QAC3H,IAAI,CAAC,OAAO,CAAC,YAAY,EAAE,CAAC,KAAK,EAAE,SAAS,CAAC,CAAC;IAChD,CAAC;IAEO,qBAAqB,CAAC,UAAyF;QACrH,IAAI,IAAI,CAAC,kBAAkB,CAAC,GAAG,CAAC,UAAU,CAAC;YAAE,OAAO;QACpD,IAAI,CAAC,kBAAkB,CAAC,GAAG,CAAC,UAAU,CAAC,CAAC;QACxC,IAAI,CAAC,OAAO,CAAC,MAAM,CAAC,IAAI,CACtB,4BAA4B,UAAU,oGAAoG,CAC3I,CAAC;IACJ,CAAC;IAEO,KAAK,CAAC,OAAO,CAAC,SAAiB;QACrC,IAAI,CAAC;YACH,OAAO,MAAM,IAAI,CAAC,OAAO,CAAC,KAAM,CAAC,GAAG,CAAsB,IAAI,CAAC,GAAG,CAAC,SAAS,CAAC,CAAC,CAAC;QACjF,CAAC;QAAC,OAAO,KAAK,EAAE,CAAC;YACf,IAAI,CAAC,gBAAgB,CAAC,KAAK,EAAE,SAAS,CAAC,CAAC;YACxC,OAAO,IAAI,CAAC;QACd,CAAC;IACH,CAAC;IAEO,KAAK,CAAC,OAAO,CAAC,SAAiB,EAAE,KAA0B,EAAE,UAAkB;QACrF,IAAI,CAAC;YACH,MAAM,IAAI,CAAC,OAAO,CAAC,KAAM,CAAC,GAAG,CAAC,IAAI,CAAC,GAAG,CAAC,SAAS,CAAC,EAAE,KAAK,EAAE,UAAU,CAAC,CAAC;QACxE,CAAC;QAAC,OAAO,KAAK,EAAE,CAAC;YACf,IAAI,CAAC,gBAAgB,CAAC,KAAK,EAAE,SAAS,CAAC,CAAC;QAC1C,CAAC;IACH,CAAC;IAED,qFAAqF;IAC7E,KAAK,CAAC,WAAW,CAAC,SAAiB;QACzC,IAAI,CAAC;YACH,MAAM,IAAI,CAAC,OAAO,CAAC,KAAM,CAAC,GAAG,CAAC,IAAI,CAAC,GAAG,CAAC,SAAS,CAAC,CAAC,CAAC;QACrD,CAAC;QAAC,OAAO,KAAK,EAAE,CAAC;YACf,IAAI,CAAC,gBAAgB,CAAC,KAAK,EAAE,SAAS,CAAC,CAAC;QAC1C,CAAC;IACH,CAAC;IAED,4NAA4N;IACpN,KAAK,CAAC,YAAY,CAAC,SAAiB;QAC1C,IAAI,CAAC;YACH,MAAM,IAAI,CAAC,OAAO,CAAC,KAAM,CAAC,GAAG,CAAC,IAAI,CAAC,QAAQ,CAAC,SAAS,CAAC,CAAC,CAAC;QAC1D,CAAC;QAAC,OAAO,KAAK,EAAE,CAAC;YACf,IAAI,CAAC,gBAAgB,CAAC,KAAK,EAAE,SAAS,CAAC,CAAC;QAC1C,CAAC;IACH,CAAC;IAED;;;;;;;;;;;;;;;;OAgBG;IACK,KAAK,CAAC,gBAAgB,CAAC,SAAiB,EAAE,KAAyB;QACzE,IAAI,CAAC,KAAK;YAAE,OAAO;QACnB,IAAI,CAAC,IAAI,CAAC,OAAO,CAAC,KAAM,CAAC,YAAY,EAAE,CAAC;YACtC,IAAI,CAAC,qBAAqB,CAAC,cAAc,CAAC,CAAC;YAC3C,OAAO;QACT,CAAC;QACD,IAAI,CAAC;YACH,MAAM,IAAI,CAAC,OAAO,CAAC,KAAM,CAAC,YAAY,CAAC,IAAI,CAAC,QAAQ,CAAC,SAAS,CAAC,EAAE,KAAK,CAAC,CAAC;QAC1E,CAAC;QAAC,OAAO,KAAK,EAAE,CAAC;YACf,IAAI,CAAC,gBAAgB,CAAC,KAAK,EAAE,SAAS,CAAC,CAAC;QAC1C,CAAC;IACH,CAAC;IAEO,KAAK,CAAC,gBAAgB,CAAC,SAAiB;QAC9C,MAAM,IAAI,CAAC,OAAO,CAAC,SAAS,EAAE,EAAE,GAAG,oBAAY,EAAE,EAAE,IAAI,CAAC,eAAe,CAAC,CAAC,CAAC,CAAC,CAAC;QAC5E,IAAI,CAAC,mBAAmB,CAAC,MAAM,CAAC,SAAS,CAAC,CAAC;IAC7C,CAAC;IAED;;;;;OAKG;IACK,KAAK,CAAC,eAAe,CAAC,SAAiB,EAAE,KAAc;QAC7D,IAAI,KAAiC,CAAC;QACtC,IAAI,CAAC;YACH,KAAK,GAAG,MAAM,IAAI,CAAC,OAAO,CAAC,KAAM,CAAC,GAAG,CAAsB,IAAI,CAAC,GAAG,CAAC,SAAS,CAAC,CAAC,CAAC;QAClF,CAAC;QAAC,OAAO,KAAK,EAAE,CAAC;YACf,IAAI,CAAC,gBAAgB,CAAC,KAAK,EAAE,SAAS,CAAC,CAAC;YACxC,IAAI,IAAI,CAAC,OAAO,CAAC,oBAAoB;gBAAE,OAAO,EAAE,OAAO,EAAE,IAAI,EAAE,OAAO,EAAE,KAAK,EAAE,CAAC;YAChF,MAAM,IAAI,qBAAqB,CAAC,SAAS,EAAE,KAAK,CAAC,CAAC;QACpD,CAAC;QAED,IAAI,CAAC,KAAK,EAAE,MAAM;YAAE,OAAO,EAAE,OAAO,EAAE,IAAI,EAAE,OAAO,EAAE,KAAK,EAAE,CAAC;QAE7D,MAAM,SAAS,GAAG,IAAI,CAAC,qBAAqB,CAAC,SAAS,EAAE,KAAK,CAAC,SAAS,IAAI,CAAC,EAAE,KAAK,CAAC,WAAW,CAAC,CAAC;QACjG,IAAI,IAAI,CAAC,GAAG,EAAE,GAAG,KAAK,CAAC,WAAW,GAAG,SAAS,EAAE,CAAC;YAC/C,OAAO,EAAE,OAAO,EAAE,KAAK,EAAE,OAAO,EAAE,KAAK,EAAE,CAAC;QAC5C,CAAC;QAED,IAAI,CAAC,KAAK;YAAE,OAAO,EAAE,OAAO,EAAE,IAAI,EAAE,OAAO,EAAE,KAAK,EAAE,CAAC;QAErD,IAAI,IAAI,CAAC,OAAO,CAAC,KAAM,CAAC,UAAU,EAAE,CAAC;YACnC,IAAI,CAAC;gBACH,MAAM,KAAK,GAAG,MAAM,IAAI,CAAC,OAAO,CAAC,KAAM,CAAC,UAAU,CAAC,IAAI,CAAC,QAAQ,CAAC,SAAS,CAAC,EAAE,IAAI,CAAC,eAAe,EAAE,CAAC,CAAC;gBACrG,IAAI,KAAK;oBAAE,IAAI,CAAC,OAAO,CAAC,aAAa,EAAE,CAAC,WAAW,EAAE,SAAS,CAAC,CAAC;gBAChE,OAAO,EAAE,OAAO,EAAE,KAAK,KAAK,IAAI,EAAE,OAAO,EAAE,KAAK,KAAK,IAAI,EAAE,UAAU,EAAE,KAAK,IAAI,SAAS,EAAE,CAAC;YAC9F,CAAC;YAAC,OAAO,KAAK,EAAE,CAAC;gBACf,IAAI,CAAC,gBAAgB,CAAC,KAAK,EAAE,SAAS,CAAC,CAAC;gBACxC,IAAI,CAAC,IAAI,CAAC,OAAO,CAAC,oBAAoB;oBAAE,MAAM,IAAI,qBAAqB,CAAC,SAAS,EAAE,KAAK,CAAC,CAAC;gBAC1F,6FAA6F;gBAC7F,IAAI,CAAC,OAAO,CAAC,aAAa,EAAE,CAAC,WAAW,EAAE,SAAS,CAAC,CAAC;gBACrD,OAAO,EAAE,OAAO,EAAE,IAAI,EAAE,OAAO,EAAE,IAAI,EAAE,CAAC;YAC1C,CAAC;QACH,CAAC;QAED,IAAI,CAAC,qBAAqB,CAAC,YAAY,CAAC,CAAC;QACzC,IAAI,CAAC,OAAO,CAAC,aAAa,EAAE,CAAC,WAAW,EAAE,SAAS,CAAC,CAAC;QACrD,OAAO,EAAE,OAAO,EAAE,IAAI,EAAE,OAAO,EAAE,IAAI,EAAE,CAAC;IAC1C,CAAC;IAEO,KAAK,CAAC,kBAAkB,CAAI,SAAiB,EAAE,EAAoB;QACzE,MAAM,OAAO,GAAG,IAAI,CAAC,UAAU,CAAC,SAAS,CAAC,CAAC;QAC3C,OAAO,CAAC,UAAU,EAAE,CAAC;QAErB,MAAM,IAAI,GAAG,MAAM,IAAI,CAAC,eAAe,CAAC,SAAS,EAAE,IAAI,CAAC,CAAC;QACzD,IAAI,CAAC,IAAI,CAAC,OAAO,EAAE,CAAC;YAClB,OAAO,CAAC,eAAe,EAAE,CAAC;YAC1B,MAAM,IAAI,KAAK,CAAC,0CAA0C,SAAS,EAAE,CAAC,CAAC;QACzE,CAAC;QAED,IAAI,CAAC;YACH,MAAM,MAAM,GAAG,MAAM,IAAI,CAAC,kBAAkB,CAAC,EAAE,CAAC,CAAC;YACjD,OAAO,CAAC,cAAc,EAAE,CAAC;YACzB,IAAI,IAAI,CAAC,OAAO,EAAE,CAAC;gBACjB,MAAM,IAAI,CAAC,gBAAgB,CAAC,SAAS,CAAC,CAAC;gBACvC,MAAM,IAAI,CAAC,gBAAgB,CAAC,SAAS,EAAE,IAAI,CAAC,UAAU,CAAC,CAAC;gBACxD,IAAI,CAAC,OAAO,CAAC,aAAa,EAAE,CAAC,QAAQ,EAAE,SAAS,CAAC,CAAC;YACpD,CAAC;iBAAM,IAAI,IAAI,CAAC,OAAO,CAAC,QAAQ,KAAK,WAAW,EAAE,CAAC;gBACjD,MAAM,IAAI,CAAC,wBAAwB,CAAC,SAAS,EAAE,IAAI,CAAC,CAAC;YACvD,CAAC;iBAAM,CAAC;gBACN,MAAM,IAAI,CAAC,gBAAgB,CAAC,SAAS,CAAC,CAAC;YACzC,CAAC;YACD,OAAO,MAAM,CAAC;QAChB,CAAC;QAAC,OAAO,KAAK,EAAE,CAAC;YACf,OAAO,CAAC,aAAa,EAAE,CAAC;YACxB,IAAI,IAAI,CAAC,OAAO,EAAE,CAAC;gBACjB,MAAM,IAAI,CAAC,iCAAiC,CAAC,SAAS,CAAC,CAAC;gBACxD,MAAM,IAAI,CAAC,gBAAgB,CAAC,SAAS,EAAE,IAAI,CAAC,UAAU,CAAC,CAAC;YAC1D,CAAC;iBAAM,IAAI,IAAI,CAAC,OAAO,CAAC,QAAQ,KAAK,WAAW,EAAE,CAAC;gBACjD,MAAM,IAAI,CAAC,wBAAwB,CAAC,SAAS,EAAE,KAAK,CAAC,CAAC;YACxD,CAAC;iBAAM,CAAC;gBACN,MAAM,IAAI,CAAC,gCAAgC,CAAC,SAAS,CAAC,CAAC;YACzD,CAAC;YACD,MAAM,KAAK,CAAC;QACd,CAAC;IACH,CAAC;IAEO,KAAK,CAAC,gCAAgC,CAAC,SAAiB;QAC9D,IAAI,IAAI,CAAC,OAAO,CAAC,KAAM,CAAC,mBAAmB,EAAE,CAAC;YAC5C,IAAI,CAAC;gBACH,MAAM,IAAI,GAAG,MAAM,IAAI,CAAC,OAAO,CAAC,KAAM,CAAC,mBAAmB,CAAC,IAAI,CAAC,GAAG,CAAC,SAAS,CAAC,EAAE;oBAC9E,UAAU,EAAE,IAAI,CAAC,eAAe,CAAC,CAAC,CAAC;oBACnC,gBAAgB,EAAE,IAAI,CAAC,OAAO,CAAC,gBAAgB;oBAC/C,GAAG,EAAE,IAAI,CAAC,GAAG,EAAE;iBAChB,CAAC,CAAC;gBACH,IAAI,CAAC,OAAO,CAAC,MAAM,CAAC,IAAI,CAAC,mCAAmC,SAAS,qBAAqB,IAAI,CAAC,QAAQ,EAAE,CAAC,CAAC;gBAC3G,IAAI,IAAI,CAAC,MAAM,IAAI,IAAI,CAAC,QAAQ,KAAK,IAAI,CAAC,OAAO,CAAC,gBAAgB,EAAE,CAAC;oBACnE,IAAI,CAAC,OAAO,CAAC,MAAM,CAAC,IAAI,CAAC,yCAAyC,SAAS,EAAE,CAAC,CAAC;oBAC/E,IAAI,CAAC,OAAO,CAAC,aAAa,EAAE,CAAC,MAAM,EAAE,SAAS,CAAC,CAAC;gBAClD,CAAC;gBACD,OAAO;YACT,CAAC;YAAC,OAAO,KAAK,EAAE,CAAC;gBACf,IAAI,CAAC,gBAAgB,CAAC,KAAK,EAAE,SAAS,CAAC,CAAC;gBACxC,4CAA4C;YAC9C,CAAC;QACH,CAAC;aAAM,CAAC;YACN,IAAI,CAAC,qBAAqB,CAAC,qBAAqB,CAAC,CAAC;QACpD,CAAC;QAED,MAAM,OAAO,GAAG,CAAC,MAAM,IAAI,CAAC,OAAO,CAAC,SAAS,CAAC,CAAC,IAAI,EAAE,GAAG,oBAAY,EAAE,CAAC;QACvE,MAAM,IAAI,GAAwB;YAChC,QAAQ,EAAE,OAAO,CAAC,QAAQ,GAAG,CAAC;YAC9B,WAAW,EAAE,IAAI,CAAC,GAAG,EAAE;YACvB,MAAM,EAAE,OAAO,CAAC,QAAQ,GAAG,CAAC,IAAI,IAAI,CAAC,OAAO,CAAC,gBAAgB;YAC7D,sEAAsE;YACtE,oEAAoE;YACpE,qEAAqE;YACrE,iEAAiE;YACjE,kDAAkD;YAClD,SAAS,EAAE,OAAO,CAAC,SAAS,IAAI,CAAC;YACjC,SAAS,EAAE,OAAO,CAAC,SAAS,IAAI,CAAC;YACjC,WAAW,EAAE,OAAO,CAAC,WAAW,IAAI,CAAC;SACtC,CAAC;QACF,MAAM,IAAI,CAAC,OAAO,CAAC,SAAS,EAAE,IAAI,EAAE,IAAI,CAAC,eAAe,CAAC,CAAC,CAAC,CAAC,CAAC;QAC7D,IAAI,CAAC,OAAO,CAAC,MAAM,CAAC,IAAI,CAAC,mCAAmC,SAAS,qBAAqB,IAAI,CAAC,QAAQ,EAAE,CAAC,CAAC;QAE3G,IAAI,IAAI,CAAC,MAAM,IAAI,CAAC,OAAO,CAAC,MAAM,EAAE,CAAC;YACnC,IAAI,CAAC,OAAO,CAAC,MAAM,CAAC,IAAI,CAAC,yCAAyC,SAAS,EAAE,CAAC,CAAC;YAC/E,IAAI,CAAC,OAAO,CAAC,aAAa,EAAE,CAAC,MAAM,EAAE,SAAS,CAAC,CAAC;QAClD,CAAC;IACH,CAAC;IAED;;;;OAIG;IACK,KAAK,CAAC,wBAAwB,CAAC,SAAiB,EAAE,OAAgB;QACxE,IAAI,IAAI,CAAC,OAAO,CAAC,KAAM,CAAC,mBAAmB,EAAE,CAAC;YAC5C,IAAI,CAAC;gBACH,MAAM,IAAI,GAAG,MAAM,IAAI,CAAC,OAAO,CAAC,KAAM,CAAC,mBAAmB,CAAC,IAAI,CAAC,GAAG,CAAC,SAAS,CAAC,EAAE;oBAC9E,OAAO;oBACP,KAAK,EAAE,IAAI,CAAC,OAAO,CAAC,cAAc;oBAClC,YAAY,EAAE,IAAI,CAAC,OAAO,CAAC,YAAY;oBACvC,kBAAkB,EAAE,IAAI,CAAC,OAAO,CAAC,kBAAkB;oBACnD,UAAU,EAAE,IAAI,CAAC,eAAe,CAAC,CAAC,CAAC;oBACnC,GAAG,EAAE,IAAI,CAAC,GAAG,EAAE;iBAChB,CAAC,CAAC;gBACH,IAAI,IAAI,CAAC,SAAS,EAAE,CAAC;oBACnB,IAAI,CAAC,OAAO,CAAC,MAAM,CAAC,IAAI,CACtB,yCAAyC,SAAS,gBAAgB,CAAC,IAAI,CAAC,SAAS,GAAG,GAAG,CAAC,CAAC,OAAO,CAAC,CAAC,CAAC,UAAU,IAAI,CAAC,WAAW,SAAS,CACvI,CAAC;oBACF,IAAI,CAAC,OAAO,CAAC,aAAa,EAAE,CAAC,MAAM,EAAE,SAAS,CAAC,CAAC;gBAClD,CAAC;gBACD,OAAO;YACT,CAAC;YAAC,OAAO,KAAK,EAAE,CAAC;gBACf,IAAI,CAAC,gBAAgB,CAAC,KAAK,EAAE,SAAS,CAAC,CAAC;gBACxC,4CAA4C;YAC9C,CAAC;QACH,CAAC;aAAM,CAAC;YACN,IAAI,CAAC,qBAAqB,CAAC,qBAAqB,CAAC,CAAC;QACpD,CAAC;QAED,MAAM,OAAO,GAAG,CAAC,MAAM,IAAI,CAAC,OAAO,CAAC,SAAS,CAAC,CAAC,IAAI,EAAE,GAAG,oBAAY,EAAE,CAAC;QACvE,MAAM,KAAK,GAAG,IAAI,CAAC,OAAO,CAAC,cAAc,CAAC;QAC1C,MAAM,SAAS,GAAG,CAAC,OAAO,CAAC,SAAS,IAAI,CAAC,CAAC,GAAG,KAAK,GAAG,CAAC,OAAO,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,GAAG,CAAC,CAAC,GAAG,KAAK,CAAC,CAAC;QACrF,MAAM,WAAW,GAAG,CAAC,OAAO,CAAC,WAAW,IAAI,CAAC,CAAC,GAAG,CAAC,CAAC;QACnD,MAAM,MAAM,GAAG,WAAW,IAAI,IAAI,CAAC,OAAO,CAAC,YAAY,IAAI,SAAS,IAAI,IAAI,CAAC,OAAO,CAAC,kBAAkB,CAAC;QAExG,MAAM,IAAI,GAAwB;YAChC,QAAQ,EAAE,OAAO,CAAC,QAAQ;YAC1B,WAAW,EAAE,IAAI,CAAC,GAAG,EAAE;YACvB,MAAM;YACN,SAAS,EAAE,OAAO,CAAC,SAAS,IAAI,CAAC;YACjC,SAAS;YACT,WAAW;SACZ,CAAC;QACF,MAAM,IAAI,CAAC,OAAO,CAAC,SAAS,EAAE,IAAI,EAAE,IAAI,CAAC,eAAe,CAAC,CAAC,CAAC,CAAC,CAAC;QAE7D,IAAI,MAAM,IAAI,CAAC,OAAO,CAAC,MAAM,EAAE,CAAC;YAC9B,IAAI,CAAC,OAAO,CAAC,MAAM,CAAC,IAAI,CACtB,yCAAyC,SAAS,gBAAgB,CAAC,SAAS,GAAG,GAAG,CAAC,CAAC,OAAO,CAAC,CAAC,CAAC,UAAU,WAAW,SAAS,CAC7H,CAAC;YACF,IAAI,CAAC,OAAO,CAAC,aAAa,EAAE,CAAC,MAAM,EAAE,SAAS,CAAC,CAAC;QAClD,CAAC;IACH,CAAC;IAEO,KAAK,CAAC,iCAAiC,CAAC,SAAiB;QAC/D,MAAM,OAAO,GAAG,CAAC,MAAM,IAAI,CAAC,OAAO,CAAC,SAAS,CAAC,CAAC,IAAI,EAAE,GAAG,oBAAY,EAAE,MAAM,EAAE,IAAI,EAAE,CAAC;QACrF,MAAM,SAAS,GAAG,CAAC,OAAO,CAAC,SAAS,IAAI,CAAC,CAAC,GAAG,CAAC,CAAC;QAC/C,MAAM,IAAI,GAAwB;YAChC,QAAQ,EAAE,IAAI,CAAC,GAAG,CAAC,OAAO,CAAC,QAAQ,EAAE,IAAI,CAAC,OAAO,CAAC,gBAAgB,CAAC;YACnE,WAAW,EAAE,IAAI,CAAC,GAAG,EAAE;YACvB,MAAM,EAAE,IAAI;YACZ,SAAS;YACT,SAAS,EAAE,OAAO,CAAC,SAAS,IAAI,CAAC;YACjC,WAAW,EAAE,OAAO,CAAC,WAAW,IAAI,CAAC;SACtC,CAAC;QACF,MAAM,IAAI,CAAC,OAAO,CAAC,SAAS,EAAE,IAAI,EAAE,IAAI,CAAC,eAAe,CAAC,SAAS,CAAC,CAAC,CAAC;QACrE,IAAI,CAAC,OAAO,CAAC,MAAM,CAAC,IAAI,CAAC,yCAAyC,SAAS,0BAA0B,SAAS,GAAG,CAAC,GAAG,CAAC,CAAC;QACvH,IAAI,CAAC,OAAO,CAAC,aAAa,EAAE,CAAC,MAAM,EAAE,SAAS,CAAC,CAAC;IAClD,CAAC;IAED,0EAA0E;IAElE,KAAK,CAAC,YAAY,CAAI,SAAiB,EAAE,EAAoB;QACnE,MAAM,OAAO,GAAG,IAAI,CAAC,UAAU,CAAC,SAAS,CAAC,CAAC;QAC3C,OAAO,CAAC,UAAU,EAAE,CAAC;QAErB,MAAM,IAAI,GAAG,IAAI,CAAC,cAAc,CAAC,SAAS,CAAC,CAAC;QAC5C,IAAI,CAAC,IAAI,CAAC,OAAO,EAAE,CAAC;YAClB,OAAO,CAAC,eAAe,EAAE,CAAC;YAC1B,MAAM,IAAI,KAAK,CAAC,0CAA0C,SAAS,EAAE,CAAC,CAAC;QACzE,CAAC;QAED,IAAI,CAAC;YACH,MAAM,MAAM,GAAG,MAAM,IAAI,CAAC,kBAAkB,CAAC,EAAE,CAAC,CAAC;YACjD,OAAO,CAAC,cAAc,EAAE,CAAC;YACzB,IAAI,IAAI,CAAC,OAAO,EAAE,CAAC;gBACjB,IAAI,CAAC,UAAU,CAAC,SAAS,CAAC,CAAC;YAC7B,CAAC;iBAAM,IAAI,IAAI,CAAC,OAAO,CAAC,QAAQ,KAAK,WAAW,EAAE,CAAC;gBACjD,IAAI,CAAC,sBAAsB,CAAC,SAAS,EAAE,IAAI,CAAC,CAAC;YAC/C,CAAC;iBAAM,CAAC;gBACN,IAAI,CAAC,UAAU,CAAC,SAAS,CAAC,CAAC;YAC7B,CAAC;YACD,OAAO,MAAM,CAAC;QAChB,CAAC;QAAC,OAAO,KAAK,EAAE,CAAC;YACf,OAAO,CAAC,aAAa,EAAE,CAAC;YACxB,IAAI,IAAI,CAAC,OAAO,EAAE,CAAC;gBACjB,IAAI,CAAC,2BAA2B,CAAC,SAAS,CAAC,CAAC;YAC9C,CAAC;iBAAM,IAAI,IAAI,CAAC,OAAO,CAAC,QAAQ,KAAK,WAAW,EAAE,CAAC;gBACjD,IAAI,CAAC,sBAAsB,CAAC,SAAS,EAAE,KAAK,CAAC,CAAC;YAChD,CAAC;iBAAM,CAAC;gBACN,IAAI,CAAC,0BAA0B,CAAC,SAAS,CAAC,CAAC;YAC7C,CAAC;YACD,MAAM,KAAK,CAAC;QACd,CAAC;IACH,CAAC;IAED,gFAAgF;IACxE,aAAa,CAAC,SAAiB;QACrC,IAAI,CAAC,IAAI,CAAC,cAAc,CAAC,GAAG,CAAC,SAAS,CAAC;YAAE,OAAO,IAAI,CAAC;QAErD,MAAM,SAAS,GAAG,IAAI,CAAC,UAAU,CAAC,GAAG,CAAC,SAAS,CAAC,IAAI,CAAC,CAAC;QACtD,MAAM,WAAW,GAAG,IAAI,CAAC,eAAe,CAAC,GAAG,CAAC,SAAS,CAAC,IAAI,CAAC,CAAC;QAC7D,MAAM,SAAS,GAAG,IAAI,CAAC,qBAAqB,CAAC,SAAS,EAAE,SAAS,EAAE,WAAW,CAAC,CAAC;QAEhF,IAAI,IAAI,CAAC,GAAG,EAAE,GAAG,WAAW,GAAG,SAAS,EAAE,CAAC;YACzC,IAAI,CAAC,OAAO,CAAC,MAAM,CAAC,IAAI,CAAC,uCAAuC,SAAS,EAAE,CAAC,CAAC;YAC7E,OAAO,KAAK,CAAC;QACf,CAAC;QACD,OAAO,IAAI,CAAC;IACd,CAAC;IAED,oFAAoF;IAC5E,cAAc,CAAC,SAAiB;QACtC,IAAI,CAAC,IAAI,CAAC,cAAc,CAAC,GAAG,CAAC,SAAS,CAAC;YAAE,OAAO,EAAE,OAAO,EAAE,IAAI,EAAE,OAAO,EAAE,KAAK,EAAE,CAAC;QAElF,MAAM,SAAS,GAAG,IAAI,CAAC,UAAU,CAAC,GAAG,CAAC,SAAS,CAAC,IAAI,CAAC,CAAC;QACtD,MAAM,WAAW,GAAG,IAAI,CAAC,eAAe,CAAC,GAAG,CAAC,SAAS,CAAC,IAAI,CAAC,CAAC;QAC7D,MAAM,SAAS,GAAG,IAAI,CAAC,qBAAqB,CAAC,SAAS,EAAE,SAAS,EAAE,WAAW,CAAC,CAAC;QAEhF,IAAI,IAAI,CAAC,GAAG,EAAE,GAAG,WAAW,GAAG,SAAS,EAAE,CAAC;YACzC,IAAI,CAAC,OAAO,CAAC,MAAM,CAAC,IAAI,CAAC,uCAAuC,SAAS,EAAE,CAAC,CAAC;YAC7E,OAAO,EAAE,OAAO,EAAE,KAAK,EAAE,OAAO,EAAE,KAAK,EAAE,CAAC;QAC5C,CAAC;QAED,IAAI,IAAI,CAAC,eAAe,CAAC,GAAG,CAAC,SAAS,CAAC,EAAE,CAAC;YACxC,oFAAoF;YACpF,OAAO,EAAE,OAAO,EAAE,KAAK,EAAE,OAAO,EAAE,KAAK,EAAE,CAAC;QAC5C,CAAC;QAED,IAAI,CAAC,eAAe,CAAC,GAAG,CAAC,SAAS,CAAC,CAAC;QACpC,IAAI,CAAC,cAAc,CAAC,GAAG,CAAC,SAAS,EAAE,IAAI,CAAC,GAAG,EAAE,CAAC,CAAC;QAC/C,IAAI,CAAC,OAAO,CAAC,aAAa,EAAE,CAAC,WAAW,EAAE,SAAS,CAAC,CAAC;QACrD,OAAO,EAAE,OAAO,EAAE,IAAI,EAAE,OAAO,EAAE,IAAI,EAAE,CAAC;IAC1C,CAAC;IAEO,0BAA0B,CAAC,SAAiB;QAClD,MAAM,QAAQ,GAAG,CAAC,IAAI,CAAC,QAAQ,CAAC,GAAG,CAAC,SAAS,CAAC,IAAI,CAAC,CAAC,GAAG,CAAC,CAAC;QACzD,IAAI,CAAC,QAAQ,CAAC,GAAG,CAAC,SAAS,EAAE,QAAQ,CAAC,CAAC;QACvC,IAAI,CAAC,eAAe,CAAC,GAAG,CAAC,SAAS,EAAE,IAAI,CAAC,GAAG,EAAE,CAAC,CAAC;QAChD,IAAI,CAAC,OAAO,CAAC,MAAM,CAAC,IAAI,CAAC,mCAAmC,SAAS,qBAAqB,QAAQ,EAAE,CAAC,CAAC;QAEtG,IAAI,QAAQ,IAAI,IAAI,CAAC,OAAO,CAAC,gBAAgB,IAAI,CAAC,IAAI,CAAC,cAAc,CAAC,GAAG,CAAC,SAAS,CAAC,EAAE,CAAC;YACrF,IAAI,CAAC,cAAc,CAAC,GAAG,CAAC,SAAS,CAAC,CAAC;YACnC,IAAI,CAAC,UAAU,CAAC,GAAG,CAAC,SAAS,EAAE,CAAC,CAAC,CAAC;YAClC,IAAI,CAAC,OAAO,CAAC,MAAM,CAAC,IAAI,CAAC,yCAAyC,SAAS,EAAE,CAAC,CAAC;YAC/E,IAAI,CAAC,OAAO,CAAC,aAAa,EAAE,CAAC,MAAM,EAAE,SAAS,CAAC,CAAC;QAClD,CAAC;IACH,CAAC;IAED,kIAAkI;IAC1H,sBAAsB,CAAC,SAAiB,EAAE,OAAgB;QAChE,MAAM,KAAK,GAAG,IAAI,CAAC,OAAO,CAAC,cAAc,CAAC;QAC1C,MAAM,IAAI,GAAG,CAAC,IAAI,CAAC,UAAU,CAAC,GAAG,CAAC,SAAS,CAAC,IAAI,CAAC,CAAC,GAAG,KAAK,GAAG,CAAC,OAAO,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,GAAG,CAAC,CAAC,GAAG,KAAK,CAAC,CAAC;QAC7F,MAAM,KAAK,GAAG,CAAC,IAAI,CAAC,YAAY,CAAC,GAAG,CAAC,SAAS,CAAC,IAAI,CAAC,CAAC,GAAG,CAAC,CAAC;QAC1D,IAAI,CAAC,UAAU,CAAC,GAAG,CAAC,SAAS,EAAE,IAAI,CAAC,CAAC;QACrC,IAAI,CAAC,YAAY,CAAC,GAAG,CAAC,SAAS,EAAE,KAAK,CAAC,CAAC;QAExC,IAAI,KAAK,IAAI,IAAI,CAAC,OAAO,CAAC,YAAY,IAAI,IAAI,IAAI,IAAI,CAAC,OAAO,CAAC,kBAAkB,IAAI,CAAC,IAAI,CAAC,cAAc,CAAC,GAAG,CAAC,SAAS,CAAC,EAAE,CAAC;YACzH,IAAI,CAAC,cAAc,CAAC,GAAG,CAAC,SAAS,CAAC,CAAC;YACnC,IAAI,CAAC,UAAU,CAAC,GAAG,CAAC,SAAS,EAAE,CAAC,CAAC,CAAC;YAClC,IAAI,CAAC,eAAe,CAAC,GAAG,CAAC,SAAS,EAAE,IAAI,CAAC,GAAG,EAAE,CAAC,CAAC;YAChD,IAAI,CAAC,OAAO,CAAC,MAAM,CAAC,IAAI,CACtB,yCAAyC,SAAS,gBAAgB,CAAC,IAAI,GAAG,GAAG,CAAC,CAAC,OAAO,CAAC,CAAC,CAAC,UAAU,KAAK,SAAS,CAClH,CAAC;YACF,IAAI,CAAC,OAAO,CAAC,aAAa,EAAE,CAAC,MAAM,EAAE,SAAS,CAAC,CAAC;QAClD,CAAC;IACH,CAAC;IAEO,2BAA2B,CAAC,SAAiB;QACnD,IAAI,CAAC,eAAe,CAAC,MAAM,CAAC,SAAS,CAAC,CAAC;QACvC,IAAI,CAAC,cAAc,CAAC,MAAM,CAAC,SAAS,CAAC,CAAC;QACtC,MAAM,SAAS,GAAG,CAAC,IAAI,CAAC,UAAU,CAAC,GAAG,CAAC,SAAS,CAAC,IAAI,CAAC,CAAC,GAAG,CAAC,CAAC;QAC5D,IAAI,CAAC,UAAU,CAAC,GAAG,CAAC,SAAS,EAAE,SAAS,CAAC,CAAC;QAC1C,IAAI,CAAC,eAAe,CAAC,GAAG,CAAC,SAAS,EAAE,IAAI,CAAC,GAAG,EAAE,CAAC,CAAC;QAChD,IAAI,CAAC,OAAO,CAAC,MAAM,CAAC,IAAI,CAAC,yCAAyC,SAAS,0BAA0B,SAAS,GAAG,CAAC,GAAG,CAAC,CAAC;QACvH,IAAI,CAAC,OAAO,CAAC,aAAa,EAAE,CAAC,MAAM,EAAE,SAAS,CAAC,CAAC;IAClD,CAAC;IAEO,UAAU,CAAC,SAAiB;QAClC,MAAM,OAAO,GAAG,IAAI,CAAC,cAAc,CAAC,GAAG,CAAC,SAAS,CAAC,CAAC;QACnD,IAAI,CAAC,QAAQ,CAAC,MAAM,CAAC,SAAS,CAAC,CAAC;QAChC,IAAI,CAAC,eAAe,CAAC,MAAM,CAAC,SAAS,CAAC,CAAC;QACvC,IAAI,CAAC,cAAc,CAAC,MAAM,CAAC,SAAS,CAAC,CAAC;QACtC,IAAI,CAAC,UAAU,CAAC,MAAM,CAAC,SAAS,CAAC,CAAC;QAClC,IAAI,CAAC,eAAe,CAAC,MAAM,CAAC,SAAS,CAAC,CAAC;QACvC,IAAI,CAAC,cAAc,CAAC,MAAM,CAAC,SAAS,CAAC,CAAC;QACtC,IAAI,CAAC,UAAU,CAAC,MAAM,CAAC,SAAS,CAAC,CAAC;QAClC,IAAI,CAAC,YAAY,CAAC,MAAM,CAAC,SAAS,CAAC,CAAC;QACpC,IAAI,CAAC,mBAAmB,CAAC,MAAM,CAAC,SAAS,CAAC,CAAC;QAC3C,IAAI,OAAO,EAAE,CAAC;YACZ,IAAI,CAAC,OAAO,CAAC,aAAa,EAAE,CAAC,QAAQ,EAAE,SAAS,CAAC,CAAC;QACpD,CAAC;QACD,IAAI,CAAC,OAAO,CAAC,MAAM,CAAC,GAAG,CAAC,wCAAwC,SAAS,EAAE,CAAC,CAAC;IAC/E,CAAC;IAED;;;;;;OAMG;IACK,OAAO;QACb,MAAM,GAAG,GAAG,IAAI,CAAC,GAAG,EAAE,CAAC;QACvB,MAAM,UAAU,GAAG,IAAI,CAAC,GAAG,CAAC,IAAI,CAAC,OAAO,CAAC,YAAY,EAAE,IAAI,CAAC,GAAG,CAAC,CAAC;QACjE,KAAK,MAAM,CAAC,SAAS,EAAE,SAAS,CAAC,IAAI,IAAI,CAAC,cAAc,CAAC,OAAO,EAAE,EAAE,CAAC;YACnE,IAAI,GAAG,GAAG,SAAS,GAAG,UAAU,EAAE,CAAC;gBACjC,IAAI,CAAC,eAAe,CAAC,MAAM,CAAC,SAAS,CAAC,CAAC;gBACvC,IAAI,CAAC,cAAc,CAAC,MAAM,CAAC,SAAS,CAAC,CAAC;gBACtC,IAAI,CAAC,OAAO,CAAC,MAAM,CAAC,IAAI,CACtB,wCAAwC,SAAS,0EAA0E,UAAU,qCAAqC,CAC3K,CAAC;YACJ,CAAC;QACH,CAAC;IACH,CAAC;IAED,4EAA4E;IAEpE,kBAAkB,CAAI,EAAoB;QAChD,IAAI,CAAC,IAAI,CAAC,OAAO,CAAC,OAAO;YAAE,OAAO,EAAE,EAAE,CAAC;QAEvC,OAAO,IAAI,OAAO,CAAI,CAAC,OAAO,EAAE,MAAM,EAAE,EAAE;YACxC,MAAM,KAAK,GAAG,UAAU,CAAC,GAAG,EAAE,CAAC,MAAM,CAAC,IAAI,KAAK,CAAC,6BAA6B,IAAI,CAAC,OAAO,CAAC,OAAO,IAAI,CAAC,CAAC,EAAE,IAAI,CAAC,OAAO,CAAC,OAAO,CAAC,CAAC;YAC/H,EAAE,EAAE,CAAC,IAAI,CACP,CAAC,KAAK,EAAE,EAAE;gBACR,YAAY,CAAC,KAAK,CAAC,CAAC;gBACpB,OAAO,CAAC,KAAK,CAAC,CAAC;YACjB,CAAC,EACD,CAAC,KAAK,EAAE,EAAE;gBACR,YAAY,CAAC,KAAK,CAAC,CAAC;gBACpB,MAAM,CAAC,KAAK,CAAC,CAAC;YAChB,CAAC,CACF,CAAC;QACJ,CAAC,CAAC,CAAC;IACL,CAAC;CACF;AAzuBD,wCAyuBC"}
|
|
1
|
+
{"version":3,"file":"circuit-breaker.js","sourceRoot":"","sources":["../src/circuit-breaker.ts"],"names":[],"mappings":";;;AACA,mCAA0F;AAW1F;;;;;GAKG;AACH,MAAa,qBAAsB,SAAQ,KAAK;IAG9C,YAAY,SAAiB,EAAE,KAAc;QAC3C,KAAK,CAAC,oDAAoD,SAAS,EAAE,CAAC,CAAC;QACvE,IAAI,CAAC,IAAI,GAAG,uBAAuB,CAAC;QACpC,IAAI,CAAC,KAAK,GAAG,KAAK,CAAC;IACrB,CAAC;CACF;AARD,sDAQC;AA8GD,MAAM,UAAU,GAAkB,EAAE,IAAI,EAAE,GAAG,EAAE,GAAE,CAAC,EAAE,GAAG,EAAE,GAAG,EAAE,GAAE,CAAC,EAAE,CAAC;AAwBpE;;;;;;;;;;;;;;GAcG;AACH,MAAa,cAAc;IAwDzB,YAAY,UAAiC,EAAE;QA/B9B,uBAAkB,GAAG,IAAI,GAAG,EAAU,CAAC;QAChD,sCAAiC,GAAG,KAAK,CAAC;QACjC,oBAAe,GAAG,IAAI,GAAG,EAAU,CAAC;QAC7C,wBAAmB,GAAG,KAAK,CAAC;QAC5B,+BAA0B,GAAG,KAAK,CAAC;QAC1B,YAAO,GAAG,IAAI,GAAG,EAA0B,CAAC;QAC5C,wBAAmB,GAAG,IAAI,GAAG,EAAqE,CAAC;QACpH;;;;;;;;;WASG;QACc,eAAU,GAAG,IAAI,GAAG,EAA8F,CAAC;QAEpI,mBAAmB;QACF,aAAQ,GAAG,IAAI,GAAG,EAAkB,CAAC;QACrC,oBAAe,GAAG,IAAI,GAAG,EAAkB,CAAC;QAC5C,mBAAc,GAAG,IAAI,GAAG,EAAU,CAAC;QACnC,eAAU,GAAG,IAAI,GAAG,EAAkB,CAAC;QACvC,oBAAe,GAAG,IAAI,GAAG,EAAU,CAAC;QACpC,mBAAc,GAAG,IAAI,GAAG,EAAkB,CAAC;QAC3C,eAAU,GAAG,IAAI,GAAG,EAAkB,CAAC;QACvC,iBAAY,GAAG,IAAI,GAAG,EAAkB,CAAC;QAKxD,MAAM,YAAY,GAAG,OAAO,CAAC,YAAY,IAAI,KAAK,CAAC;QACnD,IAAI,CAAC,OAAO,GAAG;YACb,QAAQ,EAAE,OAAO,CAAC,QAAQ,IAAI,aAAa;YAC3C,gBAAgB,EAAE,OAAO,CAAC,gBAAgB,IAAI,CAAC;YAC/C,kBAAkB,EAAE,OAAO,CAAC,kBAAkB,IAAI,GAAG;YACrD,YAAY,EAAE,OAAO,CAAC,YAAY,IAAI,EAAE;YACxC,cAAc,EAAE,OAAO,CAAC,cAAc,IAAI,GAAG;YAC7C,YAAY;YACZ,eAAe,EAAE,OAAO,CAAC,eAAe,IAAI,IAAI;YAChD,OAAO,EAAE,OAAO,CAAC,OAAO;YACxB,YAAY,EAAE,OAAO,CAAC,YAAY,IAAI,IAAI,CAAC,GAAG,CAAC,OAAO,CAAC,OAAO,IAAI,KAAK,EAAE,YAAY,CAAC;YACtF,iBAAiB,EAAE,OAAO,CAAC,iBAAiB,IAAI,CAAC;YACjD,eAAe,EAAE,OAAO,CAAC,eAAe,IAAI,YAAY,GAAG,CAAC;YAC5D,MAAM,EAAE,OAAO,CAAC,MAAM,IAAI,GAAG;YAC7B,aAAa,EAAE,OAAO,CAAC,aAAa;YACpC,KAAK,EAAE,OAAO,CAAC,KAAK;YACpB,oBAAoB,EAAE,OAAO,CAAC,oBAAoB,IAAI,IAAI;YAC1D,YAAY,EAAE,OAAO,CAAC,YAAY;YAClC,MAAM,EAAE,OAAO,CAAC,MAAM,IAAI,UAAU;YACpC,aAAa,EAAE,OAAO,CAAC,aAAa;YACpC,kBAAkB,EAAE,OAAO,CAAC,kBAAkB,IAAI,IAAI;SACvD,CAAC;QAEF,IAAI,CAAC,WAAW,GAAG,CAAC,CAAC,OAAO,CAAC,KAAK,CAAC;QAEnC,IAAI,CAAC,aAAa,GAAG,WAAW,CAAC,GAAG,EAAE,CAAC,IAAI,CAAC,OAAO,EAAE,EAAE,IAAI,CAAC,OAAO,CAAC,eAAe,CAAC,CAAC;QACrF,IAAI,CAAC,aAAa,CAAC,KAAK,EAAE,EAAE,CAAC;IAC/B,CAAC;IAED,6FAA6F;IAC7F,OAAO;QACL,aAAa,CAAC,IAAI,CAAC,aAAa,CAAC,CAAC;IACpC,CAAC;IAED,KAAK,CAAC,OAAO,CAAI,SAAiB,EAAE,EAAoB;QACtD,IAAI,CAAC,kBAAkB,CAAC,SAAS,CAAC,CAAC;QACnC,OAAO,IAAI,CAAC,WAAW,CAAC,CAAC,CAAC,IAAI,CAAC,kBAAkB,CAAC,SAAS,EAAE,EAAE,CAAC,CAAC,CAAC,CAAC,IAAI,CAAC,YAAY,CAAC,SAAS,EAAE,EAAE,CAAC,CAAC;IACtG,CAAC;IAED;;;;;;OAMG;IACK,kBAAkB,CAAC,SAAiB;QAC1C,IAAI,CAAC,IAAI,CAAC,0BAA0B,IAAI,CAAC,SAAS,CAAC,MAAM,GAAG,GAAG,IAAI,SAAS,CAAC,QAAQ,CAAC,GAAG,CAAC,CAAC,EAAE,CAAC;YAC5F,IAAI,CAAC,0BAA0B,GAAG,IAAI,CAAC;YACvC,MAAM,KAAK,GAAG,SAAS,CAAC,MAAM,GAAG,GAAG,CAAC,CAAC,CAAC,GAAG,SAAS,CAAC,KAAK,CAAC,CAAC,EAAE,GAAG,CAAC,GAAG,CAAC,CAAC,CAAC,SAAS,CAAC;YACjF,IAAI,CAAC,OAAO,CAAC,MAAM,CAAC,IAAI,CACtB,mCAAmC,KAAK,oBACtC,SAAS,CAAC,MAAM,GAAG,GAAG,CAAC,CAAC,CAAC,GAAG,SAAS,CAAC,MAAM,QAAQ,CAAC,CAAC,CAAC,cACzD,uNAAuN,CACxN,CAAC;QACJ,CAAC;QAED,IAAI,IAAI,CAAC,OAAO,CAAC,aAAa,KAAK,SAAS,IAAI,CAAC,IAAI,CAAC,mBAAmB,IAAI,CAAC,IAAI,CAAC,eAAe,CAAC,GAAG,CAAC,SAAS,CAAC,EAAE,CAAC;YAClH,IAAI,CAAC,eAAe,CAAC,GAAG,CAAC,SAAS,CAAC,CAAC;YACpC,IAAI,IAAI,CAAC,eAAe,CAAC,IAAI,GAAG,IAAI,CAAC,OAAO,CAAC,aAAa,EAAE,CAAC;gBAC3D,IAAI,CAAC,mBAAmB,GAAG,IAAI,CAAC;gBAChC,IAAI,CAAC,OAAO,CAAC,MAAM,CAAC,IAAI,CACtB,4BAA4B,IAAI,CAAC,eAAe,CAAC,IAAI,uDAAuD,IAAI,CAAC,OAAO,CAAC,aAAa,8HAA8H,CACrQ,CAAC;YACJ,CAAC;QACH,CAAC;IACH,CAAC;IAED;;;;;;;;OAQG;IACH,UAAU,CAAC,SAAiB;QAC1B,IAAI,IAAI,CAAC,WAAW,EAAE,CAAC;YACrB,IAAI,CAAC,IAAI,CAAC,iCAAiC,EAAE,CAAC;gBAC5C,IAAI,CAAC,iCAAiC,GAAG,IAAI,CAAC;gBAC9C,IAAI,CAAC,OAAO,CAAC,MAAM,CAAC,IAAI,CACtB,kMAAkM,CACnM,CAAC;YACJ,CAAC;YACD,MAAM,MAAM,GAAG,IAAI,CAAC,UAAU,CAAC,GAAG,CAAC,SAAS,CAAC,CAAC;YAC9C,IAAI,MAAM,IAAI,IAAI,CAAC,OAAO,CAAC,kBAAkB,GAAG,CAAC,IAAI,IAAI,CAAC,GAAG,EAAE,GAAG,MAAM,CAAC,SAAS,EAAE,CAAC;gBACnF,OAAO,KAAK,CAAC;YACf,CAAC;YACD,OAAO,IAAI,CAAC;QACd,CAAC;QACD,OAAO,IAAI,CAAC,aAAa,CAAC,SAAS,CAAC,CAAC;IACvC,CAAC;IAED;;;;;OAKG;IACH,KAAK,CAAC,eAAe,CAAC,SAAiB;QACrC,IAAI,CAAC,IAAI,CAAC,WAAW;YAAE,OAAO,IAAI,CAAC,aAAa,CAAC,SAAS,CAAC,CAAC;QAC5D,MAAM,IAAI,GAAG,MAAM,IAAI,CAAC,eAAe,CAAC,SAAS,EAAE,KAAK,CAAC,CAAC;QAC1D,OAAO,IAAI,CAAC,OAAO,CAAC;IACtB,CAAC;IAED,aAAa,CAAC,SAAiB;QAC7B,IAAI,CAAC,kBAAkB,CAAC,SAAS,CAAC,CAAC;QACnC,IAAI,IAAI,CAAC,WAAW,EAAE,CAAC;YACrB,IAAI,IAAI,CAAC,OAAO,CAAC,QAAQ,KAAK,WAAW,EAAE,CAAC;gBAC1C,KAAK,IAAI,CAAC,wBAAwB,CAAC,SAAS,EAAE,KAAK,CAAC,CAAC;YACvD,CAAC;iBAAM,CAAC;gBACN,KAAK,IAAI,CAAC,gCAAgC,CAAC,SAAS,CAAC,CAAC;YACxD,CAAC;QACH,CAAC;aAAM,IAAI,IAAI,CAAC,OAAO,CAAC,QAAQ,KAAK,WAAW,EAAE,CAAC;YACjD,IAAI,CAAC,sBAAsB,CAAC,SAAS,EAAE,KAAK,CAAC,CAAC;QAChD,CAAC;aAAM,CAAC;YACN,IAAI,CAAC,0BAA0B,CAAC,SAAS,CAAC,CAAC;QAC7C,CAAC;IACH,CAAC;IAED,aAAa,CAAC,SAAiB;QAC7B,IAAI,CAAC,kBAAkB,CAAC,SAAS,CAAC,CAAC;QACnC,IAAI,IAAI,CAAC,WAAW,EAAE,CAAC;YACrB,KAAK,IAAI,CAAC,yBAAyB,CAAC,SAAS,CAAC,CAAC;QACjD,CAAC;aAAM,CAAC;YACN,IAAI,CAAC,UAAU,CAAC,SAAS,CAAC,CAAC;QAC7B,CAAC;IACH,CAAC;IAED;;;;;;;;;OASG;IACK,KAAK,CAAC,yBAAyB,CAAC,SAAiB;QACvD,MAAM,KAAK,GAAG,MAAM,IAAI,CAAC,OAAO,CAAC,SAAS,CAAC,CAAC;QAC5C,MAAM,IAAI,CAAC,gBAAgB,CAAC,SAAS,CAAC,CAAC;QACvC,IAAI,KAAK,EAAE,MAAM,EAAE,CAAC;YAClB,IAAI,CAAC,OAAO,CAAC,aAAa,EAAE,CAAC,QAAQ,EAAE,SAAS,CAAC,CAAC;QACpD,CAAC;IACH,CAAC;IAED,KAAK,CAAC,QAAQ,CAAC,SAAiB;QAC9B,IAAI,IAAI,CAAC,WAAW,EAAE,CAAC;YACrB,OAAO,CAAC,MAAM,IAAI,CAAC,OAAO,CAAC,SAAS,CAAC,CAAC,IAAI,EAAE,GAAG,oBAAY,EAAE,CAAC;QAChE,CAAC;QACD,MAAM,QAAQ,GAAG,IAAI,CAAC,QAAQ,CAAC,GAAG,CAAC,SAAS,CAAC,IAAI,CAAC,CAAC;QACnD,MAAM,WAAW,GAAG,IAAI,CAAC,eAAe,CAAC,GAAG,CAAC,SAAS,CAAC,IAAI,CAAC,CAAC;QAC7D,MAAM,SAAS,GAAG,IAAI,CAAC,UAAU,CAAC,GAAG,CAAC,SAAS,CAAC,IAAI,CAAC,CAAC;QACtD,MAAM,SAAS,GAAG,IAAI,CAAC,UAAU,CAAC,GAAG,CAAC,SAAS,CAAC,IAAI,CAAC,CAAC;QACtD,MAAM,WAAW,GAAG,IAAI,CAAC,YAAY,CAAC,GAAG,CAAC,SAAS,CAAC,IAAI,CAAC,CAAC;QAC1D,OAAO,EAAE,QAAQ,EAAE,WAAW,EAAE,MAAM,EAAE,IAAI,CAAC,cAAc,CAAC,GAAG,CAAC,SAAS,CAAC,EAAE,SAAS,EAAE,SAAS,EAAE,WAAW,EAAE,CAAC;IAClH,CAAC;IAED,KAAK,CAAC,KAAK,CAAC,SAAiB;QAC3B,IAAI,IAAI,CAAC,WAAW,EAAE,CAAC;YACrB,MAAM,KAAK,GAAG,MAAM,IAAI,CAAC,OAAO,CAAC,SAAS,CAAC,CAAC;YAC5C,MAAM,IAAI,CAAC,WAAW,CAAC,SAAS,CAAC,CAAC;YAClC,MAAM,IAAI,CAAC,YAAY,CAAC,SAAS,CAAC,CAAC;YACnC,IAAI,CAAC,mBAAmB,CAAC,MAAM,CAAC,SAAS,CAAC,CAAC;YAC3C,IAAI,CAAC,UAAU,CAAC,MAAM,CAAC,SAAS,CAAC,CAAC;YAClC,IAAI,KAAK,EAAE,MAAM,EAAE,CAAC;gBAClB,IAAI,CAAC,OAAO,CAAC,aAAa,EAAE,CAAC,QAAQ,EAAE,SAAS,CAAC,CAAC;YACpD,CAAC;YACD,IAAI,CAAC,OAAO,CAAC,MAAM,CAAC,GAAG,CAAC,wCAAwC,SAAS,EAAE,CAAC,CAAC;QAC/E,CAAC;aAAM,CAAC;YACN,IAAI,CAAC,UAAU,CAAC,SAAS,CAAC,CAAC;QAC7B,CAAC;IACH,CAAC;IAED,wPAAwP;IACxP,UAAU,CAAC,SAAiB;QAC1B,MAAM,CAAC,GAAG,IAAI,CAAC,OAAO,CAAC,GAAG,CAAC,SAAS,CAAC,CAAC;QACtC,OAAO,CAAC,CAAC,CAAC,CAAC,EAAE,GAAG,CAAC,EAAE,CAAC,CAAC,CAAC,IAAA,oBAAY,GAAE,CAAC;IACvC,CAAC;IAED,aAAa;QACX,MAAM,MAAM,GAAmC,EAAE,CAAC;QAClD,KAAK,MAAM,CAAC,SAAS,EAAE,CAAC,CAAC,IAAI,IAAI,CAAC,OAAO,CAAC,OAAO,EAAE;YAAE,MAAM,CAAC,SAAS,CAAC,GAAG,EAAE,GAAG,CAAC,EAAE,CAAC;QAClF,OAAO,MAAM,CAAC;IAChB,CAAC;IAEO,UAAU,CAAC,SAAiB;QAClC,IAAI,CAAC,GAAG,IAAI,CAAC,OAAO,CAAC,GAAG,CAAC,SAAS,CAAC,CAAC;QACpC,IAAI,CAAC,CAAC,EAAE,CAAC;YACP,CAAC,GAAG,IAAA,oBAAY,GAAE,CAAC;YACnB,IAAI,CAAC,OAAO,CAAC,GAAG,CAAC,SAAS,EAAE,CAAC,CAAC,CAAC;QACjC,CAAC;QACD,OAAO,CAAC,CAAC;IACX,CAAC;IAED,0EAA0E;IAElE,aAAa,CAAC,SAAiB;QACrC,MAAM,MAAM,GAAG,IAAI,CAAC,OAAO,CAAC,YAAY,GAAG,IAAI,CAAC,GAAG,CAAC,IAAI,CAAC,OAAO,CAAC,iBAAiB,EAAE,SAAS,CAAC,CAAC;QAC/F,OAAO,IAAI,CAAC,GAAG,CAAC,MAAM,EAAE,IAAI,CAAC,OAAO,CAAC,eAAe,CAAC,CAAC;IACxD,CAAC;IAED;;;;;;;;;OASG;IACK,qBAAqB,CAAC,SAAiB,EAAE,SAAiB,EAAE,WAAmB;QACrF,MAAM,MAAM,GAAG,IAAI,CAAC,mBAAmB,CAAC,GAAG,CAAC,SAAS,CAAC,CAAC;QACvD,IAAI,MAAM,IAAI,MAAM,CAAC,SAAS,KAAK,SAAS,IAAI,MAAM,CAAC,WAAW,KAAK,WAAW,EAAE,CAAC;YACnF,OAAO,MAAM,CAAC,KAAK,CAAC;QACtB,CAAC;QACD,MAAM,MAAM,GAAG,IAAI,CAAC,aAAa,CAAC,SAAS,CAAC,CAAC;QAC7C,IAAI,KAAK,GAAG,MAAM,CAAC;QACnB,IAAI,IAAI,CAAC,OAAO,CAAC,MAAM,EAAE,CAAC;YACxB,MAAM,KAAK,GAAG,MAAM,GAAG,IAAI,CAAC,OAAO,CAAC,MAAM,CAAC;YAC3C,KAAK,GAAG,MAAM,GAAG,CAAC,IAAI,CAAC,MAAM,EAAE,GAAG,CAAC,GAAG,CAAC,CAAC,GAAG,KAAK,CAAC;QACnD,CAAC;QACD,IAAI,CAAC,mBAAmB,CAAC,GAAG,CAAC,SAAS,EAAE,EAAE,SAAS,EAAE,WAAW,EAAE,KAAK,EAAE,CAAC,CAAC;QAC3E,OAAO,KAAK,CAAC;IACf,CAAC;IAED;;;;;;;OAOG;IACK,cAAc,CAAC,SAAiB,EAAE,KAA6D;QACrG,MAAM,SAAS,GAAG,KAAK,CAAC,SAAS,IAAI,CAAC,CAAC;QACvC,MAAM,SAAS,GAAG,IAAI,CAAC,qBAAqB,CAAC,SAAS,EAAE,SAAS,EAAE,KAAK,CAAC,WAAW,CAAC,CAAC;QACtF,IAAI,CAAC,UAAU,CAAC,GAAG,CAAC,SAAS,EAAE;YAC7B,WAAW,EAAE,KAAK,CAAC,WAAW;YAC9B,SAAS;YACT,SAAS,EAAE,KAAK,CAAC,WAAW,GAAG,SAAS;YACxC,WAAW,EAAE,IAAI,CAAC,GAAG,EAAE;SACxB,CAAC,CAAC;IACL,CAAC;IAED,wEAAwE;IAEhE,GAAG,CAAC,SAAiB;QAC3B,OAAO,mBAAmB,SAAS,EAAE,CAAC;IACxC,CAAC;IAEO,QAAQ,CAAC,SAAiB;QAChC,OAAO,mBAAmB,SAAS,QAAQ,CAAC;IAC9C,CAAC;IAEO,eAAe,CAAC,SAAiB;QACvC,uEAAuE;QACvE,kEAAkE;QAClE,sEAAsE;QACtE,gEAAgE;QAChE,uEAAuE;QACvE,kCAAkC;QAClC,OAAO,IAAI,CAAC,GAAG,CAAC,CAAC,EAAE,IAAI,CAAC,IAAI,CAAC,CAAC,IAAI,CAAC,aAAa,CAAC,SAAS,CAAC,GAAG,CAAC,CAAC,GAAG,IAAI,CAAC,CAAC,CAAC;IAC5E,CAAC;IAEO,eAAe;QACrB,OAAO,IAAI,CAAC,GAAG,CAAC,CAAC,EAAE,IAAI,CAAC,IAAI,CAAC,IAAI,CAAC,OAAO,CAAC,YAAY,GAAG,IAAI,CAAC,CAAC,CAAC;IAClE,CAAC;IAEO,gBAAgB,CAAC,KAAc,EAAE,SAAiB;QACxD,IAAI,CAAC,UAAU,CAAC,SAAS,CAAC,CAAC,gBAAgB,EAAE,CAAC;QAC9C,IAAI,CAAC,OAAO,CAAC,MAAM,CAAC,IAAI,CAAC,8CAA8C,SAAS,KAAM,KAAe,EAAE,OAAO,IAAI,KAAK,EAAE,CAAC,CAAC;QAC3H,IAAI,CAAC,OAAO,CAAC,YAAY,EAAE,CAAC,KAAK,EAAE,SAAS,CAAC,CAAC;IAChD,CAAC;IAEO,qBAAqB,CAAC,UAAyF;QACrH,IAAI,IAAI,CAAC,kBAAkB,CAAC,GAAG,CAAC,UAAU,CAAC;YAAE,OAAO;QACpD,IAAI,CAAC,kBAAkB,CAAC,GAAG,CAAC,UAAU,CAAC,CAAC;QACxC,IAAI,CAAC,OAAO,CAAC,MAAM,CAAC,IAAI,CACtB,4BAA4B,UAAU,oGAAoG,CAC3I,CAAC;IACJ,CAAC;IAEO,KAAK,CAAC,OAAO,CAAC,SAAiB;QACrC,IAAI,CAAC;YACH,OAAO,MAAM,IAAI,CAAC,OAAO,CAAC,KAAM,CAAC,GAAG,CAAsB,IAAI,CAAC,GAAG,CAAC,SAAS,CAAC,CAAC,CAAC;QACjF,CAAC;QAAC,OAAO,KAAK,EAAE,CAAC;YACf,IAAI,CAAC,gBAAgB,CAAC,KAAK,EAAE,SAAS,CAAC,CAAC;YACxC,OAAO,IAAI,CAAC;QACd,CAAC;IACH,CAAC;IAEO,KAAK,CAAC,OAAO,CAAC,SAAiB,EAAE,KAA0B,EAAE,UAAkB;QACrF,IAAI,CAAC;YACH,MAAM,IAAI,CAAC,OAAO,CAAC,KAAM,CAAC,GAAG,CAAC,IAAI,CAAC,GAAG,CAAC,SAAS,CAAC,EAAE,KAAK,EAAE,UAAU,CAAC,CAAC;QACxE,CAAC;QAAC,OAAO,KAAK,EAAE,CAAC;YACf,IAAI,CAAC,gBAAgB,CAAC,KAAK,EAAE,SAAS,CAAC,CAAC;QAC1C,CAAC;IACH,CAAC;IAED,qFAAqF;IAC7E,KAAK,CAAC,WAAW,CAAC,SAAiB;QACzC,IAAI,CAAC;YACH,MAAM,IAAI,CAAC,OAAO,CAAC,KAAM,CAAC,GAAG,CAAC,IAAI,CAAC,GAAG,CAAC,SAAS,CAAC,CAAC,CAAC;QACrD,CAAC;QAAC,OAAO,KAAK,EAAE,CAAC;YACf,IAAI,CAAC,gBAAgB,CAAC,KAAK,EAAE,SAAS,CAAC,CAAC;QAC1C,CAAC;IACH,CAAC;IAED,4NAA4N;IACpN,KAAK,CAAC,YAAY,CAAC,SAAiB;QAC1C,IAAI,CAAC;YACH,MAAM,IAAI,CAAC,OAAO,CAAC,KAAM,CAAC,GAAG,CAAC,IAAI,CAAC,QAAQ,CAAC,SAAS,CAAC,CAAC,CAAC;QAC1D,CAAC;QAAC,OAAO,KAAK,EAAE,CAAC;YACf,IAAI,CAAC,gBAAgB,CAAC,KAAK,EAAE,SAAS,CAAC,CAAC;QAC1C,CAAC;IACH,CAAC;IAED;;;;;;;;;;;;;;;;OAgBG;IACK,KAAK,CAAC,gBAAgB,CAAC,SAAiB,EAAE,KAAyB;QACzE,IAAI,CAAC,KAAK;YAAE,OAAO;QACnB,IAAI,CAAC,IAAI,CAAC,OAAO,CAAC,KAAM,CAAC,YAAY,EAAE,CAAC;YACtC,IAAI,CAAC,qBAAqB,CAAC,cAAc,CAAC,CAAC;YAC3C,OAAO;QACT,CAAC;QACD,IAAI,CAAC;YACH,MAAM,IAAI,CAAC,OAAO,CAAC,KAAM,CAAC,YAAY,CAAC,IAAI,CAAC,QAAQ,CAAC,SAAS,CAAC,EAAE,KAAK,CAAC,CAAC;QAC1E,CAAC;QAAC,OAAO,KAAK,EAAE,CAAC;YACf,IAAI,CAAC,gBAAgB,CAAC,KAAK,EAAE,SAAS,CAAC,CAAC;QAC1C,CAAC;IACH,CAAC;IAEO,KAAK,CAAC,gBAAgB,CAAC,SAAiB;QAC9C,MAAM,IAAI,CAAC,OAAO,CAAC,SAAS,EAAE,EAAE,GAAG,oBAAY,EAAE,EAAE,IAAI,CAAC,eAAe,CAAC,CAAC,CAAC,CAAC,CAAC;QAC5E,IAAI,CAAC,mBAAmB,CAAC,MAAM,CAAC,SAAS,CAAC,CAAC;QAC3C,IAAI,CAAC,UAAU,CAAC,MAAM,CAAC,SAAS,CAAC,CAAC;IACpC,CAAC;IAED;;;;;OAKG;IACK,KAAK,CAAC,eAAe,CAAC,SAAiB,EAAE,KAAc;QAC7D,MAAM,MAAM,GAAG,IAAI,CAAC,UAAU,CAAC,GAAG,CAAC,SAAS,CAAC,CAAC;QAC9C,IAAI,MAAM,IAAI,IAAI,CAAC,OAAO,CAAC,kBAAkB,GAAG,CAAC,EAAE,CAAC;YAClD,MAAM,GAAG,GAAG,IAAI,CAAC,GAAG,EAAE,CAAC;YACvB,IAAI,GAAG,IAAI,MAAM,CAAC,SAAS,EAAE,CAAC;gBAC5B,4DAA4D;gBAC5D,mEAAmE;gBACnE,IAAI,CAAC,UAAU,CAAC,MAAM,CAAC,SAAS,CAAC,CAAC;YACpC,CAAC;iBAAM,IAAI,GAAG,GAAG,MAAM,CAAC,WAAW,GAAG,IAAI,CAAC,OAAO,CAAC,kBAAkB,EAAE,CAAC;gBACtE,OAAO,EAAE,OAAO,EAAE,KAAK,EAAE,OAAO,EAAE,KAAK,EAAE,CAAC;YAC5C,CAAC;YACD,sEAAsE;YACtE,oEAAoE;YACpE,mEAAmE;QACrE,CAAC;QAED,IAAI,KAAiC,CAAC;QACtC,IAAI,CAAC;YACH,KAAK,GAAG,MAAM,IAAI,CAAC,OAAO,CAAC,KAAM,CAAC,GAAG,CAAsB,IAAI,CAAC,GAAG,CAAC,SAAS,CAAC,CAAC,CAAC;QAClF,CAAC;QAAC,OAAO,KAAK,EAAE,CAAC;YACf,IAAI,CAAC,gBAAgB,CAAC,KAAK,EAAE,SAAS,CAAC,CAAC;YACxC,IAAI,CAAC,UAAU,CAAC,MAAM,CAAC,SAAS,CAAC,CAAC,CAAC,yCAAyC;YAC5E,IAAI,IAAI,CAAC,OAAO,CAAC,oBAAoB;gBAAE,OAAO,EAAE,OAAO,EAAE,IAAI,EAAE,OAAO,EAAE,KAAK,EAAE,CAAC;YAChF,MAAM,IAAI,qBAAqB,CAAC,SAAS,EAAE,KAAK,CAAC,CAAC;QACpD,CAAC;QAED,IAAI,CAAC,KAAK,EAAE,MAAM,EAAE,CAAC;YACnB,IAAI,CAAC,UAAU,CAAC,MAAM,CAAC,SAAS,CAAC,CAAC;YAClC,OAAO,EAAE,OAAO,EAAE,IAAI,EAAE,OAAO,EAAE,KAAK,EAAE,aAAa,EAAE,KAAK,EAAE,CAAC;QACjE,CAAC;QAED,MAAM,SAAS,GAAG,IAAI,CAAC,qBAAqB,CAAC,SAAS,EAAE,KAAK,CAAC,SAAS,IAAI,CAAC,EAAE,KAAK,CAAC,WAAW,CAAC,CAAC;QACjG,IAAI,IAAI,CAAC,GAAG,EAAE,GAAG,KAAK,CAAC,WAAW,GAAG,SAAS,EAAE,CAAC;YAC/C,IAAI,CAAC,cAAc,CAAC,SAAS,EAAE,KAAK,CAAC,CAAC;YACtC,OAAO,EAAE,OAAO,EAAE,KAAK,EAAE,OAAO,EAAE,KAAK,EAAE,aAAa,EAAE,KAAK,EAAE,CAAC;QAClE,CAAC;QAED,IAAI,CAAC,KAAK;YAAE,OAAO,EAAE,OAAO,EAAE,IAAI,EAAE,OAAO,EAAE,KAAK,EAAE,aAAa,EAAE,KAAK,EAAE,CAAC;QAE3E,IAAI,IAAI,CAAC,OAAO,CAAC,KAAM,CAAC,UAAU,EAAE,CAAC;YACnC,IAAI,CAAC;gBACH,MAAM,KAAK,GAAG,MAAM,IAAI,CAAC,OAAO,CAAC,KAAM,CAAC,UAAU,CAAC,IAAI,CAAC,QAAQ,CAAC,SAAS,CAAC,EAAE,IAAI,CAAC,eAAe,EAAE,CAAC,CAAC;gBACrG,IAAI,KAAK;oBAAE,IAAI,CAAC,OAAO,CAAC,aAAa,EAAE,CAAC,WAAW,EAAE,SAAS,CAAC,CAAC;gBAChE,OAAO,EAAE,OAAO,EAAE,KAAK,KAAK,IAAI,EAAE,OAAO,EAAE,KAAK,KAAK,IAAI,EAAE,UAAU,EAAE,KAAK,IAAI,SAAS,EAAE,aAAa,EAAE,KAAK,EAAE,CAAC;YACpH,CAAC;YAAC,OAAO,KAAK,EAAE,CAAC;gBACf,IAAI,CAAC,gBAAgB,CAAC,KAAK,EAAE,SAAS,CAAC,CAAC;gBACxC,IAAI,CAAC,IAAI,CAAC,OAAO,CAAC,oBAAoB;oBAAE,MAAM,IAAI,qBAAqB,CAAC,SAAS,EAAE,KAAK,CAAC,CAAC;gBAC1F,6FAA6F;gBAC7F,IAAI,CAAC,OAAO,CAAC,aAAa,EAAE,CAAC,WAAW,EAAE,SAAS,CAAC,CAAC;gBACrD,OAAO,EAAE,OAAO,EAAE,IAAI,EAAE,OAAO,EAAE,IAAI,EAAE,aAAa,EAAE,KAAK,EAAE,CAAC;YAChE,CAAC;QACH,CAAC;QAED,IAAI,CAAC,qBAAqB,CAAC,YAAY,CAAC,CAAC;QACzC,IAAI,CAAC,OAAO,CAAC,aAAa,EAAE,CAAC,WAAW,EAAE,SAAS,CAAC,CAAC;QACrD,OAAO,EAAE,OAAO,EAAE,IAAI,EAAE,OAAO,EAAE,IAAI,EAAE,aAAa,EAAE,KAAK,EAAE,CAAC;IAChE,CAAC;IAEO,KAAK,CAAC,kBAAkB,CAAI,SAAiB,EAAE,EAAoB;QACzE,MAAM,OAAO,GAAG,IAAI,CAAC,UAAU,CAAC,SAAS,CAAC,CAAC;QAC3C,OAAO,CAAC,UAAU,EAAE,CAAC;QAErB,MAAM,IAAI,GAAG,MAAM,IAAI,CAAC,eAAe,CAAC,SAAS,EAAE,IAAI,CAAC,CAAC;QACzD,IAAI,CAAC,IAAI,CAAC,OAAO,EAAE,CAAC;YAClB,OAAO,CAAC,eAAe,EAAE,CAAC;YAC1B,MAAM,IAAI,KAAK,CAAC,0CAA0C,SAAS,EAAE,CAAC,CAAC;QACzE,CAAC;QAED,IAAI,CAAC;YACH,MAAM,MAAM,GAAG,MAAM,IAAI,CAAC,kBAAkB,CAAC,EAAE,CAAC,CAAC;YACjD,OAAO,CAAC,cAAc,EAAE,CAAC;YACzB,IAAI,IAAI,CAAC,OAAO,EAAE,CAAC;gBACjB,MAAM,IAAI,CAAC,gBAAgB,CAAC,SAAS,CAAC,CAAC;gBACvC,MAAM,IAAI,CAAC,gBAAgB,CAAC,SAAS,EAAE,IAAI,CAAC,UAAU,CAAC,CAAC;gBACxD,IAAI,CAAC,OAAO,CAAC,aAAa,EAAE,CAAC,QAAQ,EAAE,SAAS,CAAC,CAAC;YACpD,CAAC;iBAAM,IAAI,IAAI,CAAC,OAAO,CAAC,QAAQ,KAAK,WAAW,EAAE,CAAC;gBACjD,MAAM,IAAI,CAAC,wBAAwB,CAAC,SAAS,EAAE,IAAI,CAAC,CAAC;YACvD,CAAC;iBAAM,CAAC;gBACN,kEAAkE;gBAClE,uDAAuD;gBACvD,2DAA2D;gBAC3D,EAAE;gBACF,gEAAgE;gBAChE,kEAAkE;gBAClE,oDAAoD;gBACpD,gEAAgE;gBAChE,iDAAiD;gBACjD,MAAM,CAAC,GAAG,IAAI,CAAC,aAAa,CAAC;gBAC7B,MAAM,aAAa,GAAG,CAAC,KAAK,SAAS,IAAI,CAAC,CAAC,KAAK,IAAI,IAAI,CAAC,CAAC,CAAC,CAAC,MAAM,IAAI,CAAC,CAAC,QAAQ,KAAK,CAAC,CAAC,CAAC,CAAC;gBACzF,IAAI,CAAC,aAAa,EAAE,CAAC;oBACnB,MAAM,IAAI,CAAC,gBAAgB,CAAC,SAAS,CAAC,CAAC;gBACzC,CAAC;YACH,CAAC;YACD,OAAO,MAAM,CAAC;QAChB,CAAC;QAAC,OAAO,KAAK,EAAE,CAAC;YACf,OAAO,CAAC,aAAa,EAAE,CAAC;YACxB,IAAI,IAAI,CAAC,OAAO,EAAE,CAAC;gBACjB,MAAM,IAAI,CAAC,iCAAiC,CAAC,SAAS,CAAC,CAAC;gBACxD,MAAM,IAAI,CAAC,gBAAgB,CAAC,SAAS,EAAE,IAAI,CAAC,UAAU,CAAC,CAAC;YAC1D,CAAC;iBAAM,IAAI,IAAI,CAAC,OAAO,CAAC,QAAQ,KAAK,WAAW,EAAE,CAAC;gBACjD,MAAM,IAAI,CAAC,wBAAwB,CAAC,SAAS,EAAE,KAAK,CAAC,CAAC;YACxD,CAAC;iBAAM,CAAC;gBACN,MAAM,IAAI,CAAC,gCAAgC,CAAC,SAAS,CAAC,CAAC;YACzD,CAAC;YACD,MAAM,KAAK,CAAC;QACd,CAAC;IACH,CAAC;IAEO,KAAK,CAAC,gCAAgC,CAAC,SAAiB;QAC9D,IAAI,IAAI,CAAC,OAAO,CAAC,KAAM,CAAC,mBAAmB,EAAE,CAAC;YAC5C,IAAI,CAAC;gBACH,MAAM,IAAI,GAAG,MAAM,IAAI,CAAC,OAAO,CAAC,KAAM,CAAC,mBAAmB,CAAC,IAAI,CAAC,GAAG,CAAC,SAAS,CAAC,EAAE;oBAC9E,UAAU,EAAE,IAAI,CAAC,eAAe,CAAC,CAAC,CAAC;oBACnC,gBAAgB,EAAE,IAAI,CAAC,OAAO,CAAC,gBAAgB;oBAC/C,GAAG,EAAE,IAAI,CAAC,GAAG,EAAE;iBAChB,CAAC,CAAC;gBACH,IAAI,CAAC,OAAO,CAAC,MAAM,CAAC,IAAI,CAAC,mCAAmC,SAAS,qBAAqB,IAAI,CAAC,QAAQ,EAAE,CAAC,CAAC;gBAC3G,IAAI,IAAI,CAAC,MAAM;oBAAE,IAAI,CAAC,cAAc,CAAC,SAAS,EAAE,IAAI,CAAC,CAAC;gBACtD,IAAI,IAAI,CAAC,MAAM,IAAI,IAAI,CAAC,QAAQ,KAAK,IAAI,CAAC,OAAO,CAAC,gBAAgB,EAAE,CAAC;oBACnE,IAAI,CAAC,OAAO,CAAC,MAAM,CAAC,IAAI,CAAC,yCAAyC,SAAS,EAAE,CAAC,CAAC;oBAC/E,IAAI,CAAC,OAAO,CAAC,aAAa,EAAE,CAAC,MAAM,EAAE,SAAS,CAAC,CAAC;gBAClD,CAAC;gBACD,OAAO;YACT,CAAC;YAAC,OAAO,KAAK,EAAE,CAAC;gBACf,IAAI,CAAC,gBAAgB,CAAC,KAAK,EAAE,SAAS,CAAC,CAAC;gBACxC,4CAA4C;YAC9C,CAAC;QACH,CAAC;aAAM,CAAC;YACN,IAAI,CAAC,qBAAqB,CAAC,qBAAqB,CAAC,CAAC;QACpD,CAAC;QAED,MAAM,OAAO,GAAG,CAAC,MAAM,IAAI,CAAC,OAAO,CAAC,SAAS,CAAC,CAAC,IAAI,EAAE,GAAG,oBAAY,EAAE,CAAC;QACvE,MAAM,IAAI,GAAwB;YAChC,QAAQ,EAAE,OAAO,CAAC,QAAQ,GAAG,CAAC;YAC9B,WAAW,EAAE,IAAI,CAAC,GAAG,EAAE;YACvB,MAAM,EAAE,OAAO,CAAC,QAAQ,GAAG,CAAC,IAAI,IAAI,CAAC,OAAO,CAAC,gBAAgB;YAC7D,sEAAsE;YACtE,oEAAoE;YACpE,qEAAqE;YACrE,iEAAiE;YACjE,kDAAkD;YAClD,SAAS,EAAE,OAAO,CAAC,SAAS,IAAI,CAAC;YACjC,SAAS,EAAE,OAAO,CAAC,SAAS,IAAI,CAAC;YACjC,WAAW,EAAE,OAAO,CAAC,WAAW,IAAI,CAAC;SACtC,CAAC;QACF,MAAM,IAAI,CAAC,OAAO,CAAC,SAAS,EAAE,IAAI,EAAE,IAAI,CAAC,eAAe,CAAC,CAAC,CAAC,CAAC,CAAC;QAC7D,IAAI,IAAI,CAAC,MAAM;YAAE,IAAI,CAAC,cAAc,CAAC,SAAS,EAAE,IAAI,CAAC,CAAC;QACtD,IAAI,CAAC,OAAO,CAAC,MAAM,CAAC,IAAI,CAAC,mCAAmC,SAAS,qBAAqB,IAAI,CAAC,QAAQ,EAAE,CAAC,CAAC;QAE3G,IAAI,IAAI,CAAC,MAAM,IAAI,CAAC,OAAO,CAAC,MAAM,EAAE,CAAC;YACnC,IAAI,CAAC,OAAO,CAAC,MAAM,CAAC,IAAI,CAAC,yCAAyC,SAAS,EAAE,CAAC,CAAC;YAC/E,IAAI,CAAC,OAAO,CAAC,aAAa,EAAE,CAAC,MAAM,EAAE,SAAS,CAAC,CAAC;QAClD,CAAC;IACH,CAAC;IAED;;;;OAIG;IACK,KAAK,CAAC,wBAAwB,CAAC,SAAiB,EAAE,OAAgB;QACxE,IAAI,IAAI,CAAC,OAAO,CAAC,KAAM,CAAC,mBAAmB,EAAE,CAAC;YAC5C,IAAI,CAAC;gBACH,MAAM,IAAI,GAAG,MAAM,IAAI,CAAC,OAAO,CAAC,KAAM,CAAC,mBAAmB,CAAC,IAAI,CAAC,GAAG,CAAC,SAAS,CAAC,EAAE;oBAC9E,OAAO;oBACP,KAAK,EAAE,IAAI,CAAC,OAAO,CAAC,cAAc;oBAClC,YAAY,EAAE,IAAI,CAAC,OAAO,CAAC,YAAY;oBACvC,kBAAkB,EAAE,IAAI,CAAC,OAAO,CAAC,kBAAkB;oBACnD,UAAU,EAAE,IAAI,CAAC,eAAe,CAAC,CAAC,CAAC;oBACnC,GAAG,EAAE,IAAI,CAAC,GAAG,EAAE;iBAChB,CAAC,CAAC;gBACH,IAAI,IAAI,CAAC,MAAM;oBAAE,IAAI,CAAC,cAAc,CAAC,SAAS,EAAE,IAAI,CAAC,CAAC;gBACtD,IAAI,IAAI,CAAC,SAAS,EAAE,CAAC;oBACnB,IAAI,CAAC,OAAO,CAAC,MAAM,CAAC,IAAI,CACtB,yCAAyC,SAAS,gBAAgB,CAAC,IAAI,CAAC,SAAS,GAAG,GAAG,CAAC,CAAC,OAAO,CAAC,CAAC,CAAC,UAAU,IAAI,CAAC,WAAW,SAAS,CACvI,CAAC;oBACF,IAAI,CAAC,OAAO,CAAC,aAAa,EAAE,CAAC,MAAM,EAAE,SAAS,CAAC,CAAC;gBAClD,CAAC;gBACD,OAAO;YACT,CAAC;YAAC,OAAO,KAAK,EAAE,CAAC;gBACf,IAAI,CAAC,gBAAgB,CAAC,KAAK,EAAE,SAAS,CAAC,CAAC;gBACxC,4CAA4C;YAC9C,CAAC;QACH,CAAC;aAAM,CAAC;YACN,IAAI,CAAC,qBAAqB,CAAC,qBAAqB,CAAC,CAAC;QACpD,CAAC;QAED,MAAM,OAAO,GAAG,CAAC,MAAM,IAAI,CAAC,OAAO,CAAC,SAAS,CAAC,CAAC,IAAI,EAAE,GAAG,oBAAY,EAAE,CAAC;QACvE,MAAM,KAAK,GAAG,IAAI,CAAC,OAAO,CAAC,cAAc,CAAC;QAC1C,MAAM,SAAS,GAAG,CAAC,OAAO,CAAC,SAAS,IAAI,CAAC,CAAC,GAAG,KAAK,GAAG,CAAC,OAAO,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,GAAG,CAAC,CAAC,GAAG,KAAK,CAAC,CAAC;QACrF,MAAM,WAAW,GAAG,CAAC,OAAO,CAAC,WAAW,IAAI,CAAC,CAAC,GAAG,CAAC,CAAC;QACnD,MAAM,MAAM,GAAG,WAAW,IAAI,IAAI,CAAC,OAAO,CAAC,YAAY,IAAI,SAAS,IAAI,IAAI,CAAC,OAAO,CAAC,kBAAkB,CAAC;QAExG,MAAM,IAAI,GAAwB;YAChC,QAAQ,EAAE,OAAO,CAAC,QAAQ;YAC1B,WAAW,EAAE,IAAI,CAAC,GAAG,EAAE;YACvB,MAAM;YACN,SAAS,EAAE,OAAO,CAAC,SAAS,IAAI,CAAC;YACjC,SAAS;YACT,WAAW;SACZ,CAAC;QACF,MAAM,IAAI,CAAC,OAAO,CAAC,SAAS,EAAE,IAAI,EAAE,IAAI,CAAC,eAAe,CAAC,CAAC,CAAC,CAAC,CAAC;QAC7D,IAAI,MAAM;YAAE,IAAI,CAAC,cAAc,CAAC,SAAS,EAAE,IAAI,CAAC,CAAC;QAEjD,IAAI,MAAM,IAAI,CAAC,OAAO,CAAC,MAAM,EAAE,CAAC;YAC9B,IAAI,CAAC,OAAO,CAAC,MAAM,CAAC,IAAI,CACtB,yCAAyC,SAAS,gBAAgB,CAAC,SAAS,GAAG,GAAG,CAAC,CAAC,OAAO,CAAC,CAAC,CAAC,UAAU,WAAW,SAAS,CAC7H,CAAC;YACF,IAAI,CAAC,OAAO,CAAC,aAAa,EAAE,CAAC,MAAM,EAAE,SAAS,CAAC,CAAC;QAClD,CAAC;IACH,CAAC;IAEO,KAAK,CAAC,iCAAiC,CAAC,SAAiB;QAC/D,MAAM,OAAO,GAAG,CAAC,MAAM,IAAI,CAAC,OAAO,CAAC,SAAS,CAAC,CAAC,IAAI,EAAE,GAAG,oBAAY,EAAE,MAAM,EAAE,IAAI,EAAE,CAAC;QACrF,MAAM,SAAS,GAAG,CAAC,OAAO,CAAC,SAAS,IAAI,CAAC,CAAC,GAAG,CAAC,CAAC;QAC/C,MAAM,IAAI,GAAwB;YAChC,QAAQ,EAAE,IAAI,CAAC,GAAG,CAAC,OAAO,CAAC,QAAQ,EAAE,IAAI,CAAC,OAAO,CAAC,gBAAgB,CAAC;YACnE,WAAW,EAAE,IAAI,CAAC,GAAG,EAAE;YACvB,MAAM,EAAE,IAAI;YACZ,SAAS;YACT,SAAS,EAAE,OAAO,CAAC,SAAS,IAAI,CAAC;YACjC,WAAW,EAAE,OAAO,CAAC,WAAW,IAAI,CAAC;SACtC,CAAC;QACF,MAAM,IAAI,CAAC,OAAO,CAAC,SAAS,EAAE,IAAI,EAAE,IAAI,CAAC,eAAe,CAAC,SAAS,CAAC,CAAC,CAAC;QACrE,IAAI,CAAC,cAAc,CAAC,SAAS,EAAE,IAAI,CAAC,CAAC,CAAC,kDAAkD;QACxF,IAAI,CAAC,OAAO,CAAC,MAAM,CAAC,IAAI,CAAC,yCAAyC,SAAS,0BAA0B,SAAS,GAAG,CAAC,GAAG,CAAC,CAAC;QACvH,IAAI,CAAC,OAAO,CAAC,aAAa,EAAE,CAAC,MAAM,EAAE,SAAS,CAAC,CAAC;IAClD,CAAC;IAED,0EAA0E;IAElE,KAAK,CAAC,YAAY,CAAI,SAAiB,EAAE,EAAoB;QACnE,MAAM,OAAO,GAAG,IAAI,CAAC,UAAU,CAAC,SAAS,CAAC,CAAC;QAC3C,OAAO,CAAC,UAAU,EAAE,CAAC;QAErB,MAAM,IAAI,GAAG,IAAI,CAAC,cAAc,CAAC,SAAS,CAAC,CAAC;QAC5C,IAAI,CAAC,IAAI,CAAC,OAAO,EAAE,CAAC;YAClB,OAAO,CAAC,eAAe,EAAE,CAAC;YAC1B,MAAM,IAAI,KAAK,CAAC,0CAA0C,SAAS,EAAE,CAAC,CAAC;QACzE,CAAC;QAED,IAAI,CAAC;YACH,MAAM,MAAM,GAAG,MAAM,IAAI,CAAC,kBAAkB,CAAC,EAAE,CAAC,CAAC;YACjD,OAAO,CAAC,cAAc,EAAE,CAAC;YACzB,IAAI,IAAI,CAAC,OAAO,EAAE,CAAC;gBACjB,IAAI,CAAC,UAAU,CAAC,SAAS,CAAC,CAAC;YAC7B,CAAC;iBAAM,IAAI,IAAI,CAAC,OAAO,CAAC,QAAQ,KAAK,WAAW,EAAE,CAAC;gBACjD,IAAI,CAAC,sBAAsB,CAAC,SAAS,EAAE,IAAI,CAAC,CAAC;YAC/C,CAAC;iBAAM,CAAC;gBACN,IAAI,CAAC,UAAU,CAAC,SAAS,CAAC,CAAC;YAC7B,CAAC;YACD,OAAO,MAAM,CAAC;QAChB,CAAC;QAAC,OAAO,KAAK,EAAE,CAAC;YACf,OAAO,CAAC,aAAa,EAAE,CAAC;YACxB,IAAI,IAAI,CAAC,OAAO,EAAE,CAAC;gBACjB,IAAI,CAAC,2BAA2B,CAAC,SAAS,CAAC,CAAC;YAC9C,CAAC;iBAAM,IAAI,IAAI,CAAC,OAAO,CAAC,QAAQ,KAAK,WAAW,EAAE,CAAC;gBACjD,IAAI,CAAC,sBAAsB,CAAC,SAAS,EAAE,KAAK,CAAC,CAAC;YAChD,CAAC;iBAAM,CAAC;gBACN,IAAI,CAAC,0BAA0B,CAAC,SAAS,CAAC,CAAC;YAC7C,CAAC;YACD,MAAM,KAAK,CAAC;QACd,CAAC;IACH,CAAC;IAED,gFAAgF;IACxE,aAAa,CAAC,SAAiB;QACrC,IAAI,CAAC,IAAI,CAAC,cAAc,CAAC,GAAG,CAAC,SAAS,CAAC;YAAE,OAAO,IAAI,CAAC;QAErD,MAAM,SAAS,GAAG,IAAI,CAAC,UAAU,CAAC,GAAG,CAAC,SAAS,CAAC,IAAI,CAAC,CAAC;QACtD,MAAM,WAAW,GAAG,IAAI,CAAC,eAAe,CAAC,GAAG,CAAC,SAAS,CAAC,IAAI,CAAC,CAAC;QAC7D,MAAM,SAAS,GAAG,IAAI,CAAC,qBAAqB,CAAC,SAAS,EAAE,SAAS,EAAE,WAAW,CAAC,CAAC;QAEhF,IAAI,IAAI,CAAC,GAAG,EAAE,GAAG,WAAW,GAAG,SAAS,EAAE,CAAC;YACzC,IAAI,CAAC,OAAO,CAAC,MAAM,CAAC,IAAI,CAAC,uCAAuC,SAAS,EAAE,CAAC,CAAC;YAC7E,OAAO,KAAK,CAAC;QACf,CAAC;QACD,OAAO,IAAI,CAAC;IACd,CAAC;IAED,oFAAoF;IAC5E,cAAc,CAAC,SAAiB;QACtC,IAAI,CAAC,IAAI,CAAC,cAAc,CAAC,GAAG,CAAC,SAAS,CAAC;YAAE,OAAO,EAAE,OAAO,EAAE,IAAI,EAAE,OAAO,EAAE,KAAK,EAAE,CAAC;QAElF,MAAM,SAAS,GAAG,IAAI,CAAC,UAAU,CAAC,GAAG,CAAC,SAAS,CAAC,IAAI,CAAC,CAAC;QACtD,MAAM,WAAW,GAAG,IAAI,CAAC,eAAe,CAAC,GAAG,CAAC,SAAS,CAAC,IAAI,CAAC,CAAC;QAC7D,MAAM,SAAS,GAAG,IAAI,CAAC,qBAAqB,CAAC,SAAS,EAAE,SAAS,EAAE,WAAW,CAAC,CAAC;QAEhF,IAAI,IAAI,CAAC,GAAG,EAAE,GAAG,WAAW,GAAG,SAAS,EAAE,CAAC;YACzC,IAAI,CAAC,OAAO,CAAC,MAAM,CAAC,IAAI,CAAC,uCAAuC,SAAS,EAAE,CAAC,CAAC;YAC7E,OAAO,EAAE,OAAO,EAAE,KAAK,EAAE,OAAO,EAAE,KAAK,EAAE,CAAC;QAC5C,CAAC;QAED,IAAI,IAAI,CAAC,eAAe,CAAC,GAAG,CAAC,SAAS,CAAC,EAAE,CAAC;YACxC,oFAAoF;YACpF,OAAO,EAAE,OAAO,EAAE,KAAK,EAAE,OAAO,EAAE,KAAK,EAAE,CAAC;QAC5C,CAAC;QAED,IAAI,CAAC,eAAe,CAAC,GAAG,CAAC,SAAS,CAAC,CAAC;QACpC,IAAI,CAAC,cAAc,CAAC,GAAG,CAAC,SAAS,EAAE,IAAI,CAAC,GAAG,EAAE,CAAC,CAAC;QAC/C,IAAI,CAAC,OAAO,CAAC,aAAa,EAAE,CAAC,WAAW,EAAE,SAAS,CAAC,CAAC;QACrD,OAAO,EAAE,OAAO,EAAE,IAAI,EAAE,OAAO,EAAE,IAAI,EAAE,CAAC;IAC1C,CAAC;IAEO,0BAA0B,CAAC,SAAiB;QAClD,MAAM,QAAQ,GAAG,CAAC,IAAI,CAAC,QAAQ,CAAC,GAAG,CAAC,SAAS,CAAC,IAAI,CAAC,CAAC,GAAG,CAAC,CAAC;QACzD,IAAI,CAAC,QAAQ,CAAC,GAAG,CAAC,SAAS,EAAE,QAAQ,CAAC,CAAC;QACvC,IAAI,CAAC,eAAe,CAAC,GAAG,CAAC,SAAS,EAAE,IAAI,CAAC,GAAG,EAAE,CAAC,CAAC;QAChD,IAAI,CAAC,OAAO,CAAC,MAAM,CAAC,IAAI,CAAC,mCAAmC,SAAS,qBAAqB,QAAQ,EAAE,CAAC,CAAC;QAEtG,IAAI,QAAQ,IAAI,IAAI,CAAC,OAAO,CAAC,gBAAgB,IAAI,CAAC,IAAI,CAAC,cAAc,CAAC,GAAG,CAAC,SAAS,CAAC,EAAE,CAAC;YACrF,IAAI,CAAC,cAAc,CAAC,GAAG,CAAC,SAAS,CAAC,CAAC;YACnC,IAAI,CAAC,UAAU,CAAC,GAAG,CAAC,SAAS,EAAE,CAAC,CAAC,CAAC;YAClC,IAAI,CAAC,OAAO,CAAC,MAAM,CAAC,IAAI,CAAC,yCAAyC,SAAS,EAAE,CAAC,CAAC;YAC/E,IAAI,CAAC,OAAO,CAAC,aAAa,EAAE,CAAC,MAAM,EAAE,SAAS,CAAC,CAAC;QAClD,CAAC;IACH,CAAC;IAED,kIAAkI;IAC1H,sBAAsB,CAAC,SAAiB,EAAE,OAAgB;QAChE,MAAM,KAAK,GAAG,IAAI,CAAC,OAAO,CAAC,cAAc,CAAC;QAC1C,MAAM,IAAI,GAAG,CAAC,IAAI,CAAC,UAAU,CAAC,GAAG,CAAC,SAAS,CAAC,IAAI,CAAC,CAAC,GAAG,KAAK,GAAG,CAAC,OAAO,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,GAAG,CAAC,CAAC,GAAG,KAAK,CAAC,CAAC;QAC7F,MAAM,KAAK,GAAG,CAAC,IAAI,CAAC,YAAY,CAAC,GAAG,CAAC,SAAS,CAAC,IAAI,CAAC,CAAC,GAAG,CAAC,CAAC;QAC1D,IAAI,CAAC,UAAU,CAAC,GAAG,CAAC,SAAS,EAAE,IAAI,CAAC,CAAC;QACrC,IAAI,CAAC,YAAY,CAAC,GAAG,CAAC,SAAS,EAAE,KAAK,CAAC,CAAC;QAExC,IAAI,KAAK,IAAI,IAAI,CAAC,OAAO,CAAC,YAAY,IAAI,IAAI,IAAI,IAAI,CAAC,OAAO,CAAC,kBAAkB,IAAI,CAAC,IAAI,CAAC,cAAc,CAAC,GAAG,CAAC,SAAS,CAAC,EAAE,CAAC;YACzH,IAAI,CAAC,cAAc,CAAC,GAAG,CAAC,SAAS,CAAC,CAAC;YACnC,IAAI,CAAC,UAAU,CAAC,GAAG,CAAC,SAAS,EAAE,CAAC,CAAC,CAAC;YAClC,IAAI,CAAC,eAAe,CAAC,GAAG,CAAC,SAAS,EAAE,IAAI,CAAC,GAAG,EAAE,CAAC,CAAC;YAChD,IAAI,CAAC,OAAO,CAAC,MAAM,CAAC,IAAI,CACtB,yCAAyC,SAAS,gBAAgB,CAAC,IAAI,GAAG,GAAG,CAAC,CAAC,OAAO,CAAC,CAAC,CAAC,UAAU,KAAK,SAAS,CAClH,CAAC;YACF,IAAI,CAAC,OAAO,CAAC,aAAa,EAAE,CAAC,MAAM,EAAE,SAAS,CAAC,CAAC;QAClD,CAAC;IACH,CAAC;IAEO,2BAA2B,CAAC,SAAiB;QACnD,IAAI,CAAC,eAAe,CAAC,MAAM,CAAC,SAAS,CAAC,CAAC;QACvC,IAAI,CAAC,cAAc,CAAC,MAAM,CAAC,SAAS,CAAC,CAAC;QACtC,MAAM,SAAS,GAAG,CAAC,IAAI,CAAC,UAAU,CAAC,GAAG,CAAC,SAAS,CAAC,IAAI,CAAC,CAAC,GAAG,CAAC,CAAC;QAC5D,IAAI,CAAC,UAAU,CAAC,GAAG,CAAC,SAAS,EAAE,SAAS,CAAC,CAAC;QAC1C,IAAI,CAAC,eAAe,CAAC,GAAG,CAAC,SAAS,EAAE,IAAI,CAAC,GAAG,EAAE,CAAC,CAAC;QAChD,IAAI,CAAC,OAAO,CAAC,MAAM,CAAC,IAAI,CAAC,yCAAyC,SAAS,0BAA0B,SAAS,GAAG,CAAC,GAAG,CAAC,CAAC;QACvH,IAAI,CAAC,OAAO,CAAC,aAAa,EAAE,CAAC,MAAM,EAAE,SAAS,CAAC,CAAC;IAClD,CAAC;IAEO,UAAU,CAAC,SAAiB;QAClC,MAAM,OAAO,GAAG,IAAI,CAAC,cAAc,CAAC,GAAG,CAAC,SAAS,CAAC,CAAC;QACnD,IAAI,CAAC,QAAQ,CAAC,MAAM,CAAC,SAAS,CAAC,CAAC;QAChC,IAAI,CAAC,eAAe,CAAC,MAAM,CAAC,SAAS,CAAC,CAAC;QACvC,IAAI,CAAC,cAAc,CAAC,MAAM,CAAC,SAAS,CAAC,CAAC;QACtC,IAAI,CAAC,UAAU,CAAC,MAAM,CAAC,SAAS,CAAC,CAAC;QAClC,IAAI,CAAC,eAAe,CAAC,MAAM,CAAC,SAAS,CAAC,CAAC;QACvC,IAAI,CAAC,cAAc,CAAC,MAAM,CAAC,SAAS,CAAC,CAAC;QACtC,IAAI,CAAC,UAAU,CAAC,MAAM,CAAC,SAAS,CAAC,CAAC;QAClC,IAAI,CAAC,YAAY,CAAC,MAAM,CAAC,SAAS,CAAC,CAAC;QACpC,IAAI,CAAC,mBAAmB,CAAC,MAAM,CAAC,SAAS,CAAC,CAAC;QAC3C,IAAI,OAAO,EAAE,CAAC;YACZ,IAAI,CAAC,OAAO,CAAC,aAAa,EAAE,CAAC,QAAQ,EAAE,SAAS,CAAC,CAAC;QACpD,CAAC;QACD,IAAI,CAAC,OAAO,CAAC,MAAM,CAAC,GAAG,CAAC,wCAAwC,SAAS,EAAE,CAAC,CAAC;IAC/E,CAAC;IAED;;;;;;OAMG;IACK,OAAO;QACb,MAAM,GAAG,GAAG,IAAI,CAAC,GAAG,EAAE,CAAC;QACvB,MAAM,UAAU,GAAG,IAAI,CAAC,GAAG,CAAC,IAAI,CAAC,OAAO,CAAC,YAAY,EAAE,IAAI,CAAC,GAAG,CAAC,CAAC;QACjE,KAAK,MAAM,CAAC,SAAS,EAAE,SAAS,CAAC,IAAI,IAAI,CAAC,cAAc,CAAC,OAAO,EAAE,EAAE,CAAC;YACnE,IAAI,GAAG,GAAG,SAAS,GAAG,UAAU,EAAE,CAAC;gBACjC,IAAI,CAAC,eAAe,CAAC,MAAM,CAAC,SAAS,CAAC,CAAC;gBACvC,IAAI,CAAC,cAAc,CAAC,MAAM,CAAC,SAAS,CAAC,CAAC;gBACtC,IAAI,CAAC,OAAO,CAAC,MAAM,CAAC,IAAI,CACtB,wCAAwC,SAAS,0EAA0E,UAAU,qCAAqC,CAC3K,CAAC;YACJ,CAAC;QACH,CAAC;IACH,CAAC;IAED,4EAA4E;IAEpE,kBAAkB,CAAI,EAAoB;QAChD,IAAI,CAAC,IAAI,CAAC,OAAO,CAAC,OAAO;YAAE,OAAO,EAAE,EAAE,CAAC;QAEvC,OAAO,IAAI,OAAO,CAAI,CAAC,OAAO,EAAE,MAAM,EAAE,EAAE;YACxC,MAAM,KAAK,GAAG,UAAU,CAAC,GAAG,EAAE,CAAC,MAAM,CAAC,IAAI,KAAK,CAAC,6BAA6B,IAAI,CAAC,OAAO,CAAC,OAAO,IAAI,CAAC,CAAC,EAAE,IAAI,CAAC,OAAO,CAAC,OAAO,CAAC,CAAC;YAC/H,EAAE,EAAE,CAAC,IAAI,CACP,CAAC,KAAK,EAAE,EAAE;gBACR,YAAY,CAAC,KAAK,CAAC,CAAC;gBACpB,OAAO,CAAC,KAAK,CAAC,CAAC;YACjB,CAAC,EACD,CAAC,KAAK,EAAE,EAAE;gBACR,YAAY,CAAC,KAAK,CAAC,CAAC;gBACpB,MAAM,CAAC,KAAK,CAAC,CAAC;YAChB,CAAC,CACF,CAAC;QACJ,CAAC,CAAC,CAAC;IACL,CAAC;CACF;AAxzBD,wCAwzBC"}
|
package/package.json
CHANGED
|
@@ -1,71 +1,71 @@
|
|
|
1
|
-
{
|
|
2
|
-
"name": "redeye-breaker",
|
|
3
|
-
"version": "0.
|
|
4
|
-
"description": "A circuit breaker for Node.js with an optional Redis-backed distributed mode, so breaker state can be shared across multiple app instances instead of being tracked per-process.",
|
|
5
|
-
"main": "dist/index.js",
|
|
6
|
-
"types": "dist/index.d.ts",
|
|
7
|
-
"files": [
|
|
8
|
-
"dist"
|
|
9
|
-
],
|
|
10
|
-
"exports": {
|
|
11
|
-
".": {
|
|
12
|
-
"types": "./dist/index.d.ts",
|
|
13
|
-
"default": "./dist/index.js"
|
|
14
|
-
},
|
|
15
|
-
"./redis-store": {
|
|
16
|
-
"types": "./dist/stores/redis-store.d.ts",
|
|
17
|
-
"default": "./dist/stores/redis-store.js"
|
|
18
|
-
}
|
|
19
|
-
},
|
|
20
|
-
"typesVersions": {
|
|
21
|
-
"*": {
|
|
22
|
-
"redis-store": [
|
|
23
|
-
"dist/stores/redis-store.d.ts"
|
|
24
|
-
]
|
|
25
|
-
}
|
|
26
|
-
},
|
|
27
|
-
"scripts": {
|
|
28
|
-
"build": "tsc -p tsconfig.json",
|
|
29
|
-
"test": "jest",
|
|
30
|
-
"test:integration": "jest --config jest.integration.config.js",
|
|
31
|
-
"lint": "tsc --noEmit",
|
|
32
|
-
"prepublishOnly": "npm run lint && npm run test && npm run build"
|
|
33
|
-
},
|
|
34
|
-
"keywords": [
|
|
35
|
-
"circuit-breaker",
|
|
36
|
-
"resilience",
|
|
37
|
-
"redis",
|
|
38
|
-
"distributed-systems",
|
|
39
|
-
"fault-tolerance",
|
|
40
|
-
"microservices"
|
|
41
|
-
],
|
|
42
|
-
"author": "Laurels Echichinwo",
|
|
43
|
-
"license": "MIT",
|
|
44
|
-
"repository": {
|
|
45
|
-
"type": "git",
|
|
46
|
-
"url": "git+https://github.com/laurells/redeye.git"
|
|
47
|
-
},
|
|
48
|
-
"bugs": {
|
|
49
|
-
"url": "https://github.com/laurells/redeye/issues"
|
|
50
|
-
},
|
|
51
|
-
"homepage": "https://github.com/laurells/redeye#readme",
|
|
52
|
-
"peerDependencies": {
|
|
53
|
-
"ioredis": ">=5.0.0"
|
|
54
|
-
},
|
|
55
|
-
"peerDependenciesMeta": {
|
|
56
|
-
"ioredis": {
|
|
57
|
-
"optional": true
|
|
58
|
-
}
|
|
59
|
-
},
|
|
60
|
-
"devDependencies": {
|
|
61
|
-
"@types/jest": "^29.5.12",
|
|
62
|
-
"@types/node": "^20.11.0",
|
|
63
|
-
"ioredis": "^5.4.1",
|
|
64
|
-
"jest": "^29.7.0",
|
|
65
|
-
"ts-jest": "^29.1.2",
|
|
66
|
-
"typescript": "^5.4.5"
|
|
67
|
-
},
|
|
68
|
-
"engines": {
|
|
69
|
-
"node": ">=18"
|
|
70
|
-
}
|
|
71
|
-
}
|
|
1
|
+
{
|
|
2
|
+
"name": "redeye-breaker",
|
|
3
|
+
"version": "0.4.0",
|
|
4
|
+
"description": "A circuit breaker for Node.js with an optional Redis-backed distributed mode, so breaker state can be shared across multiple app instances instead of being tracked per-process.",
|
|
5
|
+
"main": "dist/index.js",
|
|
6
|
+
"types": "dist/index.d.ts",
|
|
7
|
+
"files": [
|
|
8
|
+
"dist"
|
|
9
|
+
],
|
|
10
|
+
"exports": {
|
|
11
|
+
".": {
|
|
12
|
+
"types": "./dist/index.d.ts",
|
|
13
|
+
"default": "./dist/index.js"
|
|
14
|
+
},
|
|
15
|
+
"./redis-store": {
|
|
16
|
+
"types": "./dist/stores/redis-store.d.ts",
|
|
17
|
+
"default": "./dist/stores/redis-store.js"
|
|
18
|
+
}
|
|
19
|
+
},
|
|
20
|
+
"typesVersions": {
|
|
21
|
+
"*": {
|
|
22
|
+
"redis-store": [
|
|
23
|
+
"dist/stores/redis-store.d.ts"
|
|
24
|
+
]
|
|
25
|
+
}
|
|
26
|
+
},
|
|
27
|
+
"scripts": {
|
|
28
|
+
"build": "tsc -p tsconfig.json",
|
|
29
|
+
"test": "jest",
|
|
30
|
+
"test:integration": "jest --config jest.integration.config.js",
|
|
31
|
+
"lint": "tsc --noEmit",
|
|
32
|
+
"prepublishOnly": "npm run lint && npm run test && npm run build"
|
|
33
|
+
},
|
|
34
|
+
"keywords": [
|
|
35
|
+
"circuit-breaker",
|
|
36
|
+
"resilience",
|
|
37
|
+
"redis",
|
|
38
|
+
"distributed-systems",
|
|
39
|
+
"fault-tolerance",
|
|
40
|
+
"microservices"
|
|
41
|
+
],
|
|
42
|
+
"author": "Laurels Echichinwo",
|
|
43
|
+
"license": "MIT",
|
|
44
|
+
"repository": {
|
|
45
|
+
"type": "git",
|
|
46
|
+
"url": "git+https://github.com/laurells/redeye.git"
|
|
47
|
+
},
|
|
48
|
+
"bugs": {
|
|
49
|
+
"url": "https://github.com/laurells/redeye/issues"
|
|
50
|
+
},
|
|
51
|
+
"homepage": "https://github.com/laurells/redeye#readme",
|
|
52
|
+
"peerDependencies": {
|
|
53
|
+
"ioredis": ">=5.0.0"
|
|
54
|
+
},
|
|
55
|
+
"peerDependenciesMeta": {
|
|
56
|
+
"ioredis": {
|
|
57
|
+
"optional": true
|
|
58
|
+
}
|
|
59
|
+
},
|
|
60
|
+
"devDependencies": {
|
|
61
|
+
"@types/jest": "^29.5.12",
|
|
62
|
+
"@types/node": "^20.11.0",
|
|
63
|
+
"ioredis": "^5.4.1",
|
|
64
|
+
"jest": "^29.7.0",
|
|
65
|
+
"ts-jest": "^29.1.2",
|
|
66
|
+
"typescript": "^5.4.5"
|
|
67
|
+
},
|
|
68
|
+
"engines": {
|
|
69
|
+
"node": ">=18"
|
|
70
|
+
}
|
|
71
|
+
}
|