@nodii/retry 0.0.0 → 0.1.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/README.md CHANGED
@@ -1 +1,214 @@
1
- Placeholder claim for @nodii/retry. Real releases come from cognion-nucleus/nodii-libs via npm trusted publishing.
1
+ # @nodii/retry
2
+
3
+ The retry / exponential-backoff / jitter policy for the Nodii stack, as a real package.
4
+
5
+ Retry+backoff+jitter was hand-rolled **14 times inside `ts/*` itself** — not in the services, in
6
+ the shared libraries. Every service inherited whichever variant its lib happened to implement, so
7
+ two consumers of the same fleet behaved differently under the same failure.
8
+
9
+ Zero runtime dependencies. Injectable clock and RNG throughout — a suite exercising a 30s cap
10
+ finishes in microseconds.
11
+
12
+ ---
13
+
14
+ ## The survey
15
+
16
+ Every row is a real implementation on `origin/main` @ `8af8e137`. `FLEET_POLICIES` in
17
+ `src/fleet-policies.ts` encodes each one, and `tests/fleet-parity.test.ts` asserts this package
18
+ reproduces it.
19
+
20
+ | # | Site | Base | Mult | Cap | Jitter | Max attempts | Retryable |
21
+ |---|------|------|------|-----|--------|--------------|-----------|
22
+ | 1 | `approval/consumer.ts:161` (BullMQ `backoffStrategy`) | 250ms | ×2 | 60s | none | 5 (`consumerMaxRetries`) | any handler throw |
23
+ | 2 | `approval/streams-reader.ts:389` | 200ms | — | — | none | ∞ | any loop throw |
24
+ | 3 | `onboarding/submit/client.ts:37` | 1000ms | ×2 (literal `[1000,2000]`) | — | none | 3 | network/timeout + 5xx |
25
+ | 4 | `onboarding/submit/signup-client.ts:44` | 1000ms | ×2 (literal `[1000,2000]`) | — | none | 3 | network/timeout + 5xx |
26
+ | 5 | `outbox-dispatcher/sql-builders.ts:39` (SQL cooldown) | 60s | **linear** ×n | — | none | 8 (`publishMaxAttempts`) | every publish failure |
27
+ | 6 | `outbox-dispatcher/drainer.ts:421` | 200ms | — | — | none | ∞ | any drain throw |
28
+ | 7 | `replica-consumer/start.ts:242` (cold start) | 1000ms | ×2 | 30s | **±20%** | **0 = indefinite** | every bootstrap failure |
29
+ | 8 | `replica-consumer/define-replica.ts:294` + `init.ts:17` | 100ms | ×2 | 30s | `jitterPct: 25` | 5 | **n/a — DEAD CONFIG** |
30
+ | 9 | `replica-consumer/worker/streams-worker.ts:1120` | — (PEL redelivery) | — | — | none | 5 | all apply errors → DLQ |
31
+ | 10 | `role-catalog/publish.ts:203` | 100ms | ×2 | 30s | **±12.5%** | 30 | gRPC `UNAVAILABLE` only |
32
+ | 11 | `saga/participant-worker.ts:351` | 250ms | — | — | none | ∞ | any loop throw |
33
+ | 12 | `telemetry/worker/streams-worker.ts:776` | — (PEL redelivery) | — | — | none | 3 | poison-tagged only |
34
+ | 13 | `telemetry/worker/outbox-dispatcher.ts:262` | 200ms | — | — | none | ∞ | any drain throw |
35
+ | 14 | `telemetry/worker/single-active-lease.ts:714` | `max(1s, ttl/2)` | — | — | none | ∞ | lease not held / Redis down |
36
+
37
+ ## The divergences
38
+
39
+ ### Accidental
40
+
41
+ 1. **`jitterPct` carries two different UNITS inside one package.** `replica-consumer/start.ts`
42
+ writes `0.2` (a fraction); `define-replica.ts` and `init.ts` write `25` (a percent) for a field
43
+ of the same name in the same library. Read as a fraction, `25` is ±2500% jitter. This package
44
+ rejects `pct > 1` outright so the ambiguity cannot survive migration.
45
+ 2. **`retryBackoff` is DEAD CONFIG.** `git grep retryBackoff -- ts/` returns three hits: two
46
+ constructors and one type. Nothing reads it. `replica-consumer` declares an exponential
47
+ 100ms→30s ±25% apply backoff that has never executed — the real spacing is Redis PEL redelivery
48
+ plus the XCLAIM reaper. Do not "migrate" it; migrating would ACTIVATE a delay that has never run
49
+ in production.
50
+ 3. **Three structurally identical loop-error sleeps, two constants.** `approval/streams-reader`,
51
+ `outbox-dispatcher/drainer` and `telemetry/worker/outbox-dispatcher` all sleep 200ms; `saga/
52
+ participant-worker` sleeps 250ms. Same failure mode (Redis blip in a stream-read loop), same
53
+ remedy, arbitrary difference. None has jitter or growth, so a sustained Redis outage means every
54
+ pod issues 4–5 commands/second forever.
55
+ 4. **Both jitter sites can exceed their own declared cap.** Both apply jitter AFTER the cap and
56
+ never re-clamp: `role-catalog` sleeps up to 33 750ms against a declared 30 000ms cap;
57
+ `replica-consumer` cold start up to 36 000ms against the same declared cap. A cap the code may
58
+ exceed is not a cap.
59
+ 5. **`approval`'s declared 250ms base is never slept.** `2 ** attemptsMade * 250` against
60
+ BullMQ's post-incremented counter means the first retry waits 500ms. Off by one doubling.
61
+ 6. **Three jitter magnitudes (12.5% / 20% / 25%) for one concept**, none of them derived from
62
+ anything.
63
+ 7. **`single-active-lease` self-heals on a fixed `ttl/2` tick with zero jitter.** This is the one
64
+ genuine thundering-herd shape in the survey: after a Redis blip every replica of a service
65
+ retries the acquire path in lockstep.
66
+
67
+ ### Deliberate (leave alone)
68
+
69
+ 1. **`replica-consumer` cold start retries indefinitely (`maxAttempts: 0`).** Giving up means
70
+ silently dropping a backfill. Correct.
71
+ 2. **`outbox-dispatcher`'s cooldown is LINEAR, not exponential** — and `publishMaxAttempts` caps
72
+ RE-CLAIM, never deletes the row (D401). A row is source of truth; the 24h sweep raises the M17
73
+ page. Correct, and the reason `growth: "linear"` exists in this package.
74
+ 3. **`onboarding` does not auto-retry 429.** Honouring a `Retry-After` is the caller's decision.
75
+ Correct.
76
+ 4. **`role-catalog` retries `UNAVAILABLE` only, fail-fast on `FAILED_PRECONDITION`.** A version
77
+ mismatch will never heal by waiting. Correct.
78
+ 5. **`telemetry`'s retry ledger is bumped only for poison-tagged errors**; infra errors (fence
79
+ lost, store down) leave the entry in the PEL without spending budget. Correct — otherwise five
80
+ Redis blips silently dead-letter a good event.
81
+ 6. **`replica-consumer` treats `UnknownAggregateKind` as process-fatal, not retryable.** A silent
82
+ drop there is data loss. Correct.
83
+
84
+ ---
85
+
86
+ ## Why equal jitter is the default
87
+
88
+ `"equal"` — `d/2 + U(0, d/2)`.
89
+
90
+ - **Full jitter** (`U(0, d)`) minimizes contention but has no floor: any attempt can return ≈0. In
91
+ this fleet almost every retry waits on a *single dependency that is booting* (role-catalog
92
+ publishing at boot against a cold tenant-service; replica cold-start backfill against the
93
+ snapshot producer), not on a contended shared resource. Full jitter's benefit is *herd
94
+ dispersal*; its cost — losing the monotonically growing floor — is what actually bites, because
95
+ a near-zero delay both burns an attempt from a finite budget and adds load to something already
96
+ struggling.
97
+ - **Equal jitter** keeps an exponentially growing floor (`d/2`) while still dispersing enough to
98
+ break lockstep between the N pods of a service that boot together — which is exactly the herd
99
+ case here (an ECS rolling deploy). AWS's own analysis finds full and equal near-identical on
100
+ total work and completion time; equal keeps the floor.
101
+ - **Decorrelated jitter** (`min(cap, U(base, 3·prev))`) is the strongest for pure contention, but
102
+ it is *stateful* and also non-monotone, and nothing in the fleet does it today — defaulting to it
103
+ would silently change all 14 sites. Offered, never defaulted. (`bullmqBackoffStrategy` rejects it
104
+ outright: BullMQ's `backoffStrategy` is stateless and has no previous delay to thread.)
105
+ - **`{ kind: "proportional", pct }`** exists because it is what the two live jitter sites actually
106
+ do. `role-catalog`'s `randomFn()*0.25 + 0.875` is *exactly* `1 + 0.125·(2r−1)`, so their
107
+ migration is byte-preserving for the same RNG draw.
108
+ - **`"none"`** exists because 11 sites have no jitter and their migration must be
109
+ behaviour-preserving.
110
+
111
+ Order of operations — the part the fleet gets wrong:
112
+
113
+ ```
114
+ grow → cap → jitter → clamp (unless allowJitterAboveCap) → floor at 0 → round to integer ms
115
+ ```
116
+
117
+ `allowJitterAboveCap: true` reproduces the live overshoot exactly, for a byte-preserving migration.
118
+ Default `false`, i.e. the declared cap holds.
119
+
120
+ ---
121
+
122
+ ## The BullMQ off-by-one
123
+
124
+ BullMQ increments `job.attemptsMade` **before** the backoff is computed and before `failed` fires.
125
+ So after the k-th attempt fails, `attemptsMade === k`, and "was that the last one?" is:
126
+
127
+ ```ts
128
+ isFinalAttemptFromAttemptsMade(attemptsMade, maxAttempts); // attemptsMade + 1 >= maxAttempts
129
+ ```
130
+
131
+ The naive `attemptsMade >= maxAttempts` disagrees at exactly one point (`attemptsMade === max - 1`)
132
+ and runs one attempt too many. The 1-based in-process ledger convention (`retries.inc(id)`) is a
133
+ **different** function:
134
+
135
+ ```ts
136
+ isFinalAttempt(attempt, maxAttempts); // attempt >= maxAttempts
137
+ ```
138
+
139
+ Two named functions, deliberately — there is no single `isLastAttempt(n, max)` for a call site to
140
+ guess at. `maxAttempts === 0` means unbounded; nothing is ever final.
141
+
142
+ ---
143
+
144
+ ## The boundary with `@nodii/idempotency`
145
+
146
+ They answer different questions and must not be merged:
147
+
148
+ | | `@nodii/idempotency` (D242) | `@nodii/retry` |
149
+ |---|---|---|
150
+ | Question | should the KEY be cached or released? | should the CALLER re-invoke? |
151
+ | Kind | storage / replay | control flow |
152
+ | Side | server | client |
153
+ | Keyed by | the completed OUTCOME | the thrown ERROR |
154
+
155
+ `transport` is a **superset** of "retryable": `RESOURCE_EXHAUSTED` is transport, yet a caller
156
+ honouring a `Retry-After` may decline to retry (`@nodii/onboarding` does exactly this for 429). And
157
+ `CANCELLED` / `UNKNOWN` / `UNIMPLEMENTED` / `DATA_LOSS` are deliberately unmapped by D242 and fall
158
+ back to `defaultClass` — a default that exists to answer the *storage* question and must not
159
+ silently become a retry verdict.
160
+
161
+ So this package ships **no gRPC-status table, no HTTP-status table, and no dependency on
162
+ `@nodii/idempotency`** — a change to D242's map cannot silently change anyone's retry behaviour.
163
+ Coupling is explicit and **one-way**:
164
+
165
+ ```ts
166
+ shouldRetry: shouldRetryFromOutcomeClassifier(myD242Classifier); // transport ⇒ may retry
167
+ ```
168
+
169
+ The reverse direction is forbidden and not offered: declining to retry does not make an outcome
170
+ deterministic, and caching a key on that basis would replay a transport failure as a business
171
+ result. The bridge fails **closed** on any unrecognised class.
172
+
173
+ ---
174
+
175
+ ## Usage
176
+
177
+ ```ts
178
+ import { retry, NonRetryableError, bullmqBackoffStrategy } from "@nodii/retry";
179
+
180
+ const res = await retry(
181
+ async (ctx) => {
182
+ const r = await fetch(url);
183
+ if (r.status === 403) throw new NonRetryableError("bot challenge failed");
184
+ if (r.status >= 500) throw new Error(`server ${r.status}`);
185
+ return r.json();
186
+ },
187
+ {
188
+ policy: { maxAttempts: 3, baseDelayMs: 1000, multiplier: 2, jitter: "equal" },
189
+ shouldRetry: (err) => !(err instanceof TypeError),
190
+ onRetry: ({ attempt, delayMs, error }) =>
191
+ logger.warn("retrying", { attempt, delay_ms: delayMs, error: String(error) }),
192
+ },
193
+ );
194
+ ```
195
+
196
+ ```ts
197
+ // BullMQ — the post-increment handled in one place
198
+ new Worker(queue, handler, {
199
+ settings: {
200
+ backoffStrategy: bullmqBackoffStrategy({
201
+ baseDelayMs: 500, multiplier: 2, maxDelayMs: 60_000, jitter: "equal",
202
+ }),
203
+ },
204
+ });
205
+ ```
206
+
207
+ Everything is injectable: `sleep`, `random`, `signal`. `backoffSequence(policy, n)` gives the
208
+ deterministic schedule for tests and for operators reasoning about "how long until we give up".
209
+
210
+ ## Migration
211
+
212
+ Call sites are **not** migrated in this release. `FLEET_POLICIES[<id>].migration` states, per site,
213
+ whether the migration is behaviour-preserving or a deliberate change, and `tests/fleet-parity.
214
+ test.ts` proves the preserving ones really are.
@@ -0,0 +1,38 @@
1
+ import { type RandomFn } from "../jitter.js";
2
+ import { type RetryPolicy } from "../policy.js";
3
+ export interface BullmqBackoffOptions {
4
+ random?: RandomFn;
5
+ /**
6
+ * Added to BullMQ's `attemptsMade` before it is used as the 1-based retry
7
+ * index.
8
+ *
9
+ * `0` (default, CORRECT): `attemptsMade === 1` → retry index 1 → `baseDelayMs`.
10
+ *
11
+ * `1`: reproduces `ts/approval/src/consumer.ts` byte-for-byte — retry index
12
+ * `attemptsMade + 1`, so the first retry waits `base · multiplier`. Set this
13
+ * ONLY to make an approval migration behaviour-preserving, and say so in the
14
+ * migration note. The honest alternative is `attemptOffset: 0` with
15
+ * `baseDelayMs: 500`, which produces the same sequence from a truthful base.
16
+ */
17
+ attemptOffset?: 0 | 1;
18
+ }
19
+ /**
20
+ * Build a BullMQ `settings.backoffStrategy` from a policy.
21
+ *
22
+ * ```ts
23
+ * new Worker(queue, handler, {
24
+ * settings: {
25
+ * backoffStrategy: bullmqBackoffStrategy({
26
+ * baseDelayMs: 500, multiplier: 2, maxDelayMs: 60_000, jitter: "equal",
27
+ * }),
28
+ * },
29
+ * });
30
+ * ```
31
+ *
32
+ * The returned function accepts BullMQ's full `(attemptsMade, type, err, job)`
33
+ * signature but only consults the first argument, so it is assignable wherever
34
+ * BullMQ expects a `BackoffStrategy` without importing bullmq's types (this
35
+ * package has zero dependencies).
36
+ */
37
+ export declare function bullmqBackoffStrategy(policy?: Partial<RetryPolicy>, opts?: BullmqBackoffOptions): (attemptsMade: number) => number;
38
+ //# sourceMappingURL=bullmq.d.ts.map
@@ -0,0 +1 @@
1
+ {"version":3,"file":"bullmq.d.ts","sourceRoot":"","sources":["../../src/adapters/bullmq.ts"],"names":[],"mappings":"AAcA,OAAO,EAAE,KAAK,QAAQ,EAA2B,MAAM,cAAc,CAAC;AACtE,OAAO,EAAE,KAAK,WAAW,EAAiB,MAAM,cAAc,CAAC;AAE/D,MAAM,WAAW,oBAAoB;IACnC,MAAM,CAAC,EAAE,QAAQ,CAAC;IAClB;;;;;;;;;;;OAWG;IACH,aAAa,CAAC,EAAE,CAAC,GAAG,CAAC,CAAC;CACvB;AAED;;;;;;;;;;;;;;;;;GAiBG;AACH,wBAAgB,qBAAqB,CACnC,MAAM,CAAC,EAAE,OAAO,CAAC,WAAW,CAAC,EAC7B,IAAI,GAAE,oBAAyB,GAC9B,CAAC,YAAY,EAAE,MAAM,KAAK,MAAM,CAmBlC"}
@@ -0,0 +1,52 @@
1
+ // BullMQ adapter — the post-increment off-by-one, contained in one place.
2
+ //
3
+ // BullMQ v5 increments `job.attemptsMade` BEFORE it computes the backoff and
4
+ // BEFORE the `failed` handler runs. So a `backoffStrategy` receiving
5
+ // `attemptsMade === 1` is being asked for the delay before the SECOND attempt,
6
+ // i.e. the FIRST retry.
7
+ //
8
+ // `ts/approval/src/consumer.ts:161` computes `Math.min(60_000, 2 ** attempts *
9
+ // 250)` from that argument, which is `250 · 2^attemptsMade` — one doubling
10
+ // AHEAD of the canonical `base · multiplier^(retryIndex - 1)`. Its declared
11
+ // 250ms base is therefore never actually slept: approval's first retry waits
12
+ // 500ms. That is an off-by-one in approval, and this adapter does NOT
13
+ // reproduce it silently — see `attemptOffset`.
14
+ import { jitteredDelayForAttempt } from "../jitter.js";
15
+ import { resolvePolicy } from "../policy.js";
16
+ /**
17
+ * Build a BullMQ `settings.backoffStrategy` from a policy.
18
+ *
19
+ * ```ts
20
+ * new Worker(queue, handler, {
21
+ * settings: {
22
+ * backoffStrategy: bullmqBackoffStrategy({
23
+ * baseDelayMs: 500, multiplier: 2, maxDelayMs: 60_000, jitter: "equal",
24
+ * }),
25
+ * },
26
+ * });
27
+ * ```
28
+ *
29
+ * The returned function accepts BullMQ's full `(attemptsMade, type, err, job)`
30
+ * signature but only consults the first argument, so it is assignable wherever
31
+ * BullMQ expects a `BackoffStrategy` without importing bullmq's types (this
32
+ * package has zero dependencies).
33
+ */
34
+ export function bullmqBackoffStrategy(policy, opts = {}) {
35
+ const resolved = resolvePolicy(policy);
36
+ const random = opts.random ?? Math.random;
37
+ const offset = opts.attemptOffset ?? 0;
38
+ // `"decorrelated"` is a recurrence over the PREVIOUS delay, and BullMQ hands
39
+ // us only an index — there is no previous delay to thread. Refuse rather
40
+ // than silently emitting the seeded first delay on every attempt.
41
+ if (resolved.jitter === "decorrelated") {
42
+ throw new Error('bullmqBackoffStrategy: jitter "decorrelated" is stateful (min(cap, U(base, 3·previous))) and BullMQ\'s backoffStrategy is stateless — use "equal", "full", "none", or proportional jitter');
43
+ }
44
+ return (attemptsMade) => {
45
+ // Guard: BullMQ has been observed to pass 0 on some paths. A retry index
46
+ // must be >= 1 or `baseDelayForAttempt` throws inside a worker's failure
47
+ // path, which would mask the real error.
48
+ const retryIndex = Math.max(1, Math.floor(attemptsMade) + offset);
49
+ return jitteredDelayForAttempt(resolved, retryIndex, { random });
50
+ };
51
+ }
52
+ //# sourceMappingURL=bullmq.js.map
@@ -0,0 +1 @@
1
+ {"version":3,"file":"bullmq.js","sourceRoot":"","sources":["../../src/adapters/bullmq.ts"],"names":[],"mappings":"AAAA,0EAA0E;AAC1E,EAAE;AACF,6EAA6E;AAC7E,qEAAqE;AACrE,+EAA+E;AAC/E,wBAAwB;AACxB,EAAE;AACF,+EAA+E;AAC/E,2EAA2E;AAC3E,4EAA4E;AAC5E,6EAA6E;AAC7E,sEAAsE;AACtE,+CAA+C;AAE/C,OAAO,EAAiB,uBAAuB,EAAE,MAAM,cAAc,CAAC;AACtE,OAAO,EAAoB,aAAa,EAAE,MAAM,cAAc,CAAC;AAmB/D;;;;;;;;;;;;;;;;;GAiBG;AACH,MAAM,UAAU,qBAAqB,CACnC,MAA6B,EAC7B,OAA6B,EAAE;IAE/B,MAAM,QAAQ,GAAG,aAAa,CAAC,MAAM,CAAC,CAAC;IACvC,MAAM,MAAM,GAAG,IAAI,CAAC,MAAM,IAAI,IAAI,CAAC,MAAM,CAAC;IAC1C,MAAM,MAAM,GAAG,IAAI,CAAC,aAAa,IAAI,CAAC,CAAC;IACvC,6EAA6E;IAC7E,yEAAyE;IACzE,kEAAkE;IAClE,IAAI,QAAQ,CAAC,MAAM,KAAK,cAAc,EAAE,CAAC;QACvC,MAAM,IAAI,KAAK,CACb,2LAA2L,CAC5L,CAAC;IACJ,CAAC;IACD,OAAO,CAAC,YAAoB,EAAU,EAAE;QACtC,yEAAyE;QACzE,yEAAyE;QACzE,yCAAyC;QACzC,MAAM,UAAU,GAAG,IAAI,CAAC,GAAG,CAAC,CAAC,EAAE,IAAI,CAAC,KAAK,CAAC,YAAY,CAAC,GAAG,MAAM,CAAC,CAAC;QAClE,OAAO,uBAAuB,CAAC,QAAQ,EAAE,UAAU,EAAE,EAAE,MAAM,EAAE,CAAC,CAAC;IACnE,CAAC,CAAC;AACJ,CAAC"}
@@ -0,0 +1,27 @@
1
+ /**
2
+ * Convention (b): `attempt` is 1-based and IS the attempt being judged.
3
+ *
4
+ * `isFinalAttempt(1, 1)` → `true` (a budget of 1 means no retries at all).
5
+ */
6
+ export declare function isFinalAttempt(attempt: number, maxAttempts: number): boolean;
7
+ /**
8
+ * Convention (a): BullMQ's POST-INCREMENTED `job.attemptsMade`.
9
+ *
10
+ * After the first attempt fails, `attemptsMade === 1`. With
11
+ * `maxAttempts === 3`, attempts 1 and 2 must retry and attempt 3 is terminal:
12
+ *
13
+ * | attemptsMade | attemptsMade + 1 | >= 3 ? | verdict |
14
+ * |--------------|------------------|--------|----------|
15
+ * | 1 | 2 | no | retry |
16
+ * | 2 | 3 | YES | TERMINAL |
17
+ *
18
+ * `attemptsMade === 2` is the boundary. Using `attemptsMade >= maxAttempts`
19
+ * here burns a 4th attempt against a budget of 3.
20
+ */
21
+ export declare function isFinalAttemptFromAttemptsMade(attemptsMade: number, maxAttempts: number): boolean;
22
+ /**
23
+ * Retries still available after the 1-based `attempt` (convention (b)).
24
+ * `Number.POSITIVE_INFINITY` when unbounded.
25
+ */
26
+ export declare function remainingAttempts(attempt: number, maxAttempts: number): number;
27
+ //# sourceMappingURL=attempts.d.ts.map
@@ -0,0 +1 @@
1
+ {"version":3,"file":"attempts.d.ts","sourceRoot":"","sources":["../src/attempts.ts"],"names":[],"mappings":"AAgCA;;;;GAIG;AACH,wBAAgB,cAAc,CAAC,OAAO,EAAE,MAAM,EAAE,WAAW,EAAE,MAAM,GAAG,OAAO,CAG5E;AAED;;;;;;;;;;;;;GAaG;AACH,wBAAgB,8BAA8B,CAC5C,YAAY,EAAE,MAAM,EACpB,WAAW,EAAE,MAAM,GAClB,OAAO,CAGT;AAED;;;GAGG;AACH,wBAAgB,iBAAiB,CAC/B,OAAO,EAAE,MAAM,EACf,WAAW,EAAE,MAAM,GAClB,MAAM,CAGR"}
@@ -0,0 +1,69 @@
1
+ // Attempt-counting semantics — the off-by-one this fleet has already been
2
+ // burned by.
3
+ //
4
+ // There are TWO counter conventions live in ts/* right now and they are NOT
5
+ // interchangeable:
6
+ //
7
+ // (a) POST-INCREMENTED (BullMQ `job.attemptsMade`): after the k-th attempt
8
+ // fails, `attemptsMade === k`. The library sees it AFTER the increment,
9
+ // so "was that the last one?" is `attemptsMade + 1 >= maxAttempts`.
10
+ // -> ts/approval/src/consumer.ts:621 gets this right.
11
+ //
12
+ // (b) PRE-INCREMENTED / 1-BASED (an in-process ledger the worker bumps
13
+ // itself, e.g. `retries.inc(id)`): the value IS the index of the attempt
14
+ // currently being judged, so the test is `attempt >= maxAttempts`.
15
+ // -> ts/replica-consumer/src/worker/streams-worker.ts:1121 (`>=`) and
16
+ // ts/telemetry/src/worker/streams-worker.ts:776 (`<`, complementary)
17
+ // both use this convention.
18
+ //
19
+ // Getting (a) wrong by using (b)'s test means a permanent failure retries
20
+ // forever (off by one too many) or a transient one is dropped on its first
21
+ // blip (off by one too few). Both have shipped in this fleet before. These are
22
+ // two NAMED functions precisely so a call site cannot silently pick the wrong
23
+ // convention — there is no single `isLastAttempt(n, max)` to guess at.
24
+ /**
25
+ * `maxAttempts === 0` means UNBOUNDED (`@nodii/replica-consumer` cold-start
26
+ * default). Nothing is ever the final attempt.
27
+ */
28
+ function unbounded(maxAttempts) {
29
+ return maxAttempts === 0;
30
+ }
31
+ /**
32
+ * Convention (b): `attempt` is 1-based and IS the attempt being judged.
33
+ *
34
+ * `isFinalAttempt(1, 1)` → `true` (a budget of 1 means no retries at all).
35
+ */
36
+ export function isFinalAttempt(attempt, maxAttempts) {
37
+ if (unbounded(maxAttempts))
38
+ return false;
39
+ return attempt >= maxAttempts;
40
+ }
41
+ /**
42
+ * Convention (a): BullMQ's POST-INCREMENTED `job.attemptsMade`.
43
+ *
44
+ * After the first attempt fails, `attemptsMade === 1`. With
45
+ * `maxAttempts === 3`, attempts 1 and 2 must retry and attempt 3 is terminal:
46
+ *
47
+ * | attemptsMade | attemptsMade + 1 | >= 3 ? | verdict |
48
+ * |--------------|------------------|--------|----------|
49
+ * | 1 | 2 | no | retry |
50
+ * | 2 | 3 | YES | TERMINAL |
51
+ *
52
+ * `attemptsMade === 2` is the boundary. Using `attemptsMade >= maxAttempts`
53
+ * here burns a 4th attempt against a budget of 3.
54
+ */
55
+ export function isFinalAttemptFromAttemptsMade(attemptsMade, maxAttempts) {
56
+ if (unbounded(maxAttempts))
57
+ return false;
58
+ return attemptsMade + 1 >= maxAttempts;
59
+ }
60
+ /**
61
+ * Retries still available after the 1-based `attempt` (convention (b)).
62
+ * `Number.POSITIVE_INFINITY` when unbounded.
63
+ */
64
+ export function remainingAttempts(attempt, maxAttempts) {
65
+ if (unbounded(maxAttempts))
66
+ return Number.POSITIVE_INFINITY;
67
+ return Math.max(0, maxAttempts - attempt);
68
+ }
69
+ //# sourceMappingURL=attempts.js.map
@@ -0,0 +1 @@
1
+ {"version":3,"file":"attempts.js","sourceRoot":"","sources":["../src/attempts.ts"],"names":[],"mappings":"AAAA,0EAA0E;AAC1E,aAAa;AACb,EAAE;AACF,4EAA4E;AAC5E,mBAAmB;AACnB,EAAE;AACF,4EAA4E;AAC5E,6EAA6E;AAC7E,yEAAyE;AACzE,2DAA2D;AAC3D,EAAE;AACF,wEAAwE;AACxE,8EAA8E;AAC9E,wEAAwE;AACxE,2EAA2E;AAC3E,6EAA6E;AAC7E,oCAAoC;AACpC,EAAE;AACF,0EAA0E;AAC1E,2EAA2E;AAC3E,+EAA+E;AAC/E,8EAA8E;AAC9E,uEAAuE;AAEvE;;;GAGG;AACH,SAAS,SAAS,CAAC,WAAmB;IACpC,OAAO,WAAW,KAAK,CAAC,CAAC;AAC3B,CAAC;AAED;;;;GAIG;AACH,MAAM,UAAU,cAAc,CAAC,OAAe,EAAE,WAAmB;IACjE,IAAI,SAAS,CAAC,WAAW,CAAC;QAAE,OAAO,KAAK,CAAC;IACzC,OAAO,OAAO,IAAI,WAAW,CAAC;AAChC,CAAC;AAED;;;;;;;;;;;;;GAaG;AACH,MAAM,UAAU,8BAA8B,CAC5C,YAAoB,EACpB,WAAmB;IAEnB,IAAI,SAAS,CAAC,WAAW,CAAC;QAAE,OAAO,KAAK,CAAC;IACzC,OAAO,YAAY,GAAG,CAAC,IAAI,WAAW,CAAC;AACzC,CAAC;AAED;;;GAGG;AACH,MAAM,UAAU,iBAAiB,CAC/B,OAAe,EACf,WAAmB;IAEnB,IAAI,SAAS,CAAC,WAAW,CAAC;QAAE,OAAO,MAAM,CAAC,iBAAiB,CAAC;IAC5D,OAAO,IAAI,CAAC,GAAG,CAAC,CAAC,EAAE,WAAW,GAAG,OAAO,CAAC,CAAC;AAC5C,CAAC"}
@@ -0,0 +1,41 @@
1
+ /** Context handed to a {@link ShouldRetry} predicate. */
2
+ export interface ShouldRetryContext {
3
+ /** 1-based index of the attempt that just FAILED. */
4
+ attempt: number;
5
+ /** The resolved attempt budget (`0` = unbounded). */
6
+ maxAttempts: number;
7
+ }
8
+ /**
9
+ * The seam. Return `false` to make the error terminal — `retry()` stops
10
+ * immediately and rethrows without sleeping.
11
+ */
12
+ export type ShouldRetry = (error: unknown, ctx: ShouldRetryContext) => boolean;
13
+ /**
14
+ * DEFAULT predicate: retry everything.
15
+ *
16
+ * Chosen because it is what 11 of the 14 surveyed sites do — every worker-loop
17
+ * and consumer site retries any thrown error and relies on the attempt budget
18
+ * (plus a DLQ) for termination. The two selective sites (role-catalog:
19
+ * UNAVAILABLE only; onboarding: network + 5xx only) pass their own predicate,
20
+ * which is precisely why this is a seam.
21
+ */
22
+ export declare const retryAll: ShouldRetry;
23
+ /** Never retry — a one-shot with a policy attached. */
24
+ export declare const retryNone: ShouldRetry;
25
+ /** All predicates must agree before an error is retried. */
26
+ export declare function allOf(...predicates: ShouldRetry[]): ShouldRetry;
27
+ /** Any predicate agreeing is enough to retry. */
28
+ export declare function anyOf(...predicates: ShouldRetry[]): ShouldRetry;
29
+ /**
30
+ * The ONE-WAY bridge to an `@nodii/idempotency` `OutcomeClassifier`.
31
+ *
32
+ * `classify` is structurally typed (`(response, error) => string`) so this
33
+ * package takes no dependency on @nodii/idempotency; pass that lib's
34
+ * classifier directly.
35
+ *
36
+ * `transport` → may retry. Anything else (`application`, or any future class)
37
+ * → terminal. Fail-CLOSED on an unrecognised class: an unknown outcome class
38
+ * must not license an unbounded retry loop against a money path.
39
+ */
40
+ export declare function shouldRetryFromOutcomeClassifier(classify: (response: unknown, error: unknown) => string): ShouldRetry;
41
+ //# sourceMappingURL=classify.d.ts.map
@@ -0,0 +1 @@
1
+ {"version":3,"file":"classify.d.ts","sourceRoot":"","sources":["../src/classify.ts"],"names":[],"mappings":"AAyCA,yDAAyD;AACzD,MAAM,WAAW,kBAAkB;IACjC,qDAAqD;IACrD,OAAO,EAAE,MAAM,CAAC;IAChB,qDAAqD;IACrD,WAAW,EAAE,MAAM,CAAC;CACrB;AAED;;;GAGG;AACH,MAAM,MAAM,WAAW,GAAG,CAAC,KAAK,EAAE,OAAO,EAAE,GAAG,EAAE,kBAAkB,KAAK,OAAO,CAAC;AAE/E;;;;;;;;GAQG;AACH,eAAO,MAAM,QAAQ,EAAE,WAAwB,CAAC;AAEhD,uDAAuD;AACvD,eAAO,MAAM,SAAS,EAAE,WAAyB,CAAC;AAElD,4DAA4D;AAC5D,wBAAgB,KAAK,CAAC,GAAG,UAAU,EAAE,WAAW,EAAE,GAAG,WAAW,CAE/D;AAED,iDAAiD;AACjD,wBAAgB,KAAK,CAAC,GAAG,UAAU,EAAE,WAAW,EAAE,GAAG,WAAW,CAE/D;AAED;;;;;;;;;;GAUG;AACH,wBAAgB,gCAAgC,CAC9C,QAAQ,EAAE,CAAC,QAAQ,EAAE,OAAO,EAAE,KAAK,EAAE,OAAO,KAAK,MAAM,GACtD,WAAW,CAEb"}
@@ -0,0 +1,75 @@
1
+ // The terminal-vs-transient SEAM.
2
+ //
3
+ // ── Boundary with @nodii/idempotency (D242) ─────────────────────────────────
4
+ //
5
+ // @nodii/idempotency has an `OutcomeClassifier` that answers a DIFFERENT
6
+ // question, and the two must not be merged:
7
+ //
8
+ // idempotency — "given a COMPLETED attempt's outcome, should the
9
+ // idempotency KEY be cached (application) or released
10
+ // (transport)?" Storage decision. Server side. Keyed by
11
+ // OUTCOME class. Governs replay.
12
+ //
13
+ // retry — "given a THROWN error, should the CALLER re-invoke?"
14
+ // Control-flow decision. Client side. Keyed by the ERROR.
15
+ // Governs whether a next attempt happens at all.
16
+ //
17
+ // They are duals, not synonyms, and `transport` is a SUPERSET of "retryable":
18
+ //
19
+ // - RESOURCE_EXHAUSTED(8) is TRANSPORT (key released, a retry may
20
+ // re-execute) but a caller under a tight budget, or one honouring a
21
+ // Retry-After, may legitimately decline to retry. @nodii/onboarding does
22
+ // exactly this: 429 is not auto-retried.
23
+ // - ALREADY_EXISTS(6) is APPLICATION and also non-retryable — agreement here
24
+ // is a coincidence of that particular code, not a general rule.
25
+ // - CANCELLED(1) / UNKNOWN(2) / UNIMPLEMENTED(12) / DATA_LOSS(15) are
26
+ // deliberately UNMAPPED by D242 and fall back to idempotency's
27
+ // `defaultClass`. That default exists to answer the STORAGE question; it
28
+ // must not silently become a retry verdict.
29
+ //
30
+ // So: this package ships NO gRPC-status table, NO HTTP-status table, and has
31
+ // NO dependency (compile-time or otherwise) on @nodii/idempotency. A change to
32
+ // D242's map therefore cannot silently change anyone's retry behaviour. A
33
+ // service that wants the two coupled writes the coupling EXPLICITLY, one
34
+ // direction only, via `shouldRetryFromOutcomeClassifier` below.
35
+ //
36
+ // The permitted direction is one-way: an idempotency classification may inform
37
+ // a retry decision. The reverse (deriving an outcome class from a retry
38
+ // verdict) is forbidden and not offered — declining to retry does not make an
39
+ // outcome deterministic, and caching a key on that basis would replay a
40
+ // transport failure as if it were a business result.
41
+ /**
42
+ * DEFAULT predicate: retry everything.
43
+ *
44
+ * Chosen because it is what 11 of the 14 surveyed sites do — every worker-loop
45
+ * and consumer site retries any thrown error and relies on the attempt budget
46
+ * (plus a DLQ) for termination. The two selective sites (role-catalog:
47
+ * UNAVAILABLE only; onboarding: network + 5xx only) pass their own predicate,
48
+ * which is precisely why this is a seam.
49
+ */
50
+ export const retryAll = () => true;
51
+ /** Never retry — a one-shot with a policy attached. */
52
+ export const retryNone = () => false;
53
+ /** All predicates must agree before an error is retried. */
54
+ export function allOf(...predicates) {
55
+ return (error, ctx) => predicates.every((p) => p(error, ctx));
56
+ }
57
+ /** Any predicate agreeing is enough to retry. */
58
+ export function anyOf(...predicates) {
59
+ return (error, ctx) => predicates.some((p) => p(error, ctx));
60
+ }
61
+ /**
62
+ * The ONE-WAY bridge to an `@nodii/idempotency` `OutcomeClassifier`.
63
+ *
64
+ * `classify` is structurally typed (`(response, error) => string`) so this
65
+ * package takes no dependency on @nodii/idempotency; pass that lib's
66
+ * classifier directly.
67
+ *
68
+ * `transport` → may retry. Anything else (`application`, or any future class)
69
+ * → terminal. Fail-CLOSED on an unrecognised class: an unknown outcome class
70
+ * must not license an unbounded retry loop against a money path.
71
+ */
72
+ export function shouldRetryFromOutcomeClassifier(classify) {
73
+ return (error) => classify(undefined, error) === "transport";
74
+ }
75
+ //# sourceMappingURL=classify.js.map
@@ -0,0 +1 @@
1
+ {"version":3,"file":"classify.js","sourceRoot":"","sources":["../src/classify.ts"],"names":[],"mappings":"AAAA,kCAAkC;AAClC,EAAE;AACF,+EAA+E;AAC/E,EAAE;AACF,yEAAyE;AACzE,4CAA4C;AAC5C,EAAE;AACF,oEAAoE;AACpE,wEAAwE;AACxE,4EAA4E;AAC5E,mDAAmD;AACnD,EAAE;AACF,wEAAwE;AACxE,4EAA4E;AAC5E,mEAAmE;AACnE,EAAE;AACF,8EAA8E;AAC9E,EAAE;AACF,oEAAoE;AACpE,wEAAwE;AACxE,6EAA6E;AAC7E,6CAA6C;AAC7C,+EAA+E;AAC/E,oEAAoE;AACpE,wEAAwE;AACxE,mEAAmE;AACnE,6EAA6E;AAC7E,gDAAgD;AAChD,EAAE;AACF,6EAA6E;AAC7E,+EAA+E;AAC/E,0EAA0E;AAC1E,yEAAyE;AACzE,gEAAgE;AAChE,EAAE;AACF,+EAA+E;AAC/E,wEAAwE;AACxE,8EAA8E;AAC9E,wEAAwE;AACxE,qDAAqD;AAgBrD;;;;;;;;GAQG;AACH,MAAM,CAAC,MAAM,QAAQ,GAAgB,GAAG,EAAE,CAAC,IAAI,CAAC;AAEhD,uDAAuD;AACvD,MAAM,CAAC,MAAM,SAAS,GAAgB,GAAG,EAAE,CAAC,KAAK,CAAC;AAElD,4DAA4D;AAC5D,MAAM,UAAU,KAAK,CAAC,GAAG,UAAyB;IAChD,OAAO,CAAC,KAAK,EAAE,GAAG,EAAE,EAAE,CAAC,UAAU,CAAC,KAAK,CAAC,CAAC,CAAC,EAAE,EAAE,CAAC,CAAC,CAAC,KAAK,EAAE,GAAG,CAAC,CAAC,CAAC;AAChE,CAAC;AAED,iDAAiD;AACjD,MAAM,UAAU,KAAK,CAAC,GAAG,UAAyB;IAChD,OAAO,CAAC,KAAK,EAAE,GAAG,EAAE,EAAE,CAAC,UAAU,CAAC,IAAI,CAAC,CAAC,CAAC,EAAE,EAAE,CAAC,CAAC,CAAC,KAAK,EAAE,GAAG,CAAC,CAAC,CAAC;AAC/D,CAAC;AAED;;;;;;;;;;GAUG;AACH,MAAM,UAAU,gCAAgC,CAC9C,QAAuD;IAEvD,OAAO,CAAC,KAAK,EAAE,EAAE,CAAC,QAAQ,CAAC,SAAS,EAAE,KAAK,CAAC,KAAK,WAAW,CAAC;AAC/D,CAAC"}
@@ -0,0 +1,32 @@
1
+ /**
2
+ * Marker for "do not retry this, whatever the policy says".
3
+ *
4
+ * Thrown from INSIDE the operation when the operation itself knows the failure
5
+ * is terminal (`@nodii/onboarding`'s 403 bot-challenge, `@nodii/role-catalog`'s
6
+ * FAILED_PRECONDITION). `retry()` rethrows it as-is without consulting
7
+ * `shouldRetry` — the operation's own verdict wins over the caller's predicate.
8
+ *
9
+ * The original error is on `.cause`; the marker is rethrown rather than
10
+ * unwrapped so `instanceof NonRetryableError` stays a usable signal at the
11
+ * boundary. Callers that want the underlying error read `.cause`.
12
+ */
13
+ export declare class NonRetryableError extends Error {
14
+ readonly cause: unknown;
15
+ constructor(message: string, cause?: unknown);
16
+ }
17
+ export declare function isNonRetryableError(err: unknown): err is NonRetryableError;
18
+ /**
19
+ * Optional wrapper for "the attempt budget is spent".
20
+ *
21
+ * OFF by default: `retry()` rethrows the LAST error so a caller's existing
22
+ * `instanceof OnboardingServerError` checks keep working (which is what
23
+ * `@nodii/onboarding` does today). Opt in with `wrapExhausted: true` when you
24
+ * want the `@nodii/role-catalog` shape instead — a distinct typed error that
25
+ * carries the attempt count.
26
+ */
27
+ export declare class RetryExhaustedError extends Error {
28
+ readonly cause: unknown;
29
+ readonly attempts: number;
30
+ constructor(attempts: number, cause: unknown);
31
+ }
32
+ //# sourceMappingURL=errors.d.ts.map
@@ -0,0 +1 @@
1
+ {"version":3,"file":"errors.d.ts","sourceRoot":"","sources":["../src/errors.ts"],"names":[],"mappings":"AAAA;;;;;;;;;;;GAWG;AACH,qBAAa,iBAAkB,SAAQ,KAAK;IAC1C,SAAkB,KAAK,EAAE,OAAO,CAAC;gBACrB,OAAO,EAAE,MAAM,EAAE,KAAK,CAAC,EAAE,OAAO;CAK7C;AAED,wBAAgB,mBAAmB,CAAC,GAAG,EAAE,OAAO,GAAG,GAAG,IAAI,iBAAiB,CAE1E;AAED;;;;;;;;GAQG;AACH,qBAAa,mBAAoB,SAAQ,KAAK;IAC5C,SAAkB,KAAK,EAAE,OAAO,CAAC;IACjC,QAAQ,CAAC,QAAQ,EAAE,MAAM,CAAC;gBACd,QAAQ,EAAE,MAAM,EAAE,KAAK,EAAE,OAAO;CAQ7C"}
package/dist/errors.js ADDED
@@ -0,0 +1,44 @@
1
+ /**
2
+ * Marker for "do not retry this, whatever the policy says".
3
+ *
4
+ * Thrown from INSIDE the operation when the operation itself knows the failure
5
+ * is terminal (`@nodii/onboarding`'s 403 bot-challenge, `@nodii/role-catalog`'s
6
+ * FAILED_PRECONDITION). `retry()` rethrows it as-is without consulting
7
+ * `shouldRetry` — the operation's own verdict wins over the caller's predicate.
8
+ *
9
+ * The original error is on `.cause`; the marker is rethrown rather than
10
+ * unwrapped so `instanceof NonRetryableError` stays a usable signal at the
11
+ * boundary. Callers that want the underlying error read `.cause`.
12
+ */
13
+ export class NonRetryableError extends Error {
14
+ cause;
15
+ constructor(message, cause) {
16
+ super(message);
17
+ this.name = "NonRetryableError";
18
+ this.cause = cause;
19
+ }
20
+ }
21
+ export function isNonRetryableError(err) {
22
+ return err instanceof NonRetryableError;
23
+ }
24
+ /**
25
+ * Optional wrapper for "the attempt budget is spent".
26
+ *
27
+ * OFF by default: `retry()` rethrows the LAST error so a caller's existing
28
+ * `instanceof OnboardingServerError` checks keep working (which is what
29
+ * `@nodii/onboarding` does today). Opt in with `wrapExhausted: true` when you
30
+ * want the `@nodii/role-catalog` shape instead — a distinct typed error that
31
+ * carries the attempt count.
32
+ */
33
+ export class RetryExhaustedError extends Error {
34
+ cause;
35
+ attempts;
36
+ constructor(attempts, cause) {
37
+ const detail = cause instanceof Error ? cause.message : String(cause ?? "unknown error");
38
+ super(`retry exhausted after ${attempts} attempt(s): ${detail}`);
39
+ this.name = "RetryExhaustedError";
40
+ this.attempts = attempts;
41
+ this.cause = cause;
42
+ }
43
+ }
44
+ //# sourceMappingURL=errors.js.map
@@ -0,0 +1 @@
1
+ {"version":3,"file":"errors.js","sourceRoot":"","sources":["../src/errors.ts"],"names":[],"mappings":"AAAA;;;;;;;;;;;GAWG;AACH,MAAM,OAAO,iBAAkB,SAAQ,KAAK;IACxB,KAAK,CAAU;IACjC,YAAY,OAAe,EAAE,KAAe;QAC1C,KAAK,CAAC,OAAO,CAAC,CAAC;QACf,IAAI,CAAC,IAAI,GAAG,mBAAmB,CAAC;QAChC,IAAI,CAAC,KAAK,GAAG,KAAK,CAAC;IACrB,CAAC;CACF;AAED,MAAM,UAAU,mBAAmB,CAAC,GAAY;IAC9C,OAAO,GAAG,YAAY,iBAAiB,CAAC;AAC1C,CAAC;AAED;;;;;;;;GAQG;AACH,MAAM,OAAO,mBAAoB,SAAQ,KAAK;IAC1B,KAAK,CAAU;IACxB,QAAQ,CAAS;IAC1B,YAAY,QAAgB,EAAE,KAAc;QAC1C,MAAM,MAAM,GACV,KAAK,YAAY,KAAK,CAAC,CAAC,CAAC,KAAK,CAAC,OAAO,CAAC,CAAC,CAAC,MAAM,CAAC,KAAK,IAAI,eAAe,CAAC,CAAC;QAC5E,KAAK,CAAC,yBAAyB,QAAQ,gBAAgB,MAAM,EAAE,CAAC,CAAC;QACjE,IAAI,CAAC,IAAI,GAAG,qBAAqB,CAAC;QAClC,IAAI,CAAC,QAAQ,GAAG,QAAQ,CAAC;QACzB,IAAI,CAAC,KAAK,GAAG,KAAK,CAAC;IACrB,CAAC;CACF"}