@magic6/dsh-llm-resilience 0.0.0-stage → 0.5.3

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/CHANGELOG.md ADDED
@@ -0,0 +1,66 @@
1
+ # Changelog
2
+
3
+ All notable changes to this project are documented here. The format follows
4
+ [Keep a Changelog](https://keepachangelog.com/en/1.1.0/) and the project uses
5
+ [Semantic Versioning](https://semver.org/).
6
+
7
+ ## [0.5.3] - 2026-11-16
8
+
9
+ ### Changed
10
+
11
+ - npm package renamed to `@magic6/dsh-llm-resilience`; npm requires a scoped
12
+ package for the first automated publish.
13
+ - README: update the npm installation command.
14
+
15
+ ## [0.5.2] - 2026-11-16
16
+
17
+ ### Added
18
+
19
+ - npm distribution: install with `dsh plugin --profile web add dsh-llm-resilience`.
20
+
21
+ ### Changed
22
+
23
+ - README: document npm installation and use absolute links for the language switch.
24
+
25
+ ## [0.5.1] - 2026-11-16
26
+
27
+ ### Changed
28
+
29
+ - README: add a visible English / Simplified Chinese language switch.
30
+
31
+ ## [0.5.0] - 2026-11-16
32
+
33
+ First public release. The plugin was built from a concrete failure mode observed in the
34
+ author's deployment of **OpenDesign Go** (intermittent tool-call stream and API connection
35
+ failures); it is an independent project and is not affiliated with OpenDesign.
36
+
37
+ ### Added
38
+
39
+ - Synthetic provider route (`fallback/auto`) with an ordered chain of real provider/model
40
+ targets.
41
+ - Per-target retry budget, configurable multiplier (including 1.5), jitter and delay cap.
42
+ - Error classification: entitlement/membership/quota 500 → fallback; transient errors →
43
+ retry; invalid requests → fail; `IMAGE_OFFLOAD_REQUIRED` and `CONTEXT_WINDOW_EXCEEDED`
44
+ pass through unchanged.
45
+ - Streaming-safe tool-call buffering; malformed model-generated tool calls can be retried
46
+ without duplicate output.
47
+ - Step-replay continuation with rotation across the whole chain; chain-tail failures are
48
+ retried in place.
49
+ - Persistent circuit breaker per `provider + model`, half-open probes (foreground and
50
+ background), multi-process state file with tombstones.
51
+ - Global deadline and attempt cap.
52
+ - Session observability and a settings-page chain editor with live breaker state.
53
+ - 90 `node --test` cases and isolated DSH end-to-end smoke coverage.
54
+
55
+ ### Changed
56
+
57
+ - Package and repository renamed from `dsh-llm-fallback` to `dsh-llm-resilience`. The
58
+ profile entry id remains `llm-fallback` for settings compatibility.
59
+ - Default breaker state file is now
60
+ `$DSH_HOME/storages/llm-resilience/state.json`. Set `stateFile` explicitly to reuse an
61
+ older breaker file.
62
+ - Replay envelopes written under the old package name are still accepted.
63
+ - README and DESIGN rewritten for the public release; added a Chinese README, security
64
+ policy, contribution guide and CI.
65
+
66
+ Earlier internal versions are preserved in the private development history.
package/DESIGN.md ADDED
@@ -0,0 +1,265 @@
1
+ # Design
2
+
3
+ ## 1. Status
4
+
5
+ `0.5.0` is the first public release. The adapter, per-target retry, fallback chain,
6
+ persistent breaker, half-open probing, global deadline/attempt cap, session observability,
7
+ multi-process state file, background probing and the settings chain editor are implemented
8
+ and tested. Tool **execution** retries remain out of scope because the host does not expose
9
+ a safe replay hook (see §8).
10
+
11
+ `targets: []` (or all targets `enabled: false`) leaves the synthetic route registered but
12
+ dormant. Invalid configuration fails at load time; it never silently no-ops.
13
+
14
+ ## 2. Goals and non-goals
15
+
16
+ ### Goals
17
+
18
+ - Configure retry count, initial delay, multiplier, cap and jitter per provider/target.
19
+ - One fallback chain, with an **independent** retry budget per target. Switching targets
20
+ restarts backoff from that target's initial delay.
21
+ - Independent circuit breaker per target: lock after N consecutive failures, probe once
22
+ when the lock expires, restore only after a real success.
23
+ - Retry broken model-generated tool calls (`MALFORMED_RESPONSE`, invalid tool JSON); do not
24
+ retry tool execution by default.
25
+ - A settings page that edits the chain and shows the retry schedule.
26
+ - Preserve the original failure: when every target fails, throw the last real failure; do
27
+ not fabricate success.
28
+
29
+ ### Non-goals
30
+
31
+ - Not a replacement for `@deepseek-ai/dsh-llm-retry`, which still owns step-level retries.
32
+ - No retry of side-effecting tool execution (bash, file writes, network calls).
33
+ - No modification of the DSH core; the integration point is a provider wrapper.
34
+ - No infinite retry: a global deadline and attempt cap are mandatory.
35
+ - Do not disguise membership expiry or quota exhaustion as transient faults.
36
+
37
+ ## 3. Why a provider wrapper
38
+
39
+ DSH's `RequestErrorAction` has only `{ kind: 'retry' }`; it cannot switch provider/model.
40
+ Changing provider/model inside the `agent/request` waterfall would violate the "route is
41
+ fixed when the step is prepared" contract. The only clean plugin-level seam is a provider
42
+ adapter: register a synthetic route and resolve the real target inside `stream()`.
43
+
44
+ ## 4. Architecture
45
+
46
+ ```
47
+ agent loop / compaction / other callers
48
+ │ provider = "fallback", model = "auto"
49
+ ▼
50
+ dsh-llm registry
51
+ ▼
52
+ dsh-llm-resilience adapter (wrapper)
53
+ │
54
+ ├─ target 0: primary-gateway/your-model (retry budget + breaker)
55
+ ├─ target 1: backup-provider/your-model (retry budget + breaker)
56
+ └─ target 2: local/your-model (retry budget + breaker)
57
+ ```
58
+
59
+ - The wrapper implements `LlmAdapter`; `stream(options)` calls
60
+ `ctx.llm.stream({ ...options, provider, model })` per target.
61
+ - The wrapper's own outer `providerRetryPolicy` keeps only the codes needed for "failure
62
+ after content was delivered" (`SERVER` / `TRANSPORT` / `MALFORMED_RESPONSE`, etc.) and a
63
+ small step-replay budget. `FALLBACK_EXHAUSTED` and `ALL_TARGETS_LOCKED` are deliberately
64
+ not in that list, so the whole chain is not replayed blindly.
65
+ - Breaker state is keyed by `provider + model`, not by provider alone.
66
+ - Configuration lives in the profile patch; runtime state lives in `stateFile` (default
67
+ `$DSH_HOME/storages/llm-resilience/state.json`), written atomically.
68
+
69
+ ## 5. Request flow
70
+
71
+ ```
72
+ attempts = planAttempts(targets, breaker, now)
73
+ if every target is open/probing: throw ALL_TARGETS_LOCKED (with lockedUntil per target)
74
+
75
+ for attempt of attempts (chain order):
76
+ if state == open: skip
77
+ if state == half-open:
78
+ if !breaker.beginProbe(key): skip
79
+ probe = minimal chat (maxTokens=1, timeoutMs)
80
+ if probe fails: breaker.probeFailed(key); continue to next target
81
+ # A successful probe does not close the breaker immediately:
82
+ # the real request must succeed before recordSuccess().
83
+ for retry = 0..target.maxRetries:
84
+ stream = ctx.llm.stream({ ...options, provider, model })
85
+ try:
86
+ yield chunks (text/reasoning immediately; tool-call buffered until block-end)
87
+ if finish is success: breaker.recordSuccess(key); return
88
+ failure = finish.failure
89
+ catch error:
90
+ failure = error.failure ?? { code: 'UNKNOWN', message: error.message }
91
+ action = classifyFailure(failure, codeActions)
92
+ if action == 'retry' and retry < target.maxRetries:
93
+ await delay(schedule[retry]); continue
94
+ if action == 'fail':
95
+ do not record a breaker failure; throw the original failure
96
+ # action == 'fallback' or retries exhausted
97
+ breaker.recordFailure(key); break // next target
98
+ throw the last failure
99
+ ```
100
+
101
+ ### Streaming invariants
102
+
103
+ - Switch targets inside a request **only before yielding visible content**.
104
+ - Text/reasoning deltas are forwarded immediately.
105
+ - Tool-call block-start/delta chunks are buffered until `block-end`. A malformed tool call
106
+ (no `block-end`) can therefore be dropped and retried safely.
107
+ - After visible content has been delivered, rethrow the original failure and let the host's
108
+ step-level retry replay the step.
109
+ - Replay continuation uses `sessionId + request fingerprint` (message count + last
110
+ role/id). On replay, the plan starts at the next target and **rotates**:
111
+ `[...attempts.slice(next), ...attempts.slice(0, next)]`. The hint only changes order; it
112
+ never deletes a target. If the failed target is the chain tail, retry it in place.
113
+
114
+ ## 6. Error classification
115
+
116
+ Default table (see `lib/policy.js`; override with `codeActions`):
117
+
118
+ | Error | Action | Why |
119
+ |:--|:--|:--|
120
+ | `SERVER` + membership/entitlement/quota/billing/credit/plan | fallback | deterministic account state |
121
+ | `SERVER` / `TIMEOUT` / `TRANSPORT` / `RATE_LIMIT` / `EMPTY_RESPONSE` / `STREAM_CLOSED` | retry | transient |
122
+ | `MALFORMED_RESPONSE` | retry | broken model output; retrying the request is safe |
123
+ | `AUTH` / `INVALID_CREDENTIAL` / `MISSING_CREDENTIAL` | fallback | another provider may work |
124
+ | `QUOTA` / `ACCOUNT_QUOTA` | fallback | switch provider/quota |
125
+ | `NO_ADAPTER` / `UNKNOWN_MODEL` / `MODEL_NOT_FOUND` | fallback | target unavailable |
126
+ | `ABORTED` / `CANCELLED` | fail | caller cancelled |
127
+ | `UNSUPPORTED_OPTION` / `INVALID_REQUEST` / `UNSUPPORTED_CONTENT` | fail | request itself is invalid |
128
+ | `IMAGE_OFFLOAD_REQUIRED` | fail (rethrow) | host image-offload recovery |
129
+ | `CONTEXT_WINDOW_EXCEEDED` | fail (rethrow) | host compaction recovery |
130
+ | unknown | fallback | the last target still throws the original failure |
131
+
132
+ `IMAGE_OFFLOAD_REQUIRED` and `CONTEXT_WINDOW_EXCEEDED` cannot be overridden: they are part
133
+ of DSH's request-level recovery protocol and carry metadata the host needs.
134
+
135
+ ## 7. Backoff and global budget
136
+
137
+ ```
138
+ delay(attempt) = min(initialDelayMs * multiplier^(attempt-1), maxDelayMs)
139
+ actualDelay = min(round(delay * (1 - jitter + 2*jitter*random)), maxDelayMs)
140
+ ```
141
+
142
+ - `multiplier` defaults to 2 and accepts values such as 1.5.
143
+ - Each target has its own `maxRetries` / `initialDelayMs` / `multiplier` / `maxDelayMs` /
144
+ `jitterRatio`.
145
+ - `limits.maxTotalMs` (default 600000) is a deadline: each wait is clipped to the remaining
146
+ time, and the deadline is checked again after the wait.
147
+ - `limits.maxAttempts` (default 50) caps the number of upstream requests.
148
+ - Exhaustion throws `DEADLINE_EXCEEDED` / `ATTEMPT_LIMIT` with the attempt count, elapsed
149
+ time and last real failure.
150
+
151
+ Do not give entitlement/quota 500s a large retry budget: they should fall back on the first
152
+ failure.
153
+
154
+ ## 8. Tool calls
155
+
156
+ Two different things must not be conflated:
157
+
158
+ 1. **Broken model output** (`MALFORMED_RESPONSE`, invalid tool JSON, concatenated tool-call
159
+ arguments): this is an LLM request failure, not a tool execution failure. Because tool-call
160
+ chunks are buffered until `block-end`, the host has not seen anything when the stream is
161
+ malformed; the buffer can be dropped and the LLM request retried. Controlled by
162
+ `toolCalls.retryMalformed` (default `true`).
163
+ 2. **Tool execution failure** (bash non-zero exit, file write failure, network failure): it
164
+ may have side effects and is not retried automatically. In DSH, a bash non-zero exit is a
165
+ successful tool result carrying `[exit code: N]`; the model decides what to do.
166
+
167
+ Cordis 4.1's `tools/execute` waterfall calls `next()` once; calling it again throws
168
+ `next() called multiple times`. The plugin therefore does not ship a `retryExecution` switch.
169
+ Automatically replaying a tool body requires a retry hook in the DSH core/executor.
170
+
171
+ ## 9. Circuit breaker and state file
172
+
173
+ State machine per `provider + model`:
174
+
175
+ ```
176
+ closed --(failureThreshold consecutive failures)--> open
177
+ open --(now >= lockedUntil)--> half-open
178
+ half-open --(beginProbe wins + probe succeeds)--> closed on next real success
179
+ half-open --(probe fails)--> open for another lockMs
180
+ ```
181
+
182
+ - Open/probing targets are skipped; half-open targets require the single probe slot.
183
+ - `enabled: false` targets are filtered before validation and planning; they keep their
184
+ configuration but do not participate in the chain.
185
+ - The probe is a minimal chat request (`maxTokens: 1`, short prompt, `timeoutMs`). Do not
186
+ use `/v1/models` as a health check: a gateway can fail `/models` because of entitlement
187
+ while chat is fine, or vice versa.
188
+ - `breaker.probe.background: true` makes the current request skip a half-open target and
189
+ probes in the background; the next request uses the result.
190
+ - `stateFile` persistence uses `stateFile.lock` + read-merge-write (temp file + rename).
191
+ Merge rules: later lock / more consecutive failures / more lock count wins. A successful
192
+ clear writes a tombstone; older locks cannot be resurrected. Keys this process has never
193
+ seen are not deleted.
194
+ - When every target is open/probing, throw `ALL_TARGETS_LOCKED` with each `lockedUntil`.
195
+
196
+ ## 10. Security invariants
197
+
198
+ - Never switch targets after visible content has been yielded.
199
+ - Never auto-retry side-effecting tool execution.
200
+ - A target's provider must not equal the wrapper route (no recursion); validated at load.
201
+ - Never write API keys or passwords to configuration or logs; `stateFile` stores only
202
+ counters and timestamps.
203
+ - Tests use a temporary `DSH_HOME` and temporary ports. Do not restart a live profile as
204
+ part of the test suite.
205
+ - Multi-process merge only adds/updates; only a timestamped tombstone can clear a lock.
206
+ - When all targets fail, throw the last real failure; never fabricate success or swallow
207
+ errors.
208
+ - Before falling back, check the target's capabilities (context window, modalities,
209
+ reasoning effort); `resolveModel` takes the minimum context window and the intersection of
210
+ modalities/effort ranges. Incompatible requests are not silently attempted.
211
+
212
+ ## 11. Tests and verification
213
+
214
+ Unit tests (`node --test`, 90 cases) cover:
215
+
216
+ - nominal backoff (2x / 1.5x / cap), jitter bounds, per-target reset;
217
+ - classification: entitlement 500 → fallback, transient → retry, invalid → fail, overrides;
218
+ - chain validation: empty, duplicate, self-reference, invalid numbers;
219
+ - breaker threshold/lock/half-open/probe mutex/success recovery/lock extension/snapshot;
220
+ - adapter execution: first success does not touch the second target; transient retries use
221
+ the target's curve; exhausted retries move to the next target and reset backoff;
222
+ - fallback on entitlement 500 without spending the retry budget;
223
+ - failure after text delivery rethrows the original code and does not switch targets;
224
+ - tool-call buffering: malformed calls are retried, complete calls are not duplicated;
225
+ - step replay: same fingerprint starts at the next target, rotates to cover the whole chain,
226
+ and retries the tail in place; a different fingerprint discards the hint;
227
+ - breaker persistence across processes, including sub-threshold counts, tombstones and
228
+ concurrent writes;
229
+ - global deadline/attempt caps, background probing, runtime snapshots and session records;
230
+ - settings-client pure functions: field round-trips, unit switching (minutes ↔ ms), retry
231
+ schedule parity with the host, validation and catalogue checks;
232
+ - replay envelope wrap/unwrap, including the legacy `dsh-llm-fallback` kind.
233
+
234
+ End-to-end smoke (isolated `DSH_HOME`, temporary web profile, fake OpenAI-compatible server):
235
+
236
+ - entitlement 500 on target 1 → fallback to target 2, output normal;
237
+ - two transient 500s on target 3 → retry with 100/200 ms backoff, third attempt succeeds;
238
+ - all targets fail → `FALLBACK_EXHAUSTED` with the last real failure preserved;
239
+ - malformed tool call → buffer dropped and retried, host sees only the successful attempt;
240
+ - breaker threshold writes `stateFile`; a new process skips the locked target;
241
+ - lock expiry + failed probe → lock again;
242
+ - text delivered then transport failure → host replays the step, skips the failed target,
243
+ then succeeds;
244
+ - settings card renders in a temporary web profile; add/edit/save/reload round-trips.
245
+
246
+ ## 12. Known limitations and open questions
247
+
248
+ 1. **Wrapper vs core patch.** The wrapper is verified on an isolated DSH. Nested
249
+ `ctx.llm.stream()` passes through the `llm/stream` waterfall again; the session-title
250
+ WeakSet does not mistake the wrapped object for an agent-loop request, and
251
+ `session-checkpoint-policy` flushes once more (harmless). The core is not modified.
252
+ 2. **Partial output.** Once visible content is delivered, the plugin relies on step replay
253
+ plus target continuation rather than buffering the whole stream. Buffering would remove
254
+ in-request fallback after the first token but delay the first token; it is not the default.
255
+ 3. **Model catalogue.** The synthetic route exposes one model, `auto`. `resolveModel` takes
256
+ the minimum context window, modality intersection and reasoning effort intersection of
257
+ the chain. The settings page shows the real model for each target.
258
+ 4. **All targets locked.** `ALL_TARGETS_LOCKED` is thrown; the plugin does not force one
259
+ last request into a locked provider. Whether to allow a "last resort even if locked"
260
+ target is an open question.
261
+ 5. **Tool execution retry.** Blocked by cordis `next()` single-call semantics; requires a
262
+ core/executor hook.
263
+ 6. **Replay envelope identity.** The envelope kind changed with the package rename in 0.5.0;
264
+ the unwrapper accepts the legacy kind so in-flight sessions from the private development
265
+ line keep working. Public users start on the new kind.
package/LICENSE ADDED
@@ -0,0 +1,21 @@
1
+ MIT License
2
+
3
+ Copyright (c) 2026 T-splines
4
+
5
+ Permission is hereby granted, free of charge, to any person obtaining a copy
6
+ of this software and associated documentation files (the "Software"), to deal
7
+ in the Software without restriction, including without limitation the rights
8
+ to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
9
+ copies of the Software, and to permit persons to whom the Software is
10
+ furnished to do so, subject to the following conditions:
11
+
12
+ The above copyright notice and this permission notice shall be included in all
13
+ copies or substantial portions of the Software.
14
+
15
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16
+ IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17
+ FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18
+ AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19
+ LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20
+ OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
21
+ SOFTWARE.
package/README.md CHANGED
@@ -1,3 +1,306 @@
1
- # Temporary Holding Version
1
+ # dsh-llm-resilience
2
2
 
3
- This version is a temporary placeholder for this package. An operational version to replace this has been submitted for review and is awaiting a staged release.
3
+ **English** | [简体中文](https://github.com/T-splines/dsh-llm-resilience/blob/main/README.zh-CN.md)
4
+
5
+ **Make a flaky LLM gateway behave like a reliable one — per-target retries, streaming-safe fallback chains and persistent circuit breaking for DeepSeek Harness.**
6
+
7
+ This plugin was built from a concrete failure mode we hit in daily use. **OpenDesign Go**'s
8
+ tool-call stream and API connections were intermittently unstable: a malformed tool-call
9
+ stream, a transport drop, or a membership/quota error surfaced as a generic 500 could end
10
+ an entire agent step. DSH's built-in `retryPolicy` is provider-level — it can back off on
11
+ the same route, but it cannot switch provider/model, and it has no circuit breaker. In our
12
+ deployment this plugin moved OpenDesign Go from **barely usable** to **reliably usable**:
13
+ safe retries absorb transient faults, a second provider takes over when the gateway is out
14
+ of quota or broken, and a persistent breaker stops hammering a dead upstream.
15
+
16
+ > This is an independent plugin. It is not affiliated with or endorsed by OpenDesign or
17
+ > DeepSeek; OpenDesign Go is mentioned only as the deployment that motivated the design.
18
+ > The failure pattern is not unique to that service — any gateway that intermittently
19
+ > breaks tool-call streams or returns quota errors as generic 500s benefits from the same
20
+ > chain.
21
+
22
+ - [Why this exists](#why-this-exists)
23
+ - [What makes it different](#what-makes-it-different)
24
+ - [What it registers](#what-it-registers)
25
+ - [Install](#install)
26
+ - [Configuration](#configuration)
27
+ - [Error classification](#error-classification)
28
+ - [Streaming, tool calls and step replay](#streaming-tool-calls-and-step-replay)
29
+ - [Circuit breaker](#circuit-breaker)
30
+ - [Settings page](#settings-page)
31
+ - [Limits and honest non-goals](#limits-and-honest-non-goals)
32
+ - [Tests](#tests)
33
+ - [Compatibility and migration](#compatibility-and-migration)
34
+
35
+ ## Why this exists
36
+
37
+ DSH's `retryPolicy` already retries at the provider level, but:
38
+
39
+ - the backoff multiplier is fixed at 2 — you cannot ask for 1.5;
40
+ - `agent/request-error` can only retry the **same route**; it cannot switch provider/model;
41
+ - there is no circuit breaker: when a provider is down, every step still hits it first;
42
+ - `MALFORMED_RESPONSE` (a broken tool-call stream) and a failed tool **execution** need
43
+ different treatment.
44
+
45
+ This plugin registers a synthetic provider route (default `fallback`, model `auto`) that
46
+ internally walks a chain of real provider/model targets:
47
+
48
+ ```
49
+ fallback/auto
50
+ ├─ target 0: primary-gateway/your-model independent retry budget + breaker
51
+ ├─ target 1: backup-provider/your-model independent retry budget + breaker
52
+ └─ target 2: local/your-model independent retry budget + breaker
53
+ ```
54
+
55
+ Every target has its own retry budget, backoff curve and breaker. Switching to the next
56
+ target resets the backoff instead of inheriting the previous target's already-doubled delay.
57
+
58
+ ## What makes it different
59
+
60
+ - **Per-target retry budgets.** `maxRetries`, `initialDelayMs`, `multiplier` (including 1.5),
61
+ `maxDelayMs` and `jitterRatio` are independent for each target.
62
+ - **Error-aware classification.** Membership/entitlement/quota errors surfaced as `SERVER`
63
+ 500 fall back immediately; ordinary 500/timeout/transport errors retry; invalid requests
64
+ fail without trying another target. `codeActions` lets you override the defaults.
65
+ - **Streaming-safe tool calls.** Text and reasoning deltas are forwarded immediately.
66
+ Tool-call block-start/delta chunks are buffered until `block-end`, so a malformed tool-call
67
+ stream (no `block-end`) can be retried without the host ever seeing duplicate output.
68
+ - **Step-replay continuation.** After content has been delivered, a failure cannot switch
69
+ targets inside the same request. The plugin remembers `sessionId + request fingerprint`;
70
+ when the host replays the step it starts at the next target, and the plan **rotates** to
71
+ cover the whole chain instead of truncating it. If the failure was the chain tail, it is
72
+ retried in place — a retryable failure is never rewritten into a terminal
73
+ `FALLBACK_EXHAUSTED`.
74
+ - **Persistent circuit breaker, per provider + model.** Consecutive failures lock a target
75
+ for a configurable window; the lock expires into a half-open probe (foreground or
76
+ background). State is written atomically to `stateFile` and merged across processes with
77
+ a lock + tombstones, so a restart does not forget a partially accumulated failure count.
78
+ - **Global budget.** `limits.maxTotalMs` (default 10 minutes) and `limits.maxAttempts`
79
+ (default 50) are enforced across targets and retries; backoff waits count toward the
80
+ deadline. No combination of targets × retries can hang a step for an unbounded time.
81
+ - **Request-level recovery codes pass through.** `IMAGE_OFFLOAD_REQUIRED` (with
82
+ `offloadImages`) and `CONTEXT_WINDOW_EXCEEDED` are re-thrown unchanged so the host's image
83
+ offload / compaction layers can recover first.
84
+ - **Settings page.** Edit the chain, reorder targets, see each retry schedule, and watch live
85
+ breaker state (`closed / open / half-open / probing`) next to version and uninstall.
86
+ - **Tested.** 90 `node --test` cases plus isolated DSH end-to-end smoke runs with a fake
87
+ OpenAI-compatible server (see [DESIGN.md](DESIGN.md)).
88
+
89
+ ## What it registers
90
+
91
+ | entry | value |
92
+ |:--|:--|
93
+ | synthetic route | `fallback` (configurable) |
94
+ | synthetic model | `auto` |
95
+ | profile entry id | `llm-fallback` (kept stable for settings compatibility) |
96
+ | package name | `dsh-llm-resilience` |
97
+ | default state file | `$DSH_HOME/storages/llm-resilience/state.json` |
98
+
99
+ `targets: []` (or every target `enabled: false`) leaves the route registered but dormant:
100
+ you can build the chain in the settings page first, then enable it.
101
+
102
+ ## Install
103
+
104
+ Requirements: DSH `0.2.2-rc.x`, Node.js 20+.
105
+
106
+ ### From npm
107
+
108
+ ```bash
109
+ dsh plugin --profile web add @magic6/dsh-llm-resilience
110
+ ```
111
+
112
+ ### From source
113
+
114
+ ```bash
115
+ git clone https://github.com/T-splines/dsh-llm-resilience.git
116
+ cd dsh-llm-resilience
117
+ dsh plugin --profile web add "$PWD"
118
+ ```
119
+
120
+ The package is a bundle (`dsh.bundle.patch`), so the `llm-fallback` entry is already mounted
121
+ by the bundle. **Override that entry** — do not `insert` a second one. In
122
+ `$DSH_HOME/profiles/<profile>/cordis.patch.yml`, replacing the whole `config`:
123
+
124
+ ```yaml
125
+ - id: llm-fallback
126
+ name: 'dsh-llm-resilience'
127
+ config:
128
+ enabled: true
129
+ route: fallback
130
+ targets:
131
+ - enabled: true
132
+ provider: primary-gateway
133
+ model: your-model
134
+ maxRetries: 5
135
+ initialDelayMs: 1500
136
+ multiplier: 2
137
+ maxDelayMs: 20000
138
+ jitterRatio: 0.3
139
+ - enabled: true
140
+ provider: backup-provider
141
+ model: your-backup-model
142
+ maxRetries: 3
143
+ initialDelayMs: 1000
144
+ breaker:
145
+ failureThreshold: 5
146
+ lockMs: 3600000
147
+ probe:
148
+ enabled: true
149
+ background: false
150
+ timeoutMs: 10000
151
+ maxTokens: 1
152
+ prompt: 'ping'
153
+ stateFile: '$DSH_HOME/storages/llm-resilience/state.json'
154
+ limits:
155
+ maxTotalMs: 600000
156
+ maxAttempts: 50
157
+ toolCalls:
158
+ retryMalformed: true
159
+ ```
160
+
161
+ Replace `primary-gateway` / `backup-provider` with real provider ids from your DSH model
162
+ catalog (the settings page shows them as read-only dropdowns). A target whose provider does
163
+ not exist fails with `NO_ADAPTER` and the chain moves on; it does not silently pretend the
164
+ target served.
165
+
166
+ Then point the agent's default model at the synthetic route:
167
+
168
+ ```yaml
169
+ - id: agent-default-model
170
+ config:
171
+ provider: fallback
172
+ model: auto
173
+ ```
174
+
175
+ Verify composition without starting an agent (`web` is the common profile name; use yours):
176
+
177
+ ```bash
178
+ dsh --profile web --dump-config | grep -A8 'id: llm-fallback'
179
+ ```
180
+
181
+ > When overriding an existing `llm-fallback` entry with `--patch`, write the **whole**
182
+ > `targets` array. DSH's config merge fills omitted array fields with an empty default, so
183
+ > overriding only `breaker` would clear the chain.
184
+
185
+ **Do not give entitlement/quota errors 10 retries.** They are deterministic provider state;
186
+ retrying the same target only wastes the budget. They should fall back on the first failure.
187
+
188
+ ## Configuration
189
+
190
+ | field | meaning |
191
+ |:--|:--|
192
+ | `enabled` | master switch; the route still exists when disabled, but rejects requests |
193
+ | `route` | synthetic provider name (default `fallback`); must not appear in `targets` |
194
+ | `targets[]` | ordered `{ provider, model, enabled, label, maxRetries, initialDelayMs, multiplier, maxDelayMs, jitterRatio }` |
195
+ | `breaker` | `failureThreshold`, `lockMs`, probe settings (`enabled`, `background`, `timeoutMs`, `maxTokens`, `prompt`), `stateFile` |
196
+ | `limits` | `maxTotalMs`, `maxAttempts`; `0` disables that dimension |
197
+ | `toolCalls.retryMalformed` | retry a broken model-generated tool call (default `true`); tool **execution** retries are not supported |
198
+ | `codeActions` | override the default action per stable error code (`retry` / `fallback` / `fail`) |
199
+
200
+ ## Error classification
201
+
202
+ | error | action | why |
203
+ |:--|:--|:--|
204
+ | `SERVER` + membership/entitlement/quota/billing/credit/plan | fallback | deterministic account state; retrying is useless |
205
+ | `SERVER` / `TIMEOUT` / `TRANSPORT` / `RATE_LIMIT` / `EMPTY_RESPONSE` / `STREAM_CLOSED` | retry | transient |
206
+ | `MALFORMED_RESPONSE` | retry | broken model output; safe to retry the whole LLM request |
207
+ | `AUTH` / `INVALID_CREDENTIAL` / `MISSING_CREDENTIAL` | fallback | another provider may work |
208
+ | `QUOTA` / `ACCOUNT_QUOTA` | fallback | switch provider/quota |
209
+ | `NO_ADAPTER` / `UNKNOWN_MODEL` / `MODEL_NOT_FOUND` | fallback | target unavailable |
210
+ | `ABORTED` / `CANCELLED` | fail | the caller cancelled |
211
+ | `UNSUPPORTED_OPTION` / `INVALID_REQUEST` / `UNSUPPORTED_CONTENT` | fail | the request itself is wrong; another target will not fix it |
212
+ | `IMAGE_OFFLOAD_REQUIRED` | fail (re-thrown) | host image-offload recovery must see it |
213
+ | `CONTEXT_WINDOW_EXCEEDED` | fail (re-thrown) | host compaction recovery must see it |
214
+ | unknown code | fallback | the last target still throws the original failure; nothing is swallowed |
215
+
216
+ `IMAGE_OFFLOAD_REQUIRED` and `CONTEXT_WINDOW_EXCEEDED` cannot be overridden: they are part
217
+ of DSH's request-level recovery protocol, and the host's listeners depend on the original
218
+ code and metadata.
219
+
220
+ ## Streaming, tool calls and step replay
221
+
222
+ - Only switch targets **before** yielding anything visible to the caller.
223
+ - Text/reasoning deltas are forwarded immediately.
224
+ - Tool-call block-start/delta chunks are buffered until `block-end`; a malformed tool call
225
+ (no `block-end`) can be dropped and retried safely.
226
+ - After visible content has been delivered, a failure is returned to the host so
227
+ `dsh-llm-retry` can replay the whole step. The plugin records a fingerprint and the
228
+ target it reached; on replay it starts at the next target. The plan **rotates** rather
229
+ than truncating, so a target before the failure point is not lost. If the failure was the
230
+ chain tail, it retries that target in place.
231
+ - The replay envelope written by the synthetic route is unwrapped into the real target's
232
+ provider/model before the request reaches the upstream adapter, preserving signatures and
233
+ ids. Envelopes written by the pre-rename `dsh-llm-fallback` package are still accepted.
234
+
235
+ ## Circuit breaker
236
+
237
+ ```
238
+ closed --(failureThreshold consecutive failures)--> open
239
+ open --(lock expires)--> half-open
240
+ half-open --(probe succeeds)--> closed on the next real success
241
+ half-open --(probe fails)--> open for another lock window
242
+ ```
243
+
244
+ - State is keyed by `provider + model`, not by provider alone.
245
+ - `stateFile` is written atomically with a lock + read-merge-write. Concurrent processes
246
+ keep the later/more-failed entry; a successful clear writes a tombstone so an older lock
247
+ cannot be resurrected.
248
+ - Unreached failure counts are persisted too (`updatedAt` is the merge timeline), so a
249
+ restart does not reset the count below the threshold.
250
+ - When every target is open/probing, the plugin throws `ALL_TARGETS_LOCKED` with each
251
+ target's `lockedUntil` — it does not force one last request into a locked provider.
252
+
253
+ ## Settings page
254
+
255
+ Plugins → Installed → `dsh-llm-resilience` → **Model fallback chain**:
256
+
257
+ - per-target rows: enabled, provider, model (read-only dropdowns from the session model
258
+ catalog), label, retry parameters, move up/down, delete, add;
259
+ - retry-plan tooltip: one line per retry, with the jitter range when `jitter > 0`;
260
+ - breaker and budget fields;
261
+ - live runtime state polled from the host (`closed / open / half-open / probing`, remaining
262
+ lock time, consecutive failures and lock count);
263
+ - non-loopback browsers get the host's memory-state view and the card says so instead of
264
+ rendering plugin defaults as if they were the deployed policy.
265
+
266
+ ## Limits and honest non-goals
267
+
268
+ - **Tool execution is not retried.** Cordis 4.1's `tools/execute` waterfall calls `next()`
269
+ once; calling it again throws `next() called multiple times`. Automatically replaying a
270
+ tool body needs a retry hook in the DSH core/executor. The plugin does not ship a switch
271
+ that silently does nothing. Retrying a malformed **model-generated** tool call is safe and
272
+ supported.
273
+ - **No infinite retries.** The global deadline and attempt cap are required; `0` disables a
274
+ dimension but is not recommended for production.
275
+ - **Does not replace `@deepseek-ai/dsh-llm-retry`.** That package still owns step-level
276
+ retries; this plugin owns provider selection and request-level resilience.
277
+ - **No streaming buffer for late fallback.** Once visible content is delivered, switching
278
+ targets would duplicate output; the plugin hands the step back to the host instead.
279
+
280
+ ## Tests
281
+
282
+ ```bash
283
+ npm test
284
+ ```
285
+
286
+ 90 cases cover backoff/jitter, error classification, chain validation, adapter execution,
287
+ streaming safety, step-replay rotation, breaker persistence and multi-process merging,
288
+ global budgets, background probing, settings-client pure functions and the replay envelope
289
+ (including the legacy package name). End-to-end smoke runs on an isolated `DSH_HOME` with a
290
+ fake OpenAI-compatible server are described in [DESIGN.md](DESIGN.md).
291
+
292
+ ## Compatibility and migration
293
+
294
+ - DSH `0.2.2-rc.x` (`engines.dsh` is `>=0.2.2-0 <0.3.0-0`), Node.js 20+.
295
+ - **Renamed from `dsh-llm-fallback` in 0.5.0.** The profile entry id remains `llm-fallback`,
296
+ so settings and existing patches keep working; update the `name:` field to
297
+ `dsh-llm-resilience`.
298
+ - The default breaker state file moved to
299
+ `$DSH_HOME/storages/llm-resilience/state.json`. Set `stateFile` explicitly if you want to
300
+ keep using an older breaker file.
301
+ - Replay envelopes from the old package name still unwrap (`dsh-llm-fallback` is accepted as
302
+ a legacy kind).
303
+
304
+ ## License
305
+
306
+ MIT. See [LICENSE](LICENSE).