@magic6/dsh-llm-resilience 0.0.0-stage → 0.5.3
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +66 -0
- package/DESIGN.md +265 -0
- package/LICENSE +21 -0
- package/README.md +305 -2
- package/README.zh-CN.md +279 -0
- package/cordis.patch.yml +46 -0
- package/lib/adapter.js +1055 -0
- package/lib/client.js +1695 -0
- package/lib/config.js +135 -0
- package/lib/index.js +87 -0
- package/lib/policy.js +549 -0
- package/lib/session-log.js +42 -0
- package/lib/state-file.js +105 -0
- package/package.json +69 -4
package/CHANGELOG.md
ADDED
|
@@ -0,0 +1,66 @@
|
|
|
1
|
+
# Changelog
|
|
2
|
+
|
|
3
|
+
All notable changes to this project are documented here. The format follows
|
|
4
|
+
[Keep a Changelog](https://keepachangelog.com/en/1.1.0/) and the project uses
|
|
5
|
+
[Semantic Versioning](https://semver.org/).
|
|
6
|
+
|
|
7
|
+
## [0.5.3] - 2026-11-16
|
|
8
|
+
|
|
9
|
+
### Changed
|
|
10
|
+
|
|
11
|
+
- npm package renamed to `@magic6/dsh-llm-resilience`; npm requires a scoped
|
|
12
|
+
package for the first automated publish.
|
|
13
|
+
- README: update the npm installation command.
|
|
14
|
+
|
|
15
|
+
## [0.5.2] - 2026-11-16
|
|
16
|
+
|
|
17
|
+
### Added
|
|
18
|
+
|
|
19
|
+
- npm distribution: install with `dsh plugin --profile web add dsh-llm-resilience`.
|
|
20
|
+
|
|
21
|
+
### Changed
|
|
22
|
+
|
|
23
|
+
- README: document npm installation and use absolute links for the language switch.
|
|
24
|
+
|
|
25
|
+
## [0.5.1] - 2026-11-16
|
|
26
|
+
|
|
27
|
+
### Changed
|
|
28
|
+
|
|
29
|
+
- README: add a visible English / Simplified Chinese language switch.
|
|
30
|
+
|
|
31
|
+
## [0.5.0] - 2026-11-16
|
|
32
|
+
|
|
33
|
+
First public release. The plugin was built from a concrete failure mode observed in the
|
|
34
|
+
author's deployment of **OpenDesign Go** (intermittent tool-call stream and API connection
|
|
35
|
+
failures); it is an independent project and is not affiliated with OpenDesign.
|
|
36
|
+
|
|
37
|
+
### Added
|
|
38
|
+
|
|
39
|
+
- Synthetic provider route (`fallback/auto`) with an ordered chain of real provider/model
|
|
40
|
+
targets.
|
|
41
|
+
- Per-target retry budget, configurable multiplier (including 1.5), jitter and delay cap.
|
|
42
|
+
- Error classification: entitlement/membership/quota 500 → fallback; transient errors →
|
|
43
|
+
retry; invalid requests → fail; `IMAGE_OFFLOAD_REQUIRED` and `CONTEXT_WINDOW_EXCEEDED`
|
|
44
|
+
pass through unchanged.
|
|
45
|
+
- Streaming-safe tool-call buffering; malformed model-generated tool calls can be retried
|
|
46
|
+
without duplicate output.
|
|
47
|
+
- Step-replay continuation with rotation across the whole chain; chain-tail failures are
|
|
48
|
+
retried in place.
|
|
49
|
+
- Persistent circuit breaker per `provider + model`, half-open probes (foreground and
|
|
50
|
+
background), multi-process state file with tombstones.
|
|
51
|
+
- Global deadline and attempt cap.
|
|
52
|
+
- Session observability and a settings-page chain editor with live breaker state.
|
|
53
|
+
- 90 `node --test` cases and isolated DSH end-to-end smoke coverage.
|
|
54
|
+
|
|
55
|
+
### Changed
|
|
56
|
+
|
|
57
|
+
- Package and repository renamed from `dsh-llm-fallback` to `dsh-llm-resilience`. The
|
|
58
|
+
profile entry id remains `llm-fallback` for settings compatibility.
|
|
59
|
+
- Default breaker state file is now
|
|
60
|
+
`$DSH_HOME/storages/llm-resilience/state.json`. Set `stateFile` explicitly to reuse an
|
|
61
|
+
older breaker file.
|
|
62
|
+
- Replay envelopes written under the old package name are still accepted.
|
|
63
|
+
- README and DESIGN rewritten for the public release; added a Chinese README, security
|
|
64
|
+
policy, contribution guide and CI.
|
|
65
|
+
|
|
66
|
+
Earlier internal versions are preserved in the private development history.
|
package/DESIGN.md
ADDED
|
@@ -0,0 +1,265 @@
|
|
|
1
|
+
# Design
|
|
2
|
+
|
|
3
|
+
## 1. Status
|
|
4
|
+
|
|
5
|
+
`0.5.0` is the first public release. The adapter, per-target retry, fallback chain,
|
|
6
|
+
persistent breaker, half-open probing, global deadline/attempt cap, session observability,
|
|
7
|
+
multi-process state file, background probing and the settings chain editor are implemented
|
|
8
|
+
and tested. Tool **execution** retries remain out of scope because the host does not expose
|
|
9
|
+
a safe replay hook (see §8).
|
|
10
|
+
|
|
11
|
+
`targets: []` (or all targets `enabled: false`) leaves the synthetic route registered but
|
|
12
|
+
dormant. Invalid configuration fails at load time; it never silently no-ops.
|
|
13
|
+
|
|
14
|
+
## 2. Goals and non-goals
|
|
15
|
+
|
|
16
|
+
### Goals
|
|
17
|
+
|
|
18
|
+
- Configure retry count, initial delay, multiplier, cap and jitter per provider/target.
|
|
19
|
+
- One fallback chain, with an **independent** retry budget per target. Switching targets
|
|
20
|
+
restarts backoff from that target's initial delay.
|
|
21
|
+
- Independent circuit breaker per target: lock after N consecutive failures, probe once
|
|
22
|
+
when the lock expires, restore only after a real success.
|
|
23
|
+
- Retry broken model-generated tool calls (`MALFORMED_RESPONSE`, invalid tool JSON); do not
|
|
24
|
+
retry tool execution by default.
|
|
25
|
+
- A settings page that edits the chain and shows the retry schedule.
|
|
26
|
+
- Preserve the original failure: when every target fails, throw the last real failure; do
|
|
27
|
+
not fabricate success.
|
|
28
|
+
|
|
29
|
+
### Non-goals
|
|
30
|
+
|
|
31
|
+
- Not a replacement for `@deepseek-ai/dsh-llm-retry`, which still owns step-level retries.
|
|
32
|
+
- No retry of side-effecting tool execution (bash, file writes, network calls).
|
|
33
|
+
- No modification of the DSH core; the integration point is a provider wrapper.
|
|
34
|
+
- No infinite retry: a global deadline and attempt cap are mandatory.
|
|
35
|
+
- Do not disguise membership expiry or quota exhaustion as transient faults.
|
|
36
|
+
|
|
37
|
+
## 3. Why a provider wrapper
|
|
38
|
+
|
|
39
|
+
DSH's `RequestErrorAction` has only `{ kind: 'retry' }`; it cannot switch provider/model.
|
|
40
|
+
Changing provider/model inside the `agent/request` waterfall would violate the "route is
|
|
41
|
+
fixed when the step is prepared" contract. The only clean plugin-level seam is a provider
|
|
42
|
+
adapter: register a synthetic route and resolve the real target inside `stream()`.
|
|
43
|
+
|
|
44
|
+
## 4. Architecture
|
|
45
|
+
|
|
46
|
+
```
|
|
47
|
+
agent loop / compaction / other callers
|
|
48
|
+
│ provider = "fallback", model = "auto"
|
|
49
|
+
▼
|
|
50
|
+
dsh-llm registry
|
|
51
|
+
▼
|
|
52
|
+
dsh-llm-resilience adapter (wrapper)
|
|
53
|
+
│
|
|
54
|
+
├─ target 0: primary-gateway/your-model (retry budget + breaker)
|
|
55
|
+
├─ target 1: backup-provider/your-model (retry budget + breaker)
|
|
56
|
+
└─ target 2: local/your-model (retry budget + breaker)
|
|
57
|
+
```
|
|
58
|
+
|
|
59
|
+
- The wrapper implements `LlmAdapter`; `stream(options)` calls
|
|
60
|
+
`ctx.llm.stream({ ...options, provider, model })` per target.
|
|
61
|
+
- The wrapper's own outer `providerRetryPolicy` keeps only the codes needed for "failure
|
|
62
|
+
after content was delivered" (`SERVER` / `TRANSPORT` / `MALFORMED_RESPONSE`, etc.) and a
|
|
63
|
+
small step-replay budget. `FALLBACK_EXHAUSTED` and `ALL_TARGETS_LOCKED` are deliberately
|
|
64
|
+
not in that list, so the whole chain is not replayed blindly.
|
|
65
|
+
- Breaker state is keyed by `provider + model`, not by provider alone.
|
|
66
|
+
- Configuration lives in the profile patch; runtime state lives in `stateFile` (default
|
|
67
|
+
`$DSH_HOME/storages/llm-resilience/state.json`), written atomically.
|
|
68
|
+
|
|
69
|
+
## 5. Request flow
|
|
70
|
+
|
|
71
|
+
```
|
|
72
|
+
attempts = planAttempts(targets, breaker, now)
|
|
73
|
+
if every target is open/probing: throw ALL_TARGETS_LOCKED (with lockedUntil per target)
|
|
74
|
+
|
|
75
|
+
for attempt of attempts (chain order):
|
|
76
|
+
if state == open: skip
|
|
77
|
+
if state == half-open:
|
|
78
|
+
if !breaker.beginProbe(key): skip
|
|
79
|
+
probe = minimal chat (maxTokens=1, timeoutMs)
|
|
80
|
+
if probe fails: breaker.probeFailed(key); continue to next target
|
|
81
|
+
# A successful probe does not close the breaker immediately:
|
|
82
|
+
# the real request must succeed before recordSuccess().
|
|
83
|
+
for retry = 0..target.maxRetries:
|
|
84
|
+
stream = ctx.llm.stream({ ...options, provider, model })
|
|
85
|
+
try:
|
|
86
|
+
yield chunks (text/reasoning immediately; tool-call buffered until block-end)
|
|
87
|
+
if finish is success: breaker.recordSuccess(key); return
|
|
88
|
+
failure = finish.failure
|
|
89
|
+
catch error:
|
|
90
|
+
failure = error.failure ?? { code: 'UNKNOWN', message: error.message }
|
|
91
|
+
action = classifyFailure(failure, codeActions)
|
|
92
|
+
if action == 'retry' and retry < target.maxRetries:
|
|
93
|
+
await delay(schedule[retry]); continue
|
|
94
|
+
if action == 'fail':
|
|
95
|
+
do not record a breaker failure; throw the original failure
|
|
96
|
+
# action == 'fallback' or retries exhausted
|
|
97
|
+
breaker.recordFailure(key); break // next target
|
|
98
|
+
throw the last failure
|
|
99
|
+
```
|
|
100
|
+
|
|
101
|
+
### Streaming invariants
|
|
102
|
+
|
|
103
|
+
- Switch targets inside a request **only before yielding visible content**.
|
|
104
|
+
- Text/reasoning deltas are forwarded immediately.
|
|
105
|
+
- Tool-call block-start/delta chunks are buffered until `block-end`. A malformed tool call
|
|
106
|
+
(no `block-end`) can therefore be dropped and retried safely.
|
|
107
|
+
- After visible content has been delivered, rethrow the original failure and let the host's
|
|
108
|
+
step-level retry replay the step.
|
|
109
|
+
- Replay continuation uses `sessionId + request fingerprint` (message count + last
|
|
110
|
+
role/id). On replay, the plan starts at the next target and **rotates**:
|
|
111
|
+
`[...attempts.slice(next), ...attempts.slice(0, next)]`. The hint only changes order; it
|
|
112
|
+
never deletes a target. If the failed target is the chain tail, retry it in place.
|
|
113
|
+
|
|
114
|
+
## 6. Error classification
|
|
115
|
+
|
|
116
|
+
Default table (see `lib/policy.js`; override with `codeActions`):
|
|
117
|
+
|
|
118
|
+
| Error | Action | Why |
|
|
119
|
+
|:--|:--|:--|
|
|
120
|
+
| `SERVER` + membership/entitlement/quota/billing/credit/plan | fallback | deterministic account state |
|
|
121
|
+
| `SERVER` / `TIMEOUT` / `TRANSPORT` / `RATE_LIMIT` / `EMPTY_RESPONSE` / `STREAM_CLOSED` | retry | transient |
|
|
122
|
+
| `MALFORMED_RESPONSE` | retry | broken model output; retrying the request is safe |
|
|
123
|
+
| `AUTH` / `INVALID_CREDENTIAL` / `MISSING_CREDENTIAL` | fallback | another provider may work |
|
|
124
|
+
| `QUOTA` / `ACCOUNT_QUOTA` | fallback | switch provider/quota |
|
|
125
|
+
| `NO_ADAPTER` / `UNKNOWN_MODEL` / `MODEL_NOT_FOUND` | fallback | target unavailable |
|
|
126
|
+
| `ABORTED` / `CANCELLED` | fail | caller cancelled |
|
|
127
|
+
| `UNSUPPORTED_OPTION` / `INVALID_REQUEST` / `UNSUPPORTED_CONTENT` | fail | request itself is invalid |
|
|
128
|
+
| `IMAGE_OFFLOAD_REQUIRED` | fail (rethrow) | host image-offload recovery |
|
|
129
|
+
| `CONTEXT_WINDOW_EXCEEDED` | fail (rethrow) | host compaction recovery |
|
|
130
|
+
| unknown | fallback | the last target still throws the original failure |
|
|
131
|
+
|
|
132
|
+
`IMAGE_OFFLOAD_REQUIRED` and `CONTEXT_WINDOW_EXCEEDED` cannot be overridden: they are part
|
|
133
|
+
of DSH's request-level recovery protocol and carry metadata the host needs.
|
|
134
|
+
|
|
135
|
+
## 7. Backoff and global budget
|
|
136
|
+
|
|
137
|
+
```
|
|
138
|
+
delay(attempt) = min(initialDelayMs * multiplier^(attempt-1), maxDelayMs)
|
|
139
|
+
actualDelay = min(round(delay * (1 - jitter + 2*jitter*random)), maxDelayMs)
|
|
140
|
+
```
|
|
141
|
+
|
|
142
|
+
- `multiplier` defaults to 2 and accepts values such as 1.5.
|
|
143
|
+
- Each target has its own `maxRetries` / `initialDelayMs` / `multiplier` / `maxDelayMs` /
|
|
144
|
+
`jitterRatio`.
|
|
145
|
+
- `limits.maxTotalMs` (default 600000) is a deadline: each wait is clipped to the remaining
|
|
146
|
+
time, and the deadline is checked again after the wait.
|
|
147
|
+
- `limits.maxAttempts` (default 50) caps the number of upstream requests.
|
|
148
|
+
- Exhaustion throws `DEADLINE_EXCEEDED` / `ATTEMPT_LIMIT` with the attempt count, elapsed
|
|
149
|
+
time and last real failure.
|
|
150
|
+
|
|
151
|
+
Do not give entitlement/quota 500s a large retry budget: they should fall back on the first
|
|
152
|
+
failure.
|
|
153
|
+
|
|
154
|
+
## 8. Tool calls
|
|
155
|
+
|
|
156
|
+
Two different things must not be conflated:
|
|
157
|
+
|
|
158
|
+
1. **Broken model output** (`MALFORMED_RESPONSE`, invalid tool JSON, concatenated tool-call
|
|
159
|
+
arguments): this is an LLM request failure, not a tool execution failure. Because tool-call
|
|
160
|
+
chunks are buffered until `block-end`, the host has not seen anything when the stream is
|
|
161
|
+
malformed; the buffer can be dropped and the LLM request retried. Controlled by
|
|
162
|
+
`toolCalls.retryMalformed` (default `true`).
|
|
163
|
+
2. **Tool execution failure** (bash non-zero exit, file write failure, network failure): it
|
|
164
|
+
may have side effects and is not retried automatically. In DSH, a bash non-zero exit is a
|
|
165
|
+
successful tool result carrying `[exit code: N]`; the model decides what to do.
|
|
166
|
+
|
|
167
|
+
Cordis 4.1's `tools/execute` waterfall calls `next()` once; calling it again throws
|
|
168
|
+
`next() called multiple times`. The plugin therefore does not ship a `retryExecution` switch.
|
|
169
|
+
Automatically replaying a tool body requires a retry hook in the DSH core/executor.
|
|
170
|
+
|
|
171
|
+
## 9. Circuit breaker and state file
|
|
172
|
+
|
|
173
|
+
State machine per `provider + model`:
|
|
174
|
+
|
|
175
|
+
```
|
|
176
|
+
closed --(failureThreshold consecutive failures)--> open
|
|
177
|
+
open --(now >= lockedUntil)--> half-open
|
|
178
|
+
half-open --(beginProbe wins + probe succeeds)--> closed on next real success
|
|
179
|
+
half-open --(probe fails)--> open for another lockMs
|
|
180
|
+
```
|
|
181
|
+
|
|
182
|
+
- Open/probing targets are skipped; half-open targets require the single probe slot.
|
|
183
|
+
- `enabled: false` targets are filtered before validation and planning; they keep their
|
|
184
|
+
configuration but do not participate in the chain.
|
|
185
|
+
- The probe is a minimal chat request (`maxTokens: 1`, short prompt, `timeoutMs`). Do not
|
|
186
|
+
use `/v1/models` as a health check: a gateway can fail `/models` because of entitlement
|
|
187
|
+
while chat is fine, or vice versa.
|
|
188
|
+
- `breaker.probe.background: true` makes the current request skip a half-open target and
|
|
189
|
+
probes in the background; the next request uses the result.
|
|
190
|
+
- `stateFile` persistence uses `stateFile.lock` + read-merge-write (temp file + rename).
|
|
191
|
+
Merge rules: later lock / more consecutive failures / more lock count wins. A successful
|
|
192
|
+
clear writes a tombstone; older locks cannot be resurrected. Keys this process has never
|
|
193
|
+
seen are not deleted.
|
|
194
|
+
- When every target is open/probing, throw `ALL_TARGETS_LOCKED` with each `lockedUntil`.
|
|
195
|
+
|
|
196
|
+
## 10. Security invariants
|
|
197
|
+
|
|
198
|
+
- Never switch targets after visible content has been yielded.
|
|
199
|
+
- Never auto-retry side-effecting tool execution.
|
|
200
|
+
- A target's provider must not equal the wrapper route (no recursion); validated at load.
|
|
201
|
+
- Never write API keys or passwords to configuration or logs; `stateFile` stores only
|
|
202
|
+
counters and timestamps.
|
|
203
|
+
- Tests use a temporary `DSH_HOME` and temporary ports. Do not restart a live profile as
|
|
204
|
+
part of the test suite.
|
|
205
|
+
- Multi-process merge only adds/updates; only a timestamped tombstone can clear a lock.
|
|
206
|
+
- When all targets fail, throw the last real failure; never fabricate success or swallow
|
|
207
|
+
errors.
|
|
208
|
+
- Before falling back, check the target's capabilities (context window, modalities,
|
|
209
|
+
reasoning effort); `resolveModel` takes the minimum context window and the intersection of
|
|
210
|
+
modalities/effort ranges. Incompatible requests are not silently attempted.
|
|
211
|
+
|
|
212
|
+
## 11. Tests and verification
|
|
213
|
+
|
|
214
|
+
Unit tests (`node --test`, 90 cases) cover:
|
|
215
|
+
|
|
216
|
+
- nominal backoff (2x / 1.5x / cap), jitter bounds, per-target reset;
|
|
217
|
+
- classification: entitlement 500 → fallback, transient → retry, invalid → fail, overrides;
|
|
218
|
+
- chain validation: empty, duplicate, self-reference, invalid numbers;
|
|
219
|
+
- breaker threshold/lock/half-open/probe mutex/success recovery/lock extension/snapshot;
|
|
220
|
+
- adapter execution: first success does not touch the second target; transient retries use
|
|
221
|
+
the target's curve; exhausted retries move to the next target and reset backoff;
|
|
222
|
+
- fallback on entitlement 500 without spending the retry budget;
|
|
223
|
+
- failure after text delivery rethrows the original code and does not switch targets;
|
|
224
|
+
- tool-call buffering: malformed calls are retried, complete calls are not duplicated;
|
|
225
|
+
- step replay: same fingerprint starts at the next target, rotates to cover the whole chain,
|
|
226
|
+
and retries the tail in place; a different fingerprint discards the hint;
|
|
227
|
+
- breaker persistence across processes, including sub-threshold counts, tombstones and
|
|
228
|
+
concurrent writes;
|
|
229
|
+
- global deadline/attempt caps, background probing, runtime snapshots and session records;
|
|
230
|
+
- settings-client pure functions: field round-trips, unit switching (minutes ↔ ms), retry
|
|
231
|
+
schedule parity with the host, validation and catalogue checks;
|
|
232
|
+
- replay envelope wrap/unwrap, including the legacy `dsh-llm-fallback` kind.
|
|
233
|
+
|
|
234
|
+
End-to-end smoke (isolated `DSH_HOME`, temporary web profile, fake OpenAI-compatible server):
|
|
235
|
+
|
|
236
|
+
- entitlement 500 on target 1 → fallback to target 2, output normal;
|
|
237
|
+
- two transient 500s on target 3 → retry with 100/200 ms backoff, third attempt succeeds;
|
|
238
|
+
- all targets fail → `FALLBACK_EXHAUSTED` with the last real failure preserved;
|
|
239
|
+
- malformed tool call → buffer dropped and retried, host sees only the successful attempt;
|
|
240
|
+
- breaker threshold writes `stateFile`; a new process skips the locked target;
|
|
241
|
+
- lock expiry + failed probe → lock again;
|
|
242
|
+
- text delivered then transport failure → host replays the step, skips the failed target,
|
|
243
|
+
then succeeds;
|
|
244
|
+
- settings card renders in a temporary web profile; add/edit/save/reload round-trips.
|
|
245
|
+
|
|
246
|
+
## 12. Known limitations and open questions
|
|
247
|
+
|
|
248
|
+
1. **Wrapper vs core patch.** The wrapper is verified on an isolated DSH. Nested
|
|
249
|
+
`ctx.llm.stream()` passes through the `llm/stream` waterfall again; the session-title
|
|
250
|
+
WeakSet does not mistake the wrapped object for an agent-loop request, and
|
|
251
|
+
`session-checkpoint-policy` flushes once more (harmless). The core is not modified.
|
|
252
|
+
2. **Partial output.** Once visible content is delivered, the plugin relies on step replay
|
|
253
|
+
plus target continuation rather than buffering the whole stream. Buffering would remove
|
|
254
|
+
in-request fallback after the first token but delay the first token; it is not the default.
|
|
255
|
+
3. **Model catalogue.** The synthetic route exposes one model, `auto`. `resolveModel` takes
|
|
256
|
+
the minimum context window, modality intersection and reasoning effort intersection of
|
|
257
|
+
the chain. The settings page shows the real model for each target.
|
|
258
|
+
4. **All targets locked.** `ALL_TARGETS_LOCKED` is thrown; the plugin does not force one
|
|
259
|
+
last request into a locked provider. Whether to allow a "last resort even if locked"
|
|
260
|
+
target is an open question.
|
|
261
|
+
5. **Tool execution retry.** Blocked by cordis `next()` single-call semantics; requires a
|
|
262
|
+
core/executor hook.
|
|
263
|
+
6. **Replay envelope identity.** The envelope kind changed with the package rename in 0.5.0;
|
|
264
|
+
the unwrapper accepts the legacy kind so in-flight sessions from the private development
|
|
265
|
+
line keep working. Public users start on the new kind.
|
package/LICENSE
ADDED
|
@@ -0,0 +1,21 @@
|
|
|
1
|
+
MIT License
|
|
2
|
+
|
|
3
|
+
Copyright (c) 2026 T-splines
|
|
4
|
+
|
|
5
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
|
6
|
+
of this software and associated documentation files (the "Software"), to deal
|
|
7
|
+
in the Software without restriction, including without limitation the rights
|
|
8
|
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
9
|
+
copies of the Software, and to permit persons to whom the Software is
|
|
10
|
+
furnished to do so, subject to the following conditions:
|
|
11
|
+
|
|
12
|
+
The above copyright notice and this permission notice shall be included in all
|
|
13
|
+
copies or substantial portions of the Software.
|
|
14
|
+
|
|
15
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
16
|
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
17
|
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|
18
|
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
19
|
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
|
20
|
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
|
21
|
+
SOFTWARE.
|
package/README.md
CHANGED
|
@@ -1,3 +1,306 @@
|
|
|
1
|
-
#
|
|
1
|
+
# dsh-llm-resilience
|
|
2
2
|
|
|
3
|
-
|
|
3
|
+
**English** | [简体中文](https://github.com/T-splines/dsh-llm-resilience/blob/main/README.zh-CN.md)
|
|
4
|
+
|
|
5
|
+
**Make a flaky LLM gateway behave like a reliable one — per-target retries, streaming-safe fallback chains and persistent circuit breaking for DeepSeek Harness.**
|
|
6
|
+
|
|
7
|
+
This plugin was built from a concrete failure mode we hit in daily use. **OpenDesign Go**'s
|
|
8
|
+
tool-call stream and API connections were intermittently unstable: a malformed tool-call
|
|
9
|
+
stream, a transport drop, or a membership/quota error surfaced as a generic 500 could end
|
|
10
|
+
an entire agent step. DSH's built-in `retryPolicy` is provider-level — it can back off on
|
|
11
|
+
the same route, but it cannot switch provider/model, and it has no circuit breaker. In our
|
|
12
|
+
deployment this plugin moved OpenDesign Go from **barely usable** to **reliably usable**:
|
|
13
|
+
safe retries absorb transient faults, a second provider takes over when the gateway is out
|
|
14
|
+
of quota or broken, and a persistent breaker stops hammering a dead upstream.
|
|
15
|
+
|
|
16
|
+
> This is an independent plugin. It is not affiliated with or endorsed by OpenDesign or
|
|
17
|
+
> DeepSeek; OpenDesign Go is mentioned only as the deployment that motivated the design.
|
|
18
|
+
> The failure pattern is not unique to that service — any gateway that intermittently
|
|
19
|
+
> breaks tool-call streams or returns quota errors as generic 500s benefits from the same
|
|
20
|
+
> chain.
|
|
21
|
+
|
|
22
|
+
- [Why this exists](#why-this-exists)
|
|
23
|
+
- [What makes it different](#what-makes-it-different)
|
|
24
|
+
- [What it registers](#what-it-registers)
|
|
25
|
+
- [Install](#install)
|
|
26
|
+
- [Configuration](#configuration)
|
|
27
|
+
- [Error classification](#error-classification)
|
|
28
|
+
- [Streaming, tool calls and step replay](#streaming-tool-calls-and-step-replay)
|
|
29
|
+
- [Circuit breaker](#circuit-breaker)
|
|
30
|
+
- [Settings page](#settings-page)
|
|
31
|
+
- [Limits and honest non-goals](#limits-and-honest-non-goals)
|
|
32
|
+
- [Tests](#tests)
|
|
33
|
+
- [Compatibility and migration](#compatibility-and-migration)
|
|
34
|
+
|
|
35
|
+
## Why this exists
|
|
36
|
+
|
|
37
|
+
DSH's `retryPolicy` already retries at the provider level, but:
|
|
38
|
+
|
|
39
|
+
- the backoff multiplier is fixed at 2 — you cannot ask for 1.5;
|
|
40
|
+
- `agent/request-error` can only retry the **same route**; it cannot switch provider/model;
|
|
41
|
+
- there is no circuit breaker: when a provider is down, every step still hits it first;
|
|
42
|
+
- `MALFORMED_RESPONSE` (a broken tool-call stream) and a failed tool **execution** need
|
|
43
|
+
different treatment.
|
|
44
|
+
|
|
45
|
+
This plugin registers a synthetic provider route (default `fallback`, model `auto`) that
|
|
46
|
+
internally walks a chain of real provider/model targets:
|
|
47
|
+
|
|
48
|
+
```
|
|
49
|
+
fallback/auto
|
|
50
|
+
├─ target 0: primary-gateway/your-model independent retry budget + breaker
|
|
51
|
+
├─ target 1: backup-provider/your-model independent retry budget + breaker
|
|
52
|
+
└─ target 2: local/your-model independent retry budget + breaker
|
|
53
|
+
```
|
|
54
|
+
|
|
55
|
+
Every target has its own retry budget, backoff curve and breaker. Switching to the next
|
|
56
|
+
target resets the backoff instead of inheriting the previous target's already-doubled delay.
|
|
57
|
+
|
|
58
|
+
## What makes it different
|
|
59
|
+
|
|
60
|
+
- **Per-target retry budgets.** `maxRetries`, `initialDelayMs`, `multiplier` (including 1.5),
|
|
61
|
+
`maxDelayMs` and `jitterRatio` are independent for each target.
|
|
62
|
+
- **Error-aware classification.** Membership/entitlement/quota errors surfaced as `SERVER`
|
|
63
|
+
500 fall back immediately; ordinary 500/timeout/transport errors retry; invalid requests
|
|
64
|
+
fail without trying another target. `codeActions` lets you override the defaults.
|
|
65
|
+
- **Streaming-safe tool calls.** Text and reasoning deltas are forwarded immediately.
|
|
66
|
+
Tool-call block-start/delta chunks are buffered until `block-end`, so a malformed tool-call
|
|
67
|
+
stream (no `block-end`) can be retried without the host ever seeing duplicate output.
|
|
68
|
+
- **Step-replay continuation.** After content has been delivered, a failure cannot switch
|
|
69
|
+
targets inside the same request. The plugin remembers `sessionId + request fingerprint`;
|
|
70
|
+
when the host replays the step it starts at the next target, and the plan **rotates** to
|
|
71
|
+
cover the whole chain instead of truncating it. If the failure was the chain tail, it is
|
|
72
|
+
retried in place — a retryable failure is never rewritten into a terminal
|
|
73
|
+
`FALLBACK_EXHAUSTED`.
|
|
74
|
+
- **Persistent circuit breaker, per provider + model.** Consecutive failures lock a target
|
|
75
|
+
for a configurable window; the lock expires into a half-open probe (foreground or
|
|
76
|
+
background). State is written atomically to `stateFile` and merged across processes with
|
|
77
|
+
a lock + tombstones, so a restart does not forget a partially accumulated failure count.
|
|
78
|
+
- **Global budget.** `limits.maxTotalMs` (default 10 minutes) and `limits.maxAttempts`
|
|
79
|
+
(default 50) are enforced across targets and retries; backoff waits count toward the
|
|
80
|
+
deadline. No combination of targets × retries can hang a step for an unbounded time.
|
|
81
|
+
- **Request-level recovery codes pass through.** `IMAGE_OFFLOAD_REQUIRED` (with
|
|
82
|
+
`offloadImages`) and `CONTEXT_WINDOW_EXCEEDED` are re-thrown unchanged so the host's image
|
|
83
|
+
offload / compaction layers can recover first.
|
|
84
|
+
- **Settings page.** Edit the chain, reorder targets, see each retry schedule, and watch live
|
|
85
|
+
breaker state (`closed / open / half-open / probing`) next to version and uninstall.
|
|
86
|
+
- **Tested.** 90 `node --test` cases plus isolated DSH end-to-end smoke runs with a fake
|
|
87
|
+
OpenAI-compatible server (see [DESIGN.md](DESIGN.md)).
|
|
88
|
+
|
|
89
|
+
## What it registers
|
|
90
|
+
|
|
91
|
+
| entry | value |
|
|
92
|
+
|:--|:--|
|
|
93
|
+
| synthetic route | `fallback` (configurable) |
|
|
94
|
+
| synthetic model | `auto` |
|
|
95
|
+
| profile entry id | `llm-fallback` (kept stable for settings compatibility) |
|
|
96
|
+
| package name | `dsh-llm-resilience` |
|
|
97
|
+
| default state file | `$DSH_HOME/storages/llm-resilience/state.json` |
|
|
98
|
+
|
|
99
|
+
`targets: []` (or every target `enabled: false`) leaves the route registered but dormant:
|
|
100
|
+
you can build the chain in the settings page first, then enable it.
|
|
101
|
+
|
|
102
|
+
## Install
|
|
103
|
+
|
|
104
|
+
Requirements: DSH `0.2.2-rc.x`, Node.js 20+.
|
|
105
|
+
|
|
106
|
+
### From npm
|
|
107
|
+
|
|
108
|
+
```bash
|
|
109
|
+
dsh plugin --profile web add @magic6/dsh-llm-resilience
|
|
110
|
+
```
|
|
111
|
+
|
|
112
|
+
### From source
|
|
113
|
+
|
|
114
|
+
```bash
|
|
115
|
+
git clone https://github.com/T-splines/dsh-llm-resilience.git
|
|
116
|
+
cd dsh-llm-resilience
|
|
117
|
+
dsh plugin --profile web add "$PWD"
|
|
118
|
+
```
|
|
119
|
+
|
|
120
|
+
The package is a bundle (`dsh.bundle.patch`), so the `llm-fallback` entry is already mounted
|
|
121
|
+
by the bundle. **Override that entry** — do not `insert` a second one. In
|
|
122
|
+
`$DSH_HOME/profiles/<profile>/cordis.patch.yml`, replacing the whole `config`:
|
|
123
|
+
|
|
124
|
+
```yaml
|
|
125
|
+
- id: llm-fallback
|
|
126
|
+
name: 'dsh-llm-resilience'
|
|
127
|
+
config:
|
|
128
|
+
enabled: true
|
|
129
|
+
route: fallback
|
|
130
|
+
targets:
|
|
131
|
+
- enabled: true
|
|
132
|
+
provider: primary-gateway
|
|
133
|
+
model: your-model
|
|
134
|
+
maxRetries: 5
|
|
135
|
+
initialDelayMs: 1500
|
|
136
|
+
multiplier: 2
|
|
137
|
+
maxDelayMs: 20000
|
|
138
|
+
jitterRatio: 0.3
|
|
139
|
+
- enabled: true
|
|
140
|
+
provider: backup-provider
|
|
141
|
+
model: your-backup-model
|
|
142
|
+
maxRetries: 3
|
|
143
|
+
initialDelayMs: 1000
|
|
144
|
+
breaker:
|
|
145
|
+
failureThreshold: 5
|
|
146
|
+
lockMs: 3600000
|
|
147
|
+
probe:
|
|
148
|
+
enabled: true
|
|
149
|
+
background: false
|
|
150
|
+
timeoutMs: 10000
|
|
151
|
+
maxTokens: 1
|
|
152
|
+
prompt: 'ping'
|
|
153
|
+
stateFile: '$DSH_HOME/storages/llm-resilience/state.json'
|
|
154
|
+
limits:
|
|
155
|
+
maxTotalMs: 600000
|
|
156
|
+
maxAttempts: 50
|
|
157
|
+
toolCalls:
|
|
158
|
+
retryMalformed: true
|
|
159
|
+
```
|
|
160
|
+
|
|
161
|
+
Replace `primary-gateway` / `backup-provider` with real provider ids from your DSH model
|
|
162
|
+
catalog (the settings page shows them as read-only dropdowns). A target whose provider does
|
|
163
|
+
not exist fails with `NO_ADAPTER` and the chain moves on; it does not silently pretend the
|
|
164
|
+
target served.
|
|
165
|
+
|
|
166
|
+
Then point the agent's default model at the synthetic route:
|
|
167
|
+
|
|
168
|
+
```yaml
|
|
169
|
+
- id: agent-default-model
|
|
170
|
+
config:
|
|
171
|
+
provider: fallback
|
|
172
|
+
model: auto
|
|
173
|
+
```
|
|
174
|
+
|
|
175
|
+
Verify composition without starting an agent (`web` is the common profile name; use yours):
|
|
176
|
+
|
|
177
|
+
```bash
|
|
178
|
+
dsh --profile web --dump-config | grep -A8 'id: llm-fallback'
|
|
179
|
+
```
|
|
180
|
+
|
|
181
|
+
> When overriding an existing `llm-fallback` entry with `--patch`, write the **whole**
|
|
182
|
+
> `targets` array. DSH's config merge fills omitted array fields with an empty default, so
|
|
183
|
+
> overriding only `breaker` would clear the chain.
|
|
184
|
+
|
|
185
|
+
**Do not give entitlement/quota errors 10 retries.** They are deterministic provider state;
|
|
186
|
+
retrying the same target only wastes the budget. They should fall back on the first failure.
|
|
187
|
+
|
|
188
|
+
## Configuration
|
|
189
|
+
|
|
190
|
+
| field | meaning |
|
|
191
|
+
|:--|:--|
|
|
192
|
+
| `enabled` | master switch; the route still exists when disabled, but rejects requests |
|
|
193
|
+
| `route` | synthetic provider name (default `fallback`); must not appear in `targets` |
|
|
194
|
+
| `targets[]` | ordered `{ provider, model, enabled, label, maxRetries, initialDelayMs, multiplier, maxDelayMs, jitterRatio }` |
|
|
195
|
+
| `breaker` | `failureThreshold`, `lockMs`, probe settings (`enabled`, `background`, `timeoutMs`, `maxTokens`, `prompt`), `stateFile` |
|
|
196
|
+
| `limits` | `maxTotalMs`, `maxAttempts`; `0` disables that dimension |
|
|
197
|
+
| `toolCalls.retryMalformed` | retry a broken model-generated tool call (default `true`); tool **execution** retries are not supported |
|
|
198
|
+
| `codeActions` | override the default action per stable error code (`retry` / `fallback` / `fail`) |
|
|
199
|
+
|
|
200
|
+
## Error classification
|
|
201
|
+
|
|
202
|
+
| error | action | why |
|
|
203
|
+
|:--|:--|:--|
|
|
204
|
+
| `SERVER` + membership/entitlement/quota/billing/credit/plan | fallback | deterministic account state; retrying is useless |
|
|
205
|
+
| `SERVER` / `TIMEOUT` / `TRANSPORT` / `RATE_LIMIT` / `EMPTY_RESPONSE` / `STREAM_CLOSED` | retry | transient |
|
|
206
|
+
| `MALFORMED_RESPONSE` | retry | broken model output; safe to retry the whole LLM request |
|
|
207
|
+
| `AUTH` / `INVALID_CREDENTIAL` / `MISSING_CREDENTIAL` | fallback | another provider may work |
|
|
208
|
+
| `QUOTA` / `ACCOUNT_QUOTA` | fallback | switch provider/quota |
|
|
209
|
+
| `NO_ADAPTER` / `UNKNOWN_MODEL` / `MODEL_NOT_FOUND` | fallback | target unavailable |
|
|
210
|
+
| `ABORTED` / `CANCELLED` | fail | the caller cancelled |
|
|
211
|
+
| `UNSUPPORTED_OPTION` / `INVALID_REQUEST` / `UNSUPPORTED_CONTENT` | fail | the request itself is wrong; another target will not fix it |
|
|
212
|
+
| `IMAGE_OFFLOAD_REQUIRED` | fail (re-thrown) | host image-offload recovery must see it |
|
|
213
|
+
| `CONTEXT_WINDOW_EXCEEDED` | fail (re-thrown) | host compaction recovery must see it |
|
|
214
|
+
| unknown code | fallback | the last target still throws the original failure; nothing is swallowed |
|
|
215
|
+
|
|
216
|
+
`IMAGE_OFFLOAD_REQUIRED` and `CONTEXT_WINDOW_EXCEEDED` cannot be overridden: they are part
|
|
217
|
+
of DSH's request-level recovery protocol, and the host's listeners depend on the original
|
|
218
|
+
code and metadata.
|
|
219
|
+
|
|
220
|
+
## Streaming, tool calls and step replay
|
|
221
|
+
|
|
222
|
+
- Only switch targets **before** yielding anything visible to the caller.
|
|
223
|
+
- Text/reasoning deltas are forwarded immediately.
|
|
224
|
+
- Tool-call block-start/delta chunks are buffered until `block-end`; a malformed tool call
|
|
225
|
+
(no `block-end`) can be dropped and retried safely.
|
|
226
|
+
- After visible content has been delivered, a failure is returned to the host so
|
|
227
|
+
`dsh-llm-retry` can replay the whole step. The plugin records a fingerprint and the
|
|
228
|
+
target it reached; on replay it starts at the next target. The plan **rotates** rather
|
|
229
|
+
than truncating, so a target before the failure point is not lost. If the failure was the
|
|
230
|
+
chain tail, it retries that target in place.
|
|
231
|
+
- The replay envelope written by the synthetic route is unwrapped into the real target's
|
|
232
|
+
provider/model before the request reaches the upstream adapter, preserving signatures and
|
|
233
|
+
ids. Envelopes written by the pre-rename `dsh-llm-fallback` package are still accepted.
|
|
234
|
+
|
|
235
|
+
## Circuit breaker
|
|
236
|
+
|
|
237
|
+
```
|
|
238
|
+
closed --(failureThreshold consecutive failures)--> open
|
|
239
|
+
open --(lock expires)--> half-open
|
|
240
|
+
half-open --(probe succeeds)--> closed on the next real success
|
|
241
|
+
half-open --(probe fails)--> open for another lock window
|
|
242
|
+
```
|
|
243
|
+
|
|
244
|
+
- State is keyed by `provider + model`, not by provider alone.
|
|
245
|
+
- `stateFile` is written atomically with a lock + read-merge-write. Concurrent processes
|
|
246
|
+
keep the later/more-failed entry; a successful clear writes a tombstone so an older lock
|
|
247
|
+
cannot be resurrected.
|
|
248
|
+
- Unreached failure counts are persisted too (`updatedAt` is the merge timeline), so a
|
|
249
|
+
restart does not reset the count below the threshold.
|
|
250
|
+
- When every target is open/probing, the plugin throws `ALL_TARGETS_LOCKED` with each
|
|
251
|
+
target's `lockedUntil` — it does not force one last request into a locked provider.
|
|
252
|
+
|
|
253
|
+
## Settings page
|
|
254
|
+
|
|
255
|
+
Plugins → Installed → `dsh-llm-resilience` → **Model fallback chain**:
|
|
256
|
+
|
|
257
|
+
- per-target rows: enabled, provider, model (read-only dropdowns from the session model
|
|
258
|
+
catalog), label, retry parameters, move up/down, delete, add;
|
|
259
|
+
- retry-plan tooltip: one line per retry, with the jitter range when `jitter > 0`;
|
|
260
|
+
- breaker and budget fields;
|
|
261
|
+
- live runtime state polled from the host (`closed / open / half-open / probing`, remaining
|
|
262
|
+
lock time, consecutive failures and lock count);
|
|
263
|
+
- non-loopback browsers get the host's memory-state view and the card says so instead of
|
|
264
|
+
rendering plugin defaults as if they were the deployed policy.
|
|
265
|
+
|
|
266
|
+
## Limits and honest non-goals
|
|
267
|
+
|
|
268
|
+
- **Tool execution is not retried.** Cordis 4.1's `tools/execute` waterfall calls `next()`
|
|
269
|
+
once; calling it again throws `next() called multiple times`. Automatically replaying a
|
|
270
|
+
tool body needs a retry hook in the DSH core/executor. The plugin does not ship a switch
|
|
271
|
+
that silently does nothing. Retrying a malformed **model-generated** tool call is safe and
|
|
272
|
+
supported.
|
|
273
|
+
- **No infinite retries.** The global deadline and attempt cap are required; `0` disables a
|
|
274
|
+
dimension but is not recommended for production.
|
|
275
|
+
- **Does not replace `@deepseek-ai/dsh-llm-retry`.** That package still owns step-level
|
|
276
|
+
retries; this plugin owns provider selection and request-level resilience.
|
|
277
|
+
- **No streaming buffer for late fallback.** Once visible content is delivered, switching
|
|
278
|
+
targets would duplicate output; the plugin hands the step back to the host instead.
|
|
279
|
+
|
|
280
|
+
## Tests
|
|
281
|
+
|
|
282
|
+
```bash
|
|
283
|
+
npm test
|
|
284
|
+
```
|
|
285
|
+
|
|
286
|
+
90 cases cover backoff/jitter, error classification, chain validation, adapter execution,
|
|
287
|
+
streaming safety, step-replay rotation, breaker persistence and multi-process merging,
|
|
288
|
+
global budgets, background probing, settings-client pure functions and the replay envelope
|
|
289
|
+
(including the legacy package name). End-to-end smoke runs on an isolated `DSH_HOME` with a
|
|
290
|
+
fake OpenAI-compatible server are described in [DESIGN.md](DESIGN.md).
|
|
291
|
+
|
|
292
|
+
## Compatibility and migration
|
|
293
|
+
|
|
294
|
+
- DSH `0.2.2-rc.x` (`engines.dsh` is `>=0.2.2-0 <0.3.0-0`), Node.js 20+.
|
|
295
|
+
- **Renamed from `dsh-llm-fallback` in 0.5.0.** The profile entry id remains `llm-fallback`,
|
|
296
|
+
so settings and existing patches keep working; update the `name:` field to
|
|
297
|
+
`dsh-llm-resilience`.
|
|
298
|
+
- The default breaker state file moved to
|
|
299
|
+
`$DSH_HOME/storages/llm-resilience/state.json`. Set `stateFile` explicitly if you want to
|
|
300
|
+
keep using an older breaker file.
|
|
301
|
+
- Replay envelopes from the old package name still unwrap (`dsh-llm-fallback` is accepted as
|
|
302
|
+
a legacy kind).
|
|
303
|
+
|
|
304
|
+
## License
|
|
305
|
+
|
|
306
|
+
MIT. See [LICENSE](LICENSE).
|