dsh-llm-fallbacks 0.3.4 → 0.3.5
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +3 -1
- package/README.zh-CN.md +3 -1
- package/dist/client/index.js +6 -2
- package/dist/commands.d.ts +4 -0
- package/dist/config.d.ts +14 -0
- package/dist/cooldown.d.ts +9 -0
- package/dist/index.d.ts +1 -1
- package/dist/index.js +129 -11
- package/dist/recovery.d.ts +92 -0
- package/dist/state.d.ts +27 -2
- package/package.json +1 -1
package/README.md
CHANGED
|
@@ -2,6 +2,7 @@
|
|
|
2
2
|
|
|
3
3
|
[English](README.md) | [中文](README.zh-CN.md)
|
|
4
4
|
|
|
5
|
+
[](https://www.npmjs.com/package/dsh-llm-fallbacks)
|
|
5
6
|
[](LICENSE)
|
|
6
7
|

|
|
7
8
|

|
|
@@ -130,6 +131,7 @@ Save the config and restart the session, then type `/fallbacks` — the read-onl
|
|
|
130
131
|
- **Time slots**: optional `fallbacks.timeSlots` rows rotate the effective root chain by wall-clock windows in the config-level `tz` timezone (default `Asia/Shanghai`) — four frozen UTC+8 presets (`liang-peak` / `liang-valley` / `glm-peak` / `glm-valley`, windows are code constants, models-only edits) or custom `start`/`end`/`days` windows. The first matching row wins; the all-day row is always last. A slot change applies on the **next** root request and is logged as a **time-slot switch** — a routing seed, never a failure decision: it consumes no cooldown and does not count against `maxSwitchesPerStep`. Failure walks keep the **fallback switch** copy (see [Time-slot presets](#time-slot-presets)).
|
|
131
132
|
- **Dispatch-time role resolution**: on a subagent's first request its role is resolved in three stages — explicit (`agentPreset` matches a declared role id) → deterministic rules (unchanged) → LLM auto-match from the declared role taxonomy (`fallbacks.roleAutoMatch`, default `true`). The resolved role's chain-head model is injected into the first request and recorded via an explicit `role → model` log line (no durable `fallbacks/switch` event is written — issue #52 stop-write); set `roleAutoMatch: false` to disable the LLM auto-match stage (the explicit `agentPreset` stage still applies — with no explicit role this reproduces the previous rules-only behavior). The settings card always renders an **Enable role auto-match** switch (default `true`) to toggle it — the schema default applies even to legacy configs that never declared the key.
|
|
132
133
|
- **Cooldown and revert**: failed / switched-away models are not re-selected during cooldown; `revertPolicy: cooldown-expiry` returns to the primary model automatically.
|
|
134
|
+
- **Half-open recovery (opt-in)**: `recovery: half-open` makes recovery evidence-driven — an expired cooldown leaves the route half-open for one logged probe instead of restoring the preference; consecutive failures escalate the suppression duration (×2 per failure, capped at 1 h); an observed completion closes the circuit and fully restores the preference. `revertPolicy: 'never'` keeps the mechanism inert; state is session-scoped in-memory (a restart resets). YAML-only — the default `timer` keeps every existing behavior byte-identical (see [docs/configuration.md](docs/configuration.md#recovery-mode-recovery-key)).
|
|
133
135
|
- **Visible behavior**: every switch is recorded in an info-level log line (from/to/role/reason) — no silent model switching. The plugin deliberately writes **no** durable `fallbacks/switch` session events (issue #52: the apply()-time event-type registration was proven ineffective, and a session containing the event refused to load after a dsh restart). Sessions written by older plugin versions that contain such events are repaired by `scripts/repair-fallbacks-switch-logs.ts`, which marks legacy events ignorable so affected sessions load again.
|
|
134
136
|
- **Safety valves**: `maxSwitchesPerStep` caps switches per step and `alwaysModeRetryCap` caps always-mode retries — chain loops cannot amplify latency.
|
|
135
137
|
- **No-config no-op**: with no chains configured the plugin behaves exactly like not being installed (`enabled` is off by default — see [Minimal configuration](#minimal-configuration)).
|
|
@@ -138,7 +140,7 @@ Save the config and restart the session, then type `/fallbacks` — the read-onl
|
|
|
138
140
|
|
|
139
141
|
In a dsh-tui profile the plugin has three operator surfaces, with a strict duty split:
|
|
140
142
|
|
|
141
|
-
- **`/fallbacks`** — what happened this session: origin, resolved role, effective chain, recent fallback switches, cooldown status. Read-only.
|
|
143
|
+
- **`/fallbacks`** — what happened this session: origin, resolved role, effective chain, recent fallback switches, cooldown status (half-open marker rows when `recovery: half-open` is active). Read-only.
|
|
142
144
|
- **`/fallbacks config`** — what is configured: composed-config readback (trigger codes, root chain, time slots, timezone, roles, role rules, cooldown, revert policy, safety valves, presets, role auto-match). Read-only apart from the one action command **`/fallbacks config revert-seed <role-id>`**, which restores a seeded role's persona to its declared seed default (a web-card action the settings seam cannot express).
|
|
143
145
|
- **`/settings`** — the edit surface. The plugin registers a **fallbacks** section with full parity to the web settings card: booleans (`enabled`, `roleAutoMatch`) render as toggles, selects (`presets`, `revertPolicy`) as pickers, and numbers (`cooldownMs`, `maxSwitchesPerStep`, `alwaysModeRetryCap`) as numeric inputs; complex structures (`rootChain`, `timeSlots`, `roles.list`, `roles.rules`) are JSON text fields and `triggerCodes` a comma-separated text field. Invalid drafts (bad JSON, non-conforming chains, malformed time-slot rows) block the save — the section never corrupts the config.
|
|
144
146
|
|
package/README.zh-CN.md
CHANGED
|
@@ -2,6 +2,7 @@
|
|
|
2
2
|
|
|
3
3
|
[English](README.md) | [中文](README.zh-CN.md)
|
|
4
4
|
|
|
5
|
+
[](https://www.npmjs.com/package/dsh-llm-fallbacks)
|
|
5
6
|
[](LICENSE)
|
|
6
7
|

|
|
7
8
|

|
|
@@ -130,6 +131,7 @@ fallbacks:
|
|
|
130
131
|
- **峰谷无忧(分时切换)**:可选的 `fallbacks.timeSlots` 行按墙钟窗口(配置级 `tz` 时区,默认 `Asia/Shanghai`)轮换 root 生效链——四个冻结的 UTC+8 预设(`liang-peak` / `liang-valley` / `glm-peak` / `glm-valley`,窗口为代码常量、仅模型链可编辑),或自定义 `start`/`end`/`days` 窗口。第一条命中的行生效;全时段行固定最后。时段切换在**下一个** root 请求生效,日志记为**分时切换**——路由种子而非失败决策:不消耗冷却、不计入 `maxSwitchesPerStep`。失败降级保留**降级切换**文案(见 [分时槽预设(分时切换)](#分时槽预设分时切换))。
|
|
131
132
|
- **派发时角色解析**:在 subagent 的首次请求上,其角色按三个阶段解析——显式(`agentPreset` 匹配已声明角色 id)→ 确定性规则(不变)→ LLM 自动匹配(从已声明角色体系中选择,`fallbacks.roleAutoMatch` 默认 `true`)。解析出的角色的链头模型注入首次请求,并以显式 `role → model` 日志行记录(不写 durable `fallbacks/switch` 事件——issue #52 停写);设 `roleAutoMatch: false` 仅关闭 LLM 自动匹配阶段(显式 `agentPreset` 阶段仍生效——无显式角色时即复现原有仅规则行为)。设置卡总是渲染「启用角色自动匹配」开关(默认 `true`)以切换之——即使是从未声明过该键的旧配置,schema 默认值同样生效。
|
|
132
133
|
- **冷却与回主**:被切离/失败的模型在冷却期内不再入选;`revertPolicy: cooldown-expiry` 冷却到期后自动回主模型。
|
|
134
|
+
- **半开恢复(可选)**:`recovery: half-open` 让恢复以证据驱动——冷却到期后路由进入 **half-open**,以一次记录探针(logged probe)放行,而不是直接恢复首选;连续失败使抑制时长按 **×2** 逐次升级、**1 小时**封顶;观察到完成即闭合回路、完全恢复首选。`revertPolicy: 'never'` 使该机制完全失效;状态为会话级内存态(重启即重置)。仅 YAML 配置——默认 `timer` 保持所有既有行为逐字节一致(见 [docs/configuration.md](docs/configuration.md#recovery-mode-recovery-key))。
|
|
133
135
|
- **行为可见**:每次切换以 info 级日志行(from/to/role/reason)记录——无静默换模型。插件**刻意不写** durable `fallbacks/switch` 会话事件(issue #52——apply() 时的事件类型注册被证伪无效,含该事件的会话在 dsh 重启后拒绝加载)。由旧版插件写入、含此类事件的会话由 `scripts/repair-fallbacks-switch-logs.ts` 修复——旧事件被标记 ignorable 后,受影响会话可重新加载。
|
|
134
136
|
- **安全阀**:`maxSwitchesPerStep` 限制每 step 切换次数、`alwaysModeRetryCap` 限制 always 模式重试——链循环不会放大延迟。
|
|
135
137
|
- **无配置回归(no-op)**:未配置任何链时行为与未安装插件完全一致——`enabled` 默认关闭(见 [最小配置](#最小配置))。
|
|
@@ -138,7 +140,7 @@ fallbacks:
|
|
|
138
140
|
|
|
139
141
|
在 dsh-tui profile 中,插件有三个操作面——职责严格区分:
|
|
140
142
|
|
|
141
|
-
- **`/fallbacks`** ——
|
|
143
|
+
- **`/fallbacks`** —— 本次会话发生了什么:来源、解析角色、生效链、最近降级切换、冷却状态(`recovery: half-open` 生效时显示 half-open 标记行)。只读。
|
|
142
144
|
- **`/fallbacks config`** —— 配置了什么:组合配置回读(触发码、根链、分时槽、时区、角色、角色规则、冷却、回主策略、安全阀、预置、角色自动匹配)。除唯一的动作命令 **`/fallbacks config revert-seed <role-id>`** 外只读——该命令把某个 seed 角色的 persona 还原为已声明的默认(设置 seam 无法表达 Web 卡的这类动作能力)。
|
|
143
145
|
- **`/settings`** —— 编辑界面。插件注册 **fallbacks** 区块,与 **Web 设置卡完全一致**:布尔(`enabled`、`roleAutoMatch`)渲染为开关、下拉(`presets`、`revertPolicy`)为选择器、数值(`cooldownMs`、`maxSwitchesPerStep`、`alwaysModeRetryCap`)为数字输入;复杂结构(`rootChain`、`timeSlots`、`roles.list`、`roles.rules`)为 JSON 文本字段,`triggerCodes` 为逗号分隔文本字段。非法草稿(JSON 解析失败、链尾不合规、分时行畸形)会阻止保存——区块绝不写入损坏配置。
|
|
144
146
|
|
package/dist/client/index.js
CHANGED
|
@@ -486,7 +486,8 @@ window.__ModuleLoader__.load({
|
|
|
486
486
|
presets: "bundled",
|
|
487
487
|
roleAutoMatch: true,
|
|
488
488
|
timeSlots: [],
|
|
489
|
-
tz: "Asia/Shanghai"
|
|
489
|
+
tz: "Asia/Shanghai",
|
|
490
|
+
recovery: "timer"
|
|
490
491
|
};
|
|
491
492
|
/**
|
|
492
493
|
* Reserved role id: legal as a rule target (`roles.rules[].role`) and as
|
|
@@ -643,6 +644,8 @@ window.__ModuleLoader__.load({
|
|
|
643
644
|
}))) throw new TypeError("fallbacks descriptor timeSlots must be an array of slot rows (kind/preset/start/end strings, chain string array, days number array)");
|
|
644
645
|
const tz = value.tz;
|
|
645
646
|
if (tz !== void 0 && typeof tz !== "string") throw new TypeError("fallbacks descriptor tz must be a string");
|
|
647
|
+
const recovery = value.recovery;
|
|
648
|
+
if (recovery !== void 0 && recovery !== "timer" && recovery !== "half-open") throw new TypeError("fallbacks descriptor recovery must be timer|half-open");
|
|
646
649
|
return {
|
|
647
650
|
enabled: enabled ?? defaultFallbacksConfig.enabled,
|
|
648
651
|
triggerCodes: triggerCodes ?? [...defaultFallbacksConfig.triggerCodes],
|
|
@@ -658,7 +661,8 @@ window.__ModuleLoader__.load({
|
|
|
658
661
|
presets: presets ?? defaultFallbacksConfig.presets,
|
|
659
662
|
roleAutoMatch: roleAutoMatch ?? defaultFallbacksConfig.roleAutoMatch,
|
|
660
663
|
timeSlots: timeSlots ?? [...defaultFallbacksConfig.timeSlots ?? []],
|
|
661
|
-
tz: tz ?? defaultFallbacksConfig.tz
|
|
664
|
+
tz: tz ?? defaultFallbacksConfig.tz,
|
|
665
|
+
recovery: recovery ?? defaultFallbacksConfig.recovery
|
|
662
666
|
};
|
|
663
667
|
}
|
|
664
668
|
/** The raw selector string a selection serializes to ('' when empty). */
|
package/dist/commands.d.ts
CHANGED
|
@@ -57,6 +57,8 @@ export interface FallbacksCooldownEntry {
|
|
|
57
57
|
readonly key: string;
|
|
58
58
|
/** Expiry epoch ms; `Infinity` for `revertPolicy: 'never'`. */
|
|
59
59
|
readonly untilEpochMs: number;
|
|
60
|
+
/** True for a half-open marker row (an expired cooldown awaiting a recovery probe). */
|
|
61
|
+
readonly halfOpen?: boolean;
|
|
60
62
|
}
|
|
61
63
|
/** The read-only diagnostic snapshot the `/fallbacks` handler renders. */
|
|
62
64
|
export interface FallbacksCommandSnapshot {
|
|
@@ -192,6 +194,7 @@ export declare const FALLBACKS_COMMAND_LOCALES: {
|
|
|
192
194
|
readonly cooldownNone: "无活跃冷却";
|
|
193
195
|
readonly cooldownLine: "{key} 冷却至 {time}";
|
|
194
196
|
readonly cooldownNever: "{key} 会话内不再回主";
|
|
197
|
+
readonly cooldownHalfOpen: "{key} half-open(等待恢复探针)";
|
|
195
198
|
readonly reason: {
|
|
196
199
|
readonly 'trigger-code': "触发码";
|
|
197
200
|
readonly 'always-cap': "always 上限";
|
|
@@ -246,6 +249,7 @@ export declare const FALLBACKS_COMMAND_LOCALES: {
|
|
|
246
249
|
readonly cooldownNone: "none active";
|
|
247
250
|
readonly cooldownLine: "{key} suppressed until {time}";
|
|
248
251
|
readonly cooldownNever: "{key} not reverting this session";
|
|
252
|
+
readonly cooldownHalfOpen: "{key} half-open (awaiting recovery probe)";
|
|
249
253
|
readonly reason: {
|
|
250
254
|
readonly 'trigger-code': "trigger-code";
|
|
251
255
|
readonly 'always-cap': "always-cap";
|
package/dist/config.d.ts
CHANGED
|
@@ -28,6 +28,8 @@
|
|
|
28
28
|
* @module dsh-llm-fallbacks/config
|
|
29
29
|
*/
|
|
30
30
|
import type { SlotRowConfig } from './time-slots.ts';
|
|
31
|
+
/** How an expired cooldown recovers the route (plan fallbacks-half-open-recovery Task 1). */
|
|
32
|
+
export type RecoveryPolicy = 'timer' | 'half-open';
|
|
31
33
|
/** How a cooled-down model comes back (spec §4). */
|
|
32
34
|
export type RevertPolicy = 'cooldown-expiry' | 'never';
|
|
33
35
|
/** A single role rule: match on provider/model patterns (spec §3). */
|
|
@@ -125,6 +127,18 @@ export interface FallbacksConfig {
|
|
|
125
127
|
* UTC+8). Not per-slot.
|
|
126
128
|
*/
|
|
127
129
|
tz?: string;
|
|
130
|
+
/**
|
|
131
|
+
* Cooldown-expiry recovery mode (plan fallbacks-half-open-recovery Task
|
|
132
|
+
* 1): `'timer'` restores the preferred candidate when the cooldown
|
|
133
|
+
* expires (today's behavior); `'half-open'` leaves the route half-open
|
|
134
|
+
* for one logged probe instead. Optional on purpose, mirroring
|
|
135
|
+
* `presets` — a required field would break library consumers that
|
|
136
|
+
* construct `FallbacksConfig` literals with the existing keys
|
|
137
|
+
* (additive, non-breaking). The value domain is guarded by the schema
|
|
138
|
+
* (`Config` in `src/schema.ts`), NOT by `validateFallbacksConfig`, and
|
|
139
|
+
* every resolved config carries a value via the schema default.
|
|
140
|
+
*/
|
|
141
|
+
recovery?: RecoveryPolicy;
|
|
128
142
|
}
|
|
129
143
|
/**
|
|
130
144
|
* Spec §4 defaults — `Config({})` must equal this (no-op install).
|
package/dist/cooldown.d.ts
CHANGED
|
@@ -24,6 +24,15 @@ export declare class CooldownStore {
|
|
|
24
24
|
suppress(key: string, untilEpochMs: number): void;
|
|
25
25
|
/** True while `key` is suppressed at time `now` (default `Date.now()`); expired entries are removed. */
|
|
26
26
|
isSuppressed(key: string, now?: number): boolean;
|
|
27
|
+
/**
|
|
28
|
+
* Raw expiry read — no lazy drop (plan fallbacks-half-open-recovery P2
|
|
29
|
+
* rule 3). `undefined` for unknown keys; `Infinity` for
|
|
30
|
+
* `revertPolicy: 'never'` entries. The half-open read path uses this to
|
|
31
|
+
* observe the lapsed `until` before the decision-path drop.
|
|
32
|
+
*/
|
|
33
|
+
peek(key: string): number | undefined;
|
|
34
|
+
/** Unfiltered map keys, in insertion order (plan P4 display sync). */
|
|
35
|
+
keys(): IterableIterator<string>;
|
|
27
36
|
/**
|
|
28
37
|
* Read-only diagnostic accessor: the active (non-expired) entries, in
|
|
29
38
|
* insertion order. This is a pure read — expired entries are FILTERED OUT
|
package/dist/index.d.ts
CHANGED
|
@@ -85,7 +85,7 @@ export type { FallbackSwitchReason, FallbacksSwitchEventData } from './events.ts
|
|
|
85
85
|
export type { AgentFallbackState, FallbackStateStore, PendingSwitch, StepFailures } from './state.ts';
|
|
86
86
|
export { resolveRole, type AgentLike, type Origin, } from './roles.ts';
|
|
87
87
|
export { annotateCandidates, createCandidateFilter, hasWildcardEntry, resolveCandidate, resolveChain, resolveChainViews, selectCandidates, type AnnotatedCandidate, type CandidateFilterOptions, type CandidateSkipReason, type FailingModel, } from './chains.ts';
|
|
88
|
-
export { defaultFallbacksConfig, detectLegacyKeys, INHERIT_ROLE_ID, ROLE_ID_PATTERN, validateFallbacksConfig, type FallbacksConfig, type FallbacksConfigLogger, type FallbacksRole, type FallbacksRoles, type FallbacksRoleRule, type FallbackStrategy, type RevertPolicy, } from './config.ts';
|
|
88
|
+
export { defaultFallbacksConfig, detectLegacyKeys, INHERIT_ROLE_ID, ROLE_ID_PATTERN, validateFallbacksConfig, type FallbacksConfig, type FallbacksConfigLogger, type FallbacksRole, type FallbacksRoles, type FallbacksRoleRule, type FallbackStrategy, type RevertPolicy, type RecoveryPolicy, } from './config.ts';
|
|
89
89
|
export { parseSelector, SelectorError, type Selector, } from './selectors.ts';
|
|
90
90
|
export { FallbacksSeedManager, type EffectiveRole, type EffectiveRolesReadback, type SeedConflict, type SeedDeclaration, type SeedDeclareOutcome, type SeedRevertFailReason, type SeedRevertOutcome, type SeedSkipReason, type SeedsIo, type SeedsWireStatus, } from './seeds.ts';
|
|
91
91
|
export { presetRoles } from './presets.ts';
|
package/dist/index.js
CHANGED
|
@@ -259,6 +259,65 @@ function resolveSlotState(config, now, tz) {
|
|
|
259
259
|
label: "all-day"
|
|
260
260
|
};
|
|
261
261
|
}
|
|
262
|
+
const ESCALATION_CAP_MS = 36e5;
|
|
263
|
+
function escalatedCooldownMs(ms, n) {
|
|
264
|
+
return Math.max(ms, Math.min(ESCALATION_CAP_MS, ms * 2 ** (n - 1)));
|
|
265
|
+
}
|
|
266
|
+
var RecoveryStore = class {
|
|
267
|
+
entries = new Map();
|
|
268
|
+
recordFailure(key) {
|
|
269
|
+
const entry = this.entries.get(key);
|
|
270
|
+
if (entry === void 0) {
|
|
271
|
+
this.entries.set(key, {
|
|
272
|
+
consecutiveFailures: 1,
|
|
273
|
+
halfOpen: false,
|
|
274
|
+
probeLogged: false
|
|
275
|
+
});
|
|
276
|
+
return 1;
|
|
277
|
+
}
|
|
278
|
+
entry.consecutiveFailures += 1;
|
|
279
|
+
entry.halfOpen = false;
|
|
280
|
+
entry.probeLogged = false;
|
|
281
|
+
return entry.consecutiveFailures;
|
|
282
|
+
}
|
|
283
|
+
markHalfOpen(key, until) {
|
|
284
|
+
const entry = this.entries.get(key);
|
|
285
|
+
if (entry === void 0) {
|
|
286
|
+
this.entries.set(key, {
|
|
287
|
+
consecutiveFailures: 0,
|
|
288
|
+
halfOpen: true,
|
|
289
|
+
probeLogged: false,
|
|
290
|
+
expiredUntil: until
|
|
291
|
+
});
|
|
292
|
+
return;
|
|
293
|
+
}
|
|
294
|
+
entry.halfOpen = true;
|
|
295
|
+
entry.probeLogged = false;
|
|
296
|
+
entry.expiredUntil = until;
|
|
297
|
+
}
|
|
298
|
+
tryMarkProbeLogged(key) {
|
|
299
|
+
const entry = this.entries.get(key);
|
|
300
|
+
if (entry === void 0 || !entry.halfOpen || entry.probeLogged) return false;
|
|
301
|
+
entry.probeLogged = true;
|
|
302
|
+
return true;
|
|
303
|
+
}
|
|
304
|
+
isHalfOpen(key) {
|
|
305
|
+
return this.entries.get(key)?.halfOpen === true;
|
|
306
|
+
}
|
|
307
|
+
close(key) {
|
|
308
|
+
const entry = this.entries.get(key);
|
|
309
|
+
if (entry === void 0 || !entry.halfOpen) return;
|
|
310
|
+
this.entries.delete(key);
|
|
311
|
+
}
|
|
312
|
+
halfOpenEntries() {
|
|
313
|
+
const out = [];
|
|
314
|
+
for (const [key, entry] of this.entries) if (entry.halfOpen && entry.expiredUntil !== void 0) out.push({
|
|
315
|
+
key,
|
|
316
|
+
untilEpochMs: entry.expiredUntil
|
|
317
|
+
});
|
|
318
|
+
return out;
|
|
319
|
+
}
|
|
320
|
+
};
|
|
262
321
|
//#endregion
|
|
263
322
|
//#region .build/host/config.js
|
|
264
323
|
const defaultFallbacksConfig = {
|
|
@@ -280,7 +339,8 @@ const defaultFallbacksConfig = {
|
|
|
280
339
|
presets: "bundled",
|
|
281
340
|
roleAutoMatch: true,
|
|
282
341
|
timeSlots: [],
|
|
283
|
-
tz: "Asia/Shanghai"
|
|
342
|
+
tz: "Asia/Shanghai",
|
|
343
|
+
recovery: "timer"
|
|
284
344
|
};
|
|
285
345
|
const INHERIT_ROLE_ID = "inherit";
|
|
286
346
|
const ROLE_ID_PATTERN = /^[a-z0-9-]{1,32}$/;
|
|
@@ -325,6 +385,7 @@ function validateFallbacksConfig(config, logger) {
|
|
|
325
385
|
if (row.start !== void 0 || row.end !== void 0 || row.days !== void 0 && row.days.length > 0) logger.warn(`llm-fallbacks: preset "${preset}" carries its own window/day fields — preset windows are fixed code constants; the row is ignored`);
|
|
326
386
|
}
|
|
327
387
|
}
|
|
388
|
+
if ((config.recovery ?? "timer") === "half-open" && config.cooldownMs >= 36e5) logger.warn(`llm-fallbacks: recovery "half-open" escalation is inert — cooldownMs (${config.cooldownMs} ms) is at or above the 1-hour escalation cap (${ESCALATION_CAP_MS} ms), so every suppression stays flat at cooldownMs`);
|
|
328
389
|
}
|
|
329
390
|
function detectLegacyKeys(source) {
|
|
330
391
|
const keys = [];
|
|
@@ -394,7 +455,8 @@ const Config = z.object({
|
|
|
394
455
|
days: z.array(z.number()),
|
|
395
456
|
chain: z.array(z.string())
|
|
396
457
|
})).default([]),
|
|
397
|
-
tz: z.string().default("Asia/Shanghai")
|
|
458
|
+
tz: z.string().default("Asia/Shanghai"),
|
|
459
|
+
recovery: z.union([z.const("timer"), z.const("half-open")]).default("timer")
|
|
398
460
|
});
|
|
399
461
|
function buildAutomatchPrompt(roles, agent) {
|
|
400
462
|
const lines = [
|
|
@@ -651,6 +713,12 @@ var CooldownStore = class {
|
|
|
651
713
|
}
|
|
652
714
|
return true;
|
|
653
715
|
}
|
|
716
|
+
peek(key) {
|
|
717
|
+
return this.entries.get(key);
|
|
718
|
+
}
|
|
719
|
+
keys() {
|
|
720
|
+
return this.entries.keys();
|
|
721
|
+
}
|
|
654
722
|
snapshot(now = Date.now()) {
|
|
655
723
|
const active = [];
|
|
656
724
|
for (const [key, until] of this.entries) {
|
|
@@ -701,7 +769,8 @@ var FallbackStateStore = class {
|
|
|
701
769
|
failed: new StepFailureSet(),
|
|
702
770
|
switchCount: 0
|
|
703
771
|
},
|
|
704
|
-
cooldown: new CooldownStore()
|
|
772
|
+
cooldown: new CooldownStore(),
|
|
773
|
+
recovery: new RecoveryStore()
|
|
705
774
|
};
|
|
706
775
|
this.states.set(agentId, state);
|
|
707
776
|
}
|
|
@@ -752,8 +821,21 @@ var FallbackStateStore = class {
|
|
|
752
821
|
suppress(state, key, untilEpochMs) {
|
|
753
822
|
state.cooldown.suppress(key, untilEpochMs);
|
|
754
823
|
}
|
|
755
|
-
isSuppressed(state, key, now = Date.now()) {
|
|
756
|
-
return state.cooldown.isSuppressed(key, now);
|
|
824
|
+
isSuppressed(state, key, now = Date.now(), recovery = "timer") {
|
|
825
|
+
if (recovery !== "half-open") return state.cooldown.isSuppressed(key, now);
|
|
826
|
+
const until = state.cooldown.peek(key);
|
|
827
|
+
if (until === void 0) return false;
|
|
828
|
+
if (until > now) return true;
|
|
829
|
+
state.cooldown.isSuppressed(key, now);
|
|
830
|
+
state.recovery.markHalfOpen(key, until);
|
|
831
|
+
return false;
|
|
832
|
+
}
|
|
833
|
+
syncRecovery(state, now, mode) {
|
|
834
|
+
if (mode !== "half-open") return;
|
|
835
|
+
for (const key of state.cooldown.keys()) this.isSuppressed(state, key, now, mode);
|
|
836
|
+
}
|
|
837
|
+
observeSuccess(state, key) {
|
|
838
|
+
state.recovery.close(key);
|
|
757
839
|
}
|
|
758
840
|
};
|
|
759
841
|
//#endregion
|
|
@@ -771,7 +853,8 @@ const CONFIG_KEYS = {
|
|
|
771
853
|
presets: true,
|
|
772
854
|
roleAutoMatch: true,
|
|
773
855
|
timeSlots: true,
|
|
774
|
-
tz: true
|
|
856
|
+
tz: true,
|
|
857
|
+
recovery: true
|
|
775
858
|
};
|
|
776
859
|
const ROLES_KEYS = {
|
|
777
860
|
list: true,
|
|
@@ -1005,6 +1088,7 @@ const FALLBACKS_COMMAND_LOCALES = {
|
|
|
1005
1088
|
cooldownNone: "无活跃冷却",
|
|
1006
1089
|
cooldownLine: "{key} 冷却至 {time}",
|
|
1007
1090
|
cooldownNever: "{key} 会话内不再回主",
|
|
1091
|
+
cooldownHalfOpen: "{key} half-open(等待恢复探针)",
|
|
1008
1092
|
reason: {
|
|
1009
1093
|
"trigger-code": "触发码",
|
|
1010
1094
|
"always-cap": "always 上限"
|
|
@@ -1059,6 +1143,7 @@ const FALLBACKS_COMMAND_LOCALES = {
|
|
|
1059
1143
|
cooldownNone: "none active",
|
|
1060
1144
|
cooldownLine: "{key} suppressed until {time}",
|
|
1061
1145
|
cooldownNever: "{key} not reverting this session",
|
|
1146
|
+
cooldownHalfOpen: "{key} half-open (awaiting recovery probe)",
|
|
1062
1147
|
reason: {
|
|
1063
1148
|
"trigger-code": "trigger-code",
|
|
1064
1149
|
"always-cap": "always-cap"
|
|
@@ -1151,6 +1236,7 @@ function formatSwitch(entry, t) {
|
|
|
1151
1236
|
return t.switchLine.replace("{from}", from).replace("{to}", to).replace("{role}", entry.role).replace("{reason}", t.reason[entry.reason] ?? entry.reason);
|
|
1152
1237
|
}
|
|
1153
1238
|
function formatCooldown(entry, t) {
|
|
1239
|
+
if (entry.halfOpen === true) return t.cooldownHalfOpen.replace("{key}", entry.key);
|
|
1154
1240
|
if (!Number.isFinite(entry.untilEpochMs)) return t.cooldownNever.replace("{key}", entry.key);
|
|
1155
1241
|
return t.cooldownLine.replace("{key}", entry.key).replace("{time}", new Date(entry.untilEpochMs).toISOString());
|
|
1156
1242
|
}
|
|
@@ -2148,7 +2234,7 @@ function apply(ctx, config = defaultFallbacksConfig) {
|
|
|
2148
2234
|
const { all, wildcard } = resolveChainViews(config.roles.list, rootTail, role, current.provider, current.model, logger.warn);
|
|
2149
2235
|
if (all.length === 0) return null;
|
|
2150
2236
|
const modelExists = hasWildcardEntry(config.roles.list, rootTail, role) ? await makeModelExists(ctx, [...new Set(all.map((candidate) => candidate.provider))]) : void 0;
|
|
2151
|
-
const cooldown = { isSuppressed: (key) => state !== void 0 && states.isSuppressed(state, key) };
|
|
2237
|
+
const cooldown = { isSuppressed: (key) => state !== void 0 && states.isSuppressed(state, key, Date.now(), source().recovery ?? "timer") };
|
|
2152
2238
|
const failed = { has: (key) => state !== void 0 && state.stepFailures.failed.has(key) };
|
|
2153
2239
|
const surviving = selectCandidates(all, wildcard, createCandidateFilter({
|
|
2154
2240
|
current,
|
|
@@ -2157,6 +2243,7 @@ function apply(ctx, config = defaultFallbacksConfig) {
|
|
|
2157
2243
|
}), modelExists);
|
|
2158
2244
|
const target = surviving[0];
|
|
2159
2245
|
if (target === void 0 || target.model === void 0) return null;
|
|
2246
|
+
if ((source().recovery ?? "timer") === "half-open" && state !== void 0 && state.recovery.tryMarkProbeLogged(selectorKey(target.provider, target.model))) logger.info("llm-fallbacks: agent \"%s\" half-open probe %s/%s (role=%s, reason=%s)", agent.id, target.provider, target.model, role, reason);
|
|
2160
2247
|
logger.info("llm-fallbacks: agent \"%s\" fallback switch (降级切换) %s/%s -> %s/%s (role=%s, reason=%s, candidates=%o)", agent.id, current.provider, current.model, target.provider, target.model, role, reason, annotateCandidates(all, surviving, {
|
|
2161
2248
|
current,
|
|
2162
2249
|
cooldown,
|
|
@@ -2178,13 +2265,22 @@ function apply(ctx, config = defaultFallbacksConfig) {
|
|
|
2178
2265
|
function commit(state, pending, turn, step) {
|
|
2179
2266
|
const config = source();
|
|
2180
2267
|
const fromKey = selectorKey(pending.from.provider, pending.from.model);
|
|
2181
|
-
const until = config.revertPolicy === "never" ? Number.POSITIVE_INFINITY : Date.now() + config.cooldownMs;
|
|
2268
|
+
const until = config.revertPolicy === "never" ? Number.POSITIVE_INFINITY : config.recovery === "half-open" ? Date.now() + escalatedCooldownMs(config.cooldownMs, state.recovery.recordFailure(fromKey)) : Date.now() + config.cooldownMs;
|
|
2182
2269
|
states.syncStep(state, turn, step);
|
|
2183
2270
|
states.writePending(state, pending);
|
|
2184
2271
|
states.suppress(state, fromKey, until);
|
|
2185
2272
|
states.recordFailure(state, fromKey);
|
|
2186
2273
|
states.recordSwitch(state);
|
|
2187
2274
|
}
|
|
2275
|
+
function failHalfOpenProbe(agentId, current) {
|
|
2276
|
+
const key = selectorKey(current.provider, current.model);
|
|
2277
|
+
if ((source().recovery ?? "timer") !== "half-open") return;
|
|
2278
|
+
if (source().revertPolicy === "never") return;
|
|
2279
|
+
if (states.peek(agentId)?.recovery.isHalfOpen(key) !== true) return;
|
|
2280
|
+
const state = states.get(agentId);
|
|
2281
|
+
const n = state.recovery.recordFailure(key);
|
|
2282
|
+
states.suppress(state, key, Date.now() + escalatedCooldownMs(source().cooldownMs, n));
|
|
2283
|
+
}
|
|
2188
2284
|
ctx.on("agent/request-error", async ({ agent, turn, step, provider, failure }, next) => {
|
|
2189
2285
|
const config = source();
|
|
2190
2286
|
if (!config.enabled || !config.triggerCodes.includes(failure.code)) return next();
|
|
@@ -2192,7 +2288,10 @@ function apply(ctx, config = defaultFallbacksConfig) {
|
|
|
2192
2288
|
if (!current.model) return next();
|
|
2193
2289
|
try {
|
|
2194
2290
|
const pending = await decide(agent, turn, step, current, "trigger-code", states.peek(agent.id));
|
|
2195
|
-
if (pending === null)
|
|
2291
|
+
if (pending === null) {
|
|
2292
|
+
failHalfOpenProbe(agent.id, current);
|
|
2293
|
+
return next();
|
|
2294
|
+
}
|
|
2196
2295
|
commit(states.get(agent.id), pending, turn, step);
|
|
2197
2296
|
return { kind: "retry" };
|
|
2198
2297
|
} catch (error) {
|
|
@@ -2262,7 +2361,10 @@ function apply(ctx, config = defaultFallbacksConfig) {
|
|
|
2262
2361
|
commit(commitState, pending, turn, step);
|
|
2263
2362
|
const appliedCap = states.applyPending(commitState, turn, step);
|
|
2264
2363
|
if (appliedCap !== void 0) return overrideConfig(seed, appliedCap.to);
|
|
2265
|
-
}
|
|
2364
|
+
} else failHalfOpenProbe(agent.id, {
|
|
2365
|
+
provider: seed.provider,
|
|
2366
|
+
model: seed.model
|
|
2367
|
+
});
|
|
2266
2368
|
}
|
|
2267
2369
|
return seed;
|
|
2268
2370
|
});
|
|
@@ -2276,6 +2378,16 @@ function apply(ctx, config = defaultFallbacksConfig) {
|
|
|
2276
2378
|
dispatchInjected.delete(agent.id);
|
|
2277
2379
|
slotWinners.delete(agent.id);
|
|
2278
2380
|
});
|
|
2381
|
+
ctx.on("session/event", (session, event) => {
|
|
2382
|
+
if (event.type !== "assistant/message") return;
|
|
2383
|
+
if ((source().recovery ?? "timer") !== "half-open") return;
|
|
2384
|
+
if (event.data.interrupted === true) return;
|
|
2385
|
+
const message = event.data.message;
|
|
2386
|
+
if (message.source.kind !== "model") return;
|
|
2387
|
+
const state = states.peek(session.id);
|
|
2388
|
+
if (state === void 0) return;
|
|
2389
|
+
states.observeSuccess(state, selectorKey(message.source.provider, message.source.model));
|
|
2390
|
+
});
|
|
2279
2391
|
ctx.effect(() => () => {
|
|
2280
2392
|
states.clear();
|
|
2281
2393
|
dispatchInjected.clear();
|
|
@@ -2286,13 +2398,19 @@ function apply(ctx, config = defaultFallbacksConfig) {
|
|
|
2286
2398
|
const config = source();
|
|
2287
2399
|
const role = resolveRole(agent, config.roles.rules, roleIds, logger.warn);
|
|
2288
2400
|
const state = states.peek(agent.id);
|
|
2401
|
+
const recovery = config.recovery ?? "timer";
|
|
2402
|
+
if (state !== void 0 && recovery === "half-open") states.syncRecovery(state, Date.now(), recovery);
|
|
2289
2403
|
return {
|
|
2290
2404
|
origin: agent.session.header?.origin ?? "root",
|
|
2291
2405
|
role,
|
|
2292
2406
|
...resolveChainForDiagnostic(config.roles.list, config.rootChain, role, logger.warn),
|
|
2293
2407
|
slot: resolveSlotState(config, new Date(), config.tz ?? "Asia/Shanghai"),
|
|
2294
2408
|
switches: recentFallbacksSwitches(agent.session.events, 5),
|
|
2295
|
-
cooldown: state === void 0 ? [] : state.cooldown.snapshot()
|
|
2409
|
+
cooldown: state === void 0 ? [] : recovery === "half-open" ? [...state.cooldown.snapshot(), ...state.recovery.halfOpenEntries().map((entry) => ({
|
|
2410
|
+
key: entry.key,
|
|
2411
|
+
untilEpochMs: entry.untilEpochMs,
|
|
2412
|
+
halfOpen: true
|
|
2413
|
+
}))] : state.cooldown.snapshot()
|
|
2296
2414
|
};
|
|
2297
2415
|
},
|
|
2298
2416
|
getConfig() {
|
|
@@ -0,0 +1,92 @@
|
|
|
1
|
+
/**
|
|
2
|
+
* Half-open recovery state machine (plan fallbacks-half-open-recovery Task 2).
|
|
3
|
+
*
|
|
4
|
+
* `RecoveryStore` tracks per-route consecutive-failure counters and half-open
|
|
5
|
+
* episode flags, keyed `${provider}/${model}` via `selectorKey`. It is held
|
|
6
|
+
* per-agent on `AgentFallbackState` and is session-scoped in-memory (same
|
|
7
|
+
* lifetime as the cooldown store; a restart resets every route to a flat
|
|
8
|
+
* first cooldown).
|
|
9
|
+
*
|
|
10
|
+
* The store is pure logic — zero imports, the cooldown.ts purity precedent —
|
|
11
|
+
* so it is unit-testable without a host (recovery.spec.ts imports this module
|
|
12
|
+
* directly).
|
|
13
|
+
*
|
|
14
|
+
* State machine (plan P2):
|
|
15
|
+
* 1. `recordFailure` — a suppression write: increments the consecutive-failure
|
|
16
|
+
* counter and clears any half-open episode (rule 1).
|
|
17
|
+
* 2. `escalatedCooldownMs` — bounded geometric escalation of the suppression
|
|
18
|
+
* duration (rule 2).
|
|
19
|
+
* 3. `markHalfOpen` — an expired cooldown leaves the route half-open for one
|
|
20
|
+
* logged probe (rule 3).
|
|
21
|
+
* 4. `tryMarkProbeLogged` — the once-per-episode admission marker, not a gate
|
|
22
|
+
* (rule 4).
|
|
23
|
+
* 5. `isHalfOpen` — probe-failure gate for the no-surviving-target path
|
|
24
|
+
* (rule 5b).
|
|
25
|
+
* 6. `close` — an observed completion closes the circuit, only from half-open
|
|
26
|
+
* (rule 6).
|
|
27
|
+
*
|
|
28
|
+
* @module dsh-llm-fallbacks/recovery
|
|
29
|
+
*/
|
|
30
|
+
/** Escalation multiplier applied per consecutive failure (rule 2). */
|
|
31
|
+
export declare const ESCALATION_MULTIPLIER = 2;
|
|
32
|
+
/** Escalation ceiling: 1 hour (rule 2). */
|
|
33
|
+
export declare const ESCALATION_CAP_MS = 3600000;
|
|
34
|
+
/**
|
|
35
|
+
* Bounded geometric escalation of the suppression duration (rule 2):
|
|
36
|
+
* `max(ms, min(ESCALATION_CAP_MS, ms * ESCALATION_MULTIPLIER ** (n - 1)))`
|
|
37
|
+
* for `n >= 1`. `n = 1` is flat `ms`; the `max(ms, ...)` floor keeps
|
|
38
|
+
* escalation from ever shortening below the configured flat cooldown when
|
|
39
|
+
* `ms >= ESCALATION_CAP_MS`; `ms = 0` degenerates to today's flat-0
|
|
40
|
+
* semantics.
|
|
41
|
+
*/
|
|
42
|
+
export declare function escalatedCooldownMs(ms: number, n: number): number;
|
|
43
|
+
/**
|
|
44
|
+
* Per-route half-open recovery state (plan P2). Pure logic — no host
|
|
45
|
+
* dependencies, no `@deepseek-ai/*` imports.
|
|
46
|
+
*/
|
|
47
|
+
export declare class RecoveryStore {
|
|
48
|
+
private entries;
|
|
49
|
+
/**
|
|
50
|
+
* Record a suppression write for `key` (rule 1): increments the
|
|
51
|
+
* consecutive-failure counter and clears any in-progress half-open episode
|
|
52
|
+
* (a failure while half-open re-suppresses escalated — rule 5). Returns the
|
|
53
|
+
* new counter value `n`; the caller computes the escalated duration with
|
|
54
|
+
* {@link escalatedCooldownMs}.
|
|
55
|
+
*/
|
|
56
|
+
recordFailure(key: string): number;
|
|
57
|
+
/**
|
|
58
|
+
* Open a half-open episode for `key` (rule 3): the lapsed cooldown leaves
|
|
59
|
+
* the route half-open for one logged probe. Resets the per-episode probe
|
|
60
|
+
* marker; the consecutive-failure counter survives the episode so a probe
|
|
61
|
+
* failure escalates to `n + 1` (rule 5). Creates the entry on first sight
|
|
62
|
+
* (mid-session mode flips can surface a cooldown entry with no recovery
|
|
63
|
+
* entry — rule 8).
|
|
64
|
+
*/
|
|
65
|
+
markHalfOpen(key: string, until: number): void;
|
|
66
|
+
/**
|
|
67
|
+
* Mark the episode's one logged admission (rule 4). Returns `true` only for
|
|
68
|
+
* the first admission of a half-open episode; later admissions while the
|
|
69
|
+
* episode is unresolved return `false` and route normally (the marker is
|
|
70
|
+
* not a gate — there is no admission limit).
|
|
71
|
+
*/
|
|
72
|
+
tryMarkProbeLogged(key: string): boolean;
|
|
73
|
+
/** True while `key` has an in-progress half-open episode (rule 5b gate). */
|
|
74
|
+
isHalfOpen(key: string): boolean;
|
|
75
|
+
/**
|
|
76
|
+
* Close the circuit for `key` (rule 6): acts only when the entry exists and
|
|
77
|
+
* is half-open — the entry is deleted (counter reset by deletion, preference
|
|
78
|
+
* fully restored). A completion observed while the route is actively
|
|
79
|
+
* suppressed, or with no entry, is a no-op: a stale in-flight success must
|
|
80
|
+
* not cancel a fresher escalated re-suppression.
|
|
81
|
+
*/
|
|
82
|
+
close(key: string): void;
|
|
83
|
+
/**
|
|
84
|
+
* The in-progress half-open episodes, in insertion order (display path —
|
|
85
|
+
* plan P4). `untilEpochMs` is the lapsed expiry epoch that drives the
|
|
86
|
+
* `/fallbacks` marker row.
|
|
87
|
+
*/
|
|
88
|
+
halfOpenEntries(): Array<{
|
|
89
|
+
key: string;
|
|
90
|
+
untilEpochMs: number;
|
|
91
|
+
}>;
|
|
92
|
+
}
|
package/dist/state.d.ts
CHANGED
|
@@ -29,6 +29,8 @@
|
|
|
29
29
|
*/
|
|
30
30
|
import type { FallbackSwitchReason } from './events.ts';
|
|
31
31
|
import { CooldownStore, StepFailureSet } from './cooldown.ts';
|
|
32
|
+
import { RecoveryStore } from './recovery.ts';
|
|
33
|
+
import type { RecoveryPolicy } from './config.ts';
|
|
32
34
|
/** Decision awaiting application at the next `agent/request` boundary (spec §5.1). */
|
|
33
35
|
export interface PendingSwitch {
|
|
34
36
|
from: {
|
|
@@ -64,6 +66,8 @@ export interface AgentFallbackState {
|
|
|
64
66
|
stepFailures: StepFailures;
|
|
65
67
|
/** `provider/model → expiry epoch ms`; lazily expired on read. */
|
|
66
68
|
cooldown: CooldownStore;
|
|
69
|
+
/** Half-open recovery bookkeeping (plan fallbacks-half-open-recovery P2). */
|
|
70
|
+
recovery: RecoveryStore;
|
|
67
71
|
}
|
|
68
72
|
/**
|
|
69
73
|
* The `Map<agent.id, AgentFallbackState>` store plus the state-machine
|
|
@@ -107,6 +111,27 @@ export declare class FallbackStateStore {
|
|
|
107
111
|
clearStepState(state: AgentFallbackState): void;
|
|
108
112
|
/** Suppress `key` until `untilEpochMs` (`Infinity` for `revertPolicy: 'never'`). */
|
|
109
113
|
suppress(state: AgentFallbackState, key: string, untilEpochMs: number): void;
|
|
110
|
-
/**
|
|
111
|
-
|
|
114
|
+
/**
|
|
115
|
+
* Lazy cooldown read (spec §5.1; plan fallbacks-half-open-recovery P2
|
|
116
|
+
* rule 3). Under `'timer'` (the default) today's body runs verbatim:
|
|
117
|
+
* expired entries are dropped on read. Under `'half-open'` the raw `until`
|
|
118
|
+
* is read via `CooldownStore.peek` (no lazy drop): absent ⇒ false;
|
|
119
|
+
* `until > now` ⇒ true; `until <= now` ⇒ the entry is dropped via the
|
|
120
|
+
* existing `isSuppressed` and the route is marked half-open, returning
|
|
121
|
+
* false. `Infinity` (`revertPolicy: 'never'`) never satisfies `until <=
|
|
122
|
+
* now` ⇒ the never short-circuit is structural on the read side.
|
|
123
|
+
*/
|
|
124
|
+
isSuppressed(state: AgentFallbackState, key: string, now?: number, recovery?: RecoveryPolicy): boolean;
|
|
125
|
+
/**
|
|
126
|
+
* Bulk half-open transition for the display path (plan P4): walks the
|
|
127
|
+
* cooldown keys applying rule 3 per key. A pure no-op under `'timer'` —
|
|
128
|
+
* internal state and output stay byte-identical in default mode.
|
|
129
|
+
*/
|
|
130
|
+
syncRecovery(state: AgentFallbackState, now: number, mode: RecoveryPolicy): void;
|
|
131
|
+
/**
|
|
132
|
+
* Observed completion ⇒ close the circuit (plan P2 rule 6): acts only when
|
|
133
|
+
* the entry exists and is half-open; otherwise a no-op (a stale in-flight
|
|
134
|
+
* success must not cancel a fresher escalated re-suppression).
|
|
135
|
+
*/
|
|
136
|
+
observeSuccess(state: AgentFallbackState, key: string): void;
|
|
112
137
|
}
|