pi-ccswitch-auto-switch 0.1.6 → 0.2.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +9 -7
- package/README.zh-CN.md +9 -7
- package/candidates.ts +52 -6
- package/classify.ts +10 -4
- package/health.ts +60 -6
- package/index.ts +61 -19
- package/package.json +1 -1
package/README.md
CHANGED
|
@@ -13,7 +13,8 @@ The extension observes real Pi requests, records sanitized health signals, and
|
|
|
13
13
|
- Endpoint circuit breakers for DNS, connection, server, and streaming failures.
|
|
14
14
|
- Endpoint platform isolation: when several models on the same endpoint (same BaseURL/provider) fail within one round, the whole endpoint is isolated so sibling models from the same platform are not tried one by one.
|
|
15
15
|
- Model-only isolation for missing models and incompatible parameters.
|
|
16
|
-
-
|
|
16
|
+
- Learns model-family content-policy constraints from real failures (`glm-4.5` and `GLM-4.6`, for example, are both `glm`) and avoids the entire family during current and future content-policy failover chains. Evidence expires after 30 days.
|
|
17
|
+
- Applies hard health, failure-domain, policy-family, input-modality, and context filters before ranking candidates by provider diversity, model equivalence, capability compatibility, and historical reliability.
|
|
17
18
|
- Independent-provider semantics: a copied vendor (e.g. `my-provider-copy`) shares the BaseURL but is a distinct provider with its own endpoint key, so isolating one never blocks the other.
|
|
18
19
|
- 90-second first-response and 120-second streaming-idle watchdogs.
|
|
19
20
|
- Exponential cooldowns, `Retry-After` support, and one persisted half-open probe lease per provider.
|
|
@@ -54,7 +55,7 @@ CC Switch should be configured normally. This extension deliberately does not wr
|
|
|
54
55
|
| `/ccswitch refresh` | Refresh Pi's model registry and the status display. |
|
|
55
56
|
| `/ccswitch reactivate <provider/model\|all>` | Clear a breaker and let the next real request verify recovery, while preserving history. |
|
|
56
57
|
| `/ccswitch disable <provider/model>` | Manually exclude a model from failover. |
|
|
57
|
-
| `/ccswitch reset <provider/model\|all>` | Delete selected health history after confirmation. |
|
|
58
|
+
| `/ccswitch reset <provider/model\|all>` | Delete selected health history after confirmation; `all` also clears learned content-policy constraints. |
|
|
58
59
|
| `/ccswitch-test` | Inspect candidate discovery without changing models. |
|
|
59
60
|
|
|
60
61
|
Examples:
|
|
@@ -90,15 +91,16 @@ The state machine waits for Pi's native retry cycle to settle before switching.
|
|
|
90
91
|
| --- | --- | --- |
|
|
91
92
|
| `401`, `403`, quota, billing | Provider | 30 minutes |
|
|
92
93
|
| `429` | Provider | `Retry-After` when present; otherwise 5 minutes |
|
|
93
|
-
| DNS, network, `5xx`, interrupted stream | Endpoint | 2 minutes |
|
|
94
|
+
| DNS, network, `408`, `5xx`, interrupted stream | Endpoint | 2 minutes |
|
|
94
95
|
| `404`, invalid model, incompatible parameters | Model | 15 minutes |
|
|
95
|
-
| Content
|
|
96
|
+
| Content-policy or sensitive-content rejection | Current model + learned model-family constraint | 2-minute model cooldown; 30-day family evidence |
|
|
97
|
+
| Context overflow | Current round only | None |
|
|
96
98
|
|
|
97
|
-
Cooldowns grow exponentially within bounded limits. Context-overflow retries only consider models with a larger context window.
|
|
99
|
+
Cooldowns grow exponentially within bounded limits. Learned policy-constrained families are excluded only after a content-policy rejection; they remain eligible during ordinary rate-limit, network, and model-configuration failovers. Context-overflow retries only consider models with a larger context window, and image requests do not move to a model that explicitly supports text only. Unknown errors are isolated to the current model and still fail over normally. A user cancellation is the only aborted turn that does not trigger failover; watchdog cancellations are recorded as timeouts.
|
|
98
100
|
|
|
99
101
|
## Data and privacy
|
|
100
102
|
|
|
101
|
-
Health state is stored in Pi's agent directory as `ccswitch-auto-switch-state.json`. The extension stores counters, timestamps, cooldowns, and redacted/truncated error summaries. It does not access credentials, authorization headers, CC Switch's database, or Pi's `auth.json`.
|
|
103
|
+
Health state is stored in Pi's agent directory as `ccswitch-auto-switch-state.json`. The extension stores counters, timestamps, cooldowns, learned model-family policy constraints, and redacted/truncated error summaries. It does not access credentials, authorization headers, CC Switch's database, or Pi's `auth.json`.
|
|
102
104
|
|
|
103
105
|
## Development
|
|
104
106
|
|
|
@@ -108,7 +110,7 @@ npm run typecheck
|
|
|
108
110
|
npm test
|
|
109
111
|
```
|
|
110
112
|
|
|
111
|
-
Tests use Node's built-in test runner and cover failure
|
|
113
|
+
Tests use Node's built-in test runner and cover the failure matrix, family-level policy avoidance, provider-first selection, input compatibility, cooldowns, state persistence, and Windows-compatible paths.
|
|
112
114
|
|
|
113
115
|
## License
|
|
114
116
|
|
package/README.zh-CN.md
CHANGED
|
@@ -12,7 +12,8 @@
|
|
|
12
12
|
- 对认证、额度、账单和限流实施 Provider 优先熔断。
|
|
13
13
|
- 对 DNS、连接、服务端和流中断实施端点熔断。
|
|
14
14
|
- 模型不存在或参数不兼容时仅隔离该模型。
|
|
15
|
-
-
|
|
15
|
+
- 内容审查失败会学习模型系列约束(例如 `glm-4.5`、`GLM-4.6` 同属 `glm`),本轮及后续内容审查故障转移会避开整个受限系列;证据 30 天后自动过期。
|
|
16
|
+
- 候选模型先经过健康、故障域、审查系列、输入模态和上下文窗口硬过滤,再按跨 Provider、模型等价性、能力兼容和历史稳定性排序。
|
|
16
17
|
- 首响应 90 秒、流式停滞 120 秒看门狗。
|
|
17
18
|
- 指数退避、`Retry-After` 支持和跨进程 half-open 单探针租约。
|
|
18
19
|
- 原子持久化状态、错误脱敏和有上限的日志。
|
|
@@ -52,7 +53,7 @@ pi install npm:pi-ccswitch-auto-switch
|
|
|
52
53
|
| `/ccswitch refresh` | 刷新 Pi 模型注册表和状态栏。 |
|
|
53
54
|
| `/ccswitch reactivate <provider/model\|all>` | 解除熔断,下一次真实请求验证恢复情况;保留历史。 |
|
|
54
55
|
| `/ccswitch disable <provider/model>` | 手动排除模型。 |
|
|
55
|
-
| `/ccswitch reset <provider/model\|all>` |
|
|
56
|
+
| `/ccswitch reset <provider/model\|all>` | 确认后删除相应健康历史;`all` 同时清除已学习的审查约束。 |
|
|
56
57
|
| `/ccswitch-test` | 仅检查候选发现,不切换模型。 |
|
|
57
58
|
|
|
58
59
|
## 状态栏
|
|
@@ -80,15 +81,16 @@ CCS ✓70/131 · ⏳61 · ⛔0 · 🔄3 · provider/model-id
|
|
|
80
81
|
| --- | --- | --- |
|
|
81
82
|
| `401`、`403`、额度或账单问题 | Provider | 30 分钟 |
|
|
82
83
|
| `429` | Provider | 优先使用 `Retry-After`,否则 5 分钟 |
|
|
83
|
-
| DNS、网络、`5xx`、流中断 | 端点 | 2 分钟 |
|
|
84
|
+
| DNS、网络、`408`、`5xx`、流中断 | 端点 | 2 分钟 |
|
|
84
85
|
| `404`、模型不存在、参数不兼容 | 模型 | 15 分钟 |
|
|
85
|
-
|
|
|
86
|
+
| 内容审查/敏感拦截 | 当前模型 + 模型系列约束 | 模型冷却 2 分钟;系列避让证据 30 天 |
|
|
87
|
+
| 上下文溢出 | 仅本轮 | 无 |
|
|
86
88
|
|
|
87
|
-
|
|
89
|
+
冷却时间会指数增长但有上限。发生内容审查时,插件只在审查故障转移链中避开已标记系列,普通限流、网络或模型配置故障仍可选择这些模型。上下文溢出时只会选择上下文窗口更大的模型;带图片的请求不会切到明确仅支持文本的模型。无法归类的异常按单模型故障处理并正常切换。用户主动取消是唯一不会触发故障转移的异常终止;看门狗取消会记录为超时。
|
|
88
90
|
|
|
89
91
|
## 数据与隐私
|
|
90
92
|
|
|
91
|
-
健康状态保存在 Pi agent 目录的 `ccswitch-auto-switch-state.json
|
|
93
|
+
健康状态保存在 Pi agent 目录的 `ccswitch-auto-switch-state.json`。其中只有计数、时间、冷却信息、模型系列审查约束和脱敏/截断的错误摘要;插件不会访问凭据、Authorization 请求头、CC Switch 数据库或 Pi 的 `auth.json`。
|
|
92
94
|
|
|
93
95
|
## 开发
|
|
94
96
|
|
|
@@ -98,7 +100,7 @@ npm run typecheck
|
|
|
98
100
|
npm test
|
|
99
101
|
```
|
|
100
102
|
|
|
101
|
-
测试使用 Node
|
|
103
|
+
测试使用 Node 内置测试运行器,覆盖失败分类矩阵、系列级审查避让、Provider 优先选择、输入兼容、冷却、状态持久化以及 Windows 路径。
|
|
102
104
|
|
|
103
105
|
## 许可证
|
|
104
106
|
|
package/candidates.ts
CHANGED
|
@@ -1,12 +1,16 @@
|
|
|
1
|
-
import type { ModelRef, ScopedModel } from './types.ts'
|
|
1
|
+
import type { FailureClass, ModelRef, ScopedModel } from './types.ts'
|
|
2
2
|
import { endpointKey, modelKey, type HealthState } from './health.ts'
|
|
3
3
|
|
|
4
4
|
export interface CandidateOptions {
|
|
5
5
|
current?: ModelRef
|
|
6
6
|
tried: Set<string>
|
|
7
|
-
failureKind?:
|
|
7
|
+
failureKind?: FailureClass
|
|
8
8
|
health: HealthState
|
|
9
9
|
avoidEndpoints?: Set<string>
|
|
10
|
+
/** 模型系列已被证实存在内容审查约束时,本轮不再选择该系列。 */
|
|
11
|
+
avoidFamilies?: ReadonlySet<string>
|
|
12
|
+
/** 本轮输入真正需要的模态;显式声明不支持的候选会被排除。 */
|
|
13
|
+
requiredInputs?: readonly string[]
|
|
10
14
|
}
|
|
11
15
|
|
|
12
16
|
export interface CandidateSnapshot {
|
|
@@ -76,22 +80,64 @@ export function chooseCandidate(models: ModelRef[], options: CandidateOptions):
|
|
|
76
80
|
const candidates = models.filter(model => {
|
|
77
81
|
if (options.tried.has(modelKey(model)) || blocked(model, options.health)) return false
|
|
78
82
|
if (options.avoidEndpoints?.has(endpointKey(model))) return false
|
|
83
|
+
if (options.avoidFamilies?.has(modelFamily(model))) return false
|
|
84
|
+
if (!supportsRequiredInputs(model, options.requiredInputs)) return false
|
|
79
85
|
if (options.failureKind === 'context_overflow' && current && (model.contextWindow ?? 0) <= (current.contextWindow ?? 0)) return false
|
|
80
86
|
return true
|
|
81
87
|
})
|
|
82
|
-
return candidates.sort((a, b) => score(a, current) - score(b, current))[0]
|
|
88
|
+
return candidates.sort((a, b) => score(a, current, options.health) - score(b, current, options.health))[0]
|
|
83
89
|
}
|
|
84
90
|
|
|
85
|
-
|
|
91
|
+
/**
|
|
92
|
+
* 返回跨 Provider 稳定的模型系列标识。内容审查通常是模型系列的能力约束,而不是某个 API Key
|
|
93
|
+
* 或单一部署的瞬时故障,因此已知系列使用显式规则;未知模型则保守地按版本号前的 id 归组。
|
|
94
|
+
*/
|
|
95
|
+
export function modelFamily(model: Pick<ModelRef, 'id' | 'name'>): string {
|
|
96
|
+
const identity = `${model.id} ${model.name ?? ''}`.toLowerCase()
|
|
97
|
+
const knownFamilies: Array<[string, RegExp]> = [
|
|
98
|
+
['glm', /(?:^|[^a-z0-9])(?:chat)?glm(?:$|[^a-z])/],
|
|
99
|
+
['qwen', /(?:^|[^a-z0-9])qwen(?:$|[^a-z])/],
|
|
100
|
+
['deepseek', /(?:^|[^a-z0-9])deepseek(?:$|[^a-z])/],
|
|
101
|
+
['kimi', /(?:^|[^a-z0-9])(?:kimi|moonshot)(?:$|[^a-z])/],
|
|
102
|
+
['gpt', /(?:^|[^a-z0-9])gpt(?:$|[^a-z])/],
|
|
103
|
+
['claude', /(?:^|[^a-z0-9])claude(?:$|[^a-z])/],
|
|
104
|
+
['gemini', /(?:^|[^a-z0-9])gemini(?:$|[^a-z])/],
|
|
105
|
+
['llama', /(?:^|[^a-z0-9])llama(?:$|[^a-z])/],
|
|
106
|
+
['mistral', /(?:^|[^a-z0-9])mistral(?:$|[^a-z])/],
|
|
107
|
+
['minimax', /(?:^|[^a-z0-9])minimax(?:$|[^a-z])/],
|
|
108
|
+
['ernie', /(?:^|[^a-z0-9])ernie(?:$|[^a-z])/],
|
|
109
|
+
['doubao', /(?:^|[^a-z0-9])doubao(?:$|[^a-z])/],
|
|
110
|
+
['hunyuan', /(?:^|[^a-z0-9])hunyuan(?:$|[^a-z])/],
|
|
111
|
+
]
|
|
112
|
+
for (const [family, pattern] of knownFamilies) if (pattern.test(identity)) return family
|
|
113
|
+
|
|
114
|
+
const leaf = model.id.toLowerCase().split('/').pop() ?? model.id.toLowerCase()
|
|
115
|
+
return leaf
|
|
116
|
+
.replace(/(?:[-_.]v)?\d+(?:[.-]\d+)*(?:[-_.].*)?$/, '')
|
|
117
|
+
.replace(/[-_.:]+$/, '') || leaf
|
|
118
|
+
}
|
|
119
|
+
|
|
120
|
+
function supportsRequiredInputs(candidate: ModelRef, required: readonly string[] | undefined): boolean {
|
|
121
|
+
if (!required?.length || !candidate.input?.length) return true
|
|
122
|
+
return required.every(input => candidate.input!.includes(input))
|
|
123
|
+
}
|
|
124
|
+
|
|
125
|
+
function score(candidate: ModelRef, current: ModelRef | undefined, health: HealthState): number {
|
|
86
126
|
if (!current) return 0
|
|
127
|
+
// 排序优先级:故障域多样性 > 原模型等价性 > 输入/推理/上下文兼容性 > 历史稳定性。
|
|
128
|
+
// 硬约束(冷却、审查系列、输入模态、上下文溢出)已在上方 filter 中处理。
|
|
87
129
|
const sameProvider = candidate.provider === current.provider ? 1000 : 0
|
|
88
130
|
const differentId = candidate.id === current.id ? 0 : 100
|
|
89
131
|
const inputMismatch = sameInputs(candidate, current) ? 0 : 20
|
|
90
132
|
const reasoningMismatch = candidate.reasoning === current.reasoning ? 0 : 10
|
|
91
133
|
const contextPenalty = (candidate.contextWindow ?? 0) < (current.contextWindow ?? 0) ? 5 : 0
|
|
92
|
-
|
|
134
|
+
const records = [health.models[modelKey(candidate)], health.providers[candidate.provider], health.endpoints[endpointKey(candidate)]]
|
|
135
|
+
const historicalPenalty = Math.min(80, records.reduce((sum, record) => sum + Math.log2((record?.totalFailures ?? 0) + 1) * 4, 0))
|
|
136
|
+
return sameProvider + differentId + inputMismatch + reasoningMismatch + contextPenalty + historicalPenalty
|
|
93
137
|
}
|
|
94
138
|
|
|
95
139
|
function sameInputs(a: ModelRef, b: ModelRef): boolean {
|
|
96
|
-
|
|
140
|
+
const left = a.input ?? []
|
|
141
|
+
const right = b.input ?? []
|
|
142
|
+
return left.length === right.length && left.every(input => right.includes(input))
|
|
97
143
|
}
|
package/classify.ts
CHANGED
|
@@ -7,24 +7,30 @@ export interface Classification {
|
|
|
7
7
|
retryAfterMs?: number
|
|
8
8
|
}
|
|
9
9
|
|
|
10
|
-
const content = /content[ _-]?filter|sensitive|guardrail|policy[_ ]?violation|
|
|
10
|
+
const content = /content[ _-]?(?:filter|policy|blocked|moderation)|sensitive|guardrail|policy[_ -]?violation|responsibleai|safety[_ -]?(?:policy|filter|violation)|security[_ -]?policy|moderation[_ -]?(?:blocked|failed)|blocked by.{0,20}(?:policy|safety|moderation)|data[_ -]?inspection[_ -]?failed|内容(?:审查|审核|安全|违规|被拦截)|安全审核|安全(?:策略|风控).{0,12}(?:阻断|拦截|拒绝)|审核(?:未通过|不通过|失败)|敏感(?:内容|词|信息)?|政治敏感|涉政|不合规|违反.{0,12}(?:安全|政策|规定)/i
|
|
11
11
|
const quota = /quota|billing|insufficient.?balance|out of budget|usage.?limit|credit.?balance|monthly.?limit/i
|
|
12
12
|
const context = /context.?window|context.?length|too many tokens|prompt is too long|input is too long|token limit/i
|
|
13
|
-
const transport = /dns|enotfound|eai_again|socket|connection|network|fetch failed|timed? ?out|timeout|stream ended|terminated|websocket/i
|
|
13
|
+
const transport = /dns|enotfound|eai_again|econn(?:reset|refused|aborted)|epipe|socket|connection|network|fetch failed|tls|certificate|timed? ?out|timeout|stream ended|terminated|websocket|und_err/i
|
|
14
|
+
const overloaded = /overload|server busy|service unavailable|temporarily unavailable|capacity|upstream|bad gateway/i
|
|
14
15
|
|
|
15
16
|
export function classifyFailure(observation: FailureObservation): Classification {
|
|
16
17
|
const text = `${observation.rawStopReason ?? ''} ${observation.message ?? ''}`
|
|
17
18
|
const status = observation.status
|
|
18
19
|
if (observation.aborted && !observation.watchdog) return { kind: 'unknown', roundOnly: true }
|
|
19
20
|
if (observation.watchdog) return { kind: 'timeout', scope: 'endpoint', roundOnly: false }
|
|
20
|
-
|
|
21
|
+
// 内容审查/敏感拦截属于该模型的失败:按 model 级记录并与冷却(2min×2ⁿ 上限 30min),
|
|
22
|
+
// chooseCandidate 会排除后自动切换到其他模型,而不是同一内容原样重试到本轮上限。
|
|
23
|
+
if (content.test(text)) return { kind: 'content_policy', scope: 'model', roundOnly: false }
|
|
21
24
|
if (context.test(text)) return { kind: 'context_overflow', roundOnly: true }
|
|
25
|
+
if (status === 413) return { kind: 'context_overflow', roundOnly: true }
|
|
22
26
|
if (status === 429) return { kind: 'rate_limit', scope: 'provider', roundOnly: false, retryAfterMs: observation.retryAfterMs }
|
|
27
|
+
if (status === 402) return { kind: 'quota', scope: 'provider', roundOnly: false }
|
|
23
28
|
if (status === 401 || status === 403) return { kind: 'auth', scope: 'provider', roundOnly: false }
|
|
24
29
|
if (quota.test(text)) return { kind: 'quota', scope: 'provider', roundOnly: false }
|
|
30
|
+
if (status === 408) return { kind: 'timeout', scope: 'endpoint', roundOnly: false }
|
|
25
31
|
if (status !== undefined && status >= 500) return { kind: 'endpoint', scope: 'endpoint', roundOnly: false }
|
|
32
|
+
if (transport.test(text) || overloaded.test(text)) return { kind: 'endpoint', scope: 'endpoint', roundOnly: false }
|
|
26
33
|
if (status === 400 || status === 404 || status === 422) return { kind: 'model_config', scope: 'model', roundOnly: false }
|
|
27
|
-
if (transport.test(text)) return { kind: 'endpoint', scope: 'endpoint', roundOnly: false }
|
|
28
34
|
return { kind: 'unknown', scope: 'model', roundOnly: false }
|
|
29
35
|
}
|
|
30
36
|
|
package/health.ts
CHANGED
|
@@ -24,12 +24,25 @@ export interface HealthRecord {
|
|
|
24
24
|
leaseUntil?: number
|
|
25
25
|
}
|
|
26
26
|
|
|
27
|
+
export interface ContentPolicyConstraint {
|
|
28
|
+
observations: number
|
|
29
|
+
lastObservedAt: number
|
|
30
|
+
avoidUntil: number
|
|
31
|
+
lastModel: string
|
|
32
|
+
lastError?: string
|
|
33
|
+
}
|
|
34
|
+
|
|
27
35
|
export interface HealthState {
|
|
28
36
|
schemaVersion: 2
|
|
29
37
|
updatedAt: number
|
|
30
38
|
models: Record<string, HealthRecord>
|
|
31
39
|
providers: Record<string, HealthRecord>
|
|
32
40
|
endpoints: Record<string, HealthRecord>
|
|
41
|
+
/**
|
|
42
|
+
* 已从真实失败中学到的模型系列内容审查约束。它不会影响普通故障选模,只在本轮已经
|
|
43
|
+
* 发生 content_policy 失败时用于避开同类系列。
|
|
44
|
+
*/
|
|
45
|
+
contentPolicyFamilies?: Record<string, ContentPolicyConstraint>
|
|
33
46
|
/** 累计成功切换次数(跨 session 持久化,用于衡量扩展有效程度) */
|
|
34
47
|
switches?: number
|
|
35
48
|
/** 最近成功切换日志(有限条,环形保留) */
|
|
@@ -64,7 +77,7 @@ function safeEndpoint(value: string): string {
|
|
|
64
77
|
}
|
|
65
78
|
|
|
66
79
|
function blank(): HealthState {
|
|
67
|
-
return { schemaVersion: 2, updatedAt: Date.now(), models: {}, providers: {}, endpoints: {}, switches: 0, switchLog: [] }
|
|
80
|
+
return { schemaVersion: 2, updatedAt: Date.now(), models: {}, providers: {}, endpoints: {}, contentPolicyFamilies: {}, switches: 0, switchLog: [] }
|
|
68
81
|
}
|
|
69
82
|
|
|
70
83
|
function redact(text: string | undefined): string | undefined {
|
|
@@ -91,6 +104,7 @@ export class HealthStore {
|
|
|
91
104
|
private state: HealthState = blank()
|
|
92
105
|
private dirty = false
|
|
93
106
|
private replaceOnFlush = false
|
|
107
|
+
private resetModels = new Set<string>()
|
|
94
108
|
|
|
95
109
|
constructor(dir = agentDir()) { this.dir = dir }
|
|
96
110
|
get file(): string { return join(this.dir, STATE_FILE) }
|
|
@@ -143,6 +157,24 @@ export class HealthStore {
|
|
|
143
157
|
this.touch()
|
|
144
158
|
}
|
|
145
159
|
|
|
160
|
+
/**
|
|
161
|
+
* 内容审查约束比限流/网络故障稳定得多,但仍设置 30 天证据有效期,避免模型策略升级后
|
|
162
|
+
* 永久污染路由。再次观察到同系列审查会增加证据计数并刷新有效期。
|
|
163
|
+
*/
|
|
164
|
+
recordContentPolicyConstraint(family: string, model: ModelRef, message?: string): void {
|
|
165
|
+
const constraints = this.state.contentPolicyFamilies ??= {}
|
|
166
|
+
const previous = constraints[family]
|
|
167
|
+
const now = Date.now()
|
|
168
|
+
constraints[family] = {
|
|
169
|
+
observations: (previous?.observations ?? 0) + 1,
|
|
170
|
+
lastObservedAt: now,
|
|
171
|
+
avoidUntil: now + 30 * 24 * 60 * 60_000,
|
|
172
|
+
lastModel: modelKey(model),
|
|
173
|
+
lastError: redact(message),
|
|
174
|
+
}
|
|
175
|
+
this.touch()
|
|
176
|
+
}
|
|
177
|
+
|
|
146
178
|
recordSuccess(model: ModelRef): void {
|
|
147
179
|
this.close('model', modelKey(model))
|
|
148
180
|
this.close('provider', model.provider)
|
|
@@ -192,9 +224,14 @@ export class HealthStore {
|
|
|
192
224
|
}
|
|
193
225
|
|
|
194
226
|
reset(target: string | 'all'): void {
|
|
195
|
-
if (target === 'all')
|
|
196
|
-
|
|
197
|
-
|
|
227
|
+
if (target === 'all') {
|
|
228
|
+
this.state = blank()
|
|
229
|
+
this.replaceOnFlush = true
|
|
230
|
+
this.resetModels.clear()
|
|
231
|
+
} else {
|
|
232
|
+
delete this.state.models[target]
|
|
233
|
+
this.resetModels.add(target)
|
|
234
|
+
}
|
|
198
235
|
this.touch()
|
|
199
236
|
}
|
|
200
237
|
|
|
@@ -221,7 +258,8 @@ export class HealthStore {
|
|
|
221
258
|
async flush(): Promise<void> {
|
|
222
259
|
if (!this.dirty) return
|
|
223
260
|
await this.withLock(async () => {
|
|
224
|
-
this.state = mergeState(await this.readDisk(), this.state)
|
|
261
|
+
if (!this.replaceOnFlush) this.state = mergeState(await this.readDisk(), this.state)
|
|
262
|
+
for (const key of this.resetModels) delete this.state.models[key]
|
|
225
263
|
await this.commit()
|
|
226
264
|
}).catch(() => {})
|
|
227
265
|
}
|
|
@@ -259,6 +297,7 @@ export class HealthStore {
|
|
|
259
297
|
await rename(temp, this.file)
|
|
260
298
|
this.dirty = false
|
|
261
299
|
this.replaceOnFlush = false
|
|
300
|
+
this.resetModels.clear()
|
|
262
301
|
}
|
|
263
302
|
private bucket(scope: HealthScope): Record<string, HealthRecord> {
|
|
264
303
|
return scope === 'model' ? this.state.models : scope === 'provider' ? this.state.providers : this.state.endpoints
|
|
@@ -314,5 +353,20 @@ function mergeState(a: HealthState, b: HealthState): HealthState {
|
|
|
314
353
|
.sort((x, y) => y.at - x.at)
|
|
315
354
|
.filter((entry, index, all) => index === 0 || all[index - 1].at !== entry.at || all[index - 1].from !== entry.from || all[index - 1].to !== entry.to)
|
|
316
355
|
.slice(0, 20)
|
|
317
|
-
|
|
356
|
+
const contentPolicyFamilies: Record<string, ContentPolicyConstraint> = {}
|
|
357
|
+
for (const family of new Set([...Object.keys(a.contentPolicyFamilies ?? {}), ...Object.keys(b.contentPolicyFamilies ?? {})])) {
|
|
358
|
+
const left = a.contentPolicyFamilies?.[family]
|
|
359
|
+
const right = b.contentPolicyFamilies?.[family]
|
|
360
|
+
if (!left) contentPolicyFamilies[family] = right!
|
|
361
|
+
else if (!right) contentPolicyFamilies[family] = left
|
|
362
|
+
else {
|
|
363
|
+
const newest = right.lastObservedAt >= left.lastObservedAt ? right : left
|
|
364
|
+
contentPolicyFamilies[family] = {
|
|
365
|
+
...newest,
|
|
366
|
+
observations: Math.max(left.observations, right.observations),
|
|
367
|
+
avoidUntil: Math.max(left.avoidUntil, right.avoidUntil),
|
|
368
|
+
}
|
|
369
|
+
}
|
|
370
|
+
}
|
|
371
|
+
return { schemaVersion: 2, updatedAt: Math.max(a.updatedAt ?? 0, b.updatedAt ?? 0), models: mergeBucket(a.models ?? {}, b.models ?? {}), providers: mergeBucket(a.providers ?? {}, b.providers ?? {}), endpoints: mergeBucket(a.endpoints ?? {}, b.endpoints ?? {}), contentPolicyFamilies, switches, switchLog: log }
|
|
318
372
|
}
|
package/index.ts
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
import type { ExtensionAPI, ExtensionContext, FailureObservation, ModelRef } from './types.ts'
|
|
2
2
|
import { classifyFailure, parseRetryAfter } from './classify.ts'
|
|
3
|
-
import { candidateSnapshot, effectiveCandidates, chooseCandidate, summarizeCandidateHealth } from './candidates.ts'
|
|
4
|
-
import { HealthStore, endpointKey, modelKey } from './health.ts'
|
|
3
|
+
import { candidateSnapshot, effectiveCandidates, chooseCandidate, modelFamily, summarizeCandidateHealth } from './candidates.ts'
|
|
4
|
+
import { HealthStore, endpointKey, modelKey, type HealthState } from './health.ts'
|
|
5
5
|
|
|
6
6
|
const FIRST_RESPONSE_TIMEOUT = 90_000
|
|
7
7
|
const STREAM_IDLE_TIMEOUT = 120_000
|
|
@@ -32,10 +32,26 @@ interface Round {
|
|
|
32
32
|
observation?: FailureObservation
|
|
33
33
|
model?: ModelRef
|
|
34
34
|
endpointFails?: EndpointFailTracker
|
|
35
|
+
/** 本轮内容审查故障转移中必须避开的模型系列(包含持久化学到的约束)。 */
|
|
36
|
+
avoidFamilies?: Set<string>
|
|
37
|
+
/** 本轮内最后一次成功切换的时间;用于刷新 ROUND_LIMIT 窗口,避免供应商内部重试耗时导致误判“达到上限” */
|
|
38
|
+
lastSwitchAt?: number
|
|
35
39
|
}
|
|
36
40
|
|
|
37
41
|
function key(model: ModelRef | undefined): string | undefined { return model && modelKey(model) }
|
|
38
42
|
function canRetry(ctx: ExtensionContext): boolean { return ctx.mode === 'tui' || ctx.mode === 'rpc' }
|
|
43
|
+
function activePolicyFamilies(state: HealthState, now = Date.now()): Set<string> {
|
|
44
|
+
return new Set(Object.entries(state.contentPolicyFamilies ?? {})
|
|
45
|
+
.filter(([, record]) => record.avoidUntil > now)
|
|
46
|
+
.map(([family]) => family))
|
|
47
|
+
}
|
|
48
|
+
|
|
49
|
+
function resolveModel(ctx: ExtensionContext, provider: string, id: string, previous?: ModelRef): ModelRef {
|
|
50
|
+
// 优先使用注册表/scope 中的完整元数据,避免 turn_end 消息只有 provider/id 时丢失 BaseURL、
|
|
51
|
+
// 上下文窗口和输入模态;previous/ctx.model 作为注册表未命中时的后备。
|
|
52
|
+
const refs = [...ctx.scopedModels.map(item => item.model), ...ctx.modelRegistry.getAvailable(), previous, ctx.model]
|
|
53
|
+
return refs.find(model => model?.provider === provider && model.id === id) ?? { provider, id }
|
|
54
|
+
}
|
|
39
55
|
|
|
40
56
|
export default function (pi: ExtensionAPI) {
|
|
41
57
|
const health = new HealthStore()
|
|
@@ -49,10 +65,9 @@ export default function (pi: ExtensionAPI) {
|
|
|
49
65
|
const armWatchdog = (ctx: ExtensionContext, ms: number, roundId: number) => {
|
|
50
66
|
clearWatchdog()
|
|
51
67
|
timer = setTimeout(() => {
|
|
52
|
-
if (!round || round.id !== roundId ||
|
|
53
|
-
|
|
54
|
-
|
|
55
|
-
}
|
|
68
|
+
if (!round || round.id !== roundId || ctx.isIdle() || round.inTool) return
|
|
69
|
+
round.watchdog = true
|
|
70
|
+
ctx.abort()
|
|
56
71
|
}, ms)
|
|
57
72
|
timer.unref?.()
|
|
58
73
|
}
|
|
@@ -69,9 +84,11 @@ export default function (pi: ExtensionAPI) {
|
|
|
69
84
|
const activeModel = round?.model ?? ctx.model
|
|
70
85
|
// 恒显完整状态:健康/总数 · 冷却 · 禁用 · 本session切换 · 当前模型(均为 0 时也显示,便于确认扩展在监控中)
|
|
71
86
|
const plain = round?.phase === 'switching' ? `CCS ↻${round.attempts}/${MAX_ATTEMPTS} ${modelTag(round.model)}` :
|
|
87
|
+
round?.phase === 'exhausted' ? `CCS ⏸切换停止 ${modelTag(round.model ?? ctx.model)}` :
|
|
72
88
|
`CCS ✓${counts.healthy}/${counts.total} · ⏳${counts.cooling} · ⛔${counts.disabled}${switchSuffix()} · ${modelTag(activeModel)}`
|
|
73
89
|
const theme = ctx.ui.theme
|
|
74
|
-
|
|
90
|
+
const tone = round?.phase === 'exhausted' ? 'error' : counts.cooling || counts.disabled ? 'warning' : 'success'
|
|
91
|
+
ctx.ui.setStatus('ccswitch-ha', theme ? theme.fg(tone, plain) : plain)
|
|
75
92
|
}
|
|
76
93
|
// 本次 session 成功切换计数:始终显示 🔄N(N=0 也显示,用于确认扩展在监控中);累计切换数在面板/自检中可见
|
|
77
94
|
const switchSuffix = () => ` · 🔄${sessionSwitches}`
|
|
@@ -86,7 +103,7 @@ export default function (pi: ExtensionAPI) {
|
|
|
86
103
|
'/ccswitch refresh — 刷新 Pi 模型注册表和状态',
|
|
87
104
|
'/ccswitch reactivate <provider/model|all> — 解除熔断,保留历史',
|
|
88
105
|
'/ccswitch disable <provider/model> — 手动禁用模型',
|
|
89
|
-
'/ccswitch reset <provider/model|all> —
|
|
106
|
+
'/ccswitch reset <provider/model|all> — 清除健康历史;all 也会清除已学习的审查约束(需确认)',
|
|
90
107
|
'/ccswitch-test — 自检候选模型,不实际切换',
|
|
91
108
|
].join('\n'), 'info')
|
|
92
109
|
}
|
|
@@ -100,16 +117,19 @@ export default function (pi: ExtensionAPI) {
|
|
|
100
117
|
const candidates = snapshot.models
|
|
101
118
|
const counts = summarizeCandidateHealth(candidates, state)
|
|
102
119
|
const now = Date.now()
|
|
120
|
+
const policyFamilies = activePolicyFamilies(state, now)
|
|
103
121
|
const rows = candidates.slice(0, 10).map(model => {
|
|
104
122
|
const modelRecord = state.models[modelKey(model)]
|
|
105
123
|
const records = [modelRecord, state.providers[model.provider], state.endpoints[endpointKey(model)]]
|
|
106
124
|
const blockedUntil = Math.max(0, ...records.flatMap(record => [record?.cooldownUntil ?? 0, record?.leaseUntil ?? 0]))
|
|
107
|
-
const
|
|
125
|
+
const policySuffix = policyFamilies.has(modelFamily(model)) ? ' · 内容审查约束' : ''
|
|
126
|
+
const baseSuffix = modelRecord?.disabled ? '手动禁用' : blockedUntil > now ? `自动冷却 ${Math.max(1, Math.ceil((blockedUntil - now) / 60_000))}m` : '健康'
|
|
127
|
+
const suffix = `${baseSuffix}${policySuffix}`
|
|
108
128
|
return `${modelKey(model)} ${suffix}`
|
|
109
129
|
})
|
|
110
130
|
if (!ctx.ui.select) {
|
|
111
131
|
const source = snapshot.source === 'scoped' ? `Pi scope ${snapshot.sourceEntries} 条` : `Pi 注册表 ${snapshot.sourceEntries} 条`
|
|
112
|
-
notify(ctx, `CCSwitch:${source} · 唯一模型 ${counts.total} · 健康 ${counts.healthy} · 自动冷却 ${counts.cooling} · 手动禁用 ${counts.disabled} · 本session切换 ${sessionSwitches} · 累计切换 ${state.switches ?? 0}`, 'info')
|
|
132
|
+
notify(ctx, `CCSwitch:${source} · 唯一模型 ${counts.total} · 健康 ${counts.healthy} · 自动冷却 ${counts.cooling} · 手动禁用 ${counts.disabled} · 审查约束系列 ${policyFamilies.size} · 本session切换 ${sessionSwitches} · 累计切换 ${state.switches ?? 0}`, 'info')
|
|
113
133
|
return
|
|
114
134
|
}
|
|
115
135
|
const scopeLabel = snapshot.source === 'scoped' ? `Pi scope:${snapshot.sourceEntries} 条` : `Pi 可用注册表:${snapshot.sourceEntries} 条`
|
|
@@ -119,7 +139,8 @@ export default function (pi: ExtensionAPI) {
|
|
|
119
139
|
return ` ${time} ${entry.from} 🔄 ${entry.to}`
|
|
120
140
|
})
|
|
121
141
|
const switchPanel = recentSwitches.length ? `\n最近切换:\n${recentSwitches.join('\n')}` : ''
|
|
122
|
-
const
|
|
142
|
+
const policySummary = policyFamilies.size ? [...policyFamilies].join(', ') : '无'
|
|
143
|
+
const action = await ctx.ui.select(`CCSwitch 健康面板\n当前:${key(ctx.model) ?? '无'}\n${scopeLabel} · 唯一模型:${counts.total} · 健康:${counts.healthy} · 自动冷却:${counts.cooling} · 手动禁用:${counts.disabled}\n熔断记录:${counts.breakerRecords} · 审查约束系列:${policySummary}\n${switchSummary}${switchPanel}\n${rows.join('\n') || '没有可用模型'}`, ['刷新', '重新激活当前模型', '禁用当前模型', '重置当前模型历史', '关闭'])
|
|
123
144
|
if (action === '刷新') await refresh(ctx)
|
|
124
145
|
if (action === '重新激活当前模型' && ctx.model) { health.reactivate(ctx.model); await health.flush(); status(ctx); notify(ctx, '已重新激活当前模型') }
|
|
125
146
|
if (action === '禁用当前模型' && ctx.model) { health.disable(modelKey(ctx.model), true); await health.flush(); status(ctx); notify(ctx, '已禁用当前模型', 'warning') }
|
|
@@ -140,7 +161,9 @@ export default function (pi: ExtensionAPI) {
|
|
|
140
161
|
}
|
|
141
162
|
const failover = async (ctx: ExtensionContext) => {
|
|
142
163
|
if (!round || !round.observation || !round.model || !canRetry(ctx)) return
|
|
143
|
-
|
|
164
|
+
// 窗口从上一次成功切换(或本轮开始)起算:供应商内部重试耗时不应消耗整轮限额
|
|
165
|
+
const windowStart = Math.max(round.startedAt, round.lastSwitchAt ?? 0)
|
|
166
|
+
if (round.attempts >= MAX_ATTEMPTS || Date.now() - windowStart >= ROUND_LIMIT) return exhaust(ctx, '达到本轮切换上限')
|
|
144
167
|
const classification = classifyFailure(round.observation)
|
|
145
168
|
if (round.observation.aborted && !round.observation.watchdog) { round.phase = 'idle'; clearWatchdog(); status(ctx); return }
|
|
146
169
|
round.phase = 'switching'
|
|
@@ -155,27 +178,44 @@ export default function (pi: ExtensionAPI) {
|
|
|
155
178
|
failTracker.isolated.add(ep)
|
|
156
179
|
await health.log(`endpoint ${ep} failed ${epFails} times this round, isolating endpoint`)
|
|
157
180
|
}
|
|
181
|
+
if (classification.kind === 'content_policy') {
|
|
182
|
+
health.recordContentPolicyConstraint(modelFamily(round.model), round.model, round.observation.message)
|
|
183
|
+
round.avoidFamilies ??= new Set()
|
|
184
|
+
for (const family of activePolicyFamilies(health.snapshot)) round.avoidFamilies.add(family)
|
|
185
|
+
}
|
|
158
186
|
if (!classification.roundOnly && classification.scope) health.recordFailure(classification.scope, classification.scope === 'model' ? modelKey(round.model) : classification.scope === 'provider' ? round.model.provider : endpointKey(round.model), classification.kind, round.observation.message, classification.retryAfterMs)
|
|
159
187
|
await health.flush()
|
|
160
188
|
await refresh(ctx)
|
|
161
189
|
const candidates = effectiveCandidates(ctx.scopedModels, ctx.modelRegistry.getAvailable())
|
|
162
|
-
|
|
190
|
+
const candidateOptions = () => ({
|
|
191
|
+
current: round!.model,
|
|
192
|
+
tried: round!.tried,
|
|
193
|
+
failureKind: classification.kind,
|
|
194
|
+
health: health.snapshot,
|
|
195
|
+
avoidEndpoints: failTracker.isolated,
|
|
196
|
+
avoidFamilies: round!.avoidFamilies,
|
|
197
|
+
requiredInputs: round!.images?.length ? ['text', 'image'] : ['text'],
|
|
198
|
+
})
|
|
199
|
+
let next = chooseCandidate(candidates, candidateOptions())
|
|
163
200
|
while (next) {
|
|
164
|
-
if (!await health.claimProvider(next)) { round.tried.add(modelKey(next)); next = chooseCandidate(candidates,
|
|
201
|
+
if (!await health.claimProvider(next)) { round.tried.add(modelKey(next)); next = chooseCandidate(candidates, candidateOptions()); continue }
|
|
165
202
|
round.attempts++
|
|
166
203
|
ctx.ui.setWorkingMessage(`模型异常,正在切换到 ${modelKey(next)}…`)
|
|
204
|
+
const previousModel = round.model
|
|
167
205
|
const set = await pi.setModel(next).catch(() => false)
|
|
168
206
|
if (!set) {
|
|
169
207
|
health.recordFailure('model', modelKey(next), 'model_config', 'Pi refused model selection')
|
|
170
208
|
round.tried.add(modelKey(next)); await health.flush()
|
|
171
|
-
next = chooseCandidate(candidates,
|
|
209
|
+
next = chooseCandidate(candidates, candidateOptions())
|
|
172
210
|
continue
|
|
173
211
|
}
|
|
174
212
|
round.model = next
|
|
175
213
|
round.phase = 'redispatching'
|
|
214
|
+
// 刷新本轮切换时间窗:成功切换后重新起算 ROUND_LIMIT,避免长重试轮被误判为“达到上限”
|
|
215
|
+
round.lastSwitchAt = Date.now()
|
|
176
216
|
// 本次 session 成功切换计数 + 持久化累计/日志(衡量扩展有效程度)
|
|
177
217
|
sessionSwitches += 1
|
|
178
|
-
const fromKey = key(
|
|
218
|
+
const fromKey = key(previousModel ?? ctx.model)
|
|
179
219
|
health.recordSwitch(fromKey ?? '', modelKey(next), classification.kind)
|
|
180
220
|
await health.flush()
|
|
181
221
|
notify(ctx, `CCSwitch:已切换至 ${modelKey(next)}(${round.attempts}/${MAX_ATTEMPTS})`, 'info')
|
|
@@ -205,7 +245,7 @@ export default function (pi: ExtensionAPI) {
|
|
|
205
245
|
if (event.source === 'extension') return { action: 'continue' }
|
|
206
246
|
clearWatchdog()
|
|
207
247
|
lastStatus = {}
|
|
208
|
-
round = { id: (round?.id ?? 0) + 1, phase: 'monitoring', startedAt: Date.now(), text: event.text, images: event.images, tried: new Set(), attempts: 0, hadTool: false, inTool: false, hadOutput: false, watchdog: false, cleanRetry: false, model: ctx.model, endpointFails: undefined }
|
|
248
|
+
round = { id: (round?.id ?? 0) + 1, phase: 'monitoring', startedAt: Date.now(), text: event.text, images: event.images, tried: new Set(), attempts: 0, hadTool: false, inTool: false, hadOutput: false, watchdog: false, cleanRetry: false, model: ctx.model, endpointFails: undefined, avoidFamilies: undefined }
|
|
209
249
|
status(ctx)
|
|
210
250
|
return { action: 'continue' }
|
|
211
251
|
})
|
|
@@ -232,7 +272,7 @@ export default function (pi: ExtensionAPI) {
|
|
|
232
272
|
const message = event.message
|
|
233
273
|
if (message?.role !== 'assistant' || !round) return
|
|
234
274
|
clearWatchdog()
|
|
235
|
-
round.model =
|
|
275
|
+
round.model = resolveModel(ctx, message.provider, message.model, round.model)
|
|
236
276
|
round.hadOutput ||= Boolean(message.content?.length)
|
|
237
277
|
if (message.stopReason === 'error' || message.stopReason === 'aborted') {
|
|
238
278
|
round.phase = 'settled-error'
|
|
@@ -248,6 +288,7 @@ export default function (pi: ExtensionAPI) {
|
|
|
248
288
|
if (!ctx.isIdle()) return
|
|
249
289
|
if (!canRetry(ctx) && round.observation && round.model) {
|
|
250
290
|
const classification = classifyFailure(round.observation)
|
|
291
|
+
if (classification.kind === 'content_policy') health.recordContentPolicyConstraint(modelFamily(round.model), round.model, round.observation.message)
|
|
251
292
|
if (!classification.roundOnly && classification.scope) {
|
|
252
293
|
const domain = classification.scope === 'model' ? modelKey(round.model) : classification.scope === 'provider' ? round.model.provider : endpointKey(round.model)
|
|
253
294
|
health.recordFailure(classification.scope, domain, classification.kind, round.observation.message, classification.retryAfterMs)
|
|
@@ -285,7 +326,8 @@ export default function (pi: ExtensionAPI) {
|
|
|
285
326
|
const snapshot = candidateSnapshot(ctx.scopedModels, ctx.modelRegistry.getAvailable())
|
|
286
327
|
const counts = summarizeCandidateHealth(snapshot.models, health.snapshot)
|
|
287
328
|
const state = health.snapshot
|
|
329
|
+
const policyFamilies = activePolicyFamilies(state)
|
|
288
330
|
const source = snapshot.source === 'scoped' ? `Pi scope ${snapshot.sourceEntries} 条` : `Pi 注册表 ${snapshot.sourceEntries} 条`
|
|
289
|
-
notify(ctx, `CCSwitch 自检:${source} · 唯一模型 ${counts.total} · 健康 ${counts.healthy} · 自动冷却 ${counts.cooling} · 手动禁用 ${counts.disabled} · 熔断记录 ${counts.breakerRecords} · 本session切换 ${sessionSwitches} · 累计切换 ${state.switches ?? 0}`, counts.total ? 'info' : 'warning')
|
|
331
|
+
notify(ctx, `CCSwitch 自检:${source} · 唯一模型 ${counts.total} · 健康 ${counts.healthy} · 自动冷却 ${counts.cooling} · 手动禁用 ${counts.disabled} · 熔断记录 ${counts.breakerRecords} · 审查约束系列 ${policyFamilies.size} · 本session切换 ${sessionSwitches} · 累计切换 ${state.switches ?? 0}`, counts.total ? 'info' : 'warning')
|
|
290
332
|
}})
|
|
291
333
|
}
|