dsh-llm-router 0.1.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/LICENSE +31 -0
- package/README.md +113 -0
- package/cordis.patch.yml +19 -0
- package/lib/index.js +491 -0
- package/package.json +55 -0
package/LICENSE
ADDED
|
@@ -0,0 +1,31 @@
|
|
|
1
|
+
MIT License
|
|
2
|
+
|
|
3
|
+
Copyright (c) 2026 zhanghao3693
|
|
4
|
+
|
|
5
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
|
6
|
+
of this software and associated documentation files (the "Software"), to deal
|
|
7
|
+
in the Software without restriction, including without limitation the rights
|
|
8
|
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
9
|
+
copies of the Software, and to permit persons to whom the Software is
|
|
10
|
+
furnished to do so, subject to the following conditions:
|
|
11
|
+
|
|
12
|
+
The above copyright notice and this permission notice shall be included in all
|
|
13
|
+
copies or substantial portions of the Software.
|
|
14
|
+
|
|
15
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
16
|
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
17
|
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|
18
|
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
19
|
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
|
20
|
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
|
21
|
+
SOFTWARE.
|
|
22
|
+
|
|
23
|
+
---
|
|
24
|
+
|
|
25
|
+
DATA DISCLAIMER
|
|
26
|
+
|
|
27
|
+
The pricing and discount dataset bundled with this plugin is compiled from
|
|
28
|
+
publicly available vendor information and is provided for informational
|
|
29
|
+
purposes only. Vendor promotions change frequently and without notice.
|
|
30
|
+
Always confirm current pricing on the official vendor page before relying on
|
|
31
|
+
it for cost decisions. See the "数据说明" section of README.md.
|
package/README.md
ADDED
|
@@ -0,0 +1,113 @@
|
|
|
1
|
+
# dsh-llm-router
|
|
2
|
+
|
|
3
|
+
一个 DeepSeek Harness(dsh)的 **`auto` 路由 provider**:注册一个名为 `auto`
|
|
4
|
+
的虚拟模型,按请求内容自动挑选后端模型,并在失败时按链回退。
|
|
5
|
+
|
|
6
|
+
## 它解决什么问题
|
|
7
|
+
|
|
8
|
+
在 dsh 里,每个 provider 对应一个固定模型。想「看图用视觉模型、长文用长上下文模型、
|
|
9
|
+
其余用便宜的」就得手动切来切去。本插件把这个选择自动化:
|
|
10
|
+
|
|
11
|
+
```
|
|
12
|
+
请求进入 auto
|
|
13
|
+
├─ 含图片 → visionBackend
|
|
14
|
+
├─ 输入超阈值字符数 → longContextBackend
|
|
15
|
+
└─ 其余 → defaultBackend
|
|
16
|
+
↓ 任一后端「产出首个内容前」失败
|
|
17
|
+
按 fallbacks 链依次重试
|
|
18
|
+
```
|
|
19
|
+
|
|
20
|
+
**回退只在「首个内容块产出之前」发生。** 一旦后端已吐出内容,它就该自己负责到底——
|
|
21
|
+
中途失败会如实报错,不会重试(否则用户会看到重复的半个回答)。
|
|
22
|
+
|
|
23
|
+
## 安装
|
|
24
|
+
|
|
25
|
+
```bash
|
|
26
|
+
dsh plugin --profile web add dsh-llm-router
|
|
27
|
+
```
|
|
28
|
+
|
|
29
|
+
安装后 dsh 里会多出 `auto` 这个 provider,把它选为当前模型即可。
|
|
30
|
+
|
|
31
|
+
## ⚠️ 安装后必须做的事:配置你自己的后端
|
|
32
|
+
|
|
33
|
+
**默认配置里的 provider 名是作者本机环境的**,别人装了不一定存在。
|
|
34
|
+
不改配置就使用,大概率路由失败(找不到对应 provider)。
|
|
35
|
+
|
|
36
|
+
在你的 profile patch 层覆盖(`~/.dsh/profiles/<profile>/cordis.patch.yml`):
|
|
37
|
+
|
|
38
|
+
```yaml
|
|
39
|
+
- id: llm-router
|
|
40
|
+
config:
|
|
41
|
+
defaultBackend:
|
|
42
|
+
provider: <你的 provider id>
|
|
43
|
+
model: <模型名>
|
|
44
|
+
visionBackend:
|
|
45
|
+
provider: <支持图片的 provider>
|
|
46
|
+
model: <模型名>
|
|
47
|
+
longContextBackend:
|
|
48
|
+
provider: <支持长上下文的 provider>
|
|
49
|
+
model: <模型名>
|
|
50
|
+
fallbacks:
|
|
51
|
+
- provider: <备选1>
|
|
52
|
+
model: <模型名>
|
|
53
|
+
- provider: <备选2>
|
|
54
|
+
model: <模型名>
|
|
55
|
+
```
|
|
56
|
+
|
|
57
|
+
完整可配项与默认值:
|
|
58
|
+
|
|
59
|
+
| 字段 | 默认值 | 说明 |
|
|
60
|
+
|---|---|---|
|
|
61
|
+
| `defaultBackend` | `deepseek-official` / `deepseek-v4-pro` | 常规请求 |
|
|
62
|
+
| `visionBackend` | `zai-coding-cn` / `glm-5v-turbo` | 含图请求(**不回退**,见下) |
|
|
63
|
+
| `longContextBackend` | `longcat` / `LongCat-2.0` | 超长输入 |
|
|
64
|
+
| `longContextCharThreshold` | `200000` | 触发长上下文的字符数 |
|
|
65
|
+
| `fallbacks` | 6 项(kimi / 智谱 / MiniMax / LongCat) | 回退链 |
|
|
66
|
+
| `advertisedContextWindow` | `1000000` | 对外宣称的上下文窗口 |
|
|
67
|
+
| `defaultReasoningEffort` | `high` | 默认推理档位 |
|
|
68
|
+
|
|
69
|
+
**视觉请求刻意不回退** —— 链上只有 `visionBackend` 一个候选。原因是其他后端不接图片,
|
|
70
|
+
回退过去只会拿到一个含义不同的报错,不如让失败原因直白。
|
|
71
|
+
|
|
72
|
+
## 路由日志
|
|
73
|
+
|
|
74
|
+
插件注册 `GET /api/llm-router/routes?sessionId=&limit=`,返回最近的路由决策
|
|
75
|
+
(内存环形缓冲,最多 200 条,不落盘):
|
|
76
|
+
|
|
77
|
+
```json
|
|
78
|
+
{ "routes": [ { "id": 12, "kind": "vision", "provider": "zai-coding-cn",
|
|
79
|
+
"model": "glm-5v-turbo", "fallbackIndex": 0,
|
|
80
|
+
"affiliate": { "name": "智谱 BigModel", "badge": "2000万Tokens" } } ],
|
|
81
|
+
"total": 12 }
|
|
82
|
+
```
|
|
83
|
+
|
|
84
|
+
`kind` ∈ `default` / `long-context` / `vision`;`fallbackIndex` 为 0 表示首个候选就成功。
|
|
85
|
+
|
|
86
|
+
## 关于 `affiliate` 字段(如实披露)
|
|
87
|
+
|
|
88
|
+
`affiliate` 仅在**已选定后端之后**填入,表示该后端对应的平台与作者有推广关系。
|
|
89
|
+
它**只为界面提示而存在,不参与任何路由决策**,且这一点是用结构保证的:
|
|
90
|
+
|
|
91
|
+
- 查询点在 `decide()` 与回退循环**定论之后**,时序上不可能影响本次选择;
|
|
92
|
+
- 选择逻辑的各函数体内不含任何返利相关标识符;
|
|
93
|
+
- 该字段不含 `weight` / `score` / `priority` 之类排序语义。
|
|
94
|
+
|
|
95
|
+
可以自行验证:把 `AFFILIATE_HINTS` 改成给所有 provider 都挂上返利,
|
|
96
|
+
`decide()` 的输出对同一组输入**逐条不变**。
|
|
97
|
+
|
|
98
|
+
## 已知边界
|
|
99
|
+
|
|
100
|
+
- **回退判定只看「首个内容前」的失败**(配额、限流、连接错、缺凭证等)。
|
|
101
|
+
已产出内容后的中断不回退,见上文说明。
|
|
102
|
+
- 路由日志是**进程内内存**,重启即清空,不适合当审计账本。
|
|
103
|
+
- `longContextCharThreshold` 按**字符数**估算,不是精确 token 数。
|
|
104
|
+
- 视觉后端不回退,故 `visionBackend` 必须配一个真实可用的多模态模型。
|
|
105
|
+
|
|
106
|
+
## 兼容性
|
|
107
|
+
|
|
108
|
+
- 需要 `@deepseek-ai/dsh`(peer dependency,`*`)。
|
|
109
|
+
- 已验证 `dsh` 0.1.5 线。
|
|
110
|
+
|
|
111
|
+
## License
|
|
112
|
+
|
|
113
|
+
MIT
|
package/cordis.patch.yml
ADDED
|
@@ -0,0 +1,19 @@
|
|
|
1
|
+
# dsh-llm-router profile patch layer
|
|
2
|
+
#
|
|
3
|
+
# 这个文件是「命令安装能否生效」的关键:`dsh plugin add` 只有在包的 package.json
|
|
4
|
+
# 声明了 `dsh.bundle.patch` 时,才会把本插件并入 profile 的 layer 栈;
|
|
5
|
+
# 否则它只被当成一个普通依赖装进去,**进程不会加载它**,用户会看到「装成功但没反应」。
|
|
6
|
+
#
|
|
7
|
+
# id 必须与 host 入口里的插件名一致(lib/index.js 中 `export const name = "llm-router"`)。
|
|
8
|
+
# name 是 npm 包名,dsh 由它解析出 host 入口(main)。
|
|
9
|
+
#
|
|
10
|
+
# 这里刻意**不写 config** —— 后端配置必须由使用者按自己的 provider 设置。
|
|
11
|
+
# 请在你的 profile patch 层用 id 定向覆盖:
|
|
12
|
+
# - id: llm-router
|
|
13
|
+
# config:
|
|
14
|
+
# defaultBackend: { provider: <你的 provider>, model: <模型名> }
|
|
15
|
+
# ...
|
|
16
|
+
# 详见 README「安装后必须做的事」。
|
|
17
|
+
- insert:
|
|
18
|
+
- id: llm-router
|
|
19
|
+
name: 'dsh-llm-router'
|
package/lib/index.js
ADDED
|
@@ -0,0 +1,491 @@
|
|
|
1
|
+
import { LlmAdapter, contentHasImage } from "@deepseek-ai/dsh-llm";
|
|
2
|
+
|
|
3
|
+
/**
|
|
4
|
+
* dsh-llm-router — the "auto" provider route.
|
|
5
|
+
*
|
|
6
|
+
* One LlmAdapter owns the route `auto` with the single model id `auto`.
|
|
7
|
+
* On every request the adapter classifies the task from the assembled
|
|
8
|
+
* GenerateOptions:
|
|
9
|
+
*
|
|
10
|
+
* - any image block -> visionBackend (the only vision-capable model)
|
|
11
|
+
* - input over the threshold -> longContextBackend (1M-token models)
|
|
12
|
+
* - otherwise -> defaultBackend
|
|
13
|
+
*
|
|
14
|
+
* and then streams the request through that backend. If a backend fails
|
|
15
|
+
* with a terminal failure BEFORE any content chunk was emitted (quota,
|
|
16
|
+
* rate limit, transport error, missing credential, ...), the adapter moves
|
|
17
|
+
* to the next candidate in the fallback chain instead of surfacing the
|
|
18
|
+
* failure. Once one byte of content has been yielded the backend owns the
|
|
19
|
+
* stream — a mid-stream failure surfaces normally, because retrying after
|
|
20
|
+
* partial output has no durable attempt boundary.
|
|
21
|
+
*
|
|
22
|
+
* Routing decisions are logged through `ctx.logger`. Backend replay state
|
|
23
|
+
* is stripped at the router boundary (the historical provider is `auto`,
|
|
24
|
+
* the serving provider is the backend — replay must not impersonate it).
|
|
25
|
+
*/
|
|
26
|
+
|
|
27
|
+
export const name = "llm-router";
|
|
28
|
+
export const inject = ["llm", "webServer"];
|
|
29
|
+
|
|
30
|
+
const PROVIDER = "auto";
|
|
31
|
+
const MODEL = "auto";
|
|
32
|
+
|
|
33
|
+
/* -------------------------------------------------------- route log */
|
|
34
|
+
|
|
35
|
+
/** In-memory circular buffer of recent routing decisions (shared across all requests). */
|
|
36
|
+
const MAX_ROUTE_LOG = 200;
|
|
37
|
+
/** @type {{ id: number, ts: number, sessionId: string, kind: string, provider: string, model: string, fallbackIndex: number, affiliate: object|null }[]} */
|
|
38
|
+
const routeLog = [];
|
|
39
|
+
let routeIdCounter = 0;
|
|
40
|
+
|
|
41
|
+
/**
|
|
42
|
+
* 返利提示表 —— **仅供参考展示,绝不参与路由决策**。
|
|
43
|
+
*
|
|
44
|
+
* 为什么内联查表,而不是运行时调用 dsh-client-ui-model-clock 的
|
|
45
|
+
* `/api/model-clock/affiliate/resolve`:
|
|
46
|
+
* 1. 本插件可能在 `tui` / `headless` profile 下运行 —— 那里没有 webServer,
|
|
47
|
+
* 调不到对方的路由;而内联表在任何 profile 下都可用。
|
|
48
|
+
* 2. 端口未知:web 场景监听 8787,换 profile 就变;靠环境变量传端口比内联表更脆弱。
|
|
49
|
+
* 3. 查表**没有失败模式** —— 不会超时、不会被限流、不依赖对端是否安装。
|
|
50
|
+
*
|
|
51
|
+
* 代价是可能与权威源漂移。权威源是 dsh-client-ui-model-clock 的
|
|
52
|
+
* `config/affiliate.json`;该校验脚本 `verify-router-hint-sync.mjs`
|
|
53
|
+
* (放在 model-clock 仓库)会逐条比对两边,不一致即报错。
|
|
54
|
+
* **改返利配置后必须重跑它。**
|
|
55
|
+
*
|
|
56
|
+
* ⚠️ 用词纪律:这是「提示」不是「推荐」。
|
|
57
|
+
* 提示只能说「该平台有注册福利」,不得暗示「应该选它」。
|
|
58
|
+
* 选择逻辑(`decide()` 的能力分类 + 配置里的 fallback 顺序)**完全不受影响** ——
|
|
59
|
+
* 返利不得影响模型选择,这是本插件的硬约束。
|
|
60
|
+
*/
|
|
61
|
+
const AFFILIATE_HINTS = new Map([
|
|
62
|
+
["zai-coding-cn", { name: "智谱 BigModel", badge: "2000万Tokens", itemId: "zhipu-bigmodel-glm53" }],
|
|
63
|
+
["longcat", { name: "LongCat AI", badge: "送2000万Tokens", itemId: "longcat-ai" }],
|
|
64
|
+
]);
|
|
65
|
+
|
|
66
|
+
/**
|
|
67
|
+
* 查该 provider 是否对应一个作者有推广关系的平台。
|
|
68
|
+
*
|
|
69
|
+
* 纯函数、无 IO —— 调用它不会产生任何失败模式。
|
|
70
|
+
* 返回 null 表示「没有推广关系」,也覆盖「没配到」的情况,两者对调用方无区别。
|
|
71
|
+
*
|
|
72
|
+
* @param {string} provider dsh 的 provider id(不是厂商名,两套命名不同)
|
|
73
|
+
* @returns {{name: string, badge: string, itemId: string}|null}
|
|
74
|
+
*/
|
|
75
|
+
function lookupAffiliate(provider) {
|
|
76
|
+
if (typeof provider !== "string" || !provider) return null;
|
|
77
|
+
return AFFILIATE_HINTS.get(provider) ?? null;
|
|
78
|
+
}
|
|
79
|
+
|
|
80
|
+
function recordRoute(sessionId, kind, provider, model, fallbackIndex, affiliate) {
|
|
81
|
+
const entry = {
|
|
82
|
+
id: ++routeIdCounter,
|
|
83
|
+
ts: Date.now(),
|
|
84
|
+
sessionId: (typeof sessionId === "string") ? sessionId : "",
|
|
85
|
+
kind,
|
|
86
|
+
provider,
|
|
87
|
+
model,
|
|
88
|
+
fallbackIndex: fallbackIndex ?? 0,
|
|
89
|
+
/**
|
|
90
|
+
* 该 backend 对应的平台是否与作者有推广关系(只用于界面提示)。
|
|
91
|
+
*
|
|
92
|
+
* 刻意与 `provider` / `model` 平级而非嵌套进 `decision` ——
|
|
93
|
+
* 它在语义上属于「记录到的附加信息」,不属于「选择结果」。
|
|
94
|
+
* 任何消费端都应只把它当作展示素材,不得据此重排或推荐模型。
|
|
95
|
+
*/
|
|
96
|
+
affiliate: affiliate ?? null,
|
|
97
|
+
};
|
|
98
|
+
routeLog.push(entry);
|
|
99
|
+
while (routeLog.length > MAX_ROUTE_LOG) routeLog.shift();
|
|
100
|
+
return entry;
|
|
101
|
+
}
|
|
102
|
+
|
|
103
|
+
/** Query recent routes, optionally filtered by sessionId (prefix match). */
|
|
104
|
+
function queryRoutes(options) {
|
|
105
|
+
const { sessionId, limit = 50 } = options ?? {};
|
|
106
|
+
let list = routeLog;
|
|
107
|
+
if (sessionId) {
|
|
108
|
+
list = list.filter((r) => r.sessionId === sessionId || r.sessionId.startsWith(sessionId));
|
|
109
|
+
}
|
|
110
|
+
return list.slice(-limit).reverse(); // newest first
|
|
111
|
+
}
|
|
112
|
+
|
|
113
|
+
/* ---------------------------------------------------------------- config */
|
|
114
|
+
|
|
115
|
+
const DEFAULT_CONFIG = {
|
|
116
|
+
defaultBackend: { provider: "deepseek-official", model: "deepseek-v4-pro" },
|
|
117
|
+
visionBackend: { provider: "zai-coding-cn", model: "glm-5v-turbo" },
|
|
118
|
+
longContextBackend: { provider: "longcat", model: "LongCat-2.0" },
|
|
119
|
+
longContextCharThreshold: 200000,
|
|
120
|
+
fallbacks: [
|
|
121
|
+
{ provider: "kimi-coding", model: "kimi-for-coding" },
|
|
122
|
+
{ provider: "zai-coding-cn", model: "glm-5-turbo" },
|
|
123
|
+
{ provider: "zai-coding-cn", model: "glm-5.2" },
|
|
124
|
+
{ provider: "zai-coding-cn", model: "glm-4.5-air" },
|
|
125
|
+
{ provider: "minimax-cn", model: "MiniMax-M2.7" },
|
|
126
|
+
{ provider: "longcat", model: "LongCat-2.0" },
|
|
127
|
+
],
|
|
128
|
+
advertisedContextWindow: 1000000,
|
|
129
|
+
defaultReasoningEffort: "high",
|
|
130
|
+
};
|
|
131
|
+
|
|
132
|
+
/** Accept `{provider, model}` objects and `"provider/model"` strings. */
|
|
133
|
+
function normalizeBackend(value, fallback) {
|
|
134
|
+
if (typeof value === "string") {
|
|
135
|
+
const slash = value.indexOf("/");
|
|
136
|
+
if (slash > 0) {
|
|
137
|
+
const provider = value.slice(0, slash).trim();
|
|
138
|
+
const model = value.slice(slash + 1).trim();
|
|
139
|
+
if (provider && model) return { provider, model };
|
|
140
|
+
}
|
|
141
|
+
return fallback;
|
|
142
|
+
}
|
|
143
|
+
if (
|
|
144
|
+
value !== null && typeof value === "object" &&
|
|
145
|
+
typeof value.provider === "string" && value.provider.length > 0 &&
|
|
146
|
+
typeof value.model === "string" && value.model.length > 0
|
|
147
|
+
) {
|
|
148
|
+
return { provider: value.provider, model: value.model };
|
|
149
|
+
}
|
|
150
|
+
return fallback;
|
|
151
|
+
}
|
|
152
|
+
|
|
153
|
+
function resolveConfig(config) {
|
|
154
|
+
const c = config ?? {};
|
|
155
|
+
const fallbacks = Array.isArray(c.fallbacks)
|
|
156
|
+
? dedupe(c.fallbacks.map((entry) => normalizeBackend(entry, null)).filter(Boolean))
|
|
157
|
+
: DEFAULT_CONFIG.fallbacks.map((entry) => ({ ...entry }));
|
|
158
|
+
return {
|
|
159
|
+
defaultBackend: normalizeBackend(c.defaultBackend, DEFAULT_CONFIG.defaultBackend),
|
|
160
|
+
visionBackend: normalizeBackend(c.visionBackend, DEFAULT_CONFIG.visionBackend),
|
|
161
|
+
longContextBackend: normalizeBackend(c.longContextBackend, DEFAULT_CONFIG.longContextBackend),
|
|
162
|
+
longContextCharThreshold:
|
|
163
|
+
Number.isFinite(c.longContextCharThreshold) && c.longContextCharThreshold > 0
|
|
164
|
+
? c.longContextCharThreshold
|
|
165
|
+
: DEFAULT_CONFIG.longContextCharThreshold,
|
|
166
|
+
fallbacks,
|
|
167
|
+
advertisedContextWindow:
|
|
168
|
+
Number.isFinite(c.advertisedContextWindow) && c.advertisedContextWindow > 0
|
|
169
|
+
? c.advertisedContextWindow
|
|
170
|
+
: DEFAULT_CONFIG.advertisedContextWindow,
|
|
171
|
+
defaultReasoningEffort:
|
|
172
|
+
typeof c.defaultReasoningEffort === "string"
|
|
173
|
+
? c.defaultReasoningEffort
|
|
174
|
+
: DEFAULT_CONFIG.defaultReasoningEffort,
|
|
175
|
+
};
|
|
176
|
+
}
|
|
177
|
+
|
|
178
|
+
/* --------------------------------------------------------- classification */
|
|
179
|
+
|
|
180
|
+
function backendKey(backend) {
|
|
181
|
+
return `${backend.provider}/${backend.model}`;
|
|
182
|
+
}
|
|
183
|
+
|
|
184
|
+
function dedupe(backends) {
|
|
185
|
+
const seen = new Set();
|
|
186
|
+
const result = [];
|
|
187
|
+
for (const backend of backends) {
|
|
188
|
+
if (!backend) continue;
|
|
189
|
+
const key = backendKey(backend);
|
|
190
|
+
if (seen.has(key)) continue;
|
|
191
|
+
seen.add(key);
|
|
192
|
+
result.push(backend);
|
|
193
|
+
}
|
|
194
|
+
return result;
|
|
195
|
+
}
|
|
196
|
+
|
|
197
|
+
function requestHasImage(options) {
|
|
198
|
+
for (const message of options.messages ?? []) {
|
|
199
|
+
if (contentHasImage(message.content ?? [])) return true;
|
|
200
|
+
}
|
|
201
|
+
return false;
|
|
202
|
+
}
|
|
203
|
+
|
|
204
|
+
function blocksChars(blocks) {
|
|
205
|
+
let chars = 0;
|
|
206
|
+
for (const block of blocks ?? []) {
|
|
207
|
+
if (block.type === "text" || block.type === "reasoning") {
|
|
208
|
+
chars += block.text?.length ?? 0;
|
|
209
|
+
} else if (block.type === "tool-call") {
|
|
210
|
+
chars += (block.name?.length ?? 0) + (block.arguments?.length ?? 0);
|
|
211
|
+
} else if (block.type === "tool-result") {
|
|
212
|
+
if (typeof block.content === "string") chars += block.content.length;
|
|
213
|
+
else if (Array.isArray(block.content)) chars += blocksChars(block.content);
|
|
214
|
+
}
|
|
215
|
+
}
|
|
216
|
+
return chars;
|
|
217
|
+
}
|
|
218
|
+
|
|
219
|
+
function estimateInputChars(options) {
|
|
220
|
+
let chars = options.system?.length ?? 0;
|
|
221
|
+
for (const message of options.messages ?? []) {
|
|
222
|
+
chars += blocksChars(message.content ?? []);
|
|
223
|
+
}
|
|
224
|
+
return chars;
|
|
225
|
+
}
|
|
226
|
+
|
|
227
|
+
/**
|
|
228
|
+
* Pick the candidate chain for one request.
|
|
229
|
+
* Vision requests do not fall back: no other configured model takes images.
|
|
230
|
+
*/
|
|
231
|
+
function decide(options, config) {
|
|
232
|
+
if (requestHasImage(options)) {
|
|
233
|
+
return { kind: "vision", backends: dedupe([config.visionBackend]) };
|
|
234
|
+
}
|
|
235
|
+
const chars = estimateInputChars(options);
|
|
236
|
+
if (chars >= config.longContextCharThreshold) {
|
|
237
|
+
return {
|
|
238
|
+
kind: "long-context",
|
|
239
|
+
backends: dedupe([config.longContextBackend, config.defaultBackend, ...config.fallbacks]),
|
|
240
|
+
};
|
|
241
|
+
}
|
|
242
|
+
return { kind: "default", backends: dedupe([config.defaultBackend, ...config.fallbacks]) };
|
|
243
|
+
}
|
|
244
|
+
|
|
245
|
+
/* ---------------------------------------------------------------- helpers */
|
|
246
|
+
|
|
247
|
+
function toFailure(error) {
|
|
248
|
+
if (error !== null && typeof error === "object" && typeof error.code === "string") {
|
|
249
|
+
return { message: String(error.message ?? error.code), code: error.code };
|
|
250
|
+
}
|
|
251
|
+
return { message: String(error?.message ?? error), code: "TRANSPORT" };
|
|
252
|
+
}
|
|
253
|
+
|
|
254
|
+
function failureFinish(code, message) {
|
|
255
|
+
return { type: "finish", reason: { kind: "error", failure: { message, code } } };
|
|
256
|
+
}
|
|
257
|
+
|
|
258
|
+
function abortedFinish() {
|
|
259
|
+
return {
|
|
260
|
+
type: "finish",
|
|
261
|
+
reason: { kind: "aborted", failure: { message: "aborted", code: "ABORTED" } },
|
|
262
|
+
};
|
|
263
|
+
}
|
|
264
|
+
|
|
265
|
+
/** Strip backend replay state: it belongs to the backend adapter, not to `auto`. */
|
|
266
|
+
function sanitizeFinish(chunk) {
|
|
267
|
+
if (chunk?.type === "finish" && chunk.replayState !== undefined) {
|
|
268
|
+
return { type: "finish", reason: chunk.reason };
|
|
269
|
+
}
|
|
270
|
+
return chunk;
|
|
271
|
+
}
|
|
272
|
+
|
|
273
|
+
/* ----------------------------------------------------------------- adapter */
|
|
274
|
+
|
|
275
|
+
class RouterAdapter extends LlmAdapter {
|
|
276
|
+
constructor(ctx, config) {
|
|
277
|
+
super();
|
|
278
|
+
this.ctx = ctx;
|
|
279
|
+
this.config = config;
|
|
280
|
+
}
|
|
281
|
+
|
|
282
|
+
providerInfo(provider) {
|
|
283
|
+
return { id: provider, name: "Auto" };
|
|
284
|
+
}
|
|
285
|
+
|
|
286
|
+
listModels(provider) {
|
|
287
|
+
return Promise.resolve([
|
|
288
|
+
{
|
|
289
|
+
provider,
|
|
290
|
+
id: MODEL,
|
|
291
|
+
name: "Auto",
|
|
292
|
+
description: "按任务自动路由:图片 / 长上下文 / 默认,失败自动切换后备模型",
|
|
293
|
+
inputModalities: ["text", "image"],
|
|
294
|
+
},
|
|
295
|
+
]);
|
|
296
|
+
}
|
|
297
|
+
|
|
298
|
+
resolveModel(provider, model) {
|
|
299
|
+
return Promise.resolve({
|
|
300
|
+
provider,
|
|
301
|
+
id: model,
|
|
302
|
+
name: model === MODEL ? "Auto" : model,
|
|
303
|
+
inputModalities: ["text", "image"],
|
|
304
|
+
context: { contextWindow: this.config.advertisedContextWindow },
|
|
305
|
+
reasoning: {
|
|
306
|
+
efforts: [
|
|
307
|
+
{ id: "off", name: "Off" },
|
|
308
|
+
{ id: "high", name: "High" },
|
|
309
|
+
{ id: "max", name: "Max" },
|
|
310
|
+
],
|
|
311
|
+
defaultEffort: this.config.defaultReasoningEffort,
|
|
312
|
+
},
|
|
313
|
+
});
|
|
314
|
+
}
|
|
315
|
+
|
|
316
|
+
/** Drop a reasoning effort the chosen backend cannot serve. */
|
|
317
|
+
async clampEffort(backendOptions) {
|
|
318
|
+
const effort = backendOptions.reasoningEffort;
|
|
319
|
+
if (effort === undefined) return;
|
|
320
|
+
try {
|
|
321
|
+
const info = await this.ctx.llm.resolveModelInfo(
|
|
322
|
+
backendOptions.provider,
|
|
323
|
+
backendOptions.model,
|
|
324
|
+
backendOptions.signal,
|
|
325
|
+
);
|
|
326
|
+
const efforts = info?.reasoning?.efforts ?? [];
|
|
327
|
+
if (!efforts.some((entry) => entry.id === effort)) {
|
|
328
|
+
this.ctx.logger?.warn(
|
|
329
|
+
"llm-router: backend %s/%s does not serve reasoning effort %s — using its own default",
|
|
330
|
+
backendOptions.provider,
|
|
331
|
+
backendOptions.model,
|
|
332
|
+
effort,
|
|
333
|
+
);
|
|
334
|
+
delete backendOptions.reasoningEffort;
|
|
335
|
+
}
|
|
336
|
+
} catch {
|
|
337
|
+
// Metadata lookup failed (e.g. unregistered route): pass the effort
|
|
338
|
+
// through and let the backend request fail or serve on its own.
|
|
339
|
+
}
|
|
340
|
+
}
|
|
341
|
+
|
|
342
|
+
async *stream(options) {
|
|
343
|
+
const decision = decide(options, this.config);
|
|
344
|
+
this.ctx.logger?.info(
|
|
345
|
+
"llm-router: %s request -> %s",
|
|
346
|
+
decision.kind,
|
|
347
|
+
decision.backends.map(backendKey).join(" -> "),
|
|
348
|
+
);
|
|
349
|
+
let lastFailure = null;
|
|
350
|
+
for (const backend of decision.backends) {
|
|
351
|
+
if (backend.provider === PROVIDER) {
|
|
352
|
+
yield failureFinish("NO_ROUTE", "llm-router: a routing loop was configured (auto -> auto)");
|
|
353
|
+
return;
|
|
354
|
+
}
|
|
355
|
+
if (options.signal?.aborted) {
|
|
356
|
+
yield abortedFinish();
|
|
357
|
+
return;
|
|
358
|
+
}
|
|
359
|
+
const backendOptions = { ...options, provider: backend.provider, model: backend.model };
|
|
360
|
+
await this.clampEffort(backendOptions);
|
|
361
|
+
let iterator;
|
|
362
|
+
try {
|
|
363
|
+
iterator = this.ctx.llm.stream(backendOptions)[Symbol.asyncIterator]();
|
|
364
|
+
} catch (error) {
|
|
365
|
+
lastFailure = toFailure(error);
|
|
366
|
+
this.ctx.logger?.warn(
|
|
367
|
+
"llm-router: backend %s failed to open (%s) — trying next",
|
|
368
|
+
backendKey(backend),
|
|
369
|
+
lastFailure.code,
|
|
370
|
+
);
|
|
371
|
+
continue;
|
|
372
|
+
}
|
|
373
|
+
let closed = false;
|
|
374
|
+
const close = async () => {
|
|
375
|
+
if (closed) return;
|
|
376
|
+
closed = true;
|
|
377
|
+
await iterator.return?.().catch(() => {});
|
|
378
|
+
};
|
|
379
|
+
let adopted = false;
|
|
380
|
+
try {
|
|
381
|
+
const first = await iterator.next();
|
|
382
|
+
if (first.done) {
|
|
383
|
+
lastFailure = { message: `llm-router: ${backendKey(backend)} produced no output`, code: "EMPTY_RESPONSE" };
|
|
384
|
+
await close();
|
|
385
|
+
continue;
|
|
386
|
+
}
|
|
387
|
+
const chunk = sanitizeFinish(first.value);
|
|
388
|
+
if (chunk.type === "finish" && (chunk.reason.kind === "error" || chunk.reason.kind === "aborted")) {
|
|
389
|
+
lastFailure = chunk.reason.failure;
|
|
390
|
+
await close();
|
|
391
|
+
this.ctx.logger?.warn(
|
|
392
|
+
"llm-router: backend %s failed cleanly (%s) — trying next",
|
|
393
|
+
backendKey(backend),
|
|
394
|
+
lastFailure.code,
|
|
395
|
+
);
|
|
396
|
+
continue;
|
|
397
|
+
}
|
|
398
|
+
adopted = true;
|
|
399
|
+
// Record which backend actually served this request.
|
|
400
|
+
const fallbackIdx = decision.backends.indexOf(backend);
|
|
401
|
+
/**
|
|
402
|
+
* 返利提示**只在「已选中」之后查询** —— 这个时序是刻意的:
|
|
403
|
+
* 查询发生在 decide() 与 fallback 循环都已定论之后,
|
|
404
|
+
* 所以它在结构上不可能影响本次选择(不是「小心不去用它」,而是「用不到」)。
|
|
405
|
+
*/
|
|
406
|
+
const affiliate = lookupAffiliate(backend.provider);
|
|
407
|
+
recordRoute(options.sessionId ?? "", decision.kind, backend.provider, backend.model, fallbackIdx, affiliate);
|
|
408
|
+
this.ctx.logger?.info(
|
|
409
|
+
"llm-router: [%s] %s -> %s/%s (fallback #%d)",
|
|
410
|
+
options.sessionId ?? "?",
|
|
411
|
+
decision.kind,
|
|
412
|
+
backend.provider,
|
|
413
|
+
backend.model,
|
|
414
|
+
fallbackIdx,
|
|
415
|
+
);
|
|
416
|
+
if (affiliate) {
|
|
417
|
+
this.ctx.logger?.info(
|
|
418
|
+
"llm-router: [%s] 提示:%s 是作者有推广关系的平台(%s)。仅为展示提示,未参与本次选择。",
|
|
419
|
+
options.sessionId ?? "?",
|
|
420
|
+
affiliate.name,
|
|
421
|
+
affiliate.badge,
|
|
422
|
+
);
|
|
423
|
+
}
|
|
424
|
+
yield chunk;
|
|
425
|
+
for await (const rest of iterator) {
|
|
426
|
+
yield sanitizeFinish(rest);
|
|
427
|
+
}
|
|
428
|
+
closed = true;
|
|
429
|
+
return;
|
|
430
|
+
} catch (error) {
|
|
431
|
+
if (adopted) throw error; // mid-stream failure after content: surface it
|
|
432
|
+
lastFailure = toFailure(error);
|
|
433
|
+
this.ctx.logger?.warn(
|
|
434
|
+
"llm-router: backend %s failed during open (%s) — trying next",
|
|
435
|
+
backendKey(backend),
|
|
436
|
+
lastFailure.code,
|
|
437
|
+
);
|
|
438
|
+
continue;
|
|
439
|
+
} finally {
|
|
440
|
+
if (!closed) await close();
|
|
441
|
+
}
|
|
442
|
+
}
|
|
443
|
+
yield failureFinish(
|
|
444
|
+
lastFailure?.code ?? "NO_ROUTE",
|
|
445
|
+
lastFailure?.message ?? "llm-router: no backend route available",
|
|
446
|
+
);
|
|
447
|
+
}
|
|
448
|
+
}
|
|
449
|
+
|
|
450
|
+
/* ----------------------------------------------------------------- plugin */
|
|
451
|
+
|
|
452
|
+
export function apply(ctx, config) {
|
|
453
|
+
const resolved = resolveConfig(config);
|
|
454
|
+
ctx.logger?.info(
|
|
455
|
+
"llm-router: auto route ready — default=%s vision=%s longContext=%s (threshold=%d chars, fallbacks=%d)",
|
|
456
|
+
backendKey(resolved.defaultBackend),
|
|
457
|
+
backendKey(resolved.visionBackend),
|
|
458
|
+
backendKey(resolved.longContextBackend),
|
|
459
|
+
resolved.longContextCharThreshold,
|
|
460
|
+
resolved.fallbacks.length,
|
|
461
|
+
);
|
|
462
|
+
// Registration is disposed with this plugin's fiber.
|
|
463
|
+
ctx.llm.registerAdapter([PROVIDER], new RouterAdapter(ctx, resolved));
|
|
464
|
+
|
|
465
|
+
// HTTP API: expose recent routing decisions for the route-badge client plugin.
|
|
466
|
+
ctx.effect(() => ctx.webServer.register({
|
|
467
|
+
kind: "prefix",
|
|
468
|
+
path: "/api/llm-router/routes",
|
|
469
|
+
handler: async (req, res) => {
|
|
470
|
+
if (req.method !== "GET" && req.method !== "HEAD") {
|
|
471
|
+
res.writeHead(405, { Allow: "GET, HEAD" });
|
|
472
|
+
res.end();
|
|
473
|
+
return;
|
|
474
|
+
}
|
|
475
|
+
const url = new URL(req.url, "http://localhost");
|
|
476
|
+
const sessionId = url.searchParams.get("sessionId") || "";
|
|
477
|
+
const limit = Math.min(200, Math.max(1, parseInt(url.searchParams.get("limit") || "50", 10) || 50));
|
|
478
|
+
const routes = queryRoutes({ sessionId, limit });
|
|
479
|
+
const body = JSON.stringify({ routes, total: routeLog.length });
|
|
480
|
+
res.writeHead(200, {
|
|
481
|
+
"Content-Type": "application/json; charset=utf-8",
|
|
482
|
+
"Content-Length": Buffer.byteLength(body),
|
|
483
|
+
"Cache-Control": "no-store",
|
|
484
|
+
"Access-Control-Allow-Origin": "*",
|
|
485
|
+
});
|
|
486
|
+
res.end(req.method === "HEAD" ? undefined : body);
|
|
487
|
+
},
|
|
488
|
+
}), "llm-router: /api/llm-router/routes");
|
|
489
|
+
}
|
|
490
|
+
|
|
491
|
+
export { RouterAdapter, decide, resolveConfig, lookupAffiliate, AFFILIATE_HINTS };
|
package/package.json
ADDED
|
@@ -0,0 +1,55 @@
|
|
|
1
|
+
{
|
|
2
|
+
"name": "dsh-llm-router",
|
|
3
|
+
"version": "0.1.0",
|
|
4
|
+
"description": "Task-aware automatic model router (auto route) for the DeepSeek Harness: picks a backend model per request (vision / long-context / default) and falls back across configured models on clean pre-content failures.",
|
|
5
|
+
"type": "module",
|
|
6
|
+
"author": "zhanghao3693",
|
|
7
|
+
"repository": {
|
|
8
|
+
"type": "git",
|
|
9
|
+
"url": "git+https://github.com/zhanghao3693/dsh-llm-router.git"
|
|
10
|
+
},
|
|
11
|
+
"homepage": "https://github.com/zhanghao3693/dsh-llm-router",
|
|
12
|
+
"bugs": {
|
|
13
|
+
"url": "https://github.com/zhanghao3693/dsh-llm-router/issues"
|
|
14
|
+
},
|
|
15
|
+
"keywords": [
|
|
16
|
+
"dsh",
|
|
17
|
+
"dsh-plugin",
|
|
18
|
+
"deepseek-harness",
|
|
19
|
+
"cordis",
|
|
20
|
+
"plugin",
|
|
21
|
+
"llm-router",
|
|
22
|
+
"model-routing",
|
|
23
|
+
"auto-route",
|
|
24
|
+
"fallback"
|
|
25
|
+
],
|
|
26
|
+
"main": "lib/index.js",
|
|
27
|
+
"exports": {
|
|
28
|
+
".": "./lib/index.js"
|
|
29
|
+
},
|
|
30
|
+
"files": [
|
|
31
|
+
"lib/index.js",
|
|
32
|
+
"cordis.patch.yml",
|
|
33
|
+
"README.md",
|
|
34
|
+
"LICENSE"
|
|
35
|
+
],
|
|
36
|
+
"dsh": {
|
|
37
|
+
"bundle": {
|
|
38
|
+
"patch": "./cordis.patch.yml"
|
|
39
|
+
},
|
|
40
|
+
"engines": {
|
|
41
|
+
"dsh": ">=0.1.0-rc.5"
|
|
42
|
+
}
|
|
43
|
+
},
|
|
44
|
+
"license": "MIT",
|
|
45
|
+
"peerDependencies": {
|
|
46
|
+
"@deepseek-ai/cordis": "*",
|
|
47
|
+
"@deepseek-ai/dsh": ">=0.1.0-rc.5",
|
|
48
|
+
"@deepseek-ai/dsh-llm": "*"
|
|
49
|
+
},
|
|
50
|
+
"peerDependenciesMeta": {
|
|
51
|
+
"@deepseek-ai/dsh": {
|
|
52
|
+
"optional": true
|
|
53
|
+
}
|
|
54
|
+
}
|
|
55
|
+
}
|