dsh-auto-effort 0.1.1 → 0.1.2
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +231 -57
- package/client.js +126 -0
- package/config.js +39 -12
- package/cordis.patch.yml +2 -1
- package/docs/DETAIL.md +51 -7
- package/docs/DETAIL.zh.md +51 -7
- package/index.js +180 -23
- package/judge.js +103 -47
- package/package.json +16 -2
- package/schema.js +41 -0
- package/state.js +12 -0
package/docs/DETAIL.md
CHANGED
|
@@ -1,5 +1,15 @@
|
|
|
1
1
|

|
|
2
2
|
|
|
3
|
+
## The switch: auto or manual
|
|
4
|
+
|
|
5
|
+
The "自动选择档位" row in the model menu decides whether the plugin acts at all:
|
|
6
|
+
|
|
7
|
+
- Off (default): the plugin does nothing; `mode` = `manual`.
|
|
8
|
+
- On: every message you send is judged; `mode` = `auto`.
|
|
9
|
+
- You pick any level by hand: the plugin steps aside and the switch turns itself off (it writes `mode: manual`). Picking the level the plugin just set is not detectable, so nothing happens.
|
|
10
|
+
|
|
11
|
+
The switch is global: once off, no conversation is auto-adjusted, and open conversations react immediately — no restart.
|
|
12
|
+
|
|
3
13
|
## Which conversations it affects
|
|
4
14
|
|
|
5
15
|
The bundle takes over an agent the moment that agent is created. So:
|
|
@@ -8,16 +18,50 @@ The bundle takes over an agent the moment that agent is created. So:
|
|
|
8
18
|
- A conversation that was **already open** when the bundle loaded is **not** judged in that run: its agent already existed and never emits `agent/created` again. Reopen it after restarting DSH and it is judged from then on.
|
|
9
19
|
- Subagent runs are never judged and never delayed.
|
|
10
20
|
|
|
11
|
-
##
|
|
21
|
+
## Inputs: one message plus one card
|
|
12
22
|
|
|
13
|
-
|
|
23
|
+
Each judgement sends two things only — no transcript, no tool output, no file contents:
|
|
14
24
|
|
|
15
|
-
|
|
16
|
-
|
|
17
|
-
|
|
25
|
+
```
|
|
26
|
+
任务便签:<the task card as of the last finished turn, at most 200 chars; empty on the first turn>
|
|
27
|
+
用户新消息:<the message you just sent>
|
|
28
|
+
```
|
|
29
|
+
|
|
30
|
+
The card is rewritten by a second, background call after the reply (it reads the old card, your message, and the first 1200 chars of the reply). It carries the goal, what has been done, and what is left. Because it runs after the reply, it never adds to your wait.
|
|
31
|
+
|
|
32
|
+
The judge prompt, verbatim:
|
|
18
33
|
|
|
19
34
|
```
|
|
20
|
-
|
|
35
|
+
你是任务难度判定器。输入是任务便签(可能为空)和用户的新消息。
|
|
36
|
+
只判断“回答这条消息需要多强的推理”,不要回答它。
|
|
37
|
+
只输出一个词:off、low、high、max,不要解释、不要标点、不要换行。
|
|
38
|
+
|
|
39
|
+
档位标准:
|
|
40
|
+
off:闲聊、确认、查资料、改配置、跑命令。
|
|
41
|
+
low:小改动、改名、整理;做法已经明确,照着改就行(改语法、改写法、统一命名)。
|
|
42
|
+
high:需要自己拿主意——分析原因、排查报错、解释原理、写较复杂代码、做设计取舍。
|
|
43
|
+
max:架构方案、多方案比较、多因素权衡、疑难 bug 的根因、长链条推理。
|
|
44
|
+
|
|
45
|
+
便签是这轮对话在做的事:
|
|
46
|
+
用户只回“开始”“好的”“继续”,就按便签里没做完的事判,别因为它短就判 off。
|
|
47
|
+
消息和便签无关(换了话题),就当便签是空的。
|
|
21
48
|
```
|
|
22
49
|
|
|
23
|
-
|
|
50
|
+
The call is capped at 8 output tokens; the first `off|low|high|max` in the answer is the verdict.
|
|
51
|
+
|
|
52
|
+
## Failure, fallback and same-turn escalation
|
|
53
|
+
|
|
54
|
+
- Timeout (3 s by default) or no valid level: the turn keeps your manual level; three failures in a row switch the plugin off and it tells you.
|
|
55
|
+
- A `content[].thinking` round-trip error: it falls back to manual immediately and stops auto-switching for that session.
|
|
56
|
+
- Judged `off`/`low` but already 3 steps deep: from step 4 the remaining steps run at `high`. It only escalates, never de-escalates, and `high` never escalates to `max`.
|
|
57
|
+
|
|
58
|
+
## What a judgement costs
|
|
59
|
+
|
|
60
|
+
Two very small requests per **user message** (one judge call, one card update after the reply; tool rounds are not re-judged):
|
|
61
|
+
|
|
62
|
+
- judge: input ≈ fixed instructions + card (≤200 chars) + your message; output ≤8 tokens, in practice one word; reasoning off.
|
|
63
|
+
- card: input ≈ card + your message + the first 1200 chars of the reply; output ≤320 tokens, usually 60–150 characters.
|
|
64
|
+
|
|
65
|
+
At the November 2026 `deepseek-flash` list price (input $0.15/M off-peak, $0.30/M peak; output $0.60/M off-peak, $1.20/M peak) that is about **$0.00003–$0.00015 per message** (¥0.0002–¥0.001), or ¥0.03–¥0.1 per 100 messages. Fill in your provider's current prices if they differ.
|
|
66
|
+
|
|
67
|
+
Judging measured about 0.77 s (bounded by `timeoutMs`, 3 s by default) and the message leaves right after it; the card update happens after the reply and is not on that path. Turning the switch off stops all of this spending.
|
package/docs/DETAIL.zh.md
CHANGED
|
@@ -1,5 +1,15 @@
|
|
|
1
1
|

|
|
2
2
|
|
|
3
|
+
## 开关:自动 / 手动
|
|
4
|
+
|
|
5
|
+
模型菜单里的「自动选择档位」决定插件是否工作:
|
|
6
|
+
|
|
7
|
+
- 关(默认):插件什么都不做;`mode` = `manual`。
|
|
8
|
+
- 开:每条你发的消息都被判定;`mode` = `auto`。
|
|
9
|
+
- 你手动选了任意档位:插件立刻让位,开关自动关掉(写回 `mode: manual`)。若你选的正好和插件刚设的一样,则看不出差异,不会让位。
|
|
10
|
+
|
|
11
|
+
开关是全局的:关掉后所有对话都不再自动改档位;老对话也立即受影响,不需要重开。
|
|
12
|
+
|
|
3
13
|
## 哪些对话会被自动调档
|
|
4
14
|
|
|
5
15
|
插件在 agent 创建的那一刻接管它的发送动作。所以:
|
|
@@ -8,16 +18,50 @@
|
|
|
8
18
|
- 插件加载时**已经开着**的对话,本次不会被判定(它的 agent 早已创建,不会再触发 `agent/created`);**重启 DSH 后重新打开**它,就开始生效;
|
|
9
19
|
- 子会话(subagent)不判定、不延迟。
|
|
10
20
|
|
|
11
|
-
##
|
|
21
|
+
## 判定输入:一句 + 一张便签
|
|
12
22
|
|
|
13
|
-
|
|
23
|
+
每轮判定只发两样东西,不带对话历史、不带工具结果、不带文件内容:
|
|
14
24
|
|
|
15
|
-
|
|
16
|
-
|
|
17
|
-
|
|
25
|
+
```
|
|
26
|
+
任务便签:<上一轮结束时的便签,最多 200 字;第一轮为空>
|
|
27
|
+
用户新消息:<你刚发的那条>
|
|
28
|
+
```
|
|
29
|
+
|
|
30
|
+
便签由第二轮请求在回答结束后后台更新(同样只看便签 + 你的消息 + 回答的前 1200 字),保留「目标 / 做到哪一步 / 还差什么」这类信息。回答一结束判定就返回,所以便签不占用你的等待时间。
|
|
31
|
+
|
|
32
|
+
判定指令(完整原文):
|
|
18
33
|
|
|
19
34
|
```
|
|
20
|
-
|
|
35
|
+
你是任务难度判定器。输入是任务便签(可能为空)和用户的新消息。
|
|
36
|
+
只判断“回答这条消息需要多强的推理”,不要回答它。
|
|
37
|
+
只输出一个词:off、low、high、max,不要解释、不要标点、不要换行。
|
|
38
|
+
|
|
39
|
+
档位标准:
|
|
40
|
+
off:闲聊、确认、查资料、改配置、跑命令。
|
|
41
|
+
low:小改动、改名、整理;做法已经明确,照着改就行(改语法、改写法、统一命名)。
|
|
42
|
+
high:需要自己拿主意——分析原因、排查报错、解释原理、写较复杂代码、做设计取舍。
|
|
43
|
+
max:架构方案、多方案比较、多因素权衡、疑难 bug 的根因、长链条推理。
|
|
44
|
+
|
|
45
|
+
便签是这轮对话在做的事:
|
|
46
|
+
用户只回“开始”“好的”“继续”,就按便签里没做完的事判,别因为它短就判 off。
|
|
47
|
+
消息和便签无关(换了话题),就当便签是空的。
|
|
21
48
|
```
|
|
22
49
|
|
|
23
|
-
|
|
50
|
+
模型输出上限 8 token,解析到的第一个 `off|low|high|max` 即为结果。
|
|
51
|
+
|
|
52
|
+
## 判定失败与同轮升档
|
|
53
|
+
|
|
54
|
+
- 超时(默认 3 秒)或没有合法档位 → 沿用你手动选的档位;连续 3 次失败 → 自动关闭并提示。
|
|
55
|
+
- 出现 `content[].thinking` 回传错误 → 立刻退回手动并停止本会话的自动切换。
|
|
56
|
+
- 判成 `off`/`low` 但这一轮已到第 3 步还没结束 → 从第 4 步起升到 `high`;只升不降,`high` 不升 `max`。
|
|
57
|
+
|
|
58
|
+
## 每次判定的开销
|
|
59
|
+
|
|
60
|
+
每条**用户消息**两次很小的请求(判定 1 次 + 回答后更新便签 1 次;工具回合不重复判定):
|
|
61
|
+
|
|
62
|
+
- 判定:输入 ≈ 固定说明 + 便签(≤200 字)+ 你的消息;输出 ≤8 token,实际 1 个词;思考关闭。
|
|
63
|
+
- 便签:输入 ≈ 便签 + 你的消息 + 回答前 1200 字;输出 ≤320 token,实际通常 60–150 字。
|
|
64
|
+
|
|
65
|
+
按 2026 年 11 月 `deepseek-flash` 官方价(输入 $0.15/M 闲时、$0.30/M 忙时;输出 $0.60/M 闲时、$1.20/M 忙时)代入,约为 **每条消息 ¥0.0002 – ¥0.001**,100 条消息约 ¥0.03 – ¥0.1。单价以 platform.deepseek.com 为准。
|
|
66
|
+
|
|
67
|
+
判定实测约 0.77 秒(上限 `timeoutMs`,默认 3 秒),之后不再等待;便签更新发生在回答之后,不占你的时间。把「自动选择档位」关掉即完全不花这笔钱。
|
package/index.js
CHANGED
|
@@ -38,19 +38,74 @@ import { appendFileSync } from 'node:fs'
|
|
|
38
38
|
import { homedir } from 'node:os'
|
|
39
39
|
import { join } from 'node:path'
|
|
40
40
|
import { resolveConfig } from './config.js'
|
|
41
|
-
import { judgeLevel } from './judge.js'
|
|
41
|
+
import { judgeLevel, summarizeCard } from './judge.js'
|
|
42
42
|
import { publish, stateOf } from './state.js'
|
|
43
43
|
|
|
44
44
|
const WRAPPED = new WeakSet()
|
|
45
45
|
|
|
46
|
+
export { Config } from './schema.js'
|
|
47
|
+
|
|
48
|
+
/**
|
|
49
|
+
* Live auto/manual switch. `mode` is declared volatile in `schema.js`, so the
|
|
50
|
+
* host hands this plugin a live reference instead of a frozen value: the
|
|
51
|
+
* composer row writes the entry config through the settings service and the
|
|
52
|
+
* reference changes in place, without reloading the plugin and without starting
|
|
53
|
+
* a new conversation.
|
|
54
|
+
*/
|
|
55
|
+
let liveMode = 'manual'
|
|
56
|
+
let settingsCtx = null
|
|
57
|
+
|
|
58
|
+
/** Volatile config fields arrive as references read through `.get()`. */
|
|
59
|
+
function liveValue(value) {
|
|
60
|
+
if (value !== null && typeof value === 'object' && typeof value.get === 'function') return value.get()
|
|
61
|
+
return value
|
|
62
|
+
}
|
|
63
|
+
|
|
64
|
+
/**
|
|
65
|
+
* The switch lives in the host settings document, so it must be read from the
|
|
66
|
+
* settings service on every use: a config edit does not re-run `apply`, and the
|
|
67
|
+
* config object captured at load time never changes.
|
|
68
|
+
*/
|
|
69
|
+
function modeNow() {
|
|
70
|
+
try {
|
|
71
|
+
const rows = settingsCtx?.settings?.describe?.() ?? []
|
|
72
|
+
const mine = rows.find(row => row.ns === 'auto-effort')
|
|
73
|
+
const value = mine?.value?.mode
|
|
74
|
+
if (value === 'auto' || value === 'manual') return value
|
|
75
|
+
} catch {
|
|
76
|
+
// Reading settings is best-effort; fall back to the value loaded at apply.
|
|
77
|
+
}
|
|
78
|
+
return liveMode
|
|
79
|
+
}
|
|
80
|
+
|
|
81
|
+
/** @returns {boolean} whether the plugin should judge user turns right now. */
|
|
82
|
+
function autoMode() {
|
|
83
|
+
return modeNow() === 'auto'
|
|
84
|
+
}
|
|
85
|
+
|
|
86
|
+
/** Turning the composer switch off is how a manual pick is recorded. */
|
|
87
|
+
async function syncSwitchOff(ctx, logger) {
|
|
88
|
+
try {
|
|
89
|
+
if (modeNow() === 'manual') return
|
|
90
|
+
await ctx.settings?.update?.('auto-effort', { mode: 'manual' })
|
|
91
|
+
liveMode = 'manual'
|
|
92
|
+
} catch (error) {
|
|
93
|
+
logger.debug(`could not switch auto off in settings: ${error instanceof Error ? error.message : String(error)}`)
|
|
94
|
+
}
|
|
95
|
+
}
|
|
96
|
+
|
|
97
|
+
|
|
46
98
|
/** The judge needs the host LLM runtime; Cordis only exposes it when declared. */
|
|
47
|
-
export const inject = ['llm']
|
|
99
|
+
export const inject = ['llm', 'settings']
|
|
48
100
|
|
|
49
101
|
/** The exact host-side failure this plugin must never cause. */
|
|
50
102
|
const THINKING_ERROR = /content\[\]\.thinking|thinking mode must be passed back/i
|
|
51
103
|
|
|
52
104
|
const FAILURES_BEFORE_MANUAL = 3
|
|
53
105
|
|
|
106
|
+
/** The background card call may run while the user already reads the answer. */
|
|
107
|
+
const CARD_TIMEOUT_MS = 10_000
|
|
108
|
+
|
|
54
109
|
function makeLogger(ctx, config) {
|
|
55
110
|
let logger = null
|
|
56
111
|
try {
|
|
@@ -112,6 +167,11 @@ function judgeRoute(judgeModel, provider) {
|
|
|
112
167
|
return { provider, model: judgeModel }
|
|
113
168
|
}
|
|
114
169
|
|
|
170
|
+
/**
|
|
171
|
+
* The judge keeps a short "task card" of the conversation (see judge.js): it is
|
|
172
|
+
* the only cross-turn context, so the judge never sees a growing transcript.
|
|
173
|
+
*/
|
|
174
|
+
|
|
115
175
|
function recordFailure(state, config, logger, agent, error) {
|
|
116
176
|
state.failures += 1
|
|
117
177
|
const detail = error instanceof Error ? error.message : String(error)
|
|
@@ -156,21 +216,43 @@ async function decide(ctx, agent, state, config, logger, message) {
|
|
|
156
216
|
publish(agent, state)
|
|
157
217
|
return
|
|
158
218
|
}
|
|
219
|
+
// The user picked a level by hand after this plugin set one: stop touching the
|
|
220
|
+
// conversation (brief requirement — a manual choice always wins).
|
|
221
|
+
if (state.level !== null && current.reasoningEffort !== state.level && !state.tripped) {
|
|
222
|
+
state.auto = false
|
|
223
|
+
state.source = 'manual'
|
|
224
|
+
state.level = current.reasoningEffort ?? state.level
|
|
225
|
+
state.reason = '你手动改了档位,本对话不再自动切换'
|
|
226
|
+
publish(agent, state)
|
|
227
|
+
logger.info(`manual pick detected (${String(current.reasoningEffort)}), auto off for this conversation`)
|
|
228
|
+
syncSwitchOff(ctx, logger)
|
|
229
|
+
return
|
|
230
|
+
}
|
|
159
231
|
const text = promptText(message)
|
|
160
232
|
if (text === null) return
|
|
161
233
|
|
|
162
234
|
state.judgedTurns += 1
|
|
163
|
-
|
|
235
|
+
// Never log message text: the log is a plain file under the user's home and a
|
|
236
|
+
// judge input can be private. Lengths are enough to debug a stale card.
|
|
237
|
+
logger.info(`turn ${state.judgedTurns} judging (${text.length} chars, card=${state.card === '' ? 'empty' : `${state.card.length} chars`})`)
|
|
238
|
+
// The turn about to run: the background card writer reads it after this turn
|
|
239
|
+
// finishes, so the assistant reply can be folded in.
|
|
240
|
+
state.pendingText = text
|
|
241
|
+
state.pendingAssistant = ''
|
|
164
242
|
const route = judgeRoute(config.judgeModel, current.provider)
|
|
243
|
+
const startedJudge = Date.now()
|
|
165
244
|
const judged = await judgeLevel({
|
|
166
245
|
llm: ctx.llm,
|
|
167
246
|
provider: route.provider,
|
|
168
247
|
model: route.model,
|
|
169
248
|
text,
|
|
249
|
+
card: state.card,
|
|
170
250
|
timeoutMs: config.timeoutMs,
|
|
171
251
|
signal: agent.lifetimeSignal ?? undefined,
|
|
172
|
-
debug: config.debug === true ? logger.debug : undefined,
|
|
252
|
+
debug: config.debug === true ? message => logger.debug(`level ${message}`) : undefined,
|
|
173
253
|
})
|
|
254
|
+
state.judgeRoute = route
|
|
255
|
+
logger.debug(`level call took ${Date.now() - startedJudge}ms`)
|
|
174
256
|
|
|
175
257
|
// Brief §2.2: the adapter only knows these four. Anything else is a judge bug.
|
|
176
258
|
if (!config.levels.includes(judged)) {
|
|
@@ -178,23 +260,7 @@ async function decide(ctx, agent, state, config, logger, message) {
|
|
|
178
260
|
}
|
|
179
261
|
// Brief §3.3: optional cap so an automatic run never escalates to maximum.
|
|
180
262
|
const target = config.requireMaxForUpgrade && judged === 'max' ? 'high' : judged
|
|
181
|
-
|
|
182
|
-
if (current.reasoningEffort === target) {
|
|
183
|
-
state.level = target
|
|
184
|
-
state.source = 'auto'
|
|
185
|
-
state.reason = `判断为 ${judged},与当前档位相同,未切换`
|
|
186
|
-
publish(agent, state)
|
|
187
|
-
return
|
|
188
|
-
}
|
|
189
|
-
|
|
190
|
-
// Brief §3.3: anti-flap. The current level must survive minDwellRounds turns.
|
|
191
|
-
if (state.judgedTurns - state.lastChangeTurn < config.minDwellRounds) {
|
|
192
|
-
state.level = current.reasoningEffort ?? state.level
|
|
193
|
-
state.source = state.level === null ? 'manual' : 'auto'
|
|
194
|
-
state.reason = `判断为 ${judged},但为防抖动本轮保持上一档(${state.level ?? '手动'})`
|
|
195
|
-
publish(agent, state)
|
|
196
|
-
return
|
|
197
|
-
}
|
|
263
|
+
state.judgedLevel = judged
|
|
198
264
|
|
|
199
265
|
// The only sanctioned way to move the level: the Agent's own selection, which
|
|
200
266
|
// the host folds into the projection the step preparation reads.
|
|
@@ -203,7 +269,6 @@ async function decide(ctx, agent, state, config, logger, message) {
|
|
|
203
269
|
state.source = 'auto'
|
|
204
270
|
state.reason = `判断为 ${judged},切换到 ${target}`
|
|
205
271
|
state.failures = 0
|
|
206
|
-
state.lastChangeTurn = state.judgedTurns
|
|
207
272
|
publish(agent, state)
|
|
208
273
|
logger.info(`turn ${state.judgedTurns}: ${judged} -> ${target}`)
|
|
209
274
|
}
|
|
@@ -230,10 +295,16 @@ function attach(ctx, agent, config, logger) {
|
|
|
230
295
|
try {
|
|
231
296
|
const previous = agent[method]
|
|
232
297
|
agent[method] = function thinkLevelDispatch(message, ...rest) {
|
|
298
|
+
// A switch turned back on revives a conversation this plugin had stopped.
|
|
299
|
+
if (config.enabled && !state.tripped && !state.auto && autoMode()) state.auto = true
|
|
233
300
|
if (!state.auto || state.tripped || !isUserTurn(message)) {
|
|
234
301
|
logger.debug(`${method} skipped (auto=${state.auto}, tripped=${state.tripped}, kind=${String(message?.source?.kind)})`)
|
|
235
302
|
return original.call(agent, message, ...rest)
|
|
236
303
|
}
|
|
304
|
+
if (!autoMode()) {
|
|
305
|
+
logger.debug(`${method} skipped (mode=${modeNow()})`)
|
|
306
|
+
return original.call(agent, message, ...rest)
|
|
307
|
+
}
|
|
237
308
|
// The controller does not await followup/steer, so returning a promise is
|
|
238
309
|
// fine: judge first, then queue exactly the message the user sent.
|
|
239
310
|
return (async () => {
|
|
@@ -284,8 +355,10 @@ function trip(payload, logger) {
|
|
|
284
355
|
*/
|
|
285
356
|
export function apply(ctx, rawConfig) {
|
|
286
357
|
const config = resolveConfig(rawConfig)
|
|
358
|
+
settingsCtx = ctx
|
|
359
|
+
liveMode = liveValue(config.mode) === 'auto' ? 'auto' : 'manual'
|
|
287
360
|
const logger = makeLogger(ctx, config)
|
|
288
|
-
logger.info(`loaded (enabled=${config.enabled}, judge=${config.judgeModel}, timeout=${config.timeoutMs}ms)`)
|
|
361
|
+
logger.info(`loaded (mode=${liveMode}, enabled=${config.enabled}, judge=${config.judgeModel}, timeout=${config.timeoutMs}ms)`)
|
|
289
362
|
|
|
290
363
|
ctx.on('agent/created', payload => {
|
|
291
364
|
const agent = payload?.agent ?? payload
|
|
@@ -296,6 +369,36 @@ export function apply(ctx, rawConfig) {
|
|
|
296
369
|
}
|
|
297
370
|
})
|
|
298
371
|
|
|
372
|
+
// Same-turn escalation (no cross-turn guessing): a turn that looked light at
|
|
373
|
+
// its start but is already several steps deep at this boundary is heavier than
|
|
374
|
+
// judged. The select lands on the NEXT step of the SAME turn only.
|
|
375
|
+
ctx.on('agent/pre-step', (payload, next) => {
|
|
376
|
+
try {
|
|
377
|
+
const agent = payload?.agent
|
|
378
|
+
const step = payload?.step
|
|
379
|
+
const state = agent !== null && typeof agent === 'object' ? stateOf(agent, config) : null
|
|
380
|
+
logger.debug(`pre-step turn=${payload?.turn} step=${step} current=${String(agent?.model?.current?.()?.reasoningEffort)} judged=${String(state?.judgedLevel)} mode=${autoMode() ? 'auto' : 'manual'}`)
|
|
381
|
+
if (config.escalateAfterSteps > 0 && state !== null && state.auto && !state.tripped && autoMode()
|
|
382
|
+
&& (state.judgedLevel === 'off' || state.judgedLevel === 'low')
|
|
383
|
+
&& typeof step === 'number' && step >= config.escalateAfterSteps) {
|
|
384
|
+
const current = agent.model.current?.()
|
|
385
|
+
const level = current?.reasoningEffort
|
|
386
|
+
if (level !== 'high' && level !== 'max' && current?.provider !== undefined && current?.model !== undefined) {
|
|
387
|
+
agent.model.select({ provider: current.provider, model: current.model, reasoningEffort: 'high' })
|
|
388
|
+
state.level = 'high'
|
|
389
|
+
state.source = 'auto'
|
|
390
|
+
state.reason = `本轮已进行 ${step} 步,比判定(${state.judgedLevel})更重,从第 ${step + 1} 步起升到 high`
|
|
391
|
+
state.escalatedTurn = state.judgedTurns
|
|
392
|
+
publish(agent, state)
|
|
393
|
+
logger.info(`turn ${state.judgedTurns} step ${step}: ${state.judgedLevel} -> high (same-turn escalation)`)
|
|
394
|
+
}
|
|
395
|
+
}
|
|
396
|
+
} catch (error) {
|
|
397
|
+
logger.debug(`pre-step escalation skipped: ${error instanceof Error ? error.message : String(error)}`)
|
|
398
|
+
}
|
|
399
|
+
return next()
|
|
400
|
+
})
|
|
401
|
+
|
|
299
402
|
for (const hook of ['agent/error', 'agent/request-error']) {
|
|
300
403
|
ctx.on(hook, payload => {
|
|
301
404
|
try {
|
|
@@ -305,4 +408,58 @@ export function apply(ctx, rawConfig) {
|
|
|
305
408
|
}
|
|
306
409
|
})
|
|
307
410
|
}
|
|
411
|
+
|
|
412
|
+
// Collect the assistant text of the running turn for the background card
|
|
413
|
+
// writer. Nothing is decided here, so a failure is never visible to the user.
|
|
414
|
+
ctx.on('agent/assistant-stream', payload => {
|
|
415
|
+
try {
|
|
416
|
+
const agent = payload?.agent
|
|
417
|
+
const state = agent !== null && typeof agent === 'object' ? stateOf(agent, config) : null
|
|
418
|
+
if (state === null) return
|
|
419
|
+
const frame = payload?.frame
|
|
420
|
+
const chunk = frame?.chunk ?? frame
|
|
421
|
+
const kind = chunk?.type ?? chunk?.kind
|
|
422
|
+
if ((kind === 'text-delta' || kind === 'text') && typeof chunk?.text === 'string') {
|
|
423
|
+
state.pendingAssistant = (state.pendingAssistant + chunk.text).slice(-4000)
|
|
424
|
+
}
|
|
425
|
+
} catch {
|
|
426
|
+
/* collecting is best effort */
|
|
427
|
+
}
|
|
428
|
+
})
|
|
429
|
+
|
|
430
|
+
// The card is rewritten OUTSIDE the user's blocking path: this listener returns
|
|
431
|
+
// before the call finishes, so the turn never waits for it.
|
|
432
|
+
ctx.on('agent/turn-stopping', payload => {
|
|
433
|
+
try {
|
|
434
|
+
const agent = payload?.agent
|
|
435
|
+
if (agent === null || typeof agent !== 'object' || !autoMode()) return
|
|
436
|
+
const state = stateOf(agent, config)
|
|
437
|
+
if (!state.auto || state.tripped || state.judgeRoute === null || state.pendingText === '') return
|
|
438
|
+
const card = state.card
|
|
439
|
+
const text = state.pendingText
|
|
440
|
+
const assistant = state.pendingAssistant
|
|
441
|
+
const route = state.judgeRoute
|
|
442
|
+
state.pendingText = ''
|
|
443
|
+
state.pendingAssistant = ''
|
|
444
|
+
const started = Date.now()
|
|
445
|
+
summarizeCard({
|
|
446
|
+
llm: ctx.llm,
|
|
447
|
+
provider: route.provider,
|
|
448
|
+
model: route.model,
|
|
449
|
+
card,
|
|
450
|
+
text,
|
|
451
|
+
assistant,
|
|
452
|
+
timeoutMs: CARD_TIMEOUT_MS,
|
|
453
|
+
debug: config.debug === true ? message => logger.debug(`card ${message.replace(/raw="[\s\S]*"/, match => `raw=<${match.length} chars>`)}`) : undefined,
|
|
454
|
+
}).then(
|
|
455
|
+
next => {
|
|
456
|
+
if (next !== '') state.card = next
|
|
457
|
+
logger.debug(`task card updated in ${Date.now() - started}ms (${state.card.length} chars)`)
|
|
458
|
+
},
|
|
459
|
+
error => logger.debug(`task card skipped: ${error instanceof Error ? error.message : String(error)}`),
|
|
460
|
+
)
|
|
461
|
+
} catch (error) {
|
|
462
|
+
logger.debug(`task card skipped: ${error instanceof Error ? error.message : String(error)}`)
|
|
463
|
+
}
|
|
464
|
+
})
|
|
308
465
|
}
|
package/judge.js
CHANGED
|
@@ -1,37 +1,57 @@
|
|
|
1
1
|
/**
|
|
2
|
-
* The judge
|
|
2
|
+
* The judge works in two independent, lightweight LLM calls per turn.
|
|
3
3
|
*
|
|
4
|
-
*
|
|
5
|
-
*
|
|
6
|
-
*
|
|
7
|
-
*
|
|
8
|
-
*
|
|
9
|
-
*
|
|
10
|
-
*
|
|
11
|
-
*
|
|
4
|
+
* 1. `judgeLevel` — the blocking call. Input is the task card (what the
|
|
5
|
+
* conversation is working on) plus the newest user message; output is one
|
|
6
|
+
* word. It runs before the message is queued, so its latency is felt by the
|
|
7
|
+
* user, which is why it stays as short as the brief §2.1 requires.
|
|
8
|
+
* 2. `summarizeCard` — the background call. It runs after the turn has started
|
|
9
|
+
* (fire-and-forget from `agent/turn-stopping`) and rewrites the task card from
|
|
10
|
+
* the user message and the assistant reply. It never blocks a turn.
|
|
11
|
+
*
|
|
12
|
+
* The task card is the only cross-turn context the judge sees: no transcript, no
|
|
13
|
+
* tool output, no file contents.
|
|
12
14
|
*/
|
|
13
15
|
import { randomUUID } from 'node:crypto'
|
|
14
16
|
|
|
15
|
-
/**
|
|
16
|
-
export const
|
|
17
|
-
|
|
18
|
-
|
|
19
|
-
|
|
17
|
+
/** Longest task card we keep, in characters. */
|
|
18
|
+
export const CARD_MAX_CHARS = 200
|
|
19
|
+
|
|
20
|
+
/** Longest assistant reply excerpt the card writer reads. */
|
|
21
|
+
const ASSISTANT_MAX_CHARS = 1200
|
|
22
|
+
|
|
23
|
+
/** Stable prefix for the blocking call: never interpolate per-turn text here. */
|
|
24
|
+
export const LEVEL_SYSTEM = [
|
|
25
|
+
'你是任务难度判定器。输入是任务便签(可能为空)和用户的新消息。',
|
|
26
|
+
'只判断“回答这条消息需要多强的推理”,不要回答它。',
|
|
27
|
+
'只输出一个词:off、low、high、max,不要解释、不要标点、不要换行。',
|
|
20
28
|
].join('\n')
|
|
21
29
|
|
|
22
|
-
/** The four-way rubric from the brief §2.3,
|
|
23
|
-
export const
|
|
24
|
-
'
|
|
25
|
-
'off',
|
|
26
|
-
'low',
|
|
27
|
-
'high',
|
|
28
|
-
'max',
|
|
30
|
+
/** The four-way rubric from the brief §2.3, plus the card rules. */
|
|
31
|
+
export const LEVEL_INSTRUCTION = [
|
|
32
|
+
'档位标准:',
|
|
33
|
+
'off:闲聊、确认、查资料、改配置、跑命令。',
|
|
34
|
+
'low:小改动、改名、整理;做法已经明确,照着改就行(改语法、改写法、统一命名)。',
|
|
35
|
+
'high:需要自己拿主意——分析原因、排查报错、解释原理、写较复杂代码、做设计取舍。',
|
|
36
|
+
'max:架构方案、多方案比较、多因素权衡、疑难 bug 的根因、长链条推理。',
|
|
29
37
|
'',
|
|
30
|
-
'
|
|
31
|
-
'off
|
|
32
|
-
'
|
|
33
|
-
|
|
34
|
-
|
|
38
|
+
'便签是这轮对话在做的事:',
|
|
39
|
+
'用户只回“开始”“好的”“继续”,就按便签里没做完的事判,别因为它短就判 off。',
|
|
40
|
+
'消息和便签无关(换了话题),就当便签是空的。',
|
|
41
|
+
].join('\n')
|
|
42
|
+
|
|
43
|
+
/** Stable prefix for the background card call. */
|
|
44
|
+
export const CARD_SYSTEM = [
|
|
45
|
+
'你维护一张给“难度判定”用的任务便签。',
|
|
46
|
+
'输入是旧便签、用户的新消息、助手对这条消息的回复。',
|
|
47
|
+
'只输出便签正文,不要标题、不要解释、不要换行。',
|
|
48
|
+
].join('\n')
|
|
49
|
+
|
|
50
|
+
export const CARD_INSTRUCTION = [
|
|
51
|
+
`不超过 ${CARD_MAX_CHARS} 字,写清:目标、对象、做到哪一步、还差什么。`,
|
|
52
|
+
'新任务重写,同一件事的追加要求合并进去。',
|
|
53
|
+
'不抄用户原话,不记寒暄和已完成的事,不写文件内容。',
|
|
54
|
+
'这一轮只是闲聊或确认,就原样输出旧便签。',
|
|
35
55
|
].join('\n')
|
|
36
56
|
|
|
37
57
|
const WORD_RE = /\b(off|low|high|max)\b/i
|
|
@@ -61,58 +81,94 @@ function combineSignals(signal, timeoutMs) {
|
|
|
61
81
|
}
|
|
62
82
|
|
|
63
83
|
/**
|
|
64
|
-
*
|
|
84
|
+
* One bounded streaming call; returns the collected text.
|
|
65
85
|
* @param {object} options
|
|
66
|
-
* @param {{ prepareCall: Function }} options.llm - host `ctx.llm` service.
|
|
67
|
-
* @param {string} options.provider - provider of the session's current route.
|
|
68
|
-
* @param {string} options.model - judge model id from config.
|
|
69
|
-
* @param {string} options.text - the newest user message, unchanged.
|
|
70
|
-
* @param {number} options.timeoutMs - end-to-end deadline.
|
|
71
|
-
* @param {AbortSignal} [options.signal] - the turn's signal.
|
|
72
|
-
* @param {(message: string) => void} [options.debug]
|
|
73
|
-
* @returns {Promise<string>} one of off|low|high|max.
|
|
74
86
|
*/
|
|
75
|
-
|
|
87
|
+
async function call({ llm, provider, model, system, prompt, maxTokens, timeoutMs, signal, debug }) {
|
|
76
88
|
if (typeof llm?.prepareCall !== 'function') throw new Error('the host llm service is unavailable')
|
|
77
89
|
const route = { provider, model }
|
|
78
90
|
const bound = combineSignals(signal, timeoutMs)
|
|
79
91
|
try {
|
|
80
|
-
let
|
|
92
|
+
let prepared
|
|
81
93
|
try {
|
|
82
|
-
|
|
94
|
+
prepared = await llm.prepareCall({ ...route, maxTokens, reasoningEffort: 'off' }, bound.signal, controls => controls)
|
|
83
95
|
} catch (error) {
|
|
84
96
|
// A route that cannot run with `off` still answers; drop the effort.
|
|
85
97
|
if (!isUnsupportedEffort(error)) throw error
|
|
86
|
-
|
|
98
|
+
prepared = await llm.prepareCall({ ...route, maxTokens }, bound.signal, controls => controls)
|
|
87
99
|
}
|
|
88
100
|
bound.signal.throwIfAborted()
|
|
89
101
|
const options = Object.freeze({
|
|
90
|
-
...
|
|
102
|
+
...prepared.config,
|
|
91
103
|
messages: [{
|
|
92
104
|
id: randomUUID(),
|
|
93
105
|
role: 'user',
|
|
94
|
-
content: [{ type: 'text', text:
|
|
106
|
+
content: [{ type: 'text', text: prompt }],
|
|
95
107
|
source: { kind: 'user' },
|
|
96
108
|
}],
|
|
97
|
-
system
|
|
109
|
+
system,
|
|
98
110
|
signal: bound.signal,
|
|
99
111
|
})
|
|
100
112
|
let answer = ''
|
|
101
113
|
let finish
|
|
102
|
-
for await (const chunk of
|
|
114
|
+
for await (const chunk of prepared.stream(options)) {
|
|
103
115
|
bound.signal.throwIfAborted()
|
|
104
116
|
if (chunk.type === 'text-delta') answer += chunk.text
|
|
105
117
|
else if (chunk.type === 'finish') finish = chunk.reason
|
|
106
118
|
}
|
|
107
119
|
bound.signal.throwIfAborted()
|
|
108
|
-
debug?.(`
|
|
120
|
+
debug?.(`raw=${JSON.stringify(answer.trim().slice(0, 160))} finish=${JSON.stringify(finish)}`)
|
|
109
121
|
if (finish && finish.kind !== 'stop' && finish.kind !== 'max-tokens') {
|
|
110
122
|
throw new Error(`judge finished with ${finish.kind}`)
|
|
111
123
|
}
|
|
112
|
-
|
|
113
|
-
if (!match) throw new Error(`judge returned no level: ${JSON.stringify(answer.slice(0, 40))}`)
|
|
114
|
-
return match[1].toLowerCase()
|
|
124
|
+
return answer
|
|
115
125
|
} finally {
|
|
116
126
|
bound.dispose()
|
|
117
127
|
}
|
|
118
128
|
}
|
|
129
|
+
|
|
130
|
+
/**
|
|
131
|
+
* The blocking call: ask for the level of this turn.
|
|
132
|
+
* @param {object} options
|
|
133
|
+
* @param {{ prepareCall: Function }} options.llm
|
|
134
|
+
* @param {string} options.provider
|
|
135
|
+
* @param {string} options.model
|
|
136
|
+
* @param {string} options.text - the newest user message, unchanged.
|
|
137
|
+
* @param {string} [options.card] - the task card produced last turn.
|
|
138
|
+
* @param {number} options.timeoutMs
|
|
139
|
+
* @param {AbortSignal} [options.signal]
|
|
140
|
+
* @param {(message: string) => void} [options.debug]
|
|
141
|
+
* @returns {Promise<string>} one of off|low|high|max.
|
|
142
|
+
*/
|
|
143
|
+
export async function judgeLevel({ llm, provider, model, text, card, timeoutMs, signal, debug }) {
|
|
144
|
+
const known = typeof card === 'string' && card.trim() !== '' ? card.trim().slice(0, CARD_MAX_CHARS) : null
|
|
145
|
+
const prompt = `${LEVEL_INSTRUCTION}\n\n任务便签:\n${known ?? '(空,这是本次对话的第一条消息)'}\n\n用户新消息:\n${text}`
|
|
146
|
+
const answer = await call({ llm, provider, model, system: LEVEL_SYSTEM, prompt, maxTokens: 8, timeoutMs, signal, debug })
|
|
147
|
+
const match = WORD_RE.exec(answer)
|
|
148
|
+
if (!match) throw new Error(`judge returned no level: ${JSON.stringify(answer.slice(0, 40))}`)
|
|
149
|
+
return match[1].toLowerCase()
|
|
150
|
+
}
|
|
151
|
+
|
|
152
|
+
/**
|
|
153
|
+
* The background call: rewrite the task card after the turn started.
|
|
154
|
+
* @param {object} options
|
|
155
|
+
* @param {{ prepareCall: Function }} options.llm
|
|
156
|
+
* @param {string} options.provider
|
|
157
|
+
* @param {string} options.model
|
|
158
|
+
* @param {string} [options.card]
|
|
159
|
+
* @param {string} options.text - the user message of the turn that just ran.
|
|
160
|
+
* @param {string} [options.assistant] - what the assistant answered.
|
|
161
|
+
* @param {number} options.timeoutMs
|
|
162
|
+
* @param {AbortSignal} [options.signal]
|
|
163
|
+
* @param {(message: string) => void} [options.debug]
|
|
164
|
+
* @returns {Promise<string>} the new card, empty when the model produced nothing.
|
|
165
|
+
*/
|
|
166
|
+
export async function summarizeCard({ llm, provider, model, card, text, assistant, timeoutMs, signal, debug }) {
|
|
167
|
+
const known = typeof card === 'string' && card.trim() !== '' ? card.trim().slice(0, CARD_MAX_CHARS) : '(空)'
|
|
168
|
+
const reply = typeof assistant === 'string' && assistant.trim() !== ''
|
|
169
|
+
? assistant.trim().slice(0, ASSISTANT_MAX_CHARS)
|
|
170
|
+
: '(这一轮没有文字回复)'
|
|
171
|
+
const prompt = `${CARD_INSTRUCTION}\n\n旧便签:\n${known}\n\n用户新消息:\n${text}\n\n助手回复(节选):\n${reply}`
|
|
172
|
+
const answer = await call({ llm, provider, model, system: CARD_SYSTEM, prompt, maxTokens: 320, timeoutMs, signal, debug })
|
|
173
|
+
return answer.trim().replace(/^card\s*[::]\s*/i, '').replace(/\s+/g, ' ').slice(0, CARD_MAX_CHARS)
|
|
174
|
+
}
|