dsh-decision-layer 0.6.1 → 0.6.3
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.en.md +3 -3
- package/README.md +3 -3
- package/lib/client.js +48 -1
- package/package.json +1 -1
- package/service/danger-gate.mjs +9 -1
- package/service/dsh.mjs +1 -0
- package/service/self-check.mjs +16 -3
- package/service/session-state.mjs +9 -0
package/README.en.md
CHANGED
|
@@ -4,7 +4,7 @@
|
|
|
4
4
|
|
|
5
5
|
`dsh-decision-layer` adds structured, explicitly scoped decisions to the DeepSeek Harness (DSH) agent loop, backed by a pluggable adjudication backend. Dangerous-call gating, output self-check, tool narrowing, and loop prevention hook into the agent loop as automatic interventions.
|
|
6
6
|
|
|
7
|
-
> **0.6.
|
|
7
|
+
> **0.6.3 is a preview release.** The plugin works, but how much real value the adjudication interventions add is still being explored — every threshold is an unvalidated heuristic default. Treat it as something to observe, not a proven guarantee of impact. See [ROADMAP.md](<ROADMAP.md>) for scope and progress.
|
|
8
8
|
|
|
9
9
|
## Installation
|
|
10
10
|
|
|
@@ -35,7 +35,7 @@ Restart `dsh web` and refresh after installing or updating: the host and client
|
|
|
35
35
|
## Features
|
|
36
36
|
|
|
37
37
|
- **Manual verdicts** — `decision_evaluate` and the `decision-layer` skill submit `noul`, `score`, or `choice` questions; `decision_check_connection` tests connectivity. Neither counts as an automatic decision or changes agent behavior.
|
|
38
|
-
- **Configurable backend** — defaults to `https://api.typesafe.ai` and `jev-latest`, with per-field URL/key/model overrides under the same protocol. No request without a key. Custom HTTP endpoints require address-bound plaintext-risk confirmation.
|
|
38
|
+
- **Configurable backend** — defaults to `https://api.typesafe.ai` and `jev-latest`, with per-field URL/key/model overrides under the same protocol. No request without a key. Custom HTTP endpoints require address-bound plaintext-risk confirmation. For the full steps to run a local ollama + `tev1` backend (adjudication stays on-machine, nothing sent out), see [Running with a local tev1](<docs/en/deploy-tev1-with-plugin.md>).
|
|
39
39
|
- **Dangerous-call gate** — before a destructive tool call, a redacted summary asks the backend for a verdict, using a **two-tier model**: only a high-confidence deny is enforced monotonically through the guard to block that call; every other case (a high-confidence allow/ask, low confidence, backend failure, or an invalid result) does not intervene and runs the host's original path. The gate is a safety net that only blocks when the model is confident of danger — it never authorizes on the host's behalf or downgrades approval. Rules are configurable.
|
|
40
40
|
- **Output self-check** — scores the final output at turn end against a rubric (0–2, a probability-weighted mean across levels, so it can be fractional). Observe-only by default; an opt-in low-score re-prompt appends one prompt on your behalf when the score is low and the model is confident, at most once per turn.
|
|
41
41
|
- **Task-completion review** — when the user spells out multiple delivery conditions (numbered / bulleted / multi-command), each is judged satisfied / unsatisfied / insufficient-evidence via a choice before turn end, with tool evidence primary and self-reported replies not counting as evidence; extraction is deterministic and never infers implicit expectations. Observe-only by default, steer configurable, coexisting with self-check and tracked as separate metrics.
|
|
@@ -50,4 +50,4 @@ Restart `dsh web` and refresh after installing or updating: the host and client
|
|
|
50
50
|
|
|
51
51
|
## Safety boundary and development
|
|
52
52
|
|
|
53
|
-
Automatic intervention only subtracts: the gate only denies, narrowing only removes tools, and neither grants new permissions. The gate defers to native DSH approval on backend failure rather than allowing; self-check, narrowing, and loop-guard errors skip intervention. The redacted summary sent to the backend carries only tool name and command/path with secret redaction — never file content. Node.js `>=20.11`; `npm test` and `npm run build:client` run local checks. Rebuild and commit `lib/client.js` with any client change. 0.6.
|
|
53
|
+
Automatic intervention only subtracts: the gate only denies, narrowing only removes tools, and neither grants new permissions. The gate defers to native DSH approval on backend failure rather than allowing; self-check, narrowing, and loop-guard errors skip intervention. The redacted summary sent to the backend carries only tool name and command/path with secret redaction — never file content. Node.js `>=20.11`; `npm test` and `npm run build:client` run local checks. Rebuild and commit `lib/client.js` with any client change. 0.6.3 is a preview: the interventions work, but their real value and thresholds are still being explored — feedback on misjudgments and feel observed in the decision log is welcome.
|
package/README.md
CHANGED
|
@@ -4,7 +4,7 @@
|
|
|
4
4
|
|
|
5
5
|
DeepSeek Harness(DSH)的结构化裁决插件。它把有明确判断标准的问题交给同协议的裁决后端,供 agent 在需要时使用,并在 agent loop 上接入危险门控、产出自检、工具收窄和防循环等自动介入。
|
|
6
6
|
|
|
7
|
-
> **0.6.
|
|
7
|
+
> **0.6.3 是体验版。** 插件功能已可用,但裁决介入能带来多少实际价值仍在探索中——各决策点的阈值都是未经验证的启发式默认值,建议先以观察心态体验,别当成经过验证的效果保证。功能与后续进度以 [ROADMAP.md](<ROADMAP.md>) 为准。
|
|
8
8
|
|
|
9
9
|
## 安装
|
|
10
10
|
|
|
@@ -35,7 +35,7 @@ dsh plugin --profile web add file:/绝对路径/dsh-decision-layer
|
|
|
35
35
|
## 功能
|
|
36
36
|
|
|
37
37
|
- **手动裁决**:`decision_evaluate` 工具与 `decision-layer` skill 支持显式提交 `noul`、`score`、`choice` 问题,`decision_check_connection` 做连通测试。手动调用和连通测试不计入自动裁决指标,也不改变 agent 行为。
|
|
38
|
-
- **可配置后端**:默认 `https://api.typesafe.ai`、`jev-latest`,URL、Key、model 可逐字段覆盖;仅支持相同的结构化裁决协议。未配置 Key 时不发送请求。自定义 HTTP
|
|
38
|
+
- **可配置后端**:默认 `https://api.typesafe.ai`、`jev-latest`,URL、Key、model 可逐字段覆盖;仅支持相同的结构化裁决协议。未配置 Key 时不发送请求。自定义 HTTP 地址需针对实际目标确认明文传输风险,地址变更须重新确认。用本地 ollama + `tev1` 作为后端(裁决不出本机、无数据外发)的完整部署步骤见 [用本地 tev1 配合插件(部署指南)](<docs/zh/deploy-tev1-with-plugin.md>)。
|
|
39
39
|
- **危险门控**:破坏性工具调用执行前用脱敏摘要请后端裁决,采用**两档模型**——只有模型高置信判定「拒绝」时才经监控守卫单调拦截该调用;其余情况(高置信放行 / 询问、低置信、后端失败或响应不合法)一律不介入,按宿主原路径执行。门控只做「确信危险才拦」的安全网,不替宿主授权、不降级审批。危险规则可增删配置。
|
|
40
40
|
- **产出自检**:回合结束用评分量表给最终产出打分(0–2,按各档位概率加权,可能是小数)。默认只提示、不打断;可开启「低分补完」,得分偏低且模型有把握时以你的名义追加一次补完提示,同一回合最多补一次。
|
|
41
41
|
- **任务完成核对**:当用户明确列出多个交付条件(编号 / 项目符号 / 多命令)时,回合结束前用单选逐项判定「已满足 / 未满足 / 证据不足」,工具证据优先、回复自述不算证据;确定性提取、不推断隐含期望。默认只观察、可配 steer,与产出自检并存、指标分列。
|
|
@@ -54,4 +54,4 @@ dsh plugin --profile web add file:/绝对路径/dsh-decision-layer
|
|
|
54
54
|
|
|
55
55
|
## 开发
|
|
56
56
|
|
|
57
|
-
服务端使用 Node.js `>=20.11`(`.mjs`,无需构建);客户端用 TypeScript + esbuild 构建。`npm test` 跑本地测试,`npm run build:client` 生成 `lib/client.js`。改动客户端后需重建并连同 `lib/client.js` 一起提交,否则 GitHub 安装拿到的是旧 bundle。0.6.
|
|
57
|
+
服务端使用 Node.js `>=20.11`(`.mjs`,无需构建);客户端用 TypeScript + esbuild 构建。`npm test` 跑本地测试,`npm run build:client` 生成 `lib/client.js`。改动客户端后需重建并连同 `lib/client.js` 一起提交,否则 GitHub 安装拿到的是旧 bundle。0.6.3 为体验版:介入能力已可用,但实际价值与阈值仍在探索,欢迎反馈决策日志观察到的误判与体感。
|
package/lib/client.js
CHANGED
|
@@ -87,6 +87,9 @@ var dictionaries = {
|
|
|
87
87
|
logScore: "\u5F97\u5206",
|
|
88
88
|
logConnection: "\u8FDE\u63A5",
|
|
89
89
|
logConfidence: "\u7F6E\u4FE1\u5EA6",
|
|
90
|
+
logCommandLabel: "\u547D\u4EE4\uFF1A",
|
|
91
|
+
logPathLabel: "\u8DEF\u5F84\uFF1A",
|
|
92
|
+
logSubjectLabel: "\u8BF7\u6C42\uFF1A",
|
|
90
93
|
logScoreHint: "\u5F97\u5206\u4E3A\u5404\u6863\u4F4D\u6309\u6982\u7387\u52A0\u6743\u7684\u7ED3\u679C\uFF0C\u53EF\u80FD\u662F\u5C0F\u6570\uFF08\u5982 1.4\uFF09\uFF0C\u6EE1\u5206\u4E3A\u9876\u6863\u53F7 2\u3002",
|
|
91
94
|
logCheckHint: "\u6BCF\u56DE\u5408\u7ED3\u675F\u540E\uFF0C\u628A\u8FD9\u8F6E AI \u7684\u6700\u7EC8\u56DE\u590D\u4EA4\u7ED9\u88C1\u51B3\u6A21\u578B\uFF0C\u6309\u300C\u4E25\u91CD\u9057\u6F0F/\u81EA\u76F8\u77DB\u76FE \u2192 \u6709\u5C0F\u9057\u6F0F \u2192 \u5B8C\u6574\u51C6\u786E\u300D\u4E09\u6863\u6253\u4E00\u4E2A 0\u20132 \u7684\u8D28\u91CF\u5206\uFF0C\u7528\u6765\u53D1\u73B0\u9057\u6F0F\u6216\u77DB\u76FE\u3002",
|
|
92
95
|
logNarrowHint: "\u6BCF\u56DE\u5408\u5F00\u59CB\u65F6\uFF0C\u628A\u8FD9\u8F6E\u4EFB\u52A1\u4E0E\u5F53\u524D\u53EF\u7528\u5DE5\u5177\u9010\u4E2A\u4EA4\u7ED9\u88C1\u51B3\u6A21\u578B\u5224\u65AD\u76F8\u5173\u6027\uFF08noul\uFF09\uFF0C\u6458\u6389\u660E\u663E\u7528\u4E0D\u4E0A\u7684\u5DE5\u5177\uFF0C\u53EA\u5728\u5F53\u524D\u5BBF\u4E3B\u5DF2\u5141\u8BB8\u7684\u96C6\u5408\u5185\u53D6\u4EA4\u96C6\u3001\u7EDD\u4E0D\u6269\u6743\uFF1B\u300C\u4EC5\u89C2\u5BDF\u300D\u53EA\u8BB0\u5F55\u5EFA\u8BAE\uFF0C\u300C\u5DF2\u751F\u6548\u300D\u624D\u771F\u6B63\u5BF9\u672C\u56DE\u5408\u9650\u5236\u5DE5\u5177\u3002",
|
|
@@ -242,6 +245,9 @@ var dictionaries = {
|
|
|
242
245
|
logScore: "score",
|
|
243
246
|
logConnection: "connection",
|
|
244
247
|
logConfidence: "confidence",
|
|
248
|
+
logCommandLabel: "command: ",
|
|
249
|
+
logPathLabel: "path: ",
|
|
250
|
+
logSubjectLabel: "request: ",
|
|
245
251
|
logScoreHint: "The score is a probability-weighted mean across the levels, so it can be fractional (e.g. 1.4); the top score is the highest level number, 2.",
|
|
246
252
|
logCheckHint: "After each turn, the AI\u2019s final reply for that turn is sent to the adjudication model, which grades its quality on three levels (severe omission/contradiction \u2192 minor gaps \u2192 complete and accurate) as a 0\u20132 score, to catch omissions or contradictions.",
|
|
247
253
|
logNarrowHint: "At the start of each turn, the current task and each available tool are sent to the adjudication model (a noul per tool) to judge relevance, dropping tools clearly not needed. It only intersects the host-allowed set and never expands it; observe merely logs the suggestion, enforced actually restricts the tools for this turn.",
|
|
@@ -371,12 +377,24 @@ function failDetail(t, entry) {
|
|
|
371
377
|
if (entry.reason === "low-confidence" && typeof entry.confidence === "number") return `${reason}\uFF08${t("logConfidence")} ${entry.confidence}\uFF09`;
|
|
372
378
|
return reason;
|
|
373
379
|
}
|
|
380
|
+
function gateArgs({ t }, entry) {
|
|
381
|
+
const value = entry.command || entry.path;
|
|
382
|
+
if (!value) return null;
|
|
383
|
+
const label = entry.command ? t("logCommandLabel") : t("logPathLabel");
|
|
384
|
+
const preview = value.length > 160 ? `${value.slice(0, 160)}\u2026` : value;
|
|
385
|
+
return /* @__PURE__ */ (0, import_jsx_runtime.jsx)("code", { className: "decision-log-cmd", title: `${label}${value}`, children: preview });
|
|
386
|
+
}
|
|
374
387
|
function renderGate({ t }, entry) {
|
|
375
388
|
const tool = /* @__PURE__ */ (0, import_jsx_runtime.jsx)("code", { className: "decision-log-tool", children: entry.tool ?? "\u2014" });
|
|
389
|
+
const args = gateArgs({ t }, entry);
|
|
376
390
|
if (entry.outcome === "error") return /* @__PURE__ */ (0, import_jsx_runtime.jsxs)(import_jsx_runtime.Fragment, { children: [
|
|
377
391
|
t("logGate"),
|
|
378
392
|
" \xB7 ",
|
|
379
393
|
tool,
|
|
394
|
+
args ? /* @__PURE__ */ (0, import_jsx_runtime.jsxs)(import_jsx_runtime.Fragment, { children: [
|
|
395
|
+
" ",
|
|
396
|
+
args
|
|
397
|
+
] }) : null,
|
|
380
398
|
" ",
|
|
381
399
|
/* @__PURE__ */ (0, import_jsx_runtime.jsxs)("span", { className: "decision-log-detail", children: [
|
|
382
400
|
t("gatePassThrough"),
|
|
@@ -394,6 +412,10 @@ function renderGate({ t }, entry) {
|
|
|
394
412
|
t("logGate"),
|
|
395
413
|
" \xB7 ",
|
|
396
414
|
tool,
|
|
415
|
+
args ? /* @__PURE__ */ (0, import_jsx_runtime.jsxs)(import_jsx_runtime.Fragment, { children: [
|
|
416
|
+
" ",
|
|
417
|
+
args
|
|
418
|
+
] }) : null,
|
|
397
419
|
" ",
|
|
398
420
|
/* @__PURE__ */ (0, import_jsx_runtime.jsxs)("span", { className: "decision-log-verdict", children: [
|
|
399
421
|
"\u2192 ",
|
|
@@ -418,6 +440,10 @@ function renderGate({ t }, entry) {
|
|
|
418
440
|
t("logGate"),
|
|
419
441
|
" \xB7 ",
|
|
420
442
|
tool,
|
|
443
|
+
args ? /* @__PURE__ */ (0, import_jsx_runtime.jsxs)(import_jsx_runtime.Fragment, { children: [
|
|
444
|
+
" ",
|
|
445
|
+
args
|
|
446
|
+
] }) : null,
|
|
421
447
|
" ",
|
|
422
448
|
/* @__PURE__ */ (0, import_jsx_runtime.jsx)("span", { className: "decision-log-verdict", children: t("gatePassThrough") }),
|
|
423
449
|
suggestion,
|
|
@@ -426,8 +452,13 @@ function renderGate({ t }, entry) {
|
|
|
426
452
|
}
|
|
427
453
|
function renderCheck({ t }, entry) {
|
|
428
454
|
const label = /* @__PURE__ */ (0, import_jsx_runtime.jsx)("span", { title: t("logCheckHint"), children: t("logCheck") });
|
|
455
|
+
const subject = entry.subject ? /* @__PURE__ */ (0, import_jsx_runtime.jsx)("code", { className: "decision-log-cmd", title: `${t("logSubjectLabel")}${entry.subject}`, children: entry.subject.length > 60 ? `${entry.subject.slice(0, 60)}\u2026` : entry.subject }) : null;
|
|
429
456
|
if (entry.outcome === "error") return /* @__PURE__ */ (0, import_jsx_runtime.jsxs)(import_jsx_runtime.Fragment, { children: [
|
|
430
457
|
label,
|
|
458
|
+
subject ? /* @__PURE__ */ (0, import_jsx_runtime.jsxs)(import_jsx_runtime.Fragment, { children: [
|
|
459
|
+
" ",
|
|
460
|
+
subject
|
|
461
|
+
] }) : null,
|
|
431
462
|
" ",
|
|
432
463
|
/* @__PURE__ */ (0, import_jsx_runtime.jsxs)("span", { className: "decision-log-detail", children: [
|
|
433
464
|
t("logEvalFailed"),
|
|
@@ -456,6 +487,10 @@ function renderCheck({ t }, entry) {
|
|
|
456
487
|
] }) : null;
|
|
457
488
|
return /* @__PURE__ */ (0, import_jsx_runtime.jsxs)(import_jsx_runtime.Fragment, { children: [
|
|
458
489
|
label,
|
|
490
|
+
subject ? /* @__PURE__ */ (0, import_jsx_runtime.jsxs)(import_jsx_runtime.Fragment, { children: [
|
|
491
|
+
" ",
|
|
492
|
+
subject
|
|
493
|
+
] }) : null,
|
|
459
494
|
" \xB7 ",
|
|
460
495
|
verdict,
|
|
461
496
|
score,
|
|
@@ -1611,6 +1646,7 @@ var css = `
|
|
|
1611
1646
|
.decision-rate-btn.decision-rate-unsure{color:#c98a1e}
|
|
1612
1647
|
.decision-rate-summary{margin-top:.5rem}
|
|
1613
1648
|
.decision-log-tool{font-size:.76rem;padding:.05rem .35rem;border-radius:5px;background:var(--dsw-alias-bg-layer-1,#f2f4f6);color:var(--dsw-alias-label-primary,#202124)}
|
|
1649
|
+
.decision-log-cmd{font-size:.74rem;padding:.05rem .35rem;border-radius:5px;background:var(--dsw-alias-bg-layer-1,#f2f4f6);color:var(--dsw-alias-label-secondary,#59636e);word-break:break-all}
|
|
1614
1650
|
.decision-log-verdict{font-weight:600}
|
|
1615
1651
|
.decision-log-deny .decision-log-verdict{color:#c0392b}
|
|
1616
1652
|
.decision-log-ok .decision-log-verdict{color:#1f8a70}
|
|
@@ -1713,9 +1749,20 @@ var css = `
|
|
|
1713
1749
|
.decision-analysis-intro{margin:0 0 .4rem}
|
|
1714
1750
|
.decision-analysis-section{border-top:1px solid var(--dsw-alias-border-l2,#eceff1);margin-top:1.1rem;padding-top:1rem}
|
|
1715
1751
|
.decision-analysis-section>h3{font-size:.92rem;margin:0 0 .6rem}
|
|
1716
|
-
.decision-analysis-cards{display:grid;grid-template-columns:repeat(auto-fit,minmax(
|
|
1752
|
+
.decision-analysis-cards{display:grid;grid-template-columns:repeat(auto-fit,minmax(14rem,1fr));gap:.7rem}
|
|
1717
1753
|
.decision-analysis-trend{max-width:26rem}
|
|
1718
1754
|
.decision-analysis-note{margin:.5rem 0 0}
|
|
1755
|
+
/* Analysis-tab cards lay each stat out as a full-width row (label on the left,
|
|
1756
|
+
value and percent on the right) instead of narrow side-by-side columns. A row
|
|
1757
|
+
never depends on column width, so a long label like \u5931\u8D25\u56DE\u9000 or a three-digit
|
|
1758
|
+
count can never wrap or misalign, at any number of cards across. The modal
|
|
1759
|
+
panel keeps the compact column layout via the base rules. */
|
|
1760
|
+
.decision-analysis .decision-card-stats{display:flex;flex-direction:column;gap:0}
|
|
1761
|
+
.decision-analysis .decision-stat{flex-direction:row;align-items:baseline;justify-content:flex-start;gap:.6rem;padding:.4rem .25rem;background:transparent;border-radius:0}
|
|
1762
|
+
.decision-analysis .decision-stat+.decision-stat{border-top:1px solid var(--dsw-alias-border-l2,#eef1f3)}
|
|
1763
|
+
.decision-analysis .decision-stat-label{order:0;flex:1 1 auto;text-align:left;line-height:1.3;white-space:nowrap}
|
|
1764
|
+
.decision-analysis .decision-stat-num{order:1;font-size:1.05rem}
|
|
1765
|
+
.decision-analysis .decision-stat-pct{order:2;min-width:2.9rem;text-align:right}
|
|
1719
1766
|
/* The analysis tab's decision log shares the page scroll (no inner scrollbar);
|
|
1720
1767
|
the modal panel's .decision-log keeps its own capped scroll via the base rule. */
|
|
1721
1768
|
.decision-analysis .decision-log{max-height:none;overflow:visible}
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "dsh-decision-layer",
|
|
3
|
-
"version": "0.6.
|
|
3
|
+
"version": "0.6.3",
|
|
4
4
|
"description": "A structured decision layer for the DeepSeek Harness agent loop: a danger-call gate, output self-check, tool narrowing, and loop guard backed by a pluggable adjudication model.",
|
|
5
5
|
"keywords": [
|
|
6
6
|
"dsh",
|
package/service/danger-gate.mjs
CHANGED
|
@@ -89,7 +89,15 @@ export function createDangerGate({ evaluate, sessions, rules = DEFAULT_DANGEROUS
|
|
|
89
89
|
if (!sessions || typeof id !== 'string') return;
|
|
90
90
|
try { sessions.record(id, 'gate', outcome); } catch { /* metrics cannot change the security decision */ }
|
|
91
91
|
if (typeof sessions.log !== 'function') return;
|
|
92
|
-
|
|
92
|
+
// Attach the same redacted, bounded command/path summary that was sent to the
|
|
93
|
+
// backend, so a human reviewing the log sees WHAT was gated (e.g. the actual
|
|
94
|
+
// `rm -rf ...`), not just the tool name. Secrets are already stripped by
|
|
95
|
+
// summarizeExecution; session-state caps the length for storage.
|
|
96
|
+
const summary = summarizeExecution(exec);
|
|
97
|
+
const args = {};
|
|
98
|
+
if (summary.command) args.command = summary.command;
|
|
99
|
+
if (summary.path) args.path = summary.path;
|
|
100
|
+
try { sessions.log(id, { kind: 'gate', outcome, tool: exec?.name, ...args, ...detail }); }
|
|
93
101
|
catch { /* logging is best effort */ }
|
|
94
102
|
};
|
|
95
103
|
// Two-tier model: the gate intervenes ONLY on a high-confidence `deny`
|
package/service/dsh.mjs
CHANGED
|
@@ -161,6 +161,7 @@ export async function apply(ctx, options = {}) {
|
|
|
161
161
|
return selfCheck.review({
|
|
162
162
|
agent: payload.agent, turn: payload.turn, signal: payload.signal,
|
|
163
163
|
output: lastAssistantText(payload.agent),
|
|
164
|
+
userText: turnUserText(payload.agent),
|
|
164
165
|
});
|
|
165
166
|
});
|
|
166
167
|
if (typeof ctx.effect === 'function') ctx.effect(installCheck);
|
package/service/self-check.mjs
CHANGED
|
@@ -10,6 +10,15 @@ export const DEFAULT_RUBRIC = Object.freeze([
|
|
|
10
10
|
|
|
11
11
|
const STEER_TEXT = '本回合自检得分偏低:请复核是否答到了用户的问题、有无遗漏或自相矛盾,并在需要时补完。';
|
|
12
12
|
|
|
13
|
+
// A short, single-line hint of the request this self-check judged, so a human
|
|
14
|
+
// reviewing the log knows WHICH turn's output was scored (not just "self-check").
|
|
15
|
+
// Best-effort over an untrusted string; whitespace collapses and length is capped.
|
|
16
|
+
function subjectOf(text) {
|
|
17
|
+
if (typeof text !== 'string') return undefined;
|
|
18
|
+
const collapsed = text.replace(/\s+/g, ' ').trim();
|
|
19
|
+
return collapsed ? collapsed.slice(0, 120) : undefined;
|
|
20
|
+
}
|
|
21
|
+
|
|
13
22
|
function boundedRubric(rubric) {
|
|
14
23
|
if (!Array.isArray(rubric)) return DEFAULT_RUBRIC;
|
|
15
24
|
const levels = rubric.filter(item => typeof item === 'string' && item.trim() && item.length <= 256).slice(0, 10);
|
|
@@ -61,21 +70,25 @@ export function createSelfCheck({ evaluate, sessions, steer, settings, minConfid
|
|
|
61
70
|
try { if (sessions.snapshot(id).enabled === false) return { evaluated: false, lowScore: false, steered: false }; }
|
|
62
71
|
catch { return { evaluated: false, lowScore: false, steered: false }; }
|
|
63
72
|
}
|
|
73
|
+
// The request this turn answered, carried onto every log row so a reviewer
|
|
74
|
+
// can tell which turn's output each self-check score belongs to.
|
|
75
|
+
const subject = subjectOf(turn?.userText);
|
|
76
|
+
const subjectDetail = subject ? { subject } : {};
|
|
64
77
|
const { mode, rubric, lowScoreThreshold, trendMinConfidence, trendRun, trendSevereRun } = await resolveSettings();
|
|
65
78
|
let answer;
|
|
66
79
|
try {
|
|
67
80
|
const result = await evaluate(scoreQuestion(output, rubric), { signal: turn.signal });
|
|
68
81
|
answer = result?.answers?.quality;
|
|
69
|
-
} catch (error) { record(turn, 'error', { reason: typeof error?.reason === 'string' ? error.reason : 'unreachable' }); return { evaluated: false, lowScore: false, steered: false }; }
|
|
82
|
+
} catch (error) { record(turn, 'error', { ...subjectDetail, reason: typeof error?.reason === 'string' ? error.reason : 'unreachable' }); return { evaluated: false, lowScore: false, steered: false }; }
|
|
70
83
|
if (!answer || answer.type !== 'score' || typeof answer.score !== 'number' || !Number.isFinite(answer.score)) {
|
|
71
|
-
record(turn, 'error', { reason: 'invalid-response' });
|
|
84
|
+
record(turn, 'error', { ...subjectDetail, reason: 'invalid-response' });
|
|
72
85
|
return { evaluated: false, lowScore: false, steered: false };
|
|
73
86
|
}
|
|
74
87
|
// A low-confidence Score is a legitimate answer, not a failure: the score
|
|
75
88
|
// stays usable and is recorded as ok/low. Confidence only gates the steer,
|
|
76
89
|
// because only interrupting the user warrants the model being sure.
|
|
77
90
|
const confidence = typeof answer.confidence === 'number' && Number.isFinite(answer.confidence) ? answer.confidence : undefined;
|
|
78
|
-
const detail = confidence === undefined ? {} : { confidence };
|
|
91
|
+
const detail = confidence === undefined ? { ...subjectDetail } : { ...subjectDetail, confidence };
|
|
79
92
|
const lowScore = answer.score <= lowScoreThreshold;
|
|
80
93
|
// The trend run advances over every evaluated score; a high-confidence low
|
|
81
94
|
// score is the only thing that grows it. Thresholds flow to session-state.
|
|
@@ -213,6 +213,15 @@ export function createSessionState({ maxSessions = 1024, maxLogEntries = 50, onL
|
|
|
213
213
|
if (typeof entry.suggestion === 'string' && entry.suggestion) record.suggestion = entry.suggestion.slice(0, 16);
|
|
214
214
|
if (typeof entry.action === 'string' && entry.action) record.action = entry.action.slice(0, 16);
|
|
215
215
|
if (typeof entry.reason === 'string' && entry.reason) record.reason = entry.reason.slice(0, 32);
|
|
216
|
+
// A short hint of the request a self-check judged, so a log row says which
|
|
217
|
+
// turn's output was scored (not just "self-check"). Already collapsed upstream.
|
|
218
|
+
if (typeof entry.subject === 'string' && entry.subject.trim()) record.subject = entry.subject.replace(/\s+/g, ' ').trim().slice(0, 120);
|
|
219
|
+
// The redacted command/path summary of a gated call, so the log row can show
|
|
220
|
+
// what was evaluated (not just the tool). Already secret-stripped upstream;
|
|
221
|
+
// kept to a bounded preview here. Newlines collapse to single spaces so one
|
|
222
|
+
// multi-line script stays on one readable row.
|
|
223
|
+
if (typeof entry.command === 'string' && entry.command.trim()) record.command = entry.command.replace(/\s+/g, ' ').trim().slice(0, 512);
|
|
224
|
+
if (typeof entry.path === 'string' && entry.path.trim()) record.path = entry.path.replace(/\s+/g, ' ').trim().slice(0, 512);
|
|
216
225
|
if (typeof entry.mode === 'string' && entry.mode) record.mode = entry.mode.slice(0, 16);
|
|
217
226
|
if (typeof entry.score === 'number' && Number.isFinite(entry.score)) record.score = Math.round(entry.score * 100) / 100;
|
|
218
227
|
if (typeof entry.confidence === 'number' && Number.isFinite(entry.confidence)) record.confidence = Math.round(entry.confidence * 100) / 100;
|