dsh-ptc-plus 0.4.10 → 0.5.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.en.md +60 -130
- package/README.md +60 -130
- package/assets/dsh-ptc-plus-banner-en.webp +0 -0
- package/assets/dsh-ptc-plus-banner-zh.webp +0 -0
- package/assets/ptc-plus-repl-workspace-en.png +0 -0
- package/assets/ptc-plus-repl-workspace-zh.png +0 -0
- package/assets/ptc-plus-settings-en.png +0 -0
- package/assets/ptc-plus-settings-zh.png +0 -0
- package/client.cjs +42336 -0
- package/cordis.patch.yml +1 -0
- package/docs/README.md +6 -2
- package/docs/adr/0005-temporary-rejected-cell-edit-transport.md +31 -198
- package/docs/adr/0008-define-agent-operating-contract.md +5 -1
- package/docs/adr/0009-bound-expensive-acceptance-default.md +2 -2
- package/docs/adr/0016-archive-resolved-review-findings.md +1 -1
- package/docs/adr/0017-track-the-latest-dsh-public-surface.md +2 -2
- package/docs/adr/0019-plugin-settings-and-kill-switch.md +1 -1
- package/docs/adr/0020-optional-cordis-tools-in-ptc-mode.md +1 -1
- package/docs/adr/0022-stage-npm-releases-with-oidc.md +4 -2
- package/docs/adr/0023-global-user-bindings.md +2 -0
- package/docs/adr/0026-persist-the-delivery-verdict-against-the-verified-candidate.md +2 -0
- package/docs/adr/0028-attach-to-the-host-ptc-execution-seam.md +25 -23
- package/docs/adr/0031-close-semantic-obligations-by-owner.md +1 -1
- package/docs/adr/0034-keep-program-values-separate-from-output-projections.md +29 -0
- package/docs/adr/0035-run-coverage-only-before-a-new-release.md +27 -0
- package/docs/adr/0036-separate-focused-development-from-delivery-verification.md +29 -0
- package/docs/adr/0037-keep-tool-hover-previews-passive.md +52 -0
- package/docs/adr/0038-freeze-session-capability-plans.md +71 -0
- package/docs/adr/0039-verify-documentation-without-runtime-suites.md +29 -0
- package/docs/adr/0040-separate-capability-configurations-from-session-observations.md +39 -0
- package/docs/adr/0041-record-cell-execution-directories.md +65 -0
- package/docs/adr/0042-persist-provider-availability-and-handback.md +41 -0
- package/docs/adr/0043-compose-capability-and-context-policies.md +37 -0
- package/docs/adr/README.md +6 -0
- package/docs/architecture.md +23 -25
- package/docs/capability-planning-components.md +93 -0
- package/docs/capability-projection.md +7 -0
- package/docs/client-ui.md +70 -23
- package/docs/client-visual-contract.md +26 -9
- package/docs/context-management.md +22 -0
- package/docs/context-optimization-plan.md +353 -0
- package/docs/context-public-interface-coverage.md +25 -0
- package/docs/durability-design.md +20 -15
- package/docs/evaluation.md +41 -6
- package/docs/evaluations/command-goat-deepseek-v4.1-flash-20261006.md +126 -0
- package/docs/evaluations/command-goat-deepseek-v4.1-flash.md +281 -0
- package/docs/installation.md +31 -10
- package/docs/program-data-plane.md +5 -3
- package/docs/publishing.md +5 -5
- package/docs/release-coverage.md +73 -0
- package/docs/release-notes.md +1 -1
- package/docs/review-procedure.md +56 -0
- package/docs/runtime-reference.md +48 -16
- package/docs/semantic-validation.md +14 -2
- package/docs/stateful-binding-updates-proposal.md +2 -2
- package/docs/user-bindings.md +1 -1
- package/docs/value-wire.md +11 -1
- package/docs/verification-optimization-work.md +26 -15
- package/docs/verification.md +30 -88
- package/index.js +123 -20
- package/internal/binding-toggle-receipt.js +24 -0
- package/internal/capability-command-receipt.js +29 -0
- package/internal/capability-environment.js +1247 -0
- package/internal/capability-plan.js +285 -0
- package/internal/capability-policy.js +96 -0
- package/internal/capability-statistics-store.js +180 -0
- package/internal/capability-statistics.js +210 -0
- package/internal/config-spec.js +79 -9
- package/internal/direct-surface-owner.js +27 -29
- package/internal/dynamic-environment-runtime.js +9 -9
- package/internal/edit-transport-owner.js +180 -139
- package/internal/execution-provider-owner.js +15 -12
- package/internal/execution-seam-compat.js +8 -8
- package/internal/kernel-worker.js +104 -32
- package/internal/native-root-dynamic.js +12 -13
- package/internal/program-bindings.js +190 -0
- package/internal/recovery-tips.js +3 -2
- package/internal/rejected-cell-editor.js +8 -8
- package/internal/repl-observation-interest.js +11 -3
- package/internal/rpc-contract.js +1 -0
- package/internal/runtime-bridge-owner.js +272 -34
- package/internal/runtime-config.js +58 -0
- package/internal/runtime-contexts.js +74 -5
- package/internal/runtime-intrinsics.js +1 -0
- package/internal/runtime-messages.js +41 -0
- package/internal/sdk-projection.js +10 -82
- package/internal/session-cell-executor.js +42 -19
- package/internal/session-events.js +34 -0
- package/internal/session-journal-recovery.js +234 -11
- package/internal/session-journal-schema.js +6 -3
- package/internal/session-journal.js +31 -2
- package/internal/session-log-view.js +1 -1
- package/internal/session-runtime.js +119 -15
- package/internal/settings-compat.js +12 -5
- package/internal/stateful-module-compiler.js +3 -2
- package/internal/stateful-module-runtime.js +7 -3
- package/internal/stateful-root-runtime.js +9 -4
- package/internal/user-binding-authoring.js +2 -7
- package/internal/user-bindings-owner.js +41 -2
- package/internal/validated-parse-repair.js +1 -1
- package/internal/value-wire.js +68 -21
- package/internal/worker-cwd-virtualization.js +20 -18
- package/internal/worker-realm-surfaces.js +3 -3
- package/package.json +47 -29
- package/src/client-authoring-view.js +67 -30
- package/src/client-binding-review.js +2 -1
- package/src/client-binding-toggle.js +18 -0
- package/src/client-capabilities-view.js +1341 -0
- package/src/client-capability-sort.js +29 -0
- package/src/client-copy.js +871 -8
- package/src/client-feature-gates.js +3 -0
- package/src/client-indicator-view.js +4 -6
- package/src/client-primitives.js +33 -2
- package/src/client-repl-view.js +20 -13
- package/src/client-settings-view.js +411 -41
- package/src/client-styles.js +108 -15
- package/src/client-tool-hover.js +185 -0
- package/src/client-tool-view.js +37 -8
- package/src/client-transport.js +44 -9
- package/src/client-workbench.js +6 -7
- package/src/client.js +76 -22
- package/client.js +0 -32479
|
@@ -6,7 +6,9 @@
|
|
|
6
6
|
|
|
7
7
|
## 日志读取与派生事实
|
|
8
8
|
|
|
9
|
-
当前 Host 的历史读取由 `sessionQuery.observeSession` 提供。runtime bridge 为一次顶层调用取得一个不可变 event/cursor/header 切点,释放 observation lease 后仍消费其不可变 events
|
|
9
|
+
当前 Host 的历史读取由 `sessionQuery.observeSession` 提供。runtime bridge 为一次顶层调用取得一个不可变 event/cursor/header 切点,释放 observation lease 后仍消费其不可变 events;editor 在自身正式 start 切点捕获目标;派生执行开始前另取一个不可变切点,实际 `Session.append` 返回的派生 start 序号拥有该执行身份。各执行的恢复、call identity、provenance 和 no-op 确认只消费其已选切点,不混入较新的 live log。观察切点与当前 surface 不同时,按该切点的公开 surface operations 重建顺序,而不是用较新的 replacement 撤销较早观察的证据。生产路径不调用 `snapshotEvents`、`eventAt` 或 `ownEvents`。
|
|
10
|
+
|
|
11
|
+
新会话的顶层 submission 独立拥有执行上下文。委派子代理即使在父 cell 或原生子执行的能力调用期间启动,也按子会话 ID 读取上述公开 observation 与 ordered surface;不能继承父会话的日志切点、持久调用序号、组合工具结算或原生子执行 handoff。同会话派生 edit 使用新 admission,并只保留已证明的根来源、派生执行序号、不可变观察切点与组合结算;不能复用已释放的根 admission。
|
|
10
12
|
|
|
11
13
|
同步 presentation 从 host-only `ptcPlusSessionLog` 单元读取。该单元与批量 journal fold 共用单事件 timeline 转移;checkpoint 仅保存关联所需的最小 call、规范 journal、身份及当前/历史 runtime records,不复制原始事件日志。pruned result 的非 content 身份保存为确定性 SHA-256 摘要。提示的已见身份、最高 ordinal、cooldown 和成功后的未解决计数在事件进入时更新;装配不扫描历史提示。checkpoint 是可丢弃的派生缓存,不证明 model knowledge、成功 initialization 或 cold replay。状态版本、schema 和 Host 的 watermark/restore 规则共同限定缓存使用,损坏或过时缓存必须重新折叠原日志。
|
|
12
14
|
|
|
@@ -28,7 +30,7 @@ PTC Plus 的正确承诺是:
|
|
|
28
30
|
- restore 命名的 durable 状态可以显式丢弃 volatile 后缀;
|
|
29
31
|
- abort、timeout、worker exit、OOM、进程恢复和损坏的历史 PTC metadata 都收缩到最大可证明 frontier;没有非空 frontier 时从空 REPL 执行当前 cell;
|
|
30
32
|
- 历史恢复拒绝不可信状态证据,不因这项插件自有历史损坏而永久拒绝后续 `run_code`;当前请求自身的 DSH validation、policy、authority、approval 和 cancellation 仍可拒绝执行;
|
|
31
|
-
- 收缩不重新 dispatch、撤销或验证历史 effect
|
|
33
|
+
- 收缩不重新 dispatch、撤销或验证历史 effect;连续收缩期间只发出一次完整 `PTC-R002`,每次实际收缩仍持久化边界。无收缩的 cell 执行后,再次收缩开始新的提示周期。失败 cell 把恢复说明与主错误一起呈现,不混入捕获的控制台输出;
|
|
32
34
|
- 可重建不是默认暴露 binding 的充分条件:恢复 frontier 还必须让 binding 的精确 provenance 保留在生成当前调用的模型可见 surface 中,或由显式选择的有界、模型可见结构化投影公开;不能只为延长隐藏状态而默认注入该投影,raw log 与 UI-only inventory 也不满足这项条件;
|
|
33
35
|
- `durableReplay: false` 时,新 kernel 从空状态开始且所有已执行 live cell 都是 volatile,作为
|
|
34
36
|
用户怀疑恢复正确性时的显式逃生模式;
|
|
@@ -42,7 +44,8 @@ PTC Plus 的正确承诺是:
|
|
|
42
44
|
|
|
43
45
|
```ts
|
|
44
46
|
{
|
|
45
|
-
version:
|
|
47
|
+
version: 11,
|
|
48
|
+
executionCwd: string | null,
|
|
46
49
|
languageSemantics: "legacy-v1" | "stateful-v1" | "protected-v1",
|
|
47
50
|
moduleTransform: "amaro@1.1.11" | "stateful-module-v1+amaro@1.1.11" |
|
|
48
51
|
"stateful-module-v2+amaro@1.1.11" | "protected-module-v1+amaro@1.1.11",
|
|
@@ -107,7 +110,7 @@ SessionRuntime 创建 kernel 时完全忽略历史 nodes、head、checkpoints
|
|
|
107
110
|
|
|
108
111
|
journal、diagnostic、source、cause、call、operation、completion 和 completion error 都使用封闭字段集合;未知、symbol 或非枚举自有字段会使 journal 无效。capability-call `args`/`value` 与 return completion `value` 都是封闭、规范化的 `ptc-value-graph/v1` envelope。诊断结构、source frame 依赖和稳定代码见[架构说明](architecture.md#journal-与恢复)。
|
|
109
112
|
|
|
110
|
-
|
|
113
|
+
当前实现写入 `version: 11` schema,必须携带绝对目录或 `null` 的 `executionCwd`。新 cell 使用 Host 已解析的当前目录,cold replay 使用各 cell 的记录值;v1–v10 迁移为 `null`,保留原 header/worker 解析基准,不使用新请求目录重解释历史。目录变化保留连续 binding 和已加载模块,详见 [ADR 0041](adr/0041-record-cell-execution-directories.md)。v10 起必须携带 `languageSemantics` 与匹配的 `moduleTransform`。v1–v8 固定迁移为 `legacy-v1` 和 legacy transform;v9 保留其语言字段,并按该版本发布时的映射补入 transform:`stateful-v1` 使用 `stateful-module-v1`,`protected-v1` 使用 `protected-module-v1`,`legacy-v1` 使用 legacy transform。当前代际新提交的源码可把 `import.meta` lowering 为记录 session module base 的 cell-local metadata;`legacy-v1` 仍使用冻结 parser 并保留历史拒绝,且此前不可能成功记录的 MetaProperty 不能被恢复时重新解释。恢复不能因当前配置或新 lowering 改变隐藏状态([ADR 0025](adr/0025-use-versioned-logical-binding-identities.md))。v1-v10 作为封闭 predecessor 输入规范化为 v11:v1-v3 的旧 `bindingMode` 精确映射到 `bindingPolicy.variableRedeclarations`,而 `functionClassRedeclarations` 固定为 `false`,因为旧 pipeline 不具备该语义;`moduleSemantics.defaultExportBinding` 固定迁移为 `legacy-variable`,使 loose 历史保留旧的可写生成 binding、strict 历史保留旧的 const 行为;v1-v6 的 `moduleSemantics.importExpressionBoundary` 固定迁移为 `legacy`,使历史写入沿用其记录时的语句 lowering,v7 保留自身记录的边界代际。v1 仍使用三个 rewrite 开关全关的历史默认值,并只在 session log 唯一证明 call identity 时迁移字符串 confirmation;v2/v3 保留自身 `rewritePolicy`。v4/v5 保留已有 binding/rewrite/module 语义;v6 与 v7 保留其记录的 `userBindingsReusePolicy`。v1-v4 的 `userBindingsFingerprint` 固定为 `null`,v5 及之后的代际保留自身记录的快照指纹;v1-v5 的 `userBindingsReusePolicy` 固定为 `fingerprint-v1`。v1-v7 的 `userBindingsShadowPolicy` 固定迁移为 `whole-entry`,`userBindingNames` 固定为 `null`,因此历史 node 继续按整条条目退出,只有 v8 起的 cell 使用 `per-name` 名称证据。无法表示为非负 event sequence 的 confirmation 必须形成 unknown boundary 并触发状态收缩。包括 `executionCwd`、`languageSemantics`、`moduleTransform`、`bindingPolicy`、`rewritePolicy`、`moduleSemantics`、`userBindingsFingerprint`、`userBindingsReusePolicy`、`userBindingsShadowPolicy`、`userBindingNames`、`diagnostics` 在内的当前必需字段缺失或策略未知时 journal 必须失效,否则会削弱最终持久值与 tentative journal 的严格一致性确认;失效 journal 不证明旧 binding,但也不单独成为后续当前调用的 availability gate。profile 后续切换任一 binding 或 AST rewrite 开关只影响新 cell,历史 node 始终按自身记录的 policy、module semantics 与 transform 重放。新旧全局绑定复用策略与覆盖粒度可共存于同一历史,重复规范化不能把旧策略或旧粒度升级为当前值;没有返回值的 completion 不能证明模块内部状态相等。
|
|
111
114
|
|
|
112
115
|
## Capability Call Transcript
|
|
113
116
|
|
|
@@ -143,8 +146,8 @@ pre-execute -> tools/execute -> post-execute
|
|
|
143
146
|
|
|
144
147
|
1. `tools/execute` 保存 execution 对象与 session context;
|
|
145
148
|
2. 执行缝的 `run` 真正进入 kernel 时创建 tentative journal;
|
|
146
|
-
3.
|
|
147
|
-
4. `tools/result` 规范化最终 `meta.dshPtcPlus`
|
|
149
|
+
3. 根成功结果由 `presentationMeta` 投影 journal,失败结果由 around hook 附加;真实派生成功的受限私有凭证见下文;
|
|
150
|
+
4. `tools/result` 规范化最终 `meta.dshPtcPlus` 或已证明的派生私有凭证,与 runtime tentative journal 比较;
|
|
148
151
|
5. 两者语义完全相同时确认 tentative 状态;
|
|
149
152
|
6. cell 已执行但 metadata 缺失、损坏或被替换时,live kernel 单调降为 volatile;
|
|
150
153
|
7. 从未进入 runtime 时,把 call id 加入 pending no-op;
|
|
@@ -165,15 +168,17 @@ pre-execute -> tools/execute -> post-execute
|
|
|
165
168
|
|
|
166
169
|
## Nested run_code
|
|
167
170
|
|
|
168
|
-
模型直接发起的 top-level `run_code`
|
|
171
|
+
模型直接发起的 top-level `run_code` 与程序内 `tools.edit_run_code` 接受后的派生执行都创建本 schema 的 cell journal。根 journal 随 `run_code` result 持久化;派生 journal 随同一根结果的私有 `dshPtcPlusPostEdit` metadata 持久化,并与 editor 及真实派生执行的两个正式 start/settle pairs、根 journal 收据和原始目标关联。`startSeq` 保留 editor 因果身份,`executionStartSeq` 拥有派生 cell 身份;每个 editor 与真实派生 sub-call 身份必须在其根 call 到最终 result 的窗口内各有且只有一个 start 和一个随后 settle,指定 start 序号还须指向这唯一 start。窗口外已结算的相同 ID 可独立复用;派生 settle 的 canonical journal 和 error 状态必须与根私有证据一致。历史顶层 editor result 仍按其原有 derived metadata 契约恢复。隔离 child binding 不创建 child PTC journal,也不产生可合并到父 heap 的 binding。
|
|
172
|
+
|
|
173
|
+
公开 Host 若从未为本次 nested success 调用 `output.presentationMeta`,runtime bridge 可在只读 `projectContent` 中捕获该次 canonical frozen value 与 tentative settlement。只在 execution token、session、parent 和受控 continuation 精确匹配,最初没有插件 journal,最终仍成功且保留同一 value 对象、原 metadata,并且之后没有调用 projector 时,插件才把此私有凭证用于真实派生 settle 和根关联。此路径不修改 Host result、metadata、content 或 contexts;最终 content 始终采用 Host 决定。缺少 capture、block/error、value replacement、已有 metadata 被删除或 projector 已被调用都不得补凭证。根最终 metadata 仍须通过原确认规则;移除它会否定恢复证据。公开依据是 Host canonical value 的冻结与对象身份、`projectContent` 的政策前观察及 `tools/result` 的政策后结算,不是缺少 metadata 本身。
|
|
169
174
|
|
|
170
175
|
父 cell 把隔离 child 当成普通 program binding call,记录 graph-encoded arguments、canonical result/error 和 settlement order。正常结算后,cold replay 返回 recorded child result,不重新执行 child 源码。若取消、超时或 worker failure 发生在调用结算前,`discarded` journal 以 `code.run` 保留 possible-effect boundary。这个结果来自所有 program binding 共用的 pending/settled 生命周期,不是 `code.run` 名称特例。它不注册第二个模型工具,也不伪造 UI、policy hook、调用树或事件。
|
|
171
176
|
|
|
172
177
|
## 日志折叠
|
|
173
178
|
|
|
174
|
-
恢复以当前请求需要的 head 为目标,沿可证明 parent 关系向更早 frontier 收缩;实现可以前向折叠 event 建图,但不得把 unknown boundary 两侧的日志位置当作状态依赖证据。恢复入口先把当前 call id 解析为持久化 `tool/call.seq`,再把该序号作为 live boundary 传入折叠器;对应的在途 call event
|
|
179
|
+
恢复以当前请求需要的 head 为目标,沿可证明 parent 关系向更早 frontier 收缩;实现可以前向折叠 event 建图,但不得把 unknown boundary 两侧的日志位置当作状态依赖证据。恢复入口先把当前 call id 解析为持久化 `tool/call.seq`,再把该序号作为 live boundary 传入折叠器;对应的在途 call event 不属于历史。当前 editor 在正式 `tool/ptc-dispatch-start` 处捕获此前已结算的可编辑目标;根结果证明收据、两个真实 dispatch pairs、canonical 结算事实和目标变换后,以实际派生 start 序号将派生 cell 折叠在根计算之后。历史 direct editor 在其 `tool/call` 处捕获目标。后续 result 不追溯改变已捕获的目标:
|
|
175
180
|
|
|
176
|
-
1.
|
|
181
|
+
1. 用正式 `message.source.callId` 关联 `tool/result` 与 `tool/call`;若 result 另带 `sourceEventSeqs`,它必须准确指向同一个 call,否则该 call 建立 untrusted boundary。历史只携带 sequence relation 的结果按其旧格式校验。每个 ordinary result 在 journal 解析前闭合一次 call settlement;同一 call 的第二个 ordinary result 被拒绝。经过结构校验的 prune replacement 是唯一不占用第二次 ordinary settlement 的替代表示;
|
|
177
182
|
2. 预收集 valid journal 中的 `confirms`;
|
|
178
183
|
3. 排除与 live boundary 序号相同的在途 call,并让已确认 no-op 的无 journal call 不改变状态;
|
|
179
184
|
4. 缺失源码、缺失/损坏 journal 或无法验证的 call/result/prune 关联建立 untrusted boundary,不作为旧 binding 的证据;
|
|
@@ -194,7 +199,7 @@ pre-execute -> tools/execute -> post-execute
|
|
|
194
199
|
|
|
195
200
|
DSH session 同时提供两个不同投影:append-only event log 保存审计、journal 与 replay evidence;ordered surface/derived request history 是模型请求实际可见的消息历史。PTC Plus 的默认状态 frontier 是“结构可重建”和“模型可知”的交集,不能用 raw log 的存在替代 surface evidence。模型可知只采用外部可验证事实,不猜模型内部记忆:可见 assistant tool-call block 必须以相同 call id、tool name 和原始 JSON arguments 精确对应 raw dispatch,或插件/宿主通过有界结构把准确 binding identity 与 provenance 放入模型上下文。只匹配 call id、可见 `tool/result`、result source relation 或 raw log 不能证明声明源码可见。若 ordered surface 缺失、格式错误或不可读取,就没有历史 executable provenance 可证明,恢复必须收缩到空 frontier;这只否定历史 binding 证据,当前有效 cell 仍从空 REPL 正常执行。
|
|
196
201
|
|
|
197
|
-
surface replacement 是否删除状态取决于模型失去了什么,而不取决于是否出现 `compaction/prune` 这个名字。只替换过长 tool result、仍保留 assistant `run_code
|
|
202
|
+
surface replacement 是否删除状态取决于模型失去了什么,而不取决于是否出现 `compaction/prune` 这个名字。只替换过长 tool result、仍保留 assistant `run_code` 调用及源码时,模型仍知道该 cell;当前派生编辑还要求原始目标的源码与发起 nested editor 的根源码同时可见。历史 direct editor 保留其原有 source relation。满足这些关系后才可继续使用对应 journal。若 replacement 或 summary 遮蔽了声明 provenance,raw journal 即使完整也不能让 binding 默认继续存在;自然语言 summary 不解析为依赖或 binding 证明。当前没有模型可见的完整 binding inventory,因此缺少精确 provenance 时按 cell 与依赖后缀收缩。
|
|
198
203
|
|
|
199
204
|
该规则也约束 live worker。structured host session 的 runtime 在每个后续 cell 前都重新读取 DSH public ordered surface,并验证所有现存 live call 的精确 assistant provenance;`replaceGeneration` 没有变化不构成继续复用的证明。generation 改变、缺失或不可读取也进入同一验证。若现存 binding 超出模型可知 frontier,先重置 worker、重建交集 frontier,再执行当前 cell。只有不带 host session 的内部 primitive session id 明确使用 live-only 连续性。Client header card 的 `dshPtcPlusBindings` 只帮助用户检查,不进入模型上下文,也不延长 binding 生命周期;相同 contraction 应让 UI inventory 进入不可确认状态。未来若新增模型可见状态投影,它必须由用户、模型或明确配置选择,有固定 schema、硬预算、append-only runtime-context 证据和明确清除条件,不能仅为延长隐藏状态而默认注入,也不能污染稳定 prompt prefix。
|
|
200
205
|
|
|
@@ -214,7 +219,7 @@ surface replacement 是否删除状态取决于模型失去了什么,而不取
|
|
|
214
219
|
- recovery divergence;
|
|
215
220
|
- durable replay 触发 volatile capability。
|
|
216
221
|
|
|
217
|
-
基础设施失败会终止当前 worker,并把 log-only recovery boundary 放入当前已结算 `tool/result` 的私有 metadata。kernel 从失败 node 的 parent 重新重放;若该 frontier 仍失败则继续向 parent 收缩,直到最大可验证 frontier 或空 REPL。触发恢复的当前 `run_code` 在候选 frontier 验证成功前不执行,验证后在同一次请求中继续。boundary 无效或历史本身无法完整折叠时,相关记录和可能依赖它的状态不得进入 frontier;这类历史 PTC metadata 损坏本身不返回永久 recovery error,而是继续收缩,最差用空 worker 执行当前 cell,并在其最终 result 持久表达新 ancestry。已经写入旧版自定义 boundary event 的日志必须先通过显式、非破坏迁移转换为 DSH 可加载的 metadata;迁移失败时不得覆盖原始日志。
|
|
222
|
+
基础设施失败会终止当前 worker,并把 log-only recovery boundary 放入当前已结算 `tool/result` 的私有 metadata。kernel 从失败 node 的 parent 重新重放;若该 frontier 仍失败则继续向 parent 收缩,直到最大可验证 frontier 或空 REPL。触发恢复的当前 `run_code` 在候选 frontier 验证成功前不执行,验证后在同一次请求中继续。boundary 的失败身份可为顶层 call,或已经由共享 timeline 证明两组 dispatch pairs 和根最终关联的真实派生 start 序号;原始 nested start 不足以证明该身份。收缩保留失败 node 的已证明 parent,之后独立的有效 carrier 仍可进入新 ancestry。boundary 无效或历史本身无法完整折叠时,相关记录和可能依赖它的状态不得进入 frontier;这类历史 PTC metadata 损坏本身不返回永久 recovery error,而是继续收缩,最差用空 worker 执行当前 cell,并在其最终 result 持久表达新 ancestry。已经写入旧版自定义 boundary event 的日志必须先通过显式、非破坏迁移转换为 DSH 可加载的 metadata;迁移失败时不得覆盖原始日志。
|
|
218
223
|
|
|
219
224
|
## State Operations
|
|
220
225
|
|
|
@@ -225,11 +230,11 @@ type StateOperation =
|
|
|
225
230
|
| { action: "delete", name: string }
|
|
226
231
|
```
|
|
227
232
|
|
|
228
|
-
- `save` 只可提交到 durable node;
|
|
233
|
+
- `save` 只可提交到 durable node;volatile 拒绝说明记录的降级原因,保留当前 live binding 并允许继续计算,不推荐无条件 restore;
|
|
229
234
|
- cell 静态判断 durable、但在 `save` 后运行时降级时,tentative save 会被删除;
|
|
230
235
|
- `delete` 可独立应用于命名索引;
|
|
231
|
-
- 无名称 `restore` 选择当前 cell 之前的最后 durable head,清除 volatile suffix;
|
|
232
|
-
- `restore` 把 head 切回已存在的 durable node,清除 volatile suffix
|
|
236
|
+
- 无名称 `restore` 选择当前 cell 之前的最后 durable head,清除 volatile suffix;没有 durable head 时恢复为空 REPL;
|
|
237
|
+
- `restore` 把 head 切回已存在的 durable node,清除 volatile suffix;这会丢弃目标之后的计算状态,但不撤销外部效果或改变启用的全局绑定配置。仍生效的 volatile 输入和 `durableReplay: false` 可使后续 cell 继续 volatile,restore 不保证下一次 save 成功;
|
|
233
238
|
- list 不写 operation,返回当前 checkpoint 名称、`mode` 和首次 `volatileReason`。
|
|
234
239
|
|
|
235
240
|
状态名称由 agent 选择,内部 node index、hash、revision 和日志位置不进入模型接口。
|
|
@@ -265,7 +270,7 @@ state: ...
|
|
|
265
270
|
help: ...
|
|
266
271
|
```
|
|
267
272
|
|
|
268
|
-
|
|
273
|
+
成功 cell 的恢复通知进入 logs;失败 cell 的恢复说明追加到主错误之后,不混入捕获的控制台输出;其他既有诊断提示仍按各自规则呈现。发出的结构化诊断进入当前 journal,可从 session log 重建。连续收缩期间只显示一次完整通知;一个无收缩的 cell 结束提示周期,之后再次收缩会显示新的完整通知。每次实际收缩的边界仍随当前 result 持久化,后续 cold start 从新分支恢复,不重复撞击同一损坏历史。
|
|
269
274
|
|
|
270
275
|
live kernel 首次进入 volatile 只更新 journal 的 `status` / `volatileReason` 和 `repl.state(list)`,不向模型投影 warning/note。该转换不要求当前任务采取行动;只有 cold recovery 已实际跳过 volatile、unknown 或 replay-abandoned 历史时才发送 `PTC-R002`。worker 在首次观察到直接 Node/OS 边界时立即通知主线程,因此后续 hard abort、timeout 或 worker exit 仍能把原因写入 discarded journal。
|
|
271
276
|
|
package/docs/evaluation.md
CHANGED
|
@@ -4,7 +4,7 @@
|
|
|
4
4
|
|
|
5
5
|
The model-backed runners support native Windows paths and WSL repositories mounted as `/mnt/<drive>/...`. They invoke the Windows DSH installation through `pwsh.exe`; the Windows process environment must resolve `dsh --version` and a drive-qualified `DSH_HOME` or user-profile `.dsh` directory. Native Linux and macOS DSH execution are not supported by these runners.
|
|
6
6
|
|
|
7
|
-
Both entrypoints validate path conversion, PowerShell startup, the installed DSH command, and the Windows DSH home before creating evaluation artifacts or installing the development plugin. A public Node preload runs a keyless real-worker probe before the DSH version command: it checks persistent bindings, syntax rejection, synchronous and awaited exceptions, and explicit return. The probe owns and cleans its worker scratch directory. A failure uses `PTC-EVAL-PREREQ` and prevents model work.
|
|
7
|
+
Both entrypoints validate path conversion, PowerShell startup, the installed DSH command, and the Windows DSH home before creating evaluation artifacts or installing the development plugin. A public Node preload runs a keyless real-worker probe before the DSH version command: it checks persistent bindings, syntax rejection, synchronous and awaited exceptions, and explicit return. The launcher consumes its probe request before creating helpers, so inherited preloads cannot recursively repeat it. The probe owns and cleans its worker scratch directory. A failure uses `PTC-EVAL-PREREQ` and prevents model work.
|
|
8
8
|
|
|
9
9
|
The actual `process.version`, `process.execPath`, DSH JavaScript entry, and observed release are recorded in manifests; subsequent DSH invocations use that exact Node executable and entry. Do not launch a probe through `npm exec node`, which can resolve a different runtime. The release is evidence, not a behavior selector or compatibility gate. Keep the selected installation and repository DSH dependencies current.
|
|
10
10
|
|
|
@@ -44,21 +44,33 @@ Self-hosting acceptance against the active PTC Plus checkout is a separate track
|
|
|
44
44
|
|
|
45
45
|
The paired run uses the latest installed DSH release, a configured model, PTC mode's code-only direct-tool projection, and `danger-full-access`. The A/B task set contains ordinary project tasks plus a cheap machine-checkable canary that directly invokes `run_code` once and exercises the transport first. After the canary pair passes, the remaining pairs run with the configured concurrency.
|
|
46
46
|
|
|
47
|
-
Resolved configurations are identical except for `ptc-plus.disabled`. Agent instructions, skills, auxiliary title generation, and local identity extensions are disabled. The model route, permission mode, neutral persona,
|
|
47
|
+
Resolved configurations are identical except for `ptc-plus.disabled`. Agent instructions, skills, auxiliary title generation, and local identity extensions are disabled. The model route, permission mode, neutral persona, fixture bytes, resolved tool surface and initial context unrelated to treatment stay paired within each run. The neutral persona uses only prompt variables every supported Host release registers, so one fixture resolves across the recorded release matrix. When capability management is explicitly enabled in the isolated profile, validated source-backed capability plans must match the resolved capability settings and are treatment context: reports retain their text and usage, while pairing compares the unchanged Host context separately. Workload descriptors must explicitly allow and bound capability plan, navigation and description sources; unknown or malformed source forms fail the audit. Recovered description subsets accumulate while visible, count toward cumulative bounds and may repeat an entry only after its formal surface removal. Statistics should remain disabled for independent manual-baseline comparisons.
|
|
48
48
|
|
|
49
49
|
The shared overlay selects `tools.mode: ptc` and explicitly projects `sandbox-policy.mode` and `approval.policy`. Before any model work, both runners validate the resolved tools row with the public `ToolRuntime.Config` schema from `@deepseek-ai/dsh-tools`, then check the selected evaluation policy. A successful config dump alone does not validate plugin settings; agreement with a runner-supplied expected string cannot bypass the Host schema. `CONFIG_ONLY` also performs this zero-model schema check. The runners do not rely on child-process environment variables to project these policies across the WSL-to-Windows execution boundary.
|
|
50
50
|
|
|
51
51
|
Each arm executes in an independently named, opaque scratch tree outside the evaluator artifact tree. Treatment names are confined to evaluator-owned records. Blind packets replace the session workspace in descriptions, source, results, and final answers, including Windows, forward-slash, and JSON-escaped spellings, before the arm map is written.
|
|
52
52
|
|
|
53
|
-
Machine budgets are per session and are enforced as failures. Machine evidence comes from
|
|
53
|
+
Machine budgets are per session and are enforced as failures. Machine evidence comes from canonical execution values, workspace postconditions, or runner-owned subprocess results. Free-form final answers are never interpreted with prose or negation matching; their correctness remains `blind-pending` until the blind packet is reviewed. The transport canary requires an actual returned string exactly equal to the fixture package name. A JSON-encoded string containing extra quotation marks, a non-string value, console output, and an absent return do not satisfy that condition. Rendered tool text combines logs and values and cannot establish the return type; neither JSON parsing nor the last output line supplies canonical evidence.
|
|
54
|
+
|
|
55
|
+
The A/B overlays load the same evaluation-only Cordis observer through the public loader. It observes `tools/result`, the frozen final outcome after DSH policy and cancellation, without altering execution, tool schemas, model messages, or durable Session events. Bounded sidecars outside the model workspace retain session/call identity, source SHA-256, return presence/type, exact bounded strings, error outcome, and a separate log count; the observer does not read provider configuration, headers, or credential variables. Synchronous writes finish before the observer returns because the public notification does not await listeners. The baseline canary reads this canonical `run_code` body; the plugin canary reads its validated journal completion wire. Canonical observations preserve `valueWire` for JSON reports and expose decoded `value` only as a non-enumerable in-memory property, so even mismatching BigInt or cyclic values remain reportable failures. Missing, corrupt, duplicate, mismatched or bounded-out canonical evidence is `incomplete`, cannot pass machine acceptance, and stops the remaining paid matrix. Older presentation-only logs remain incomplete for this oracle; later evidence does not rewrite their recorded outcomes.
|
|
56
|
+
|
|
57
|
+
The observer is a new profile entry, so each overlay registers it with the public `insert` patch operation. Configuration tests compose the overlay through DSH's public patch composer against a base without the observer; an update targeting an absent entry cannot establish successful loading.
|
|
58
|
+
|
|
59
|
+
Observer identity comes from the public Agent's `session.id`, including real delegated agents. Canonical sidecars remain under the generated run artifact directory after execution scratch cleanup, so family review can inspect child observations as well as root observations. These files are local evaluation evidence and must not be committed.
|
|
54
60
|
|
|
55
61
|
For `test-gate`, a successfully completed runner-owned subprocess is valid evidence whether the tested project exits zero or nonzero. The observed exit code is recorded, while blind semantic review decides whether the answer reports it correctly. Failure to start or complete the oracle remains a runner failure.
|
|
56
62
|
|
|
57
63
|
Acceptance metrics follow durable session-log semantics. `modelRequests` counts `step/start` events, each representing one logical model loop step. `headerEpochs` counts `request/header` events, while `headerChanges` counts the subset whose reason is `change`; initial and equal resume epochs remain distinct epochs without being changes. `historyReplacements` counts events whose `surfaceOp.op` is `replace`. Provider retries can issue multiple adapter attempts within one logical step, and the current durable log does not expose a complete physical-attempt count, so no report field infers one from headers, assistant messages, or usage.
|
|
58
64
|
|
|
59
|
-
Stable-header acceptance canonicalizes empty fields by DSH rules and compares the
|
|
65
|
+
Stable-header acceptance canonicalizes empty fields by DSH rules and compares the ordered complete tool-schema JSON across every header epoch. Current DSH delivers system text through `system/message`, independently of header changes. The logical Session format selects the evidence contract: format 3 introduced formal system messages and retired `header.system`; missing or invalid format evidence cannot authorize a clean result. Absence of system messages in a modern-format log is damage, not a reason to use the historical branch.
|
|
66
|
+
|
|
67
|
+
For modern logs, the shared system audit validates every request header with public `snapshotSessionEvent`; ordered-surface folding alone does not validate header admission. It then uses the public `foldSurface` and `deriveEventMessage` functions, with the maintained public `imageOffloadProjection`, at request headers and assistant message/attempt boundaries to compare the complete ordered system-text vector consumed by the adapter. Durable message identity, source attribution and text-block segmentation remain audit evidence but do not change equal adapter text. Appending a changed or duplicate prompt fails even when no new header is recorded. A byte-identical formal replacement preserves prompt stability; missing, malformed or uninterpretable surface evidence fails. The leading system message supplies the pi-ai prompt, with its text blocks concatenated without inserted separators; later system messages remain history and are still audited. Historical `header.system` remains bounded header evidence for explicitly recorded pre-format-3 logs; it does not certify a modern ordered surface. Both runners use the same audit for persona, SDK and treatment identification, and the A/B runner also uses it to select sessions.
|
|
68
|
+
|
|
69
|
+
Any unapproved header change fails with the first differing field. `headerPolicy.allowedTransitions` names the exact epoch and route, configuration, or capability condition; `headerPolicy.historyReplacements` declares an exact replacement count and never permits header drift or a system-message change. Hashes remain diagnostic report data rather than the correctness test.
|
|
60
70
|
|
|
61
|
-
The shared Host contract recognizes `series` as an unchanged request header beginning a distinct message series, not a new schema or an inferred provider call. Usage evidence supports both historical standalone `assistant/chunk` events and compact streams in `assistant/message` or `assistant/attempt`. An embedded stream's last usage agrees with its message's accounting; formally linked duplicate storage is counted once, conflicting evidence fails, and reported usage from attempts without a surface message remains part of
|
|
71
|
+
The shared Host contract recognizes `series` as an unchanged request header beginning a distinct message series, not a new schema or an inferred provider call. Usage evidence supports both historical standalone `assistant/chunk` events and compact streams in `assistant/message` or `assistant/attempt`. An embedded stream's last usage agrees with its message's accounting; formally linked duplicate storage is counted once, conflicting evidence fails, and reported usage from attempts without a surface message remains part of reported token traffic. Logical steps, header epochs and physical attempts remain distinct. Missing usage is not a zero-cost claim. An adapter can report initialized zero usage after a failed request; this does not prove that the provider charged nothing. Reported usage cannot establish complete billing.
|
|
72
|
+
|
|
73
|
+
Stored logs pass strict public `sessionFormatCatalog` restoration before logical auditing. The original JSONL bytes and physical format are retained separately from restored logical messages and the exact inherited-event cut. Public migrations own historical interpretation; a restoration failure cannot authorize clean evidence. In WSL, Windows logs are restored by the selected Windows Node executable so absolute-path validation uses the Host execution world. Plain `parseEvents` parses JSONL syntax only and does not establish logical provenance. Both paid runners and the Web smoke reader consume restored events. Official `message.isError` outcomes remain errors even without an outer error field.
|
|
62
74
|
|
|
63
75
|
Runtime-context audits track the binding catalog independently of recovery snapshots. They detect unchanged binding text repeated inside an otherwise changed snapshot, while allowing one reassertion after a formal surface replacement removed its evidence. Model-authored ambient declarations are not plugin catalog injections.
|
|
64
76
|
|
|
@@ -66,7 +78,28 @@ Runtime-context audits track the binding catalog independently of recovery snaps
|
|
|
66
78
|
|
|
67
79
|
The typed program-surface descriptor owns `expect.programWork`: `inspectionSymbols` limits schema expansion, and each `callLimits` entry declares a namespace/member's `maxCalls` and optional `maxCallsPerArguments`. Counts include unsuccessful calls. The fixed fixture is immutable during this task, so its native read is limited to one observation; the parent passes the observed marker into the isolated child as source data. Tree retrieval is limited to one, and find/inspect counts are bounded without requiring one exact source or cell sequence.
|
|
68
80
|
|
|
69
|
-
The shared trajectory collector validates the journal and
|
|
81
|
+
The shared trajectory collector validates the journal and exposes decoded nested arguments as `args` and successful results/completions as `value` for in-memory use. These decoded properties are non-enumerable. JSON reports preserve `argsWire` and `valueWire`, canonical PTC Value Graphs that retain undefined, omitted object properties, explicit null, bigint, cycles, shared references and supported non-JSON types. Readers of persisted reports decode these wire fields with the existing value codec; plain JSON values are not substitutes. Auditing uses structured equality, independent of source spelling or object-key order; missing or corrupt evidence fails. `programWork.observations` reports call counts and `repeatedArgumentCalls` separately from `machineMetrics.repeatedSourceCalls`. A workload needing fresh observations must explicitly allow its counts; these limits never suppress runtime calls or establish effect idempotence.
|
|
82
|
+
|
|
83
|
+
### Managed Session Continuity Cohort
|
|
84
|
+
|
|
85
|
+
Four opt-in scenarios are the model-side entry points for capability management, mid-session Host surface changes and delegated sessions. They stay outside the five-workflow default cohort ([ADR 0009](adr/0009-bound-expensive-acceptance-default.md)) and run through the same runner, fixture configuration and audit contract with an explicit selection:
|
|
86
|
+
|
|
87
|
+
```sh
|
|
88
|
+
DSH_PTC_ACCEPTANCE_SCENARIOS=managed-adaptive-surface,managed-session-resume,managed-surface-change,subagent-continuity \
|
|
89
|
+
DSH_PTC_ACCEPTANCE_PROVIDER=<provider> \
|
|
90
|
+
DSH_PTC_ACCEPTANCE_MODEL=<model> \
|
|
91
|
+
DSH_PTC_ACCEPTANCE_API_KEY_ENV=PROVIDER_API_KEY \
|
|
92
|
+
npm run test:expensive
|
|
93
|
+
```
|
|
94
|
+
|
|
95
|
+
The managed variants come from the shared overlay: a frozen capability policy with adaptive allocation over the common baseline, one closed member (`tools.write`) and one explicitly Host-disclosed member (`tools.pwsh`); the surface-change phases additionally disable the `tool-fs` module. Every phase is a separate DSH process, and later phases adopt the recorded session through the Host's public `--session-id` resume, so the cohort exercises the same configuration-change and restoration path an operator uses. Each paid phase consumes the configured route; run the cohort only with explicit consumption authorization, in release preparation or a dedicated QA pass, and expect one single-turn, one two-turn, one three-turn and one root-plus-delegated session.
|
|
96
|
+
|
|
97
|
+
- `managed-adaptive-surface` requires the managed SDK render to declare the Host-disclosed member and the always-callable edit transport while the closed member stays absent from the declaration and the runtime namespace, and the declared member stays callable.
|
|
98
|
+
- `managed-session-resume` requires a routine resume to publish no second capability plan, keep the system prompt byte-stable (zero history replacements) and restore the earlier binding without redeclaration.
|
|
99
|
+
- `managed-surface-change` adds a Host tool after the plan froze, uses it in a durable cell, then removes it and resumes again: the recorded call must still settle, the binding must survive, and no contraction (`PTC-R002`) may be produced.
|
|
100
|
+
- `subagent-continuity` delegates one arithmetic cell to a real subagent and audits the delegated session log — PTC Plus declaration, durable journal, returned product — with the same expectation model.
|
|
101
|
+
|
|
102
|
+
Runtime-context accounting for this cohort allows `capability-plan`, `capability-navigation`, `capability-descriptions`, `capability-contracts` and `runtime-availability` with explicit per-form bounds, and forbids a capability-plan update after the first publication. Host releases differ in details the scenarios deliberately do not depend on: 0.2.0-rc.2 exposes no `working_directory` tool where 0.2.1-alpha releases do, and delegated-session receipts carry different `kind` values. Assertions use stable facts only (session identity, ordered tool schemas, declared members, journal statuses, decoded completions), so a plugin defect stays distinguishable from a Host version difference: a version difference changes the live surface, not the frozen plan, the binding reconstruction or the delegated session's PTC Plus behavior.
|
|
70
103
|
|
|
71
104
|
### Guarded Edit Acceptance
|
|
72
105
|
|
|
@@ -87,6 +120,8 @@ One identity-blinded paired experiment used `opencode-go/deepseek-v4-flash`, the
|
|
|
87
120
|
|
|
88
121
|
In the module-syntax task, PTC Plus completed each repetition in one `run_code`. The other arm failed the static-import requirement in both repetitions, using eight tool calls in total. This stochastic observation is not a performance guarantee. Two PTC Plus sessions and five sessions in the other arm exceeded preset machine budgets, so the overall matrix did not pass machine acceptance. Token traffic includes input, cache-read, cache-write and output tokens. These measurements describe that experiment, not acceptance of subsequent plugin changes.
|
|
89
122
|
|
|
123
|
+
The [2026-10-06 Command Goat / DeepSeek V4.1 Flash report](evaluations/command-goat-deepseek-v4.1-flash-20261006.md) records a separate complete 60-root matrix with four formal children. Both arms completed 24/26 ordinary tasks; PTC Plus's ordinary-family reported traffic was 18.04% higher. Capability probes account for most of its overall completion advantage. The report retains the 6/8 acceptance outcome, the stopped initial pair, exact byte audits, sealed blind scores and cache/billing limits; it does not establish broad savings.
|
|
124
|
+
|
|
90
125
|
## Reporting
|
|
91
126
|
|
|
92
127
|
### Binding Workflow
|
|
@@ -0,0 +1,126 @@
|
|
|
1
|
+
# Command Goat / DeepSeek V4.1 Flash:2026-10-06 实测
|
|
2
|
+
|
|
3
|
+
本轮确认了跨 cell 状态复用、同名修订、静态模块语法和主子会话隔离的可用性,也保留了模型错误与评估链路缺陷。**没有证明普通任务的广泛成本收益**:包含子代理的 13 类普通任务 reported token traffic 高出 18.04%;全部 15 类高出 16.64%。少量任务更省、直接工具错误更少,不能抵消这一总体结果。
|
|
4
|
+
|
|
5
|
+
本报告与[此前同路线实验](command-goat-deepseek-v4.1-flash.md)分开记录。未挑选有利样本覆盖旧记录,也未因结果调整任务或预算。
|
|
6
|
+
|
|
7
|
+
## 配置与比较范围
|
|
8
|
+
|
|
9
|
+
使用 Command Goat 的 `deepseek/deepseek-v4.1-flash`,通过实验开始时 npm `latest` 安装未经修改的官方 DSH。实际记录为 DSH `0.2.0-rc.2`、Windows Node `v24.14.0`;版本仅标识本次证据。隔离 DSH home 和工作区,正常 profile、凭据存储及官方安装保持原样。
|
|
10
|
+
|
|
11
|
+
受测插件运行时来自提交 `c2858ec91dc75e0a637cebb894de4ef00b889da2`,159 份运行时文件的封存 inventory 与实验结束后的字节完全一致。后续评估观察器修复改变证据采集,不改变这批插件运行时。
|
|
12
|
+
|
|
13
|
+
A/B 固定 15 类任务,每组每类两次,共 60 主会话;正式关联 4 个子代理,两组各 2 个。13 类普通任务为主要分层,`package-name` 和 `static-import-cell` 为能力探针。两组的 provider、model、权限、提示、fixture、Git 初始历史及未提交修改相同,解析后配置唯一处理差异是 `ptc-plus.disabled`。双方都通过相同公开结果 observer 采集规范证据。
|
|
14
|
+
|
|
15
|
+
fixture 为 `ab-node-project-v1` / `1.0.0`,内容 SHA-256 为 `807580baf64e367bb2dc047c389edf471e13077d54d50ee4b3a143c38f0cebd1`。其 greeting 测试原本失败;检查任务正确报告这一失败才符合要求,不能擅自改成全绿。
|
|
16
|
+
|
|
17
|
+
任务、预算和 correctness / efficiency / clarity 各 0–3 分的 rubric 在执行前冻结。完成状态与分数分别报告,效率和表达分不能弥补错误答案。普通任务保留全部失败成本;双方均通过质量要求的配对只作次要分层。独立评分者只读取匿名执行材料和最终文件快照,评分与材料 hash 封存后再关联处理组。静态语法及运行行为仍可能提示分组,身份匿名不保证评分者无法猜测处理组。
|
|
18
|
+
|
|
19
|
+
## 主会话与真实子代理验收
|
|
20
|
+
|
|
21
|
+
冻结的八项验收 **6/8 通过**,覆盖 8 主会话和 1 真实子代理。成功项观察到跨 cell 连续计算、同名修订、静态 import、编辑派生执行和状态隔离;它们是对应路径的执行证据,不能推出任意任务均成功。
|
|
22
|
+
|
|
23
|
+
两项失败原样保留:
|
|
24
|
+
|
|
25
|
+
- 能力探索把 tree/find/inspect 集合一次性返回,11,633 字符超过 8,192 字符内容预算。工具可用不等于模型选择了有界探索。
|
|
26
|
+
- 父会话要求返回 `PARENT:42` 标量,模型实际返回包含该值的对象。子代理误用 `code.run({source})`,又在错误报告对象中引用未定义名称;查看接口和保留状态后,正确签名调用成功,返回 BigInt `2n`,四项检查最终均为 true。父会话的“三个 cell、一次 code.run”概括与实际五个 cell、两次尝试不符。恢复成功不抹去错误,也不证明父模型知道子会话中间过程。
|
|
27
|
+
|
|
28
|
+
验收日志完整处理 236,815 存储字节、738,172 解压字节、294 物理行;严格公开恢复后核对 285 个 own 逻辑事件、36 个逻辑模型步骤、27 对直接调用/结果。两个直接错误分别是预设 syntax 拒绝和子代理 ReferenceError;另有一个被 catch 的 `code.run` 参数错误。日志来源、规范 journal、正式主子身份和已结算流/用量检查通过。
|
|
29
|
+
|
|
30
|
+
| 8 主 + 1 子完整家族 | 未缓存输入 | 输出(含 reasoning) | 缓存读取 | 缓存写入 | Token traffic |
|
|
31
|
+
| --- | ---: | ---: | ---: | ---: | ---: |
|
|
32
|
+
| PTC Plus | 85,511 | 11,038 | 263,168 | 0 | 359,717 |
|
|
33
|
+
|
|
34
|
+
## 停止的首对与评估链路修复
|
|
35
|
+
|
|
36
|
+
最初 A/B 只执行一对 canary,两组都实际返回正确包名,但基线规范结果缺失,机器判定 incomplete 并停止其余 58 主会话。这两次调用的 **32,348 reported token traffic** 和原判定均保留,不回写为通过。
|
|
37
|
+
|
|
38
|
+
三处评估实现缺口已在所属入口修正:新增 profile observer 必须使用公开 `insert`,不能更新不存在的条目;真实 Agent 的公开身份来自 `agent.session.id`,而非测试替身虚构的 `sessionId`;子代理规范结果必须留在保留的 artifact 目录,不能随执行 scratch 清理而消失。公开 profile composer、真实 Agent 和官方 Windows Host 的无模型反例检查区分了修复前后,覆盖主子身份、结果来源、policy、嵌套排除、disposer、提示/schema 稳定及清理后留存。
|
|
39
|
+
|
|
40
|
+
随后重新预登记整个 60 主会话矩阵,任务、rubric、预算和插件运行时字节不变;未选择性重跑不利样本。首对记录为 `2026-10-06T091526-618Z-55431bd3`,完整矩阵记录为 `2026-10-06T100016-332Z-d7e129db`。
|
|
41
|
+
|
|
42
|
+
## A/B 完整家族成本
|
|
43
|
+
|
|
44
|
+
Token traffic 是 reported 的未缓存输入、输出、缓存读取、缓存写入之和,包含所有正式子代理的 own events,继承历史不重复计数。它不是费用账单,也不证明完整 HTTP 尝试计数。此矩阵 64 会话均有已结算步骤、完整 reported usage 和完成 turn;质量是否完成另由任务证据判断。
|
|
45
|
+
|
|
46
|
+
| 全部 15 类任务,各 2 次 | PTC Plus | 无插件 DSH PTC 模式 | PTC Plus 差异 |
|
|
47
|
+
| --- | ---: | ---: | ---: |
|
|
48
|
+
| 主 / 子会话 | 30 / 2 | 30 / 2 | — |
|
|
49
|
+
| 逻辑模型步骤 | 178 | 170 | +4.71% |
|
|
50
|
+
| 直接工具调用 | 149 | 138 | +7.97% |
|
|
51
|
+
| 直接工具错误 | 2 | 4 | -2 次 |
|
|
52
|
+
| 未缓存输入 | 399,208 | 359,509 | +11.04% |
|
|
53
|
+
| 输出(含 reasoning) | 101,659 | 102,775 | -1.09% |
|
|
54
|
+
| 缓存读取 | 1,595,264 | 1,334,784 | +19.51% |
|
|
55
|
+
| 缓存写入 | 0 | 0 | — |
|
|
56
|
+
| Token traffic | 2,096,131 | 1,797,068 | +16.64% |
|
|
57
|
+
| 根进程墙钟中位数 | 27.304 秒 | 22.868 秒 | +19.40% |
|
|
58
|
+
| 根进程墙钟合计 | 1,166.056 秒 | 1,176.105 秒 | -0.85% |
|
|
59
|
+
|
|
60
|
+
墙钟包含宿主启动、工具执行和模型等待,重叠的子代理时间不再次相加。合计略低与中位数更高同时成立,不能概括成“普遍更快”。根会话汇总会遗漏 PTC Plus 252,352、基线 240,655 traffic 的子代理成本;本表使用完整家族。
|
|
61
|
+
|
|
62
|
+
普通任务完整成本为 PTC Plus **2,026,483**、基线 **1,716,776**,差异 **+18.04%**。能力探针为 **69,648 / 80,292**,差异 **-13.26%**。预算总计 18 条超限,涉及 PTC Plus 3 个、基线 5 个根会话;它们与任务质量分别统计,整个矩阵机器验收为 fail。
|
|
63
|
+
|
|
64
|
+
## 封存盲评质量与分层结果
|
|
65
|
+
|
|
66
|
+
| 分层 | PTC Plus 通过 / 总数 | 基线通过 / 总数 | Correctness(PTC Plus / 基线) | 完整家族 traffic(PTC Plus / 基线) | 差异 |
|
|
67
|
+
| --- | ---: | ---: | ---: | ---: | ---: |
|
|
68
|
+
| 全部任务 | 28/30 | 25/30 | 83 / 72(各满分 90) | 2,096,131 / 1,797,068 | +16.64% |
|
|
69
|
+
| 普通任务(主要分层) | 24/26 | 24/26 | 71 / 69(各满分 78) | 2,026,483 / 1,716,776 | +18.04% |
|
|
70
|
+
| 能力探针 | 4/4 | 1/4 | 12 / 3(各满分 12) | 69,648 / 80,292 | -13.26% |
|
|
71
|
+
| 双方均通过的普通配对(次要分层) | 23/23 | 23/23 | 64 / 63(各满分 69) | 1,626,447 / 1,407,171 | +15.58% |
|
|
72
|
+
|
|
73
|
+
全部状态为 PTC Plus 28 通过、2 部分完成、0 失败;基线 25 通过、2 部分完成、3 失败,缺证样本为零。普通任务双方各有 2 部分完成,26 个配对 correctness 为 **3 胜、22 平、1 负**;能力探针为 **3 胜、1 平**。普通层 26 对中 8 对 traffic 更低、18 对更高;降低 traffic 且 correctness 未退步的有 7 对。收益没有广泛覆盖普通任务。
|
|
74
|
+
|
|
75
|
+
| 三项 0–3 rubric 合计 | PTC Plus | 基线 |
|
|
76
|
+
| --- | ---: | ---: |
|
|
77
|
+
| 全部 correctness / efficiency / clarity(各满分 90) | 83 / 59 / 76 | 72 / 56 / 71 |
|
|
78
|
+
| 普通 correctness / efficiency / clarity(各满分 78) | 71 / 47 / 64 | 69 / 46 / 62 |
|
|
79
|
+
|
|
80
|
+
评分是固定 rubric 下的描述性判断,不是统计显著性或普遍质量保证;“通过”允许较小解释错误,correctness 满分要求更严格。失败和部分完成有具体执行依据:基线一项包名返回包含额外引号、两项没有在 cell 中完成静态 import;PTC Plus 一项只读委派实际写了 OS 临时脚本,三份整数样本(PTC Plus 1、基线 2)虽精确分摊守恒,却额外把金额转成 Number 做诊断,违背禁止浮点金额转换的要求。不得把这些错误写成插件序列化缺陷。
|
|
81
|
+
|
|
82
|
+
| 任务(每组两次) | 通过数 PTC Plus / 基线 | Correctness(各满分 6) | Traffic PTC Plus / 基线 | 差异 |
|
|
83
|
+
| --- | ---: | ---: | ---: | ---: |
|
|
84
|
+
| package-name | 2 / 1 | 6 / 3 | 35,000 / 29,509 | +18.61% |
|
|
85
|
+
| orientation | 2 / 2 | 6 / 6 | 145,971 / 145,117 | +0.59% |
|
|
86
|
+
| test-check | 2 / 2 | 6 / 6 | 110,336 / 173,856 | -36.54% |
|
|
87
|
+
| worktree-status | 2 / 2 | 6 / 6 | 53,732 / 47,914 | +12.14% |
|
|
88
|
+
| runtime-requirement | 2 / 2 | 6 / 5 | 61,121 / 66,570 | -8.19% |
|
|
89
|
+
| code-explanation | 2 / 2 | 4 / 5 | 111,262 / 124,202 | -10.42% |
|
|
90
|
+
| small-edit | 2 / 2 | 6 / 6 | 71,809 / 63,160 | +13.69% |
|
|
91
|
+
| negative-fact | 2 / 2 | 6 / 6 | 62,291 / 53,488 | +16.46% |
|
|
92
|
+
| static-import-cell | 2 / 0 | 6 / 0 | 34,648 / 50,783 | -31.77% |
|
|
93
|
+
| numeric-analysis | 2 / 2 | 5 / 5 | 176,847 / 49,340 | +258.43% |
|
|
94
|
+
| cross-file-change | 2 / 2 | 6 / 6 | 201,548 / 170,049 | +18.52% |
|
|
95
|
+
| boundary-repair | 2 / 2 | 6 / 6 | 237,121 / 218,349 | +8.60% |
|
|
96
|
+
| successive-calculation | 2 / 2 | 5 / 4 | 92,350 / 55,416 | +66.65% |
|
|
97
|
+
| delegated-review | 1 / 2 | 4 / 4 | 648,159 / 509,807 | +27.14% |
|
|
98
|
+
| precise-integers | 1 / 0 | 5 / 4 | 53,936 / 39,508 | +36.52% |
|
|
99
|
+
|
|
100
|
+
`test-check`、`runtime-requirement`、`code-explanation` 的合计 traffic 较低;不能据此挑掉其余任务。`numeric-analysis` 的额外模块查找和复核、委派任务的宽泛探索与数千次非独立差分测试,均是本轮实际成本。两份 PTC Plus 连续计算样本第二阶段都选择重新构造数据,而非复用前一 cell 的值,因此它们不构成状态复用节省的证据;真正连续性来自前述专门验收。
|
|
101
|
+
|
|
102
|
+
|
|
103
|
+
## 日志、提示和上下文
|
|
104
|
+
|
|
105
|
+
完整矩阵逐字节检查 2,444,071 存储字节、8,677,598 解压字节、3,690 物理行;严格恢复后核对 3,626 own 逻辑事件、348 模型步骤、287 对直接调用/结果及 568 次原生分派。检查覆盖 UTF-8 / JSON、字节范围、正式来源、call/result 身份、流与已结算内容、主子 catalog/header/descriptor 关系和完整家族用量。未出现公开恢复、来源闭合或已结算用量矛盾。
|
|
106
|
+
|
|
107
|
+
直接错误逐项区分:PTC Plus 主会话误用 `fs.createHash`;PTC Plus 子会话未转义报告中的模板字面量;基线一次转义 syntax 错误和三次静态 import 拒绝。前三类是模型代码错误,静态 import 是两组的语言能力差异。没有把预留 greeting 非零退出统称工具错误,也没有把恢复后的成功抹去原错误。未观察到 provider stream failure 或畸形工具 JSON;因此本轮不能评价其修复能力。
|
|
108
|
+
|
|
109
|
+
每个会话的完整有序 system 文本和工具 schema 均保持字节稳定。PTC Plus 为 **30,830 / 3,363** 字节,基线为 **26,954 / 2,130** 字节,分别增加 **3,876 / 1,233** 字节,共 **5,109** 字节固定前缀成本。64 个 runtime snapshot 的插件状态/catalog 段均为空,未观察到此类重复注入;模型重复读取、重建数据和宽泛探索仍消耗上下文。
|
|
110
|
+
|
|
111
|
+
缓存读取占 reported 输入 PTC Plus **79.98%**、基线 **78.78%**。这些计数说明本轮有缓存读取,固定前缀说明会话内字节稳定;它们不能证明插件导致缓存提升、全部 provider 缓存键/TTL 稳定或更低账单。实验没有观察强制 compaction、冷恢复或 provider 重试,相关行为不能由本轮推出。
|
|
112
|
+
|
|
113
|
+
匿名材料在评分前扫描全部文件名、正文和 JSON 键。首次扫描发现两处截断堆栈中的临时工作区 UUID,补齐该身份的匿名投影后重新生成独立材料;原始日志和快照不变,未产生评分或替换样本。修正后的 1,687 份材料、2,946,041 字节通过 728 个已知身份字节形式扫描。源码行为仍可能泄露处理组,盲评结论有此限制。
|
|
114
|
+
|
|
115
|
+
## 证据标识与限制
|
|
116
|
+
|
|
117
|
+
- 冻结任务 SHA-256:`d4293987626bc855c9bcd1a45273ea6bcf068f88b7f1f6a203467a2ac5f3b4fd`。
|
|
118
|
+
- 冻结 rubric SHA-256:`97f5b7da37a644d8c6a4a8dadd0ec086c796832640c51f5bac07b99237eb57a1`。
|
|
119
|
+
- 插件运行时 inventory SHA-256:`f6023506c861356751f666e1d636b86cbde55d4c85c95fbd825fae0b43746903`。
|
|
120
|
+
- 匿名材料 inventory SHA-256:`050646e10ac577f8139123f40cee6dce33e02d9888e42ee8ff6b308e0d53f10e`。
|
|
121
|
+
- 合并评分 SHA-256:`930fc4a446d14ad30c94febe40ad88ae9453bc0527222af9caea409c691553d6`。
|
|
122
|
+
- 封存元数据记录的两个评分分片 SHA-256:`0a29d5ac76bb1ea7cfcf31f681b0d1f46716f52f4e5feddc47f2ca4d39bc110e`、`8bb7f49111992541ee859622cd655422598d71989907f248b40788b9dea301aa`。
|
|
123
|
+
|
|
124
|
+
原始压缩日志、严格恢复记录、逐步审计、最终文件快照、规范 sidecar、合并评分及封存元数据属于保留的本地证据,不随源码提交;上列 hash 标识本次材料,不表示读者仅凭报告就能复算全部原始日志。现有材料可重新计算合并评分和匿名材料 inventory 的哈希;原始评分分片未在保留材料中定位,两个分片哈希仅由封存元数据引用,无法重新计算。验收、停止首对和完整矩阵合计 reported traffic **4,285,264**,三者均保留。
|
|
125
|
+
|
|
126
|
+
样本只有一个 provider/model、一个 fixture、两次重复,不给出普遍性能保证。已观察到的收益是可用的连续计算与模块语言能力,以及本矩阵更少的直接工具错误;普通任务成本收益仍未成立。后续若改提示或执行路径,需要另行预登记新实验,不能改写本轮结论。评估契约见[评测说明](../evaluation.md)。
|