@ckanner/flywheel 0.28.5 → 0.29.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +17 -0
- package/README.en.md +35 -12
- package/README.md +46 -13
- package/bin/flywheel-claude-code-pro-worker.js +16 -0
- package/bin/flywheel.js +4 -1
- package/bin/runtime.js +7 -1
- package/docs/claude-code-pro.md +128 -0
- package/docs/configuration.md +214 -11
- package/docs/en/claude-code-pro.md +145 -0
- package/docs/en/configuration.md +123 -11
- package/docs/en/getting-started.md +26 -16
- package/docs/en/model-policy.md +6 -2
- package/docs/en/session-evaluation.md +120 -0
- package/docs/en/session-recovery.md +81 -0
- package/docs/en/workflow.md +41 -8
- package/docs/getting-started.md +51 -17
- package/docs/model-policy.md +6 -2
- package/docs/session-evaluation.md +120 -0
- package/docs/session-recovery.md +81 -0
- package/docs/workflow.md +92 -20
- package/npm/install.js +5 -5
- package/package.json +17 -9
|
@@ -0,0 +1,81 @@
|
|
|
1
|
+
# Interrupted calls and context quality
|
|
2
|
+
|
|
3
|
+
This guide describes the P3/P4 source implementation. P1/P2 role-history continuation and saved-receipt consumption remain available. The new public contracts require **Codex Pro 0.9.0** or **Claude Code Pro 0.4.0**. Source readiness, npm publication, account access, and measured task outcomes are separate evidence.
|
|
4
|
+
|
|
5
|
+
## Ownership
|
|
6
|
+
|
|
7
|
+
| Layer | Responsibilities |
|
|
8
|
+
|---|---|
|
|
9
|
+
| Go engine | Task state, role/scope isolation, intent, candidate and contract guards, budgets, accounting, receipt consumption, delivery |
|
|
10
|
+
| Flywheel worker v3 | Typed lifecycle frames, Go durable acknowledgements, cancellation; no task decisions |
|
|
11
|
+
| Public machine API | Independent supervisor, private durable journal, native guardian, complete raw receipts, read-only `recoverMachine` |
|
|
12
|
+
| Native Codex / Claude | Model/tool execution, native history, observed configuration and usage |
|
|
13
|
+
|
|
14
|
+
The outer worker protocol is 3. The plugin request/receipt protocol remains 1, with lifecycle version 1. DSH and Kimi retain their existing normal-history and Go saved-receipt paths; the new plugin capabilities do not imply identical P3 support for those backends.
|
|
15
|
+
|
|
16
|
+
## Durable acknowledgement before dispatch
|
|
17
|
+
|
|
18
|
+
Go prepares the task's `.sessions/` parent and saves intent before an invocation. The upstream API exclusively creates the invocation's recovery subdirectory, named from a digest of the invocation ID. Recovery never scans global native history.
|
|
19
|
+
|
|
20
|
+
Typed events include `thread-ready`, `dispatch-intent`, `dispatch-possible`, `turn-accepted`, `terminal`, and `cleanup-confirmed`. Events must reflect actual backend observations:
|
|
21
|
+
|
|
22
|
+
- **Codex** reports its actual thread/configuration before `turn/start`. Go checks the complete permission policy, cwd, model, and identity before acknowledging. A failed acknowledgement prevents the next turn boundary.
|
|
23
|
+
- **Claude** exposes native init only after prompt submission. It supports pre-dispatch intent/possible acknowledgements, but does not invent a pre-prompt thread-ready or native turn-accepted event. Its observed configuration remains a terminal admission check.
|
|
24
|
+
|
|
25
|
+
Go correlates each event, atomically saves and `fsync`s its checkpoint and state, then sends the matching ACK. Missing, early, duplicate, mismatched, or expired ACKs cannot release execution. `persistent:true` describes requested history preservation; it does not prove that an empty thread has a resumable rollout.
|
|
26
|
+
|
|
27
|
+
An independent supervisor owns the native guardian. Worker loss cancels unacknowledged steps and initiates native cleanup. The complete terminal receipt is durable before the terminal notification. A lost notification or final ACK does not rewrite a completed receipt as unexecuted.
|
|
28
|
+
|
|
29
|
+
## Read-only recovery
|
|
30
|
+
|
|
31
|
+
Restarting the same task lets Go use the original adapter version, request/run IDs, directory, and binding to call public `recoverMachine`. It reads exactly one invocation journal: no CLI startup, `thread/resume`, other-history search, or prompt resubmission.
|
|
32
|
+
|
|
33
|
+
| Result | Interpretation |
|
|
34
|
+
|---|---|
|
|
35
|
+
| `terminal` | A complete original receipt is available; validate identity, outcome, cleanup, accounting, and original candidate evidence |
|
|
36
|
+
| `pending` | No durable terminal receipt is available; this is not proof of a live process, and execution stays blocked |
|
|
37
|
+
| `unknown`, corruption, or mismatch | The outcome is unproven; preserve evidence and block replay, including after switching modes or backends |
|
|
38
|
+
|
|
39
|
+
Queries may be repeated. Cross-machine recovery requires the original data and environment and is not promised. Killing the supervisor itself can leave pending evidence. This is not an exactly-once execution guarantee.
|
|
40
|
+
|
|
41
|
+
### A successful receipt also needs original candidate evidence
|
|
42
|
+
|
|
43
|
+
At `cleanup-confirmed`, before acknowledging, Go captures the post-call candidate digest, original role guards, and Execute report. A recovered success may advance the task only with that original proof. A new workspace snapshot or rerun verification after restart cannot substitute for it.
|
|
44
|
+
|
|
45
|
+
A durable receipt without the Go candidate checkpoint may reconcile trustworthy usage while leaving business execution blocked. A proven pre-dispatch rejection may be treated as unexecuted only with confirmed cleanup and an unchanged original candidate and guard. Review/Judge contract and candidate checks remain mandatory.
|
|
46
|
+
|
|
47
|
+
## Accounting and budgets
|
|
48
|
+
|
|
49
|
+
Consumed invocations do not create another model request or call count. If an unknown invocation later yields trusted usage, Go appends an auditable correction tied to the same call, provider, original day, and lifetime totals. Recovery does not recreate an expired daily bucket.
|
|
50
|
+
|
|
51
|
+
State uses CAS. Receipts, state, and metrics must be durable before success. A metrics append followed by failed state CAS does not create a duplicate row on retry; an existing row must still pass file and directory synchronization. Readable bytes alone do not prove durability.
|
|
52
|
+
|
|
53
|
+
Unknown is not zero. Rebuilds, role overrides, new workers, and Quick-to-Full transitions never reset budgets, attempts, or unknown-call blocking. Budgets constrain new dispatches, while a completed result with complete evidence can still be consumed.
|
|
54
|
+
|
|
55
|
+
## Bounded context rebuilds
|
|
56
|
+
|
|
57
|
+
`sessions.roles` can override fresh/scoped independently for Plan, Execute, Review, and Judge. Other roles inherit the global mode. See [session configuration](configuration.md#session-policy-history-reuse-and-invocation-safety).
|
|
58
|
+
|
|
59
|
+
Optional `sessions.quality` supports:
|
|
60
|
+
|
|
61
|
+
| Field | Measured fact |
|
|
62
|
+
|---|---|
|
|
63
|
+
| `max-confirmed-turns` | Confirmed, deduplicated calls in the current role history |
|
|
64
|
+
| `max-unproductive-turns` | Consecutive Execute calls leaving the candidate digest unchanged |
|
|
65
|
+
| `max-run-input-tokens` | The last complete, attributable input token count, not an inferred context-window size |
|
|
66
|
+
|
|
67
|
+
All thresholds default to disabled. They may rebuild only after the previous call has safely ended, been accounted, and been consumed. Unknown or pending results take precedence.
|
|
68
|
+
|
|
69
|
+
A rebuild records its reason and generation and hands over current contracts, workspace, valid reports, verification, and structured feedback. It never imports another role's private transcript or issues an unmetered model-compression call. A returning Reviewer must still inspect the entire current candidate.
|
|
70
|
+
|
|
71
|
+
## Paired P4 evaluation
|
|
72
|
+
|
|
73
|
+
The repository tools pin source and use an independent frozen oracle to compare fresh, Execute-only scoped, and role-scoped policies. The corpus covers first-pass success, L1 repair, Review return, long context, stop/resume, and configuration changes, with repeated trials and rotated strategy order. Reports retain quality, omissions, regressions, repair rounds, repeated findings, elapsed time, known/unknown tokens and cache use, rebuild reasons, and recovery failures. Actual state evidence determines whether a scenario occurred.
|
|
74
|
+
|
|
75
|
+
The offline native fixture exercises the real Go engine, worker v3, and public plugin API. It validates the evaluator and safety behavior, not model quality, cache rates, or savings. Real paired results require an available native CLI, account, permissions, and model. Defaults remain unchanged until that evidence exists. See the [configuration guide](configuration.md) and the help for `scripts/prepare-session-evaluation.js` and `scripts/evaluate-sessions.js`.
|
|
76
|
+
|
|
77
|
+
## Retention and operation
|
|
78
|
+
|
|
79
|
+
Recovery journals have private permissions, exclusive creation, identity/digest checks, and atomic persistence. Existing directories never authorize redispatch. They are task audit data and may contain model output and diagnostics; keep their access and backups with the task. Do not copy credentials or delete uncertain evidence during upgrades.
|
|
80
|
+
|
|
81
|
+
There is currently no automatic journal-retention timer. Archive evidence through the project's task lifecycle after completion or explicit resolution of an unknown result. Deleting a journal is not a recovery procedure. `control status` and doctor are observation surfaces; models and skills never write `state.json`. Resume through the normal Go entry point from the original task directory so the engine can reconcile before making a new call.
|
package/docs/en/workflow.md
CHANGED
|
@@ -4,9 +4,31 @@
|
|
|
4
4
|
|
|
5
5
|
This guide describes the **current source implementation** and a recommended operating workflow. It does not assert that a particular npm version has been published. Check the registry when installing. The engine's local delivery and the host's GitHub operations are separate responsibilities.
|
|
6
6
|
|
|
7
|
+
## 0. Normal entry: describe a goal and launch
|
|
8
|
+
|
|
9
|
+
With a matching CLI/plugin release containing the adaptive workflow, use:
|
|
10
|
+
|
|
11
|
+
```text
|
|
12
|
+
/flywheel-task Fix the identified bug, add regression coverage, and complete verification. mode=auto
|
|
13
|
+
```
|
|
14
|
+
|
|
15
|
+
The current host conversation researches the code, creates or reuses the task integration worktree, prepares contracts and workflow facts, and calls `flywheel_launch`. Go validates eligibility and starts the engine; DSH associates that exact task with the initiating session's Flywheel view. There is normally no need to copy paths or repeat an authorization already given. Check `flywheel control launch --help` before relying on this new entry; older installation commands do not prove its availability.
|
|
16
|
+
|
|
17
|
+
| Choice | Selection and loop |
|
|
18
|
+
|---|---|
|
|
19
|
+
| Auto, the new entry's default | Select Quick from supported contract facts; use Full when facts are insufficient or risks remain |
|
|
20
|
+
| Quick | One milestone and packet: Execute, all required checks, one independent final Review, Git delivery |
|
|
21
|
+
| Full | Plan, packet execution/review, integrated L2/L3, Judge |
|
|
22
|
+
|
|
23
|
+
Quick requires clear scope, a known implementation pattern, complete verification mappings, and a final reviewer able to assess the whole task. Missing L2/L3 needs an explicit inapplicability reason. File count does not establish safety: authentication, permissions, payments, migrations, concurrency, and unresolved design use Full. An explicit Quick request still has to qualify. A verification gap or new risk causes a persisted upgrade to Full while preserving task identity, work, budget, and history.
|
|
24
|
+
|
|
25
|
+
The first-success Quick path has two stage-agent calls, Execute and final Review. Host preparation, internal model/tool turns, and retries are separate. Required coverage, integration/user-path tests, and applicable defect sampling still run against the final candidate; Plan/Judge are recorded as not applicable with reasons, never as fabricated passes.
|
|
26
|
+
|
|
27
|
+
Use `/flywheel-new` when the user only wants a proposal and `/flywheel-run <actual task path>` to start or resume an existing task. Preserve prior implementation and delivery authorization; honor an explicit request to review a proposal first. The sections below explain the individual checkpoints, not mandatory approval stops for the unified entry.
|
|
28
|
+
|
|
7
29
|
## 1. Choose a model for preparation
|
|
8
30
|
|
|
9
|
-
Preparation turns a requirement into a Spec, task packets, acceptance commands, boundaries, and recorded dry-run evidence. `flywheel-
|
|
31
|
+
Preparation turns a requirement into a Spec, task packets, acceptance commands, boundaries, and recorded dry-run evidence. `flywheel-task` and `flywheel-new` use the host's current conversation model. They do not switch it based on `loops.plan`; that binding controls planning and revisions after the Go engine starts.
|
|
10
32
|
|
|
11
33
|
For accounts that expose the named models, these are practical starting points:
|
|
12
34
|
|
|
@@ -73,6 +95,8 @@ A runnable task needs a reviewed `spec.md` and at least one legal `task-packet-*
|
|
|
73
95
|
- Frozen acceptance tests or other oracles, separately protected from implementation tests.
|
|
74
96
|
- Actual dry-run results, with failure output and exit codes where applicable.
|
|
75
97
|
|
|
98
|
+
The unified entry also records `workflow` in the Spec: version, requested mode, qualification facts, code evidence, risks, and reasons for inapplicable checks. See [Workflow policy and recovery](configuration.md#workflow-policy-and-recovery). A model's low-risk label alone is insufficient. Old contracts without this field remain Full.
|
|
99
|
+
|
|
76
100
|
Run the stated commands before recording their results. An intentionally failing baseline can be valid evidence; a guessed passing result cannot. Ordinary implementation and regression tests can be added by the executor. Frozen acceptance and configuration must not be rewritten to make an implementation pass.
|
|
77
101
|
|
|
78
102
|
Review the files, then commit the substantive initial contracts so packet worktrees can obtain them from Git. The following assumes the index has been checked for unrelated staged work:
|
|
@@ -121,16 +145,19 @@ flywheel control status --cwd "$PWD" --task-dir "$PWD/docs/my-feature"
|
|
|
121
145
|
|
|
122
146
|
Preflight does not infer with a model or execute acceptance commands. Its unknown and warning results still need interpretation. Starting the real task invokes the configured models. A control response's `ok` describes the operation; final task completion is in `state.status`.
|
|
123
147
|
|
|
124
|
-
|
|
148
|
+
For the new unified entry, use `control launch ... --mode auto` in place of `start`. It consumes prepared contracts and defaults to Auto. `start` and foreground `run` accept optional `--mode auto|quick|full`; omission retains the contract or persisted choice. Existing legacy state remains Full. Restarting or requesting Auto does not reclassify or downgrade an established Full task.
|
|
149
|
+
|
|
150
|
+
DSH's `flywheel_launch` and `flywheel_control` start associate the supplied `taskDir` and `cwd` with the initiating session. Calling the CLI from a shell alone does not establish a desktop association. For terminal-started tasks or manual opening, enter the actual task directory and project root. The project field may be blank only if the DSH session directory already matches; it is read through public `workingDirectory.get(session)`. A shell `cd` alone does not change that directory. The plugin does not scan or guess tasks.
|
|
125
151
|
|
|
126
152
|
### Working directory by stage
|
|
127
153
|
|
|
128
154
|
| Stage or entry | Working directory | Responsibility |
|
|
129
155
|
|---|---|---|
|
|
130
|
-
| `flywheel-new` preparation | Task integration worktree | Host creates or reuses it according to the skill |
|
|
131
|
-
| Plan | Task integration worktree | Engine uses the startup root to revise the task contracts |
|
|
156
|
+
| `flywheel-task` / `flywheel-new` preparation | Task integration worktree | Host creates or reuses it according to the skill |
|
|
157
|
+
| Full Plan / revision after upgrade | Task integration worktree | Engine uses the startup root to revise the task contracts |
|
|
132
158
|
| Execute, L1, Review | Current packet candidate workspace, normally its packet worktree | Engine creates, records, and reuses that workspace |
|
|
133
|
-
| L2/L3,
|
|
159
|
+
| Quick's applicable L2/L3 | The same packet candidate, before final Review | The engine must not validate a stale integration checkout |
|
|
160
|
+
| Full L2/L3, Judge | Task integration worktree | Engine checks the integrated milestone |
|
|
134
161
|
| Intake, Garden | Selected task's worktree | Host selects the root; Go writes proposals |
|
|
135
162
|
| Status, Stop | Original task directory and execution root | Query or stop the existing task without requiring a new worktree, branch switch, or commit |
|
|
136
163
|
|
|
@@ -138,13 +165,19 @@ Model-call permissions follow `permissions.mode`. Under `role-based`, the defaul
|
|
|
138
165
|
|
|
139
166
|
Every loop does not create its own worktree. The outer task worktree spans the deliverable; a packet workspace spans its execution, acceptance, review, and delivery. Plan commits valid contract revisions in the integration worktree and attempts to fast-forward existing packet worktrees to that checkpoint while preserving implementation work. Conflicts stop for resolution.
|
|
140
167
|
|
|
141
|
-
|
|
168
|
+
Go schedules packets serially and does not need a persistent parent model session to supervise them. `sessions.mode=scoped` permits ordinary repairs to continue the packet's Executor history, and a Reviewer may continue its own independent history for the same review scope. Every re-review still checks the complete current candidate, contract, and applicable requirements; closing old findings is insufficient. Plan history is task/milestone scoped, primary Judge and second opinion are separate, and an explicit Plan `resident-session:false` remains effective. Parallel DAG scheduling and team communication remain outside this implementation.
|
|
169
|
+
|
|
170
|
+
New configuration templates explicitly select scoped mode; old files without the setting retain fresh mode. Fresh disables general role history reuse while preserving the legacy Kimi Plan resident option. Both modes journal supported main-task Codex Pro, DSH, and Kimi calls. Other adapters, mock, and Intake/Garden retain their existing paths. History reuse does not remove verification requirements or add stages to Quick's two-call first-success path. See [Session policy](configuration.md#session-policy-history-reuse-and-invocation-safety).
|
|
171
|
+
|
|
172
|
+
Saved receipt recovery retains the original candidate, contract, and role guard evidence. Re-running sampling or verification cannot replace evidence bound to a pending receipt. Consuming an already completed result makes no new model request and does not bill twice; budgets constrain new dispatches. P3 uses public Codex/Claude lifecycle events, durable Go ACKs, and read-only invocation journals; recovered success still requires the original post-call candidate and role proof. P4 adds role overrides, bounded quality rebuilds, and paired evaluation, with thresholds disabled by default and live effects separately measured. See [recovery and quality](session-recovery.md) and [paired evaluation](session-evaluation.md).
|
|
173
|
+
|
|
174
|
+
`flywheel-verify` runs deterministic acceptance only. Full retains the existing packet L1 / milestone L2/L3 behavior; Quick also collects applicable L2/L3 in the actual candidate workspace. It does **not** call Review or Judge, create worktrees, or perform local delivery. `--only execute` remains a partial Execute/L1 loop. A Quick task's `--only review` performs final Review without delivery; a verification gap, upgrade verdict, or lost Quick eligibility immediately saves the one-way Full checkpoint. The next complete run enters Full. `--only plan` requires Full, so explicitly request `--mode full` when a Quick task needs planning. Resume the complete run to let Go handle the remaining stages and delivery.
|
|
142
175
|
|
|
143
176
|
The bundled skills check task location before preparation, run, resume, and manual loop actions. By default, development starts outside `main`, `master`, a known default branch, or detached HEAD unless the user has explicitly chosen and authorized that location. Direct CLI and GUI actions do not read or execute `SKILL.md`, create the outer task branch, or inherit those host checks. Read-only status and stopping must remain available even when development-entry checks would fail.
|
|
144
177
|
|
|
145
178
|
## 6. Understand local completion before merging the PR
|
|
146
179
|
|
|
147
|
-
A packet is delivered only after
|
|
180
|
+
A packet is delivered only after required checks and independent Review match its current candidate, and Git delivery is confirmed. The engine commits implementation changes, performs `git merge --ff-only` into the integration checkout, persists delivery state, and cleans up only when safe. Failed commits or merges preserve the workspace. Full additionally runs integrated L2/L3 and Judge. Quick runs applicable L2/L3 before a final Review that covers every requirement, acceptance sufficiency, and continued eligibility; it then persists that same candidate.
|
|
148
181
|
|
|
149
182
|
After the complete task passes, inspect the integrated changes and push them:
|
|
150
183
|
|
|
@@ -168,7 +201,7 @@ flywheel control stop --cwd "$PWD" --task-dir "$PWD/docs/my-feature"
|
|
|
168
201
|
|
|
169
202
|
Check the stop result and the actual failure before resuming from the **same** root and task path with `control start` or `flywheel-run`. A startup readiness timeout is a reason to query status and logs, not to launch a duplicate run. A stopped engine does not prove that every separately managed process is gone.
|
|
170
203
|
|
|
171
|
-
|
|
204
|
+
Supported main-task adapters retain invocation facts and terminal receipts. Unknown dispatch, outcome, or cleanup blocks further execution in both session modes, including after a backend change. Confirm the task-owned plugin/app-server/guardian processes have ended when cleanup is uncertain; a fixed delay is not evidence, and process exit alone does not establish a known model outcome. Go must reconcile a trusted receipt against the original identity and evidence. Changing to fresh, switching task directories, resetting state, or retrying through another channel cannot bypass an unresolved invocation. Avoid broad process killing that could terminate unrelated sessions.
|
|
172
205
|
|
|
173
206
|
For a delivery conflict, preserve the packet worktree, inspect Git state, resolve the conflict deliberately, then resume and let the engine revalidate evidence. Do not use forced cleanup or manual state edits as a recovery shortcut. Configuration changes require restarting the engine.
|
|
174
207
|
|
package/docs/getting-started.md
CHANGED
|
@@ -8,12 +8,20 @@
|
|
|
8
8
|
|
|
9
9
|
安装前用 `npm view @ckanner/flywheel version` 确认 registry 的 latest 已更新到所需版本;源码合并或 GitHub Release 不代表 npm 已发行。旧 0.22.1 没有本教程使用的安装与 DSH 接口,当前发行状态见[发行记录](releasing.md#3-失败恢复)。
|
|
10
10
|
|
|
11
|
+
**自适应工作流是本轮源码新增能力。** `/flywheel-task`、`control launch` 与 Auto/Quick/Full
|
|
12
|
+
需要包含这些变更的同版本 CLI/插件;先核对 `flywheel control launch --help`,
|
|
13
|
+
不要把旧发行包的基础安装成功当成新功能已生效。
|
|
14
|
+
|
|
11
15
|
订阅预设迁移命令与 `permissions.mode` 从 **0.28.2** 开始提供。升级不会自动替换已有配置;先预览 `flywheel config migrate --preset subscriptions`,确认模型和完全权限后再以 `--apply` 应用并备份。
|
|
12
16
|
|
|
13
17
|
**0.28.5 排序**:Flywheel 自定义会话视图排在内置“统计”之后,视图的打开与溢出菜单仍由 DSH 管理。
|
|
14
18
|
|
|
15
19
|
**0.28.3 修复**:下文的 `deepseek-account` 默认和留空目录读取会话当前工作目录从 0.28.3 开始提供。0.28.2 仍使用 `deepseek-official` 与最初目录。安装前核对 registry 版本;升级保留已有配置,采用账号预设需要显式迁移。
|
|
16
20
|
|
|
21
|
+
**可选 Claude 后端:**默认安装不要求 Claude。选择 `claude-code-pro` 时需额外安装 worker 能解析的配套 0.4.0 插件,并准备本机 Claude CLI/登录。具体版本、配置与源码候选边界见 [Claude Code Pro 指南](claude-code-pro.md)。
|
|
22
|
+
|
|
23
|
+
Codex Pro 0.9.0 的已审查真实源码包随本次 Flywheel 插件 bundle,开发候选安装不依赖一个尚未发布的 registry 版本;[来源与摘要](../plugins/dsh/vendor/README.md)。P3/P4 的行为与实测边界见[恢复指南](session-recovery.md)和[对照评估](session-evaluation.md)。
|
|
24
|
+
|
|
17
25
|
## 目录
|
|
18
26
|
|
|
19
27
|
1. [准备运行环境](#1-准备运行环境)
|
|
@@ -40,7 +48,7 @@
|
|
|
40
48
|
| SDK profile 的 DeepSeek 和 ChatGPT provider | 默认实现/评审使用;配置与授权见下一节 |
|
|
41
49
|
| Go 1.27 或更高 | 仅本文的 Go 示例和源码构建需要;npm 安装 Flywheel 不需要 Go 工具链 |
|
|
42
50
|
|
|
43
|
-
安装器的前提是已有 Node/npm 和桌面 DSH;模型渠道可在安装后补齐。CLI 的登录和安装使用各自正式流程。默认复用 Kimi 订阅、ChatGPT 订阅和已登录、已充值的 DeepSeek 账号;DeepSeek API 是需要显式配置的可选渠道。安装 Flywheel
|
|
51
|
+
安装器的前提是已有 Node/npm 和桌面 DSH;模型渠道可在安装后补齐。CLI 的登录和安装使用各自正式流程。默认复用 Kimi 订阅、ChatGPT 订阅和已登录、已充值的 DeepSeek 账号;DeepSeek API 是需要显式配置的可选渠道。安装 Flywheel 不会替这些工具登录、复制凭证或替换桌面程序。Full 需要所选 Plan、Execute、Review、Judge;合格 Quick 的正常路径只调用实际 Execute 路由与最终 Review,发生升级后才需要 Full 阶段。
|
|
44
52
|
|
|
45
53
|
## 2. 一条命令安装
|
|
46
54
|
|
|
@@ -58,8 +66,8 @@ npx -y @ckanner/flywheel@latest install --json
|
|
|
58
66
|
|
|
59
67
|
两条命令选择一种即可。安装器按顺序完成:
|
|
60
68
|
|
|
61
|
-
1. 把本次安装器的精确版本装到 npm 全局目录,解析适合本机的 Go 平台包、同版本 DSH/Codex Pro workers 及精确插件依赖 `dsh-subagent-codex-pro@0.
|
|
62
|
-
2. 从 Go
|
|
69
|
+
1. 把本次安装器的精确版本装到 npm 全局目录,解析适合本机的 Go 平台包、同版本 DSH/Codex Pro workers 及精确插件依赖 `dsh-subagent-codex-pro@0.9.0`,核对 CLI 版本。
|
|
70
|
+
2. 从 Go 二进制安装该版本的全部 skills(当前源码为 11 个);全局配置不存在时生成默认配置,已有配置保留。
|
|
63
71
|
3. 写入 GUI 启动器和安装记录,使用绝对 Node/CLI 路径。
|
|
64
72
|
4. 通过现有桌面 `dsh` 注册精确版本的 Flywheel 插件,并通过 `plugin list` 验证安装结果。
|
|
65
73
|
5. 执行不带 `--probe` 的本地 `doctor`,汇总工具、配置及目录检查结果。
|
|
@@ -103,7 +111,7 @@ flywheel setup --no-global
|
|
|
103
111
|
|
|
104
112
|
安装器已经调用正式接口 `dsh plugin --profile desktop add --save-exact --ignore-scripts @ckanner/dsh-plugin-flywheel@<本次精确版本>`,随后通过 `list` 核对版本。无需再分别安装 worker 或手动选择压缩包。该命令必须来自桌面安装;独立 npm DSH CLI 不能管理保留的 `desktop` profile。按照宿主提示重新载入,在顶部“对话/轨迹”所在的原生视图栏选择 **Flywheel**;未直接显示时检查该栏的更多入口。
|
|
105
113
|
|
|
106
|
-
|
|
114
|
+
这个视图用于控制和观察已经准备好的任务:看进度、策略、人工处理项、实际模型和环境,显式启动、刷新或停止。它保留 DSH 的原生对话、轨迹、输入区、模型选择器和右侧栏;需求讨论与契约准备仍在对话和 skills 中进行。新入口通过 DSH 工具 launch/start 关联具体任务;视图不扫描目录,终端另行启动的任务仍可手动打开。
|
|
107
115
|
|
|
108
116
|
### 原生 SDK profile
|
|
109
117
|
|
|
@@ -165,17 +173,20 @@ flywheel config inspect --cwd /absolute/path/to/target-repo --json
|
|
|
165
173
|
|
|
166
174
|
### 查看或补装 skills
|
|
167
175
|
|
|
168
|
-
|
|
176
|
+
一条命令安装包含所选版本的全部 skills(当前源码为 11 个),默认在 `~/.agents/skills`;使用它们仍是可选的,桌面面板和 CLI 可以独立工作。需要另外安装到目标项目时运行:
|
|
169
177
|
|
|
170
178
|
```sh
|
|
171
179
|
flywheel install-skills --project /absolute/path/to/target-repo --json
|
|
172
180
|
flywheel skills list
|
|
173
|
-
flywheel skills read flywheel-
|
|
181
|
+
flywheel skills read flywheel-task
|
|
174
182
|
```
|
|
175
183
|
|
|
176
184
|
安装输出会给出 `.agents/skills` 的准确位置。按宿主支持的发现规则启用目录,不能从文件已复制推断 DSH/Kimi 已自动发现。也可直接使用 Go CLI。
|
|
177
185
|
|
|
178
|
-
DSH
|
|
186
|
+
DSH 用 `/flywheel-task <目标>` 准备并启动,`/flywheel-new` 只准备方案,`/flywheel-run`
|
|
187
|
+
启动或恢复已有任务,`/flywheel-status` 查进度;也可从 `/` 菜单选择。Kimi 对应使用
|
|
188
|
+
`/skill:flywheel-task` 等语法。准备、启动和观察可以全部在对话完成,**不必先填写
|
|
189
|
+
Flywheel 视图**;DSH 启动工具会关联真实任务。完整流程见[工作流](workflow.md)。
|
|
179
190
|
|
|
180
191
|
快照的 `.flywheel-snapshot` 清单记录文件内容 hash、类型与模式。重跑会跳过相同内容,更新未被修改的自有快照;用户修改、新增/删除文件、链接、未标记目录或损坏清单会保留并报告冲突。每个 skill 独立采用暂存、rename 和失败回滚,不会覆盖手工修改。旧纯文本标记只有在整个目录与当前内嵌内容相同时才能安全迁移。`--force` 只用于 `--link` 开发模式,不能绕过默认保护。安装会在全局配置缺失时创建它,但不会改写已有配置。
|
|
181
192
|
|
|
@@ -194,7 +205,10 @@ flywheel doctor --cwd "$PWD" --probe --json
|
|
|
194
205
|
|
|
195
206
|
结果按 pass/fail/warn/unknown/skipped 展示;有 fail 时退出码 1,JSON 仍完整可读。即使 `ok:true`,unknown 的账号授权、订阅额度与实际推理仍未验证。Kimi/Codex 在这里只检查命令;Codex Pro 额外实际导入精确版本的公开 API,不会自动登录或试调用。Codex Pro 的完整沙箱策略只能事后审查,所以对应 sandbox 项会显示 warn,模型/配置/认证保持 unknown;`--probe` 也不会为它发送推理。DSH 同一账号、模型、effort、目录和权限只探测一次,总预算 40 秒另加清理窗口;超时后明确失败。
|
|
196
207
|
|
|
197
|
-
未指定任务时检查全部配置绑定,可能包括首个任务用不到的 Codex hard
|
|
208
|
+
未指定任务时检查全部配置绑定,可能包括首个任务用不到的 Codex hard 路由。
|
|
209
|
+
准备好任务后按 Spec/持久化策略缩小范围:Full 包含 Plan/实际 Execute route/
|
|
210
|
+
Review/Judge 与启用的第二意见,合格 Quick 跳过 Plan/Judge/第二意见。
|
|
211
|
+
它不重新分类已保存的任务,也不接收启动时另传的 `--mode` 覆盖:
|
|
198
212
|
|
|
199
213
|
```sh
|
|
200
214
|
flywheel doctor --cwd "$PWD" --task-dir docs/add --probe --json
|
|
@@ -202,21 +216,28 @@ flywheel doctor --cwd "$PWD" --task-dir docs/add --probe --json
|
|
|
202
216
|
|
|
203
217
|
尚无 packet 时按规划前状态预检默认执行路由,规划后应再检查。坏契约会报告阻断项,不能用探测通过代替 schema 校验。若 cwd、权限、provider 或沙箱检查失败,先修正相应配置,不把权限改成 full access 绕过。桌面视图的“配置与排查(按需)”提供“查看模型配置/检查安装与配置/检查 DSH 接入”,使用同一个 CLI 后端;安装、改配置或排错时按需使用,不是每次任务都必须手点的步骤。GUI 与终端环境不同的问题应在视图内复验。
|
|
204
218
|
|
|
205
|
-
|
|
219
|
+
这允许在准备契约时提前排查安装,但正式启动仍需至少一个合法 packet 和明确授权的需求。完成下一节的小任务准备后,再运行带 `--task-dir` 的检查。
|
|
206
220
|
|
|
207
221
|
## 6. 准备首个小任务
|
|
208
222
|
|
|
209
223
|
已有业务仓库应先建立任务集成分支和独立 worktree,再生成契约;启动时的 `cwd` 指向该 worktree。完整步骤见[从需求准备到 PR 合并](workflow.md),下面的小仓库示例用于首次运行验收。
|
|
210
224
|
|
|
211
|
-
正常在 DSH
|
|
225
|
+
正常在 DSH 对话发送 `/flywheel-task <实际需求>`,可加 `mode=quick` 或 `mode=full`,
|
|
226
|
+
默认 Auto。助手研究代码、创建或复用 worktree,准备 `docs/<slug>/spec.md` 与 packets,
|
|
227
|
+
保存工作流事实,再启动并关联到视图。**项目目录**是代码 worktree,**任务资料目录**
|
|
228
|
+
是其中保存契约和状态/报告的 `docs/<slug>`;工具会传递真实绝对路径,无需自行猜目录。
|
|
229
|
+
只想先审阅方案时使用 `/flywheel-new`,之后沿用原路径与授权启动。
|
|
212
230
|
|
|
213
|
-
|
|
231
|
+
首次也可使用独立小仓库区分安装与业务问题。npm 包附带的准备脚本创建 Go 加法函数任务:
|
|
232
|
+
初始实现故意错误,执行者修复并补普通单测,独立 acceptance 不可修改。该兼容示例
|
|
233
|
+
没有 workflow 字段,继续走 Full;要体验 Auto/Quick,优先使用上面的统一入口准备
|
|
234
|
+
真实资格事实,不能只换启动参数伪造事实。本示例需要 Go 1.27 或更高。
|
|
214
235
|
|
|
215
236
|
```sh
|
|
216
237
|
FLYWHEEL_PACKAGE="$(npm root --global)/@ckanner/flywheel"
|
|
217
238
|
bash "$FLYWHEEL_PACKAGE/docs/examples/create-first-task.sh" "$HOME/flywheel-demo"
|
|
218
239
|
cd "$HOME/flywheel-demo"
|
|
219
|
-
git init
|
|
240
|
+
git init -b main
|
|
220
241
|
```
|
|
221
242
|
|
|
222
243
|
如果安装时使用了自定义 prefix,用 `npm root --global --prefix "$HOME/.local"` 等实际 prefix 定位包目录;不需要克隆源码。
|
|
@@ -248,11 +269,18 @@ go test -coverprofile=coverage.out . &&
|
|
|
248
269
|
```sh
|
|
249
270
|
git add go.mod sum.go acceptance .gitignore flywheel.config.json docs/add
|
|
250
271
|
git commit -m "docs: prepare first flywheel task"
|
|
272
|
+
git worktree add -b task/add ../flywheel-demo-add
|
|
273
|
+
cd ../flywheel-demo-add
|
|
251
274
|
```
|
|
252
275
|
|
|
253
|
-
Git
|
|
276
|
+
Git 提交身份必须已经按自己的设置配置,工作区新目录不可被占用。后续在 `task/add`
|
|
277
|
+
工作区运行,packet 合回该任务分支。此时不应创建 `state.json`,它由引擎初始化。
|
|
278
|
+
coverage 输出已被忽略,所以验收运行不会把生成物误判为候选代码漂移。
|
|
254
279
|
|
|
255
|
-
|
|
280
|
+
业务仓库的 `flywheel-task`/`flywheel-new` 都使用当前 DSH 会话模型生成契约,
|
|
281
|
+
不读取 `loops.plan` 来换模型,后者只控制引擎规划/修订。也可查看
|
|
282
|
+
`flywheel template --list` 后手工准备。示例 auth 模板涉及鉴权,不能因为 packet
|
|
283
|
+
很小就标 Quick;路径与验收命令必须改为项目事实,不要照抄后直接启动。
|
|
256
284
|
|
|
257
285
|
## 7. 启动、观察、停止与恢复
|
|
258
286
|
|
|
@@ -265,9 +293,15 @@ flywheel control status --task-dir "$PWD/docs/add" --cwd "$PWD"
|
|
|
265
293
|
|
|
266
294
|
在 DSH 对话中也可使用 `/flywheel-run <任务资料绝对路径>`、`/flywheel-status <同一路径>`,需要停止时用 `/flywheel-stop <同一路径>`。把占位内容替换成准备阶段给出的真实路径;这些技能会解析正确的项目根,不依赖先填写视图。
|
|
267
295
|
|
|
268
|
-
`ok: true` 只表示控制操作成功;`running`
|
|
296
|
+
`ok: true` 只表示控制操作成功;`running` 表示引擎是否在运行,完成看 `state.status`。
|
|
297
|
+
DSH 工具 launch/start 会关联当前会话视图,确认关联后查询一次最新状态;终端
|
|
298
|
+
启动或手动打开时,仍填写同一个 `task_dir` 与实际 `repo_dir`。项目目录仅在会话
|
|
299
|
+
当前目录相同时可留空,shell `cd` 不改变会话目录。视图展示请求/生效模式、理由、
|
|
300
|
+
阶段适用性、任务状态与刷新时间;不适用的 Plan/Judge 不算通过,也不会自动轮询。
|
|
269
301
|
|
|
270
|
-
|
|
302
|
+
会话的工具/命令记录仍在时,重载可通过公开投影重建明确的任务关联;fork 不继承
|
|
303
|
+
父会话关联。手动路径草稿与旧快照只是查看缓存,重载后可能需重选或刷新。修改路径
|
|
304
|
+
忽略旧响应,不会停止旧任务。每个会话只等待一个控制请求,启动或停止期间锁定目录。
|
|
271
305
|
|
|
272
306
|
启动或停止超时、断连后,后台任务的结果可能尚未确认。保留原任务目录,先“刷新状态”和查看日志;界面在成功核对该任务前禁止继续启动/停止,也不会自动重试。已有快照只是上次结果,不能凭页面关闭、切换会话或停止按钮失去等待状态推断引擎已经结束。原生轨迹保留父会话自己的记录,worker 完整历史不自动合并到其中。
|
|
273
307
|
|
|
@@ -279,7 +313,7 @@ flywheel control status --task-dir "$PWD/docs/add" --cwd "$PWD"
|
|
|
279
313
|
flywheel run docs/add --pause-before review
|
|
280
314
|
```
|
|
281
315
|
|
|
282
|
-
`control start`
|
|
316
|
+
`control launch` 和 `control start` 不接受 `--pause-before`,暂停模式不能与 `--only` 并用。`--only execute` 会运行 Execute 子环及 L1,直至通过或遇到失败出口;它既不是 Quick,也不是一次模型调用。Quick 的 `--only review` 使用最终 Review 协议,需要升级时立即保存单向 Full checkpoint,但不自动跨到 Plan 或执行本地交付;`--only verify` 在指定 packet 时仍包含候选上适用的 L2/L3。Quick 要单步运行 Plan,需明确 `--mode full` 升级。单环命令详见各内嵌 skill。
|
|
283
317
|
|
|
284
318
|
观察人话摘要和日志:
|
|
285
319
|
|
package/docs/model-policy.md
CHANGED
|
@@ -14,7 +14,7 @@
|
|
|
14
14
|
|
|
15
15
|
| 当前调用路径 | 已核对的能力 | Flywheel 当前请求 |
|
|
16
16
|
|---|---|---|
|
|
17
|
-
| GPT-6.1 Sol/Codex Pro 0.
|
|
17
|
+
| GPT-6.1 Sol/Codex Pro 0.9.0 | 模型支持 `low/medium/high/xhigh/max`;machine API 原样传递 effort | hard 为 high |
|
|
18
18
|
| GPT-6.1 Sol、GPT-6 Astra/DSH ChatGPT provider | 模型及已核对的 provider 表均列 `low/medium/high/xhigh/max`;请求写入 `reasoning.effort` | Review、Judge 为 high |
|
|
19
19
|
| Kimi K3.1 Pro/Kimi CLI | 模型原生 `low/high/max`,默认 high,不能关闭 thinking | Plan、frontend、Garden 为 high |
|
|
20
20
|
| DeepSeek Flash/DSH rc.5 | DSH 接受 `off/low/high/max`,默认 high | 普通执行 high,Intake low |
|
|
@@ -50,7 +50,7 @@ hard 承担难实现,Judge 在 L2/L3 后独立核对整个里程碑;它们
|
|
|
50
50
|
- SDK 接受配置或 machine API 接受请求,只证明对应接口接纳,不是效果评测。
|
|
51
51
|
- 服务端实际采用的强度和质量收益,需要独立回执或实际任务测量;当前证据不能补造这些事实。
|
|
52
52
|
|
|
53
|
-
DSH `initialize` 调用配置校验,但回复只含 serverInfo。Flywheel 的 `ready.reasoningEffort` 是初始化成功后的请求回显,不是模型服务独立返回的 effort 证明。Codex Pro 回执明确区分 `requestedEffort`、`accepted` 和未知的 `effectiveEffort:null`。依据见 [DSH SDK server](https://github.com/deepseek-ai/deepseek-harness/blob/b830ec950faba0a46dda564be3e2d854afed94f1/packages/sdk/server/src/server.ts)及 [Codex Pro 0.
|
|
53
|
+
DSH `initialize` 调用配置校验,但回复只含 serverInfo。Flywheel 的 `ready.reasoningEffort` 是初始化成功后的请求回显,不是模型服务独立返回的 effort 证明。Codex Pro 回执明确区分 `requestedEffort`、`accepted` 和未知的 `effectiveEffort:null`。依据见 [DSH SDK server](https://github.com/deepseek-ai/deepseek-harness/blob/b830ec950faba0a46dda564be3e2d854afed94f1/packages/sdk/server/src/server.ts)及 [Codex Pro 0.9.0 协议](https://github.com/ckanner/dsh-subagent-codex-pro/blob/37da3d75de32b809c4fb1a631ccc15191f3d87f9/docs/machine.md)。无推理 doctor 也不能证明账号余额、真实调用或更高档位的收益。
|
|
54
54
|
|
|
55
55
|
## 5. DeepSeek 默认应使用已充值账号
|
|
56
56
|
|
|
@@ -59,3 +59,7 @@ DSH `initialize` 调用配置校验,但回复只含 serverInfo。Flywheel 的
|
|
|
59
59
|
桌面能在登录后初始化账号默认模型,但独立 SDK 调用仍需明确 provider/model。Flywheel 固定请求 `deepseek-account`/`deepseek-flash`,既采用指定账号通路,也保持“只用 Flash”的选择。认证失败或额度不足会报告;默认预设没有跨渠道后备。用户显式配置的后备链仍按原有规则执行,本轮不新增后备路径。
|
|
60
60
|
|
|
61
61
|
**发布与迁移要分开:**0.28.3 将内嵌默认改为账号,0.28.2 仍内嵌 API 默认。安装升级保留已有用户配置,采用新账号预设需升级至 0.28.3 或以后版本,再预览并显式应用迁移。完整账号/API 区别、共享登录条件和迁移步骤见[配置指南](configuration.md#deepseek-账号与-api-key-是两个-provider)。本轮没有修改用户机器上的凭证或正在运行的任务。
|
|
62
|
+
|
|
63
|
+
## 可选 Claude Code Pro
|
|
64
|
+
|
|
65
|
+
默认模型策略尚未给 Claude 分配场景。可选 `claude-code-pro` harness 接收明确的原生模型与 low/medium/high/xhigh/max 请求,具体 CLI/账号可能拒绝某项选择;此接入不新增模型排名或 effort 实测结论。配套 0.4.0 公开接口、full-access 边界及用量口径见[接入指南](claude-code-pro.md)。
|
|
@@ -0,0 +1,120 @@
|
|
|
1
|
+
# 会话质量策略与配对评估
|
|
2
|
+
|
|
3
|
+
P4 提供两部分可执行能力:按已确认事实进行有界重建,以及对相同任务运行三种历史策略的配对评估。重建阈值默认关闭;评估程序保存原始条件和结果,不把缺失用量写成零,也不预设质量、缓存率或速度收益。
|
|
4
|
+
|
|
5
|
+
配置入口见 [配置文档](configuration.md),中断恢复的生命周期和账本语义见 [恢复文档](session-recovery.md)。实现入口为 [质量策略](../internal/machine/session_quality.go)、[评估器](../scripts/evaluate-sessions.js) 与 [任务集生成器](../scripts/prepare-session-evaluation.js)。
|
|
6
|
+
|
|
7
|
+
## 有界重建
|
|
8
|
+
|
|
9
|
+
`sessions.roles` 可分别指定 `plan`、`execute`、`review`、`judge` 和 `judge-panel` 的 `fresh` / `scoped`,未配置的角色继承 `sessions.mode`。Plan 的显式 `resident-session:false` 仍然禁止续用;显式 `sessions.roles.plan:"fresh"` 也关闭旧 Kimi Plan 的续用入口。
|
|
10
|
+
|
|
11
|
+
`sessions.quality` 支持以下阈值;`0` 表示关闭,对既有默认行为没有改变。
|
|
12
|
+
|
|
13
|
+
| 配置 | 触发依据 | 能证明的内容 |
|
|
14
|
+
|---|---|---|
|
|
15
|
+
| `max-confirmed-turns` | 本代已确认、安全归属的 turn 数达到阈值 | 对话被重用的次数;不等同于质量已经下降 |
|
|
16
|
+
| `max-unproductive-turns` | Execute 连续完成后,候选成果 fingerprint 没有变化 | 磁盘候选没有进展;不推断模型思考过程 |
|
|
17
|
+
| `max-run-input-tokens` | 最近一次完整、可归属的单次运行 input 达到阈值 | 该次输入用量;不等同于原生历史占用或剩余上下文容量 |
|
|
18
|
+
|
|
19
|
+
只有上一调用已经消费、记账状态已保存、派发与终态确定、清理已确认、候选成果可核对时,才按这些阈值重建。未知派发/终态、未完成的生命周期和未确认清理先进入 P3 的恢复或现有升级路径。partial/unknown usage 不能触发输入用量阈值;但已确认 turn 数和候选无进展仍可独立触发各自阈值。未知用量继续保留,原有 hard budget 可另行阻断后续调用。重复消费同一 invocation 不重复增加计数。
|
|
20
|
+
|
|
21
|
+
重建记录原因、代数、被替代调用引用,并通过当前 Spec、packet、磁盘成果、合法报告、验证证据和结构化反馈交接。Executor 的私有历史不会复制给 Reviewer。新代重置本代质量计数,但任务 attempts、失败、耗时及预算账本继续保留。这里使用可验证的重建,不调用额外模型做未计量的历史压缩。
|
|
22
|
+
|
|
23
|
+
## 三种固定策略
|
|
24
|
+
|
|
25
|
+
评估器使用同一固定 Git commit、任务契约、模型、effort、权限、验收与质量阈值。它只改变 Execute / Review 历史策略。
|
|
26
|
+
|
|
27
|
+
| 报告中的 variant | Execute | Review | Plan / Judge / judge-panel |
|
|
28
|
+
|---|---|---|---|
|
|
29
|
+
| `fresh` | 每次 fresh | 每次 fresh | 显式 fresh |
|
|
30
|
+
| `executor-scoped` | 同 packet 范围 scoped | 每次 fresh | 显式 fresh |
|
|
31
|
+
| `role-scoped` | 同 packet 范围 scoped | Reviewer 自己的评审范围 scoped | 显式 fresh |
|
|
32
|
+
|
|
33
|
+
`role-scoped` 是设计中的“Executor 和 Reviewer 各自在自己的范围续用”,不表示所有角色共享一个 thread。每个 case 至少运行两次配对重复,每一组含三种策略;运行顺序按 case 和重复编号轮换。每个 trial 使用新的隔离 Git clone,不修改原始仓库,不把前一个 trial 的工作树或历史交给下一个 trial。
|
|
34
|
+
|
|
35
|
+
## 准备可执行任务集
|
|
36
|
+
|
|
37
|
+
先准备完整的 Flywheel 配置,显式指定 `routes.default` 和 `loops.review`,以及它们引用的渠道。使用已经能正常运行的账号与后端配置。生成器只读取这份配置,不运行模型,也不读取或复制账号凭据。
|
|
38
|
+
|
|
39
|
+
```sh
|
|
40
|
+
node scripts/prepare-session-evaluation.js \
|
|
41
|
+
--config /absolute/path/flywheel.config.json \
|
|
42
|
+
--output /absolute/path/new-session-corpus \
|
|
43
|
+
--repetitions 2
|
|
44
|
+
```
|
|
45
|
+
|
|
46
|
+
生成器输出新的本地 Git 仓库、精确 commit 和 `manifest.json`。它为所有 trial 统一关闭额外 worktree 创建,因为评估器本身已经逐 trial 隔离;并把冻结检查 `eval/` 加入保护范围。原配置文件不被覆盖。
|
|
47
|
+
|
|
48
|
+
生成的配置变化场景使用 `medium` / `high`。所选模型必须接受这两个值;不同 effort 枚举的后端应在运行前把 manifest 的 `configure` 步骤改为该后端明确支持的另一个值。三种策略使用同一份修改后的 manifest,不在某一个 trial 中临时猜测或替换。
|
|
49
|
+
|
|
50
|
+
内置任务是一个可独立验收的真实代码修改:实现纯函数 `compactRanges`,把安全整数数组变成排序后的最大闭区间,去重、支持负数和空数组,并保持输入数组不变。冻结 L1 检查输出,最终 Review 独立检查输入不变性;进程外的固定 oracle 同时检查两者。模型自己的 done-report 不是评估评分依据。
|
|
51
|
+
|
|
52
|
+
| 场景 | 实验步骤 | 观测条件 |
|
|
53
|
+
|---|---|---|
|
|
54
|
+
| `first-pass` | 从未实现代码开始正常执行 | 完成任务,且一次 Execute、一次 Review |
|
|
55
|
+
| `l1-repair` | 首次执行后,固定脚本注入错误输出;真实 L1 拒绝,再由引擎修复 | 存在 L1 失败且 Execute 多于一次 |
|
|
56
|
+
| `review-return` | 注入输出仍正确、但原地排序输入的回归;L1 通过,Review 应退回 | 存在 Review return 且 Execute 多于一次 |
|
|
57
|
+
| `long-context` | 在实际 Execute prompt 的 `cache-scope` 中加入稳定参考材料 | 实测单次 input 达到 manifest 阈值;缺失用量记 unknown |
|
|
58
|
+
| `stop-resume` | 在 Review 前由引擎正常暂停,再显式运行 | 已保存 paused 状态后继续执行;这是阶段边界恢复 |
|
|
59
|
+
| `configuration-change` | 首次 Execute 后显式改变同一 binding 的 thinking,再执行 | 观察到有效 binding fingerprint 改变 |
|
|
60
|
+
|
|
61
|
+
受控注入只修改候选代码;评估器核对注入前后的 Go state、任务契约和配置不变。它不伪造 L1、Review、receipt、metrics 或 state。阶段边界暂停不替代 P3 对活跃 turn 杀进程、ACK 故障窗口和只读恢复的独立测试。
|
|
62
|
+
|
|
63
|
+
有效配置写在隔离 trial 的候选目录外,通过 `FLYWHEEL_CONFIG` 指定。开始前,评估器在该 clone 中移除会覆盖它的根目录和任务目录配置并提交这个实验基线。因此配置变化不会伪装成模型修改受保护契约。源配置和变更前后 binding 均保留在条件记录中。
|
|
64
|
+
|
|
65
|
+
## 运行真实后端评估
|
|
66
|
+
|
|
67
|
+
构建当前 Go CLI,确认所选公开 adapter、原生 CLI 与账号能够执行任务后运行:
|
|
68
|
+
|
|
69
|
+
```sh
|
|
70
|
+
node scripts/evaluate-sessions.js \
|
|
71
|
+
--manifest /absolute/path/new-session-corpus/manifest.json \
|
|
72
|
+
--flywheel /absolute/path/flywheel \
|
|
73
|
+
--output /absolute/path/new-session-results \
|
|
74
|
+
--live
|
|
75
|
+
```
|
|
76
|
+
|
|
77
|
+
`execution:"live"` 必须显式传 `--live`,且不能使用 mock driver。评估器使用提供的模型与权限,不自动切换到其他模型、渠道或更宽权限。输出目录必须不存在,已经保存的结果不会被覆盖。
|
|
78
|
+
|
|
79
|
+
所有调用仍走真实引擎的预算、停止和恢复规则。manifest 的 `maxCalls` 是额外的观测调用上限:运行中每 50 ms 检查落盘 metrics,达到上限发送取消;已在途调用和 metrics 落盘延迟可能造成超出,因此它不是模型调用前的事务额度预留。`runTimeoutMs` 控制每个步骤,取消后保留最多 30 秒清理时间。强制杀进程、未知派发/终态或未确认清理会停止整个实验,保存已知结果,不自动重发。
|
|
80
|
+
|
|
81
|
+
## 报告如何解释
|
|
82
|
+
|
|
83
|
+
`report.json` 包含源 commit/tree、manifest/config/engine/oracle 的 SHA-256、开始结束时间、每个 trial 的固定条件、顺序、策略、原始步骤结果和配对比较。每个 trial 另存 stdout/stderr、状态快照和原始 metrics。独立 oracle 从固定 commit 复制到候选目录外,执行前后核对 hash;候选篡改 oracle 会终止实验。
|
|
84
|
+
|
|
85
|
+
| 字段 | 含义与边界 |
|
|
86
|
+
|---|---|
|
|
87
|
+
| `quality.pass`、`omissions`、`regressions` | 独立 oracle 的结果;未完成检查保留 null,不拿引擎 completed 代替质量证据 |
|
|
88
|
+
| `repairRounds` | packet attempts 减首次执行的总数;使用真实 Go 状态 |
|
|
89
|
+
| `repeatedFindings` | 规范空白后完全相同的 Review return 原因重复数;不声称语义相同 |
|
|
90
|
+
| `elapsedMs` | 引擎运行步骤耗时;不包含 clone、oracle 与注入开销 |
|
|
91
|
+
| `usage.*.known` | 完整单次 input/output/cacheRead/cacheWrite/totalTokens 的已知合计 |
|
|
92
|
+
| `observedCalls` / `missingCalls` | 各指标实际可计数和缺失的调用数;有缺失时 known 仅为下限 |
|
|
93
|
+
| `cost.knownUSD` | 已知费用合计;缺失费用单列,不把订阅账单当零 |
|
|
94
|
+
| `actions`、`rebuildReasons` | 实际创建、续用、重建次数及其可核对原因 |
|
|
95
|
+
| `recovery` | 续用数、失败或未知续用数、未知终态、未解决调用、只读核对次数 |
|
|
96
|
+
| `scenarioCoverage` | 分别记录已计划、实际观测到、未观测到和 unknown;场景名称不是成功证据 |
|
|
97
|
+
|
|
98
|
+
P3 的 `record-kind:"reconciliation"` 行按 `reconciles-invocation` 替换原调用的后验事实;不把审计行当成新模型调用,也不重复累加用量或费用。原始审计行仍完整保存。
|
|
99
|
+
|
|
100
|
+
只有三个策略都完成配对、固定条件相同且实际 role/harness/provider/channel/model 集合一致时,才计算差值。某个用量字段缺失则该差值为 null。报告没有自动生成收益百分比;需要先判断验收质量和场景是否成立,再讨论性能变化。
|
|
101
|
+
|
|
102
|
+
## 离线门禁与真实证据分开
|
|
103
|
+
|
|
104
|
+
仓库提供三层不同的证据入口:
|
|
105
|
+
|
|
106
|
+
1. `node --test npm/test/session-evaluation.test.js` 验证 manifest、隔离、未知用量、配对与停止规则,使用明确标记的 runner 协议 fixture。
|
|
107
|
+
2. 下列命令运行真实 Go CLI → worker 3 → 实际公开 Codex 0.9 包 → 原生 CLI fixture,共 36 个 trial。fixture 只实现原生协议和受控代码工作,不写 Go state/metrics/receipts;真实引擎执行 L1、Review 判定、Git 边界与恢复选择。
|
|
108
|
+
3. `--live` 使用真实后端推理,才能提供模型质量、缓存、耗时与费用方面的实际证据。
|
|
109
|
+
|
|
110
|
+
```sh
|
|
111
|
+
FLYWHEEL_EVAL_BINARY=/absolute/path/freshly-built-flywheel \
|
|
112
|
+
FLYWHEEL_EVAL_OUTPUT=/absolute/path/new-native-evaluation \
|
|
113
|
+
node scripts/test-session-evaluation.js
|
|
114
|
+
```
|
|
115
|
+
|
|
116
|
+
仓库根目录的 `npm run test:sessions` 是同一门禁入口,接受相同环境变量。
|
|
117
|
+
|
|
118
|
+
第二层要求已构建 `plugins/dsh/dist`、已安装实际固定版本 Codex 包;缺少二进制或依赖会失败,不会自动 skip。它在独立目录冻结 worker 和公开包,原生 fixture 只能访问带实验标记的 CODEX_HOME 与任务集,不接触实际账号。用量由 fixture 合成,只验证计量链路,不能作为模型节省数据。
|
|
119
|
+
|
|
120
|
+
当前实现不以离线通过宣称真实账号或 macOS 生命周期验收已完成。真实环境若拒绝权限或在 thread/start 超时,应保存阻断证据;默认阈值继续关闭,直到配对真实结果支持具体调整。
|